Scaleable Solutions

Scaleable software solutions ultimately determine what applications remain limited to a few hundred users and a couple of thousand transactions and those that can truly support millions of users and transactions. Scaleability is essentially the software equivalent of "separating the men from the boys" (and the women from the little girls too!) *. One of the criteria that many companies unfortunately do not consider when buying off the shelf software or ordering a bespoke custom solution is the medium and long term growth of their business. The applications that are chosen need to be able to grow along with growth in business and the client base - or otherwise companies end up being faced with expensive migrations to new systems that can handle their new requirements.

Our way of ensuring that this does not happen is to have a scaleable design from the ground-up, with a well-thought data model that can be distributed amongst multiple servers, and possibly federated and sharded if need be. The software itself also needs to be capable of running in a distributed fashion, with algorithms that are friendly towards parallel processing techniques and frameworks such as MapReduce. Some choices in large scale algorithm design can be counter-intuitive, as such systems tend to optimise for maximum consistent data throughput rather than unpredictable short bursts of speed followed by long delays and waits.

Naturally, the data processing itself needs to deliver high performance over huge datasets - at Charonite we are generally not happy with anything that cannot process a couple of Terabytes of data without failing. Ideally without users even noticing the underlying complexity that goes on behind the scenes!

The Obulus Platform offers a variety of techniques and technologies that facilitate the creation of scaleable applications that can be distributed over multiple servers and server farms. And when things do fail (as they inevitably will in any large scale real-life installation), the platform design makes it possible to recover gracefully and turn parts of the application on and off and even paused / resumed in a flexible manner.

Our experience has shown us that by using profiling and measurement and focusing on the processing bottlenecks that cause the vast majority of slowdowns, an application can be significantly improved even in the face of large datasets. Often, a simple tweak such as an SQL query rewrite or the introduction of a buffer layer to reduce client spoon-feeding effects, can have an enormous positive effect on performance. The use of application level logging in such cases is very important. Testing - to the point of almost becoming obsessive with shaving off milliseconds from important processes - can also be extremely helpful in pinpointing the relevant areas where improvements need to be made. Constant minor tweaks and improvements are a daily aspect of life when dealing with large scale complex installations supporting millions of users.

Have a look at our Obulus Platform presentation at: http://www.charonite.com/resources/obulus.php

* And if you're someone passionate about multi-threading, high performance processing, distributed algorithms, etc. drop us an email at jobs@charonite.com.