Scalability and performance. Why we need to stay abreast of technology

Software developers often work on personal projects outside of normal working hours, not so much because they are workaholics, but more because creating outside of their normal day-to-day encourages a developer to expand their abilities and spend time exploring new technology. As well as being interesting it means that developers become more rounded and experienced than their sometimes prescriptive workloads would otherwise demand.

More and more companies are starting to see the benefits of allowing their developers to experiment with alternative technologies on company time. The benefits are seen as an increased ability to adapt to new problems that may be presented by clients or other external factors.

For example, a new client expressed an interest in our intranet product Twine, however, they wished to use it with several tens of thousands of users. For the most part, there were no problems, however, we did run into one with our “People Picker” component.

People Picker is an AngularJS component that loads a list of all the people and would allow you to paginate, filter and search for the people you wanted. For the vast majority of our clients, this meant a very fast and responsive experience, as the results would update instantly as you typed a query.

Twine’s people picker is an AngularJS component

One problem… it didn’t scale. At around 20,000 people and upwards the slowdown was very noticeable, and this was on a fast machine running a modern browser.

We decided that the only option would be to move the pagination, filtering a searching to the server — this would mean telling the server about each action the user took and delivering the results back to the browser. Despite simply querying a database and converting the results to JSON the delay in updates while typing a search term was unacceptable, so we needed a better solution.

Some months back, I started experimenting with Go (Google’s systems development language). Go has become popular with companies looking to build small, highly reliable, microservices due to it’s easy to use networking abilities, performance, and concurrency model. As a result of our experiences, we had already begun using Go behind the scenes to deliver non-critical features in our own products — so we decided to see if Go could deliver the user experience we required.

We made use of Go’s built-in tools alongside the excellent Gorilla mux package to authenticate the user, run the query and transform it into JSON at a speed that was simply not possible with our Symfony stack. Moreover, we found that by exploiting Go’s easy to use concurrency model that we could handle many requests at once with practically no modification of our code.

This post isn’t aimed at extolling the virtues of Go, or any other technology, but rather to encourage experimentation with technology. In fact, I believe it’s a ritual that every company should actively encourage in order to succeed.

Perhaps some Erlang next?