Serializing Slow Functions

Quite often, on the Internet, we want software that does it all because ... it's better.

However, once in a while a function has to handle a very complicated or at least very consuming (works on a large amount of data) function.

If you have hundreds of hits per seconds (or even millions), you could end up eating all your resources as in: so many instances of that heavy functions are running concurrently that the computer has to swap out some memory, some network connections fail, the system is slow for everyone and everything.

A solution is to serialize the requests. With a bit of AJAX, you can send a request and wait for the reply and display it once available. The wait could be very long (minutes) if you have a heavy load (many requests in a row), or it could be very short (you were the only request in the queue and you get your results within 5 sec. which is the time the function takes to run to completion).

I'm sure there are many ways to serialize a request. I've seen a feature in Nginx to prevent more than one request at a time from going through the proxy. One advantage is that if two clients ask for the same thing, it's going to be cached so the second request can be answered from the cache once done with the first.

Another way is for your application to accept batch like requests which you record and then work on those requests one at a time. This has one potential drawback: you may get a very large number of unhandled connections lingering. Note that Nginx would have the same problem. Unfortunately, HTTP is a one way protocol. You can't send the reply to a browser that isn't already connected to your server. HTTP2 helps in that the connection remains open so the server can push to the client once the data is ready.

The great news about this is that this way you avoid the memory and other issues on your server. If you have too many requests that can't all be served (many just never get an answer) then you may have to consider getting more backend resources. However, if your bottleneck is your database, more servers may not help much. You may have to make a read-only copy of your database and do the queries on that copy instead. If one copy isn't enough, you can have more, of course. You may also check the replication capability of your database. It may be enough and more reliable.