We all would have heard at least once that we need to have certain tasks performed in the background. But what is it? Why is it even necessary? Building a feature in an application that takes a considerable amount of time to execute and keeps the user frozen from accessing the other parts is like inviting someone to home for lunch and serving them at the time of dinner. It is unacceptable for the user experience. So, we need to have the job done in the background and then keep the user notified about the status of the job. Every web server has its own default and maximum time out for a request. If that exceeds, the server throws out a 504 Gateway Timeout error. By default Apache Web Server timeout is 30seconds and the maximum is 300seconds (5 minutes). But, letting the user uninformed about the status of a request for 30 seconds (considering the default timeout) is such a bad experience. The user might even exit from using our application without waiting. So, we run the job in the background. Ideally, background tasks are "fire and forget" operations, and their execution progress has no impact on the UI or the calling process. This means that the calling process does not wait for the completion of the task. The certain tasks that can be scheduled in the background are:
Sending notifications (Email, SMS, Push) to the users.
Running a machine learning algorithm or running a query that takes a longer duration and storing the results in the cache so that it can be accessed when required.
We certainly realize that we should do use background jobs. The most common two background job processing systems are:
Celery
RQ (Redis Queue)
Celery is most widely used for it supports various brokers(we will discuss brokers shortly), available in different programming languages, etc. While RQ can be used only with Redis and is available only in Python.
The client(user) sends a request to our web server, Flask application. If the request has a background job associated, the request is passed to Redis which acts as a message broker.
A message broker is a middleware in a server-based application in which the source component (flask server) sends a message (task to be accomplished) to a consumer (celery worker). The broker stores the message in its queue till it is received by the consumer so that, the message is not lost. The message received by the worker from the broker is processed as defined by the task.
The tasks can be as simple as doing an addition of two variables to updating the database based on certain conditions and sending an email.
Celery also supports task scheduling with the help of celery beat a task scheduler. This can be used as an alternative for cron jobs for resetting the table values at a specific time every day, sending status emails at the end of the day, etc.
The background tasks and the scheduled tasks can be viewed and monitored using Flower, a Celery monitoring web-based tool.
Celery comes with a lot of configuration options, which makes it used widely across multiple use cases.
Do you not think such background scheduling tasks do improve the performance of your applications? Not sure what tasks can be scheduled in the background? Are you looking for a similar system but your tech stack is different? We are capable of doing it across various tech stacks. Are interested in making your product better. Is writing such tasks and building better software your expertise? Get in touch with us.
Comments