José Antonio Perdiguero

Author

José Antonio Perdiguero

Lead Big Data Engineer

Development

Daemons with Celery III

May 12, 2016

May 12, 2016

To improve the solution I explored in the previous entry, we need to go deeper into our knowledge of Celery.

Our goal is to change our tasks’ behavior so that, if the same type of task is currently being executed, the second task is marked as aborted (or similar) and not executed. To achieve this behavior we need to use Celery signals.

Read more

Development

Daemons with Celery II

April 27, 2016

April 27, 2016

Continuing with our previous entry about how to daemonize with Celery. We left the solution in a state where our buffer could eventually collapse due to our producer generating tasks faster that our consumer could execute them.

To solve this I propose another question: “Is foo_action_postsave task currently being executed?”

Read more

Development

Daemons with Celery I

April 13, 2016

April 13, 2016

We’re used to working with python so the examples I’ll use will be either pure python or python-based pseudocode. Python has a really simple and elegant syntax so it’s almost understandable for people who don’t know the language.

I’m going to explore a common problem in the backend of all kinds of applications that have heavy processing requirements, such as those that reprocess all links of each object from a class due to an update or recalculate an attribute that depends on huge functions, and walk through asynchronous solutions.

Read more