antoniopaez

Author

antoniopaez

Development

Queue tasks in Celery after database commit – Introducing django-transaction-hooks

April 18, 2018

April 18, 2018

At Ebury, we use Django and have followed an ongoing upgrade path from 1.3 to 1.5 to 1.7. During that time we have had an issue that was messing with us. You might be familiar with it.

We use celery for  executing asynchronous tasks and Django is our framework with PostgreSQL database.

The issue occurs when an asynchronous task makes use of an object that has been just updated, or  created. There is a dependency with the database, the object might not have the updated status when the asynchronous task starts, or not even exists yet.

We are now able to utilise the  library django-transaction-hooks, which works with Django 1.6 through 1.8, and has been merged into Django 1.9+.

What is important with this library is that adds the event “on_commit” to manage timing with database transactions. So, we  can use this for scheduling when to queue tasks for celery workers. The main advantage comes when we want to queue using an object created into an atomic transaction. Consider the following example:

Read more