Search results for

celery

Queue tasks in Celery after database commit – Introducing django-transaction-hooks

April 18, 2018

At Ebury, we use Django and have followed an ongoing upgrade path from 1.3 to 1.5 to 1.7. During that time we have had an issue that was messing with us. You might be familiar with it.

We use celery for  executing asynchronous tasks and Django is our framework with PostgreSQL database.

The issue occurs when an asynchronous task makes use of an object that has been just updated, or  created. There is a dependency with the database, the object might not have the updated status when the asynchronous task starts, or not even exists yet.

We are now able to utilise the  library django-transaction-hooks, which works with Django 1.6 through 1.8, and has been merged into Django 1.9+.

What is important with this library is that adds the event “on_commit” to manage timing with database transactions. So, we  can use this for scheduling when to queue tasks for celery workers. The main advantage comes when we want to queue using an object created into an atomic transaction. Consider the following example:

Read more

José Antonio Perdiguero
Lead Big Data Engineer

Daemons with Celery III

May 12, 2016

To improve the solution I explored in the previous entry, we need to go deeper into our knowledge of Celery.

Our goal is to change our tasks’ behavior so that, if the same type of task is currently being executed, the second task is marked as aborted (or similar) and not executed. To achieve this behavior we need to use Celery signals.

Read more

José Antonio Perdiguero
Lead Big Data Engineer

Daemons with Celery II

April 27, 2016

Continuing with our previous entry about how to daemonize with Celery. We left the solution in a state where our buffer could eventually collapse due to our producer generating tasks faster that our consumer could execute them.

To solve this I propose another question: “Is foo_action_postsave task currently being executed?”

Read more

José Antonio Perdiguero
Lead Big Data Engineer

Daemons with Celery I

April 13, 2016

We’re used to working with python so the examples I’ll use will be either pure python or python-based pseudocode. Python has a really simple and elegant syntax so it’s almost understandable for people who don’t know the language.

I’m going to explore a common problem in the backend of all kinds of applications that have heavy processing requirements, such as those that reprocess all links of each object from a class due to an update or recalculate an attribute that depends on huge functions, and walk through asynchronous solutions.

Read more

EUROPYTHON 2018 Asyncio learnings

October 23, 2018

EDINBURGH
Thanks to Ebury’s learning program, Héctor Álvarez and Jesús Gutiérrez were selected to attend EuroPython in Edinburgh.

EuroPython is a yearly conference in Europe that focuses on Python programming language and its ecosystem . This year’s sessions were held at the Edinburgh International Conference Center; an amazing building in the core of the city, just a stone’s throw away from the historical city of Edinburgh.

More than 1200 programmers and python lovers from 51 countries attended the event. With over 150 sessions in 7 tracks we were prepared to take away as much new information as possible.

From the outset, it was clear that the hottest topic in EuroPython was asynchronous programming. First integrated into 3.6 Python, and in various iterations since, it is still entirely possible to use Python without needing or even knowing about the asynchronous paradigm. However, if you are interested in the nuts and bolts of the tech involved, read on.

For the beginners out there, your central processing unit (CPU) follows a synchronous programming model, which means that things happen one by one. For example, when you create a function that performs a long-running action, it returns only when the action is finalised and it can return the result. Even when different programs that are run by your Operating System (OS), which is programmed synchronously, the OS manages them asynchronously. This is why multitask Operating Systems have been used for a long time.

The pitfall of asynchronous programming is that it’s difficult to know which of the coroutines has the execution time, and which coroutine spawned which; the event loop obviously knows but the programmer doesn’t.

The recently launched Python3.7 tries to solve that problem with inheritance of tags.

Under the asynchronous topic:

Asyncio in python 3.7 and 3.8 (Yury Selivanov)

Asyncio in production (Hrafn Eiriksson)

Asyncio in practice: We did it wrong (Lynn Root)

Here, I’ll briefly explain how the asynchronous programming works since Python 3.5.

As mentioned before, classic program code is executed in a single event line.In asynchronous programming, however, code is executed in a single loop. That loop is part of the code that orchestrates what’s executed when, inside the loop, there are a group of tasks—coroutines. Coroutines are defined with the reserved word async.

When a coroutine has been executed, it reports to the loop when it’s waiting for an external resource using the reserved word await.

When the loop detects that a coroutine is awaiting, it gives the execution time to the next coroutine, the loop then stores the memory state and where it’s waiting. When the external resource finally gives the response it fires a callback so knows that the coroutine is ready to keep working.

That’s  the theory, but now it’s time to look at the code:

 

import asyncio
import logging


logging.basicConfig(format='%(asctime)s %(message)s', datefmt='[%H:%M:%S]')
log = logging.getLogger()
log.setLevel(logging.INFO)


# define a coroutine
async def sleeper(name, delay):
    """This coroutine will wait for 2 seconds and then keep working."""
    log.info(f"{name}: START (wait for {delay}s)")
    await asyncio.sleep(delay)
    log.info(f"{name}: END (wait for {delay}s)")
    return name

if __name__ == '__main__':
    # create the loop
    loop = asyncio.get_event_loop()

    coroutine1 = sleeper('first coroutine', 2)
    coroutine2 = sleeper('second coroutine', 5)
    task1 = loop.create_task(coroutine1)
    task2 = loop.create_task(coroutine2)

    log.info("main: START run_until_complete")
    loop.run_until_complete(asyncio.wait([task1, task2]))
    log.info("main: END   run_until_complete")


Here at  Ebury, we don’t currently use asynchronous programming because Django (our framework) is not asynchronous. However, there are parts of the code that are slow (like sending an email that will delay for a second), and so in those cases we use a workaround—task executor named celery. If you want to know more about celery, follow this link: https://labs.ebury.rocks/?s=celery

Miscellaneous

Domain Driven Design, Robert SmallShire

Domain Driven Design is an approach to software development that emphasises high-fidelity modelling of the problem domain, which uses a software implementation of the domain model as a foundation for system design.

PEP 557 (data classes) versus the world (Guillaume Gelin)

Data classes are a very controversial feature, yet here it’s explained why they’re useful for us.

Getting started with mypy and type checking (Jukka Lehtosalo)

Mypy , is defined as: a static type checker for Python that aims to combine the benefits of dynamic (or “duck”) typing and static typing.

Static typing can help you find bugs faster with less testing and debugging. In large and complex projects, this can be a major time-saver.

Python decorators: Gift or poison? (Anastasiia Tymoshchuk)

A taxonomy of decorators: A-E (Andy Fundinger)

Python 2 is dead! Drag your old code into the modern age (Becky Smit)

What’s new in Python 3.7 (Stephane Wirtel)

Pythonic code vs. Performance (Łukasz Kąkol)

 

Adrián Matellanes
API Developer

QFS: Real-time streaming for trading with FIX and Redis

April 20, 2016

Ebury uses the Financial Information eXchange (FIX) Protocol to facilitate many of our electronic trading trends. FIX has become the language of global financial markets and is used extensively by banking trading platforms.

This non-proprietary, free and open standard is constantly being developed to support evolving business and regulatory needs, and is used by thousands of firms every day to complete millions of transactions.

Read more