Welcome to aiotasks’s documentation!¶

Project site	http://github.com/cr0hn/aiotasks
Author	Daniel Garcia (cr0hn) - @ggdaniel
Documentation	http://aiotasks.readthedocs.org
Last Version	1.0.0
Python versions	3.5 or above

aiotasks: A Celery like task manager that distributes Asyncio coroutines

What’s aiotasks¶

aiotasks is an asynchronous & distributed task queue / jobs queue, implemented as coroutines and based on Python asyncio framework.

Based on the Celery Task Queue ideas, but distributing coroutines and doing focus in performance, non-blocking & event-driven concepts.

aiotasks doesn’t pulling and doesn’t has active waiting for incoming jobs, instead use asyncio framework to suspend the execution until any new data are received by a new broker notification.

Note

aiotasks is still under development. Not as active as I would like (for time limitations), but the project is in active development.

If you want to contribute, take a look to the TODO.md file.

Contents¶

Install aiotasks¶

Installing aiotasks is very simple, depending on the release you want, you can install:

stable version (recomended).
development version.

Stable version¶

> python3.5 -m pip install aiotasks

Development version¶

> python3.5 -m pip install https://github
.com/cr0hn/aiotasks/archive/develop.zip

Quick Start¶

This is a quick start with aiotasks with the minimum information to start.

Note

You can find examples at project examples folder.

Running modes¶

You can run aiotasks as two ways:

Launching an aiotasks manager in an independent console / process (like Celery does), and then sending any tasks to aiotasks through the broker.
Running the standalone way: Launching the client and the server in a unique point and running both at the same time.

Running using the manager¶

Run the manager¶

> aiotasks -vvvv worker -A examples.launch_manager_tasks_and_launch_in_console

Send the tasks¶

> python examples/launch_manager_tasks_and_launch_in_console.py

Running standalone¶

> python examples/standalone_tasks_standalone.py

Defining tasks¶

This concept was ported from Celery. Defining any tasks is very simple, only need to decorate a function with task function.

from aiotasks import build_manager

manager = build_manager("redis://")

@manager.task()  # <-- DEFINITION OF TASK
async def task_01(num):  # <-- TASK SHOULD BE A **COROUTINE**
    print("Task 01 starting: {}".format(num))
    await asyncio.sleep(2, loop=manager.loop)
    print("Task 01 stopping")

Sending info to tasks¶

We can send tasks to the manager using methods:

Using delay method¶

from aiotasks import build_manager

manager = build_manager("redis://")

@manager.task()
async def task_01(num):
    await asyncio.sleep(0, loop=manager.loop)

async def generate_tasks():
    # Generates 5 tasks
    for x in range(5):
        await task_01.delay(x)  # <-- METHOD DELAY SEND A TASK

if __name__ == '__main__':
    manager.loop.run_until_complete(generate_tasks())

Using send_task¶

from aiotasks import build_manager, send_task

manager = build_manager("redis://")

@manager.task()
async def task_01(num):
    await asyncio.sleep(0, loop=manager.loop)

async def generate_tasks():
    # Generates 5 tasks
    for x in range(5):
        await send_task("task_01", args=(x, ))  # <-- SENDING A TASK

if __name__ == '__main__':
    manager.loop.run_until_complete(generate_tasks())

Sending info to tasks & wait for response¶

We can also send for a task job and wait for the response in a non-blocking mode:

from aiotasks import build_manager

manager = build_manager("redis://")

@manager.task()
async def task_01(num):
    await asyncio.sleep(0, loop=manager.loop)

async def generate_tasks():
    # Generates 5 tasks
    async with task_01.wait(x) as f:  # <-- NON-BLOCKING WAITING FOR RESPONSE
        print(f)

if __name__ == '__main__':
    manager.loop.run_until_complete(generate_tasks())

Backends¶

aiotasks supports various backends. All of them are event-driven queue systems.

Currently supported backends are:

Redis
Memory

Planned supported in near future:

RabbitMQ (AMQP)
ZeroMQ

Panned supported in more long future:

etcd
consul

How to specify a backend¶

The backends should be specified as a URI format. This is:

BACKEND_TYPE://USER:PASSWORD@IP_OR_HOST:PORT/ANY_OTHER_INFO

Redis¶

To configure the Redis backend we must to specify the BACKEND_TYPE as redis, following this format: redis://HOST:PORT/DB

Examples¶

Connect to localhost and default Redis options:

from aiotasks import build_manager

manager = build_manager("redis://")

...

Custom Redis server, with the information:

Redis password: mypassword.
Custom host: 10.0.0.1:
Redis database: 12.

from aiotasks import build_manager

manager = build_manager("redis://:mypassword@10.0.0.1/12")

...

Memory¶

Memory is a special backend type. It should be used for small workload systems or development environments.

This execution mode is useful to do small and local tasks. For example: If you’re using aiohttp and you only want to send an email in a background way, you can use the standalone way and the memory backend.

The BACKEND_TYPE type is memory and the format is: :memory://.

Example¶

from aiotasks import build_manager

manager = build_manager("memory://")

...

Architecture¶

TODO

Contributing¶

Contributors are welcome. You can find a list ot TODO tasks in the TODO.md at the project file.

All contributors will be added to the CONTRIBUTORS.md file.

Thanks in advance if you’re planning to contribute to the project! :)

API¶