# django-ox documentation
Every page of https://oxpull.com/django-ox/ in navigation order. The short index is at https://oxpull.com/django-ox/llms.txt.

---

# Source: https://oxpull.com/django-ox/

# django-ox

A database-backed worker backend for Django's Tasks framework
(`django.tasks`, Django 6.0+).

Django 6.0 ships the Tasks API but no production backend: the built-in
`ImmediateBackend` runs tasks inline and `DummyBackend` runs nothing.
django-ox stores background tasks in the database you already run and executes
them with a worker process. You get a durable queue with retries, priorities,
scheduling and a result store, and no broker to provision, secure, upgrade or
back up.

## One fewer service to run

A broker-based task queue adds a second datastore to your deployment. Redis or
RabbitMQ has to be provisioned, monitored, secured and upgraded, and it has to
be running before a single task executes. For an application that already
depends on a database, that is a full operational surface added for one
feature.

django-ox uses the database you already run. A deployment is your application,
a worker process, and one migration. Backups already cover the queue, because
the queue is a table, and there is no second datastore that can fail on its
own.

## Transactional enqueue

The queue lives in your database, so enqueueing a task is a single INSERT
on your default connection. That gives you a guarantee no broker-based
queue can offer: **the task and your data commit or roll back together.**

```python
from django.db import transaction

with transaction.atomic():
    order = Order.objects.create(...)
    send_confirmation.enqueue(order_id=order.pk)
    # If anything below raises, the order AND the task vanish together.
    charge(order)
```

With a broker, the enqueue leaves your process the moment you call it. If
the transaction then rolls back, a worker races to process an order that
does not exist. The standard workaround is wrapping every enqueue in
`transaction.on_commit()`, and remembering to, everywhere, forever. With
django-ox there is nothing to remember: a task enqueued inside
`transaction.atomic()` becomes visible to workers only when the
transaction commits, and disappears on rollback. There is no window where
business data exists without its task, or a task without its data.

Execution is at-least-once. Workers claim tasks atomically (`SKIP LOCKED`
on databases that support it, with a single-statement fast path on
PostgreSQL; an atomic compare-and-set elsewhere, including SQLite),
failed tasks retry with
exponential backoff, and a reaper returns tasks whose worker died to the
queue. Details in [Production](production.md).

## What happens when a worker dies

A worker that claims a task takes a lease on it, and the attempt is counted at
that moment. While the task runs, the worker renews the lease every
`LOCK_TIMEOUT / 3` seconds, so a slow task on a live worker is never reclaimed.
If the worker is killed, the lease goes stale. After `LOCK_TIMEOUT` (default
300 seconds) the reaper in any surviving worker takes the task back: to READY
if attempts remain, or to LOST if they are spent. LOST reads as `FAILED`
through the result API, so nothing waits forever on a worker that is not
coming back. The mechanics, and the one case worth knowing about, are in
[Production](production.md#the-lease).

## What you get

- Transactional enqueue, as above. No `on_commit` boilerplate.
- Retries with exponential backoff and the full traceback of every attempt.
- A reaper that reclaims tasks from dead workers, and a lease that keeps it
  away from live slow ones.
- Graceful drain on SIGTERM: in-flight tasks finish before the worker exits.
- Priorities (-100 to 100) and deferred tasks (`run_after`).
- [Recurring tasks](recurring-tasks.md): cron schedules declared in
  settings, no separate scheduler process.
- A result store: status, return value and errors readable through the
  standard `django.tasks` result API.
- A [prune command](configuration.md#ox_prune) to keep the table small.
- [Monitoring](monitoring.md): a queue-stats API, an `ox_health` command
  for probes and cron alerting, a Prometheus endpoint, structured log
  events, and an admin page with retry and discard.

## Install

Requires Python 3.12+ and Django 6.0+.

```
pip install django-ox
```

Add the app and point the Tasks framework at the backend:

```python
INSTALLED_APPS = [
    # ...
    "django_ox",
]

TASKS = {
    "default": {
        "BACKEND": "django_ox.backend.OxBackend",
    }
}
```

Create the tables:

```
python manage.py migrate django_ox
```

## Quickstart

Tasks are plain `django.tasks` tasks. django-ox adds nothing to learn on
the producer side.

```python
# myapp/tasks.py
from django.tasks import task


@task
def send_welcome_email(user_id): ...
```

Enqueue one:

```python
from myapp.tasks import send_welcome_email

result = send_welcome_email.enqueue(user_id=42)
```

Run a worker in a second terminal:

```
python manage.py ox_worker
```

Check on the result later:

```python
result.refresh()
result.status  # READY, RUNNING, FAILED, or SUCCESSFUL
result.return_value  # once SUCCESSFUL
result.errors  # per-attempt tracebacks, if any
```

That is the whole integration. Next steps:

- [Configuration](configuration.md) for every setting, option and command
  flag.
- [Recurring tasks](recurring-tasks.md) for cron schedules.
- [Production](production.md) for systemd units, scaling and shutdown
  semantics.

## Scope

The core is deliberately small: a durable queue, a worker, recurring
schedules, and monitoring, with nothing extra to operate. Design
decisions worth knowing before you commit:

- A queued task can be discarded before a worker claims it, and a failed
  one retried, from the admin or with `django_ox.actions`. `TASK_TIMEOUT`
  bounds how long any attempt may run; a particular running task cannot be
  interrupted on demand.
- Tasks are stored on the default database for the model; multi-database
  routing is not part of the current scope.
- Worker concurrency is a thread pool, which fits I/O-bound tasks. For
  CPU-bound work, run `--processes N --concurrency 1`, which is N worker
  processes under one supervisor. See
  [Production](production.md#threads-and-processes).

Batches and unique tasks are in [Oxpull Pro](pro.md), a paid add-on that is
not on sale yet; the waitlist is at <https://oxpull.com/>. Metrics stay in
this package: `django_ox.stats` and `ox_health` are free and stay free.

## License

BSD 3-Clause.

---

# Source: https://oxpull.com/django-ox/configuration/

# Configuration

Everything django-ox reads lives in the standard `TASKS` setting, plus
three management commands. A full entry with every option spelled out:

```python
TASKS = {
    "default": {
        "BACKEND": "django_ox.backend.OxBackend",
        "QUEUES": ["default", "emails"],
        "OPTIONS": {
            "MAX_ATTEMPTS": 3,
            "LOCK_TIMEOUT": 300,
            "BACKOFF_INITIAL": 5,
            "BACKOFF_MAX": 600,
            "TASK_TIMEOUT": None,
            "TASK_TIMEOUTS": {},
            "TASK_TIMEOUT_GRACE": 30,
            "SCHEDULES": {},  # see the Recurring tasks page
        },
    }
}
```

## The smallest working entry

Every option has a default. This is enough to run tasks:

```python
TASKS = {
    "default": {
        "BACKEND": "django_ox.backend.OxBackend",
    }
}
```

That gives you the `default` queue, three attempts, and a five second first
retry. Add options when you have a reason to.

## Backend entry

| Key | Default | Meaning |
| --- | --- | --- |
| `BACKEND` | required | `"django_ox.backend.OxBackend"`. |
| `QUEUES` | `["default"]` | Queue names tasks may be enqueued to. An empty list (`[]`) allows any queue name. Read by Django's Tasks framework itself. |
| `OPTIONS` | `{}` | Backend options, below. |

## OPTIONS

| Key | Default | Meaning |
| --- | --- | --- |
| `MAX_ATTEMPTS` | `3` | Executions a task gets before it is marked FAILED. An attempt is consumed when a worker claims the task, so a worker dying mid-run counts too and retries stay bounded. |
| `LOCK_TIMEOUT` | `300` | Seconds a RUNNING task's lock may go unrefreshed before the reaper takes the task back. A worker refreshes the lock every `LOCK_TIMEOUT / 3` seconds while it is executing, so this is a limit on how long a worker may be unresponsive, not on how long a task may run. |
| `BACKOFF_INITIAL` | `5` | Delay in seconds before the first retry. |
| `BACKOFF_MAX` | `600` | Ceiling on the retry delay, in seconds. |
| `TASK_TIMEOUT` | `None` | Seconds one attempt may run. `None` means no limit. At the deadline the worker raises `django_ox.exceptions.TaskTimeout` inside the task, on the task's own thread, and records the attempt as failed: retried on the usual backoff, or FAILED when attempts are spent. An async task is cancelled at the deadline instead. A sync task on a thread a coverage tool or debugger is watching is left alone, and `TASK_TIMEOUT_GRACE` is the whole enforcement for it. See [Task timeouts](production.md#task-timeouts). |
| `TASK_TIMEOUTS` | `{}` | Per-queue timeouts, `{"queue name": seconds}`. A queue in the mapping uses its own value instead of `TASK_TIMEOUT`; `None` as a value exempts that queue. Every key must be a queue named in `QUEUES`, unless `QUEUES` is `[]`. Per queue rather than per task because `django.tasks` gives a task no field a backend could read a timeout from, and a queue is its unit of routing. |
| `TASK_TIMEOUT_GRACE` | `30` | Seconds a timed-out attempt gets to stop. A thread still running after that is treated as stuck, which usually means it is in a call that never returns to Python, where the exception cannot land: the worker records the attempt as failed, stops claiming, drains its other tasks and exits with code 75 so its supervisor restarts it. A task that catches `TaskTimeout` has the same deadline to return or raise, and so does a task on a watched thread, where nothing was raised at all. |
| `SCHEDULES` | `{}` | Recurring task definitions. Documented on the [Recurring tasks](recurring-tasks.md) page. |

The retry delay after attempt *n* fails is
`BACKOFF_INITIAL * 2 ** (n - 1)`, capped at `BACKOFF_MAX`. With the
defaults: 5 s, 10 s, 20 s, 40 s, and so on up to 600 s. There is no
jitter.

## ox_worker

```
python manage.py ox_worker [options]
```

| Flag | Default | Meaning |
| --- | --- | --- |
| `--backend` | `default` | Backend alias from the `TASKS` setting. |
| `--queues` | all configured queues | Comma-separated queue names this worker processes. |
| `--concurrency` | `1` | Tasks executed concurrently, as a thread pool inside each worker process. |
| `--processes` | `1` | Worker processes to run. At `1` the command is the worker. Above `1` it supervises that many copies of itself, each a full worker with its own connections, lease renewal, reaper and `--concurrency` thread pool, so `--processes 2 --concurrency 4` runs eight tasks at once. See [Threads and processes](production.md#threads-and-processes). |
| `--interval` | `1.0` | Polling interval in seconds when idle. When tasks are in flight the worker wakes as soon as one finishes, so this does not bound throughput. |
| `--lock-timeout` | backend `LOCK_TIMEOUT`, or 300 | Seconds a RUNNING task's lock may go unrefreshed before the task is reclaimed. |

The command also honors Django's standard `-v/--verbosity`: at the default
verbosity it logs worker lifecycle and warnings to stderr, and `-v 2`
enables debug logging. `-v 0` attaches no log handler. With `--processes`
above 1 every flag is passed on to each worker process unchanged, including
`--settings` and `--pythonpath`, and each worker process is started the way
the supervisor was (`manage.py` by absolute path, or `python -m django`), so
the command works from any working directory.

Two intervals are derived rather than flagged:

- The reaper runs every `min(30, max(lock_timeout / 2, 1))` seconds.
- Lease renewal runs every `max(lock_timeout / 3, 0.1)` seconds, on its own
  thread, and keeps running until the last in-flight task has drained.
- Schedule dispatch (when `SCHEDULES` is configured) runs every
  `max(1, min(interval, 30))` seconds, about once a second at the default
  polling interval.

### Routing a queue to its own worker

Declare every queue on the backend, then give each worker a subset. Slow work
stops blocking fast work without a second backend or a second database.

```python
TASKS = {
    "default": {
        "BACKEND": "django_ox.backend.OxBackend",
        "QUEUES": ["default", "emails", "exports"],
    }
}
```

```
python manage.py ox_worker --queues default,emails --concurrency 4
python manage.py ox_worker --queues exports --concurrency 1 --lock-timeout 3600
```

The exports worker runs one task at a time and tolerates hour-long jobs. The
other worker keeps short work moving at four at a time. A queue with no worker
assigned to it accumulates tasks and never runs them, so make sure every queue
in `QUEUES` is covered by some worker.

### Tasks that run longer than the lock timeout

A long task is not by itself a problem. A worker refreshes the lock on the
tasks it is running every `LOCK_TIMEOUT / 3` seconds, so an hour-long task on
a healthy worker keeps its lease for the hour.

What `LOCK_TIMEOUT` bounds is how long a worker may stop refreshing before its
work is handed to somebody else. Set it above the longest pause you are
willing to tolerate from a worker: a long garbage-collection pause, a
throttled container, a slow database, a host that swapped. If a queue runs on
hardware that stalls, give it its own worker with its own timeout rather than
raising the global value and delaying recovery for everything else:

```
python manage.py ox_worker --queues exports --lock-timeout 7200
```

If you embed the worker programmatically, `django_ox.worker.Worker`
accepts `reap_interval`, `renew_interval`, `schedule_interval`,
`backoff_initial`, `backoff_max`, `task_timeout` and `task_timeout_grace`
keyword overrides, which have no flag; they win over the `OPTIONS` values.

## ox_prune

Finished task rows stay in the table until pruned; the queue table doubles
as the result store, and django-ox does not guess at your retention
needs. Run `ox_prune` on your own schedule (cron or a systemd timer;
there is an example unit on the [Production](production.md#pruning-on-a-timer)
page):

```
python manage.py ox_prune --older-than 7d
```

| Flag | Default | Meaning |
| --- | --- | --- |
| `--older-than` | `7d` | Minimum time since the task finished. Accepts `7d`, `24h`, `90m`, `45s`, or a plain number of seconds. |
| `--include-failed` | off | Also delete FAILED and LOST rows. By default they are kept, because they hold the per-attempt tracebacks and can be retried. |
| `--batch-size` | `1000` | Rows per DELETE statement, so pruning a large table never takes a long lock or builds a giant IN clause. Must be at least 1. |
| `--dry-run` | off | Report how many rows would be deleted without deleting any. |

Only SUCCESSFUL and DISCARDED rows (and, with `--include-failed`, FAILED and
LOST rows) whose `finished_at` is past the cutoff are deleted. READY and RUNNING rows are
never touched, whatever their age. Rows from the recurring-schedule tick
log are pruned with the same cutoff, always keeping each schedule's most
recent tick; that row anchors missed-tick recovery and deleting it would
make the schedule re-anchor. The latest tick row of a schedule that has
been removed from settings is kept by the same rule; such rows are
harmless and can be deleted by hand if unwanted. See
[Recurring tasks](recurring-tasks.md#missed-ticks).

## ox_health

A health check for cron alerting and container probes: exits 0 when
every enabled check passes, non-zero with a one-line reason otherwise.
With no flags it verifies only that the database answers.

```
python manage.py ox_health --max-backlog 1000 --max-age 600
```

| Flag | Default | Meaning |
| --- | --- | --- |
| `--queue` | all queues | Restrict the checks to one queue. |
| `--max-backlog` | off | Fail when more than this many READY tasks are eligible to run. Tasks deferred to a future `run_after` do not count. |
| `--max-age` | off | Fail when the oldest waiting task has waited longer than this many seconds since becoming eligible. |
| `--worker-timeout` | off | Fail when no worker has claimed a task within this many seconds, or no claim was ever recorded. |

Check semantics, probe examples, and guidance on which check fits which
alert are on the
[Monitoring](monitoring.md#health-checks-ox_health) page.

## System checks

`manage.py check` validates the setup:

- `django_ox.E001`: `django_ox` is missing from `INSTALLED_APPS`. The app
  itself registers these checks with Django, so a project that also lacks
  any other import of `django.tasks` gets no `E001` from `manage.py check`;
  the reliable symptom is `Unknown command: 'ox_worker'`, because the
  command ships with the app.
- `django_ox.E002`: a `SCHEDULES` entry is invalid (task path does not
  import, cron expression does not parse or can never fire, arguments not
  JSON-serializable, bad queue name or priority).
- `django_ox.E003`: the same schedule name is defined on more than one
  backend; schedule names must be unique across backends.
- `django_ox.E004`: `TASK_TIMEOUT`, a `TASK_TIMEOUTS` value or
  `TASK_TIMEOUT_GRACE` is not a positive, finite number of seconds, at most
  a thousand years (the first two may also be `None`, which means no limit;
  `float("inf")` does not), or `TASK_TIMEOUTS` is not a mapping keyed by
  queue name. Every bad value is reported in one run.
- `django_ox.E005`: a `TASK_TIMEOUTS` key names a queue that is not in
  `QUEUES`, so the entry would never apply.

The worker performs the same schedule and timeout validation at startup, so
a bad deploy fails loudly rather than skipping dispatches.

---

# Source: https://oxpull.com/django-ox/recurring-tasks/

# Recurring tasks

django-ox runs cron-style schedules with no separate scheduler process. You
declare them in settings, next to the backend they enqueue through, so they are
versioned and deployed with your code. The database holds a dispatch log and
nothing else. There are no schedule rows to edit by hand.

```python
TASKS = {
    "default": {
        "BACKEND": "django_ox.backend.OxBackend",
        "QUEUES": ["default", "emails"],
        "OPTIONS": {
            "SCHEDULES": {
                "nightly-report": {
                    "task": "reports.tasks.build_report",
                    "cron": "0 3 * * *",
                    "kwargs": {"full": True},
                },
                "warm-cache": {
                    "task": "core.tasks.warm_cache",
                    "cron": "*/15 * * * *",
                },
            },
        },
    }
}
```

Each tick enqueues an ordinary task. Workers claim it through the normal queue,
so retries, backoff, priorities and the result store all work as usual.

## Schedule keys

The dictionary keys are the schedule names. They must be non-empty strings of at
most 128 characters.

**Names must be unique across every backend, not just within one.** The dispatch
log is keyed by name alone, so two backends sharing a name would suppress each
other's ticks. A duplicate is rejected at worker startup and by
`manage.py check`, as `django_ox.E003`.

Each entry accepts exactly these keys. Anything else is rejected at startup:

| Key | Required | Meaning |
| --- | --- | --- |
| `task` | yes | Dotted path to a `@task` callable, e.g. `"reports.tasks.build_report"`. |
| `cron` | yes | Five-field cron expression or `@` shortcut, syntax below. |
| `args` | no | Positional arguments, as a list. Must be JSON-serializable. |
| `kwargs` | no | Keyword arguments, as a dict. Must be JSON-serializable. |
| `queue_name` | no | Queue override; defaults to the task's own queue. |
| `priority` | no | Priority override, -100 to 100. |

### Overriding the queue and priority

A schedule can put its task somewhere other than the task's own queue, which is
useful when a nightly job would otherwise sit behind interactive work:

```python
"SCHEDULES": {
    "nightly-export": {
        "task": "exports.tasks.rebuild",
        "cron": "0 2 * * *",
        "queue_name": "exports",
        "priority": -50,
        "kwargs": {"full": True},
    },
}
```

The queue must be listed in that backend's `QUEUES`, and some worker must be
processing it. Priority runs from -100 to 100, and lower runs later.

A schedule enqueues through the backend it is declared under. That holds even if
the task itself was declared against a different backend alias.

## Cron syntax

Five fields, in order: minute (0-59), hour (0-23), day of month (1-31), month
(1-12), day of week (0-7, where 0 and 7 both mean Sunday).

| Form | Example | Meaning |
| --- | --- | --- |
| Any | `*` | Every value. |
| Value | `30` | That value. |
| List | `1,15` | Any listed value. Elements can be ranges or steps. |
| Range | `9-17` | Every value in the range, inclusive. Descending ranges are rejected. |
| Step | `*/15`, `1-10/2` | Every Nth value across the range. |
| Value with step | `5/15` | From that value to the field maximum. In the minute field, `5/15` means 5, 20, 35, 50. |
| Month names | `jan`, `DEC` | Case-insensitive, three letters, month field only. |
| Weekday names | `sun`, `Mon-Fri` | Case-insensitive, three letters, day-of-week field only. |

Shortcuts work in place of a full expression: `@hourly`, `@daily` and
`@midnight`, `@weekly`, `@monthly`, `@yearly` and `@annually`.

### Common schedules

```python
"SCHEDULES": {
    # Every fifteen minutes.
    "warm-cache":     {"task": "core.tasks.warm_cache",    "cron": "*/15 * * * *"},
    # 03:00 every day.
    "nightly-report": {"task": "reports.tasks.build",      "cron": "0 3 * * *"},
    # Every weekday at 09:30.
    "weekday-digest": {"task": "core.tasks.digest",        "cron": "30 9 * * mon-fri"},
    # 00:00 on the 1st and 15th.
    "twice-monthly":  {"task": "billing.tasks.invoice",    "cron": "0 0 1,15 * *"},
    # Top of every hour.
    "hourly-sync":    {"task": "sync.tasks.pull",          "cron": "@hourly"},
    # 02:15 on the first of the month.
    "monthly-prune":  {"task": "core.tasks.prune_archive", "cron": "15 2 1 * *"},
}
```

Expressions that can never fire are rejected when the configuration loads. So
`0 0 30 2 *` (February 30th) is a startup error, not a schedule that silently
never runs.

### Day-of-month and day-of-week combine with OR

This follows vixie cron. When both fields are restricted, a day matches if
*either* one matches.

`0 0 1,15 * mon` fires on the 1st, on the 15th, and every Monday. It does not
fire only on Mondays that land on the 1st or 15th.

One deliberate divergence from vixie: a stepped star like `*/2` counts as
restricted here, which follows the croniter interpretation common in Python. So
`0 0 */2 * 1` fires on every odd day of the month *and* every Monday. Vixie
would require both.

### Timezones

Cron fields are wall-clock time in your project's timezone. With `USE_TZ = True`,
`0 3 * * *` means 03:00 local all year. With `USE_TZ = False` it is naive local
time.

Across a DST transition, a tick whose wall-clock time is missing or ambiguous is
resolved by zoneinfo fold handling. It shifts rather than raising. If a job must
not land inside a transition window, schedule it outside your zone's transition
hours.

**On the fall-back day a schedule can fire twice.** Take `30 1 * * *` in a zone
that replays 01:00 to 02:00: it fires once per pass. That is at most one extra
run per year, and it is consistent with the at-least-once delivery the rest of
the system already assumes. Hourly and more frequent schedules in the repeated
hour fire on each pass by design.

Changing `TIME_ZONE` to a zone behind the old one can also re-fire one
wall-clock label on the day you change it, because ticks are stored as distinct
UTC instants.

## Many workers, one tick

Every worker is also the scheduler. Alongside its polling, each one checks about
once a second whether a tick is due.

Coordination is a database unique constraint on (schedule name, tick time). It
works like this:

1. Every worker derives the same tick datetimes from the cron expression.
2. Dispatch wraps the tick-log `INSERT` and the task enqueue in one transaction.
3. When workers race the same tick, exactly one `INSERT` commits.
4. The losers hit the constraint and roll back, task row included.

So each tick fires exactly once, whatever the worker count, and dispatch keeps
working as long as one worker is alive. Scheduling is a property of the workers
you already run. There is nothing extra to deploy, monitor or fail over.

## Missed ticks

**If every worker was down when a tick passed, the most recent missed tick fires
once on recovery. Older ones are skipped.**

A nightly job due during an unlucky deploy still runs when workers return. A
weekend of downtime on a five-minute schedule does not replay hundreds of stale
runs.

**A new schedule never fires for times before it existed.** The first time a
worker sees a schedule with no history, it records the current tick as an anchor
and enqueues nothing. The schedule first fires at its next tick.

## Checking a schedule is live

`manage.py check` validates every schedule without starting a worker. It reports
a bad task path, an unparseable or never-firing cron expression, arguments that
are not JSON-serializable, and duplicate names across backends:

```
python manage.py check
```

To see what has actually dispatched, read the tick log:

```python
from django_ox.models import OxScheduleTick

OxScheduleTick.objects.filter(schedule_name="nightly-report").order_by(
    "-scheduled_for"
)[:5]
```

Each row is one dispatched tick. A row whose `task` is `None` is the anchor
written the first time a worker saw the schedule; it enqueued nothing and marks
the tick before the first real fire.

The recovery baseline is each schedule's most recent tick row, which is why
`ox_prune` always keeps it. See
[Configuration](configuration.md#ox_prune).

---

# Source: https://oxpull.com/django-ox/patterns/

# Common patterns

Worked examples for the things people actually reach for. Every snippet uses the
standard `django.tasks` API, so it works on any backend; the notes point out
where django-ox behaves differently from a broker.

## Send an email after signup

The common case, and the one where a database queue behaves differently from a
broker.

```python
from django.db import transaction
from django.tasks import task


@task
def send_welcome_email(user_id):
    user = User.objects.get(pk=user_id)
    send_mail("Welcome", "...", None, [user.email])


def register(request):
    with transaction.atomic():
        user = User.objects.create_user(...)
        send_welcome_email.enqueue(user.pk)
```

No `transaction.on_commit()` here. The enqueue is an `INSERT` on the same
connection, so the task becomes visible to workers only when the transaction
commits. If `create_user` is rolled back further up, the email task disappears
with it.

Pass the id, not the object. Arguments are stored as JSON, and a stale copy of a
model is a bug waiting to happen.

## Retry a flaky third-party call

Retries are automatic. Raise, and the worker schedules the next attempt with
exponential backoff.

```python
@task
def sync_to_crm(order_id):
    order = Order.objects.get(pk=order_id)
    response = httpx.post("https://crm.example.com/orders", json=order.payload)
    response.raise_for_status()  # a 5xx raises, so the task retries
```

Tune the envelope on the backend, not per task:

```python
"OPTIONS": {
    "MAX_ATTEMPTS": 5,
    "BACKOFF_INITIAL": 10,   # seconds before the second attempt
    "BACKOFF_MAX": 600,      # ceiling
}
```

Every attempt keeps its own traceback, so a task that failed four times shows
all four. After `MAX_ATTEMPTS` the task is `FAILED` and stays in the table:
`ox_prune` keeps failed rows unless you pass `--include-failed`.

## Answer a webhook fast

Do the minimum in the request, then hand off. The sender gets its `200` straight
away, however long the work behind it takes.

```python
@task
def process_payment_event(event_id): ...


@csrf_exempt
def stripe_webhook(request):
    event = WebhookEvent.objects.create(payload=json.loads(request.body))
    process_payment_event.enqueue(event.pk)
    return HttpResponse(status=200)
```

Both writes are in the same transaction, so you cannot acknowledge an event you
failed to record, or record one that never gets processed.

## Run a job and check on it later

`enqueue()` returns a result you can look up by id.

```python
result = build_report.enqueue(month="2026-08")
request.session["report_task_id"] = result.id
```

```python
from django.tasks import TaskResultStatus

result = build_report.get_result(request.session["report_task_id"])
result.refresh()

if result.status == TaskResultStatus.SUCCESSFUL:
    return redirect(result.return_value)
if result.status == TaskResultStatus.FAILED:
    return render(request, "report_failed.html", {"errors": result.errors})
return render(request, "report_pending.html", {"attempts": result.attempts})
```

`refresh()` re-reads from the database, so call it before checking status.
`status` is one of `READY`, `RUNNING`, `SUCCESSFUL` or `FAILED`, and
`is_finished` covers the last two. A task whose worker vanished without
reporting also reads as `FAILED`; [the reaper](production.md#the-reaper)
explains what that record contains. Return values are stored as JSON, so
return a URL or an id rather than a file or a model.

## Defer work to a specific time

Use `.using(run_after=...)`. It returns a copy of the task with that setting
applied.

```python
from datetime import timedelta
from django.utils import timezone

send_reminder.using(run_after=timezone.now() + timedelta(days=1)).enqueue(booking.pk)
```

Workers ignore the row until then. For anything on a repeating clock, use a
[schedule](recurring-tasks.md) instead of enqueueing the next one from inside
the task.

## Enqueuing many tasks at once

A loop of `enqueue()` calls is one `INSERT` per task. For a few dozen that is
fine; for a mailing run or a nightly fan-out it is the slow part of the
request. `django_ox.bulk.enqueue_many()` writes them in one statement per
thousand rows.

```python
from django_ox.bulk import enqueue_many

results = enqueue_many(
    send_digest,
    [((user.pk,), {}) for user in User.objects.filter(digest=True)],
)
```

Each element of the list is an `(args, kwargs)` pair, the same two values
`enqueue(*args, **kwargs)` passes to the backend. The return value is one
`TaskResult` per pair, in the order given. Queue, priority and `run_after`
belong to the task, so set them once with `.using(...)`:

```python
results = enqueue_many(
    send_digest.using(queue_name="emails", run_after=tonight),
    [((user.pk,), {}) for user in users],
)
```

The task and every argument are checked before the first row is written: a
queue the backend does not accept, a task bound to another backend, or an
argument that will not serialise to JSON raises with nothing inserted. The
rows go in one `INSERT` per 1,000 (SQLite caps the variables a statement can
bind) inside one transaction, so a call of 5,000 commits all 5,000 or none,
and inside your own `transaction.atomic()` it commits or rolls back with
the rest of your work, as `enqueue()` does.

This is a bulk insert and nothing more. Grouping the
tasks, reading their progress as one number and firing a callback when the
last one settles is a [batch](pro.md), in Oxpull Pro.

## Keep slow work off the fast queue

Give slow tasks their own queue and run a separate worker for it, so a batch of
report builds cannot delay password resets.

```python
@task(queue_name="reports")
def build_report(month): ...


@task(queue_name="emails", priority=50)
def send_password_reset(user_id): ...
```

```
python manage.py ox_worker --queues emails --concurrency 4
python manage.py ox_worker --queues reports --concurrency 1
```

Priority runs from -100 to 100, higher first, and applies within a queue rather
than across queues. List every queue you use in the backend's `QUEUES`, or set
`QUEUES: []` to accept any name.

## Clean up on a schedule

```python
"OPTIONS": {
    "SCHEDULES": {
        "expire-carts": {
            "task": "shop.tasks.expire_abandoned_carts",
            "cron": "*/30 * * * *",
        },
    },
}
```

Nothing extra to run: the workers you already have dispatch the ticks. Details
in [Recurring tasks](recurring-tasks.md).

Task rows are not deleted for you. Run `ox_prune` on your own schedule, from
cron, a systemd timer, or a django-ox schedule:

```
python manage.py ox_prune --older-than 7d
```

## Make a task safe to run twice

The one habit worth building. Execution is at-least-once, so a task retries both
when it raises and when its worker dies mid-run. Assume every task can run
again.

```python
from django.db import transaction


@task
def charge_order(order_id):
    with transaction.atomic():
        order = Order.objects.select_for_update().get(pk=order_id)
        if order.charged_at:
            return  # a previous attempt already did this
        charge(order, idempotency_key=f"order-{order.pk}")
        order.charged_at = timezone.now()
        order.save(update_fields=["charged_at"])
```

Guard on state you have written, not on a flag you set in memory. The row lock
serialises concurrent attempts, and the idempotency key covers the gap where the
charge succeeds but the transaction does not commit. Use one whenever the
external system offers it.

**Your task is not run inside a transaction.** The worker manages its own for
claiming and bookkeeping, but your function is called outside them, so
`select_for_update()` and anything else needing an open transaction must open
one, as above.

## Test without a worker

Point the test settings at Django's own backends. No django-ox tables, no worker
process.

```python
# runs tasks inline, so an assertion right after enqueue sees the effect
TASKS = {"default": {"BACKEND": "django.tasks.backends.immediate.ImmediateBackend"}}
```

```python
# records tasks without running them, for asserting what was enqueued
TASKS = {"default": {"BACKEND": "django.tasks.backends.dummy.DummyBackend"}}
```

Keep django-ox in the settings you use for integration tests, where the point is
to exercise claiming and retries for real.

## Not in the core

Batches and unique or deduplicated tasks are in [Oxpull Pro](pro.md), a paid
add-on that is not on sale yet. Metrics are in the free tier: `django_ox.stats`
and `manage.py ox_health` ship in the core. Chains and workflows are on the
Pro roadmap, undated.

---

# Source: https://oxpull.com/django-ox/migrating/

# Migrating to django-ox

Your task code stays the same. django-ox implements Django's `django.tasks` API,
so `@task` functions and `.enqueue()` calls do not change.

Three things do change: the backend in your `TASKS` setting, the worker command,
and the table the queue lives in.

Find your section below, then read [Switching over](#switching-over). That last
part is where migrations go wrong.

## From another `django.tasks` backend

Configuration only. Here it is with `django-tasks-db`, the most common one:

```python
# before
INSTALLED_APPS = ["django_tasks_db", ...]

TASKS = {
    "default": {
        "BACKEND": "django_tasks_db.DatabaseBackend",
        "QUEUES": ["default"],
    }
}
```

```python
# after
INSTALLED_APPS = ["django_ox", ...]

TASKS = {
    "default": {
        "BACKEND": "django_ox.backend.OxBackend",
        "QUEUES": ["default"],
    }
}
```

Run `python manage.py migrate django_ox` to create the table.

| | Before | After |
| --- | --- | --- |
| Worker | `manage.py db_worker` | `manage.py ox_worker` |
| Clean up finished rows | `manage.py prune_db_task_results` | `manage.py ox_prune --older-than 7d` |

Nothing else moves. Same decorator, same `.enqueue()`, same result API.

## From Celery

Celery needs a broker and its own workers. django-ox uses the database you
already have, so there is no broker to run.

```python
# before
from celery import shared_task


@shared_task
def send_confirmation(order_id): ...


send_confirmation.delay(order_id=42)
```

```python
# after
from django.tasks import task


@task
def send_confirmation(order_id): ...


send_confirmation.enqueue(order_id=42)
```

| Celery | django-ox |
| --- | --- |
| Broker URL (Redis, RabbitMQ) | none. The queue is a table on your default database. |
| `celery -A proj worker` | `manage.py ox_worker` |
| `celery -A proj beat` | nothing to run. Schedules go in `TASKS` and every worker dispatches them. See [Recurring tasks](recurring-tasks.md). |
| `.delay(...)`, `.apply_async(...)` | `.enqueue(...)` |
| `apply_async(countdown=..., eta=...)` | `run_after` |
| `autoretry_for`, `self.retry` | automatic. Tune with `MAX_ATTEMPTS`, `BACKOFF_INITIAL`, `BACKOFF_MAX`. |
| Result backend | the same table, read through the standard result API. |
| Flower | the [stats API, `ox_health`, the Prometheus endpoint and the admin page](monitoring.md) |

One difference in behaviour is worth reading before you switch. With a broker,
`enqueue` leaves your process immediately. If the surrounding transaction then
rolls back, a worker can pick up an order that no longer exists. The usual fix
is to wrap every call in `transaction.on_commit()`.

Here the enqueue is an `INSERT` on your own connection. It commits or rolls back
with the row it belongs to, so there is nothing to wrap.

Queues, priorities and `run_after` are what exist today. Celery's chains,
groups and chords, and routing across multiple brokers, are outside the
package; chains and workflows are on the [Oxpull Pro](pro.md) roadmap,
undated.

## From huey

Closer to django-ox than Celery is, since huey can already store tasks in
SQLite or Postgres. What changes is the API, and where schedules live.

```python
# before
from huey.contrib.djhuey import task, periodic_task
from huey import crontab


@task()
def send_confirmation(order_id): ...


@periodic_task(crontab(minute="0", hour="3"))
def nightly_report(): ...
```

```python
# after
from django.tasks import task


@task
def send_confirmation(order_id): ...


@task
def nightly_report(): ...
```

The schedule moves off the function and into settings:

```python
TASKS = {
    "default": {
        "BACKEND": "django_ox.backend.OxBackend",
        "OPTIONS": {
            "SCHEDULES": {
                "nightly-report": {
                    "task": "reports.tasks.nightly_report",
                    "cron": "0 3 * * *",
                },
            },
        },
    }
}
```

Schedules in settings deploy with your code, so there are no rows to edit by
hand. A typo fails at `manage.py check` instead of at dispatch time.

| huey | django-ox |
| --- | --- |
| `manage.py run_huey` | `manage.py ox_worker` |
| `@periodic_task(crontab(...))` | a `SCHEDULES` entry, same five-field cron syntax |
| `.schedule(delay=...)` | `run_after` |
| `retries`, `retry_delay` | `MAX_ATTEMPTS` and the backoff options |
| `huey.immediate` in tests | Django's `ImmediateBackend` or `DummyBackend` |

## Switching over

The two systems use different tables. Neither reads the other's rows. So if you
flip the setting and deploy, anything still queued in the old table has nothing
left to run it.

1. **Stop enqueueing to the old system.** Leave its workers running.
2. **Let it drain.** Watch until pending work hits zero. Check scheduled tasks
   too: a job due in six hours still counts.
3. **Deploy django-ox.** Run `migrate django_ox`, then switch `TASKS`.
4. **Start `ox_worker`** and check it picks up work. `manage.py ox_health` will
   tell you, and the worker logs every claim to the `django_ox` logger.
5. **Retire the old worker,** then its tables and broker.

No drain window available? Run both. Old workers keep serving the old table
while new work goes to django-ox. They cannot see each other's rows.

Both systems run tasks at least once, so your tasks should already be
idempotent. Worth confirming before you start rather than halfway through.

## Migrating away

Task functions are portable. django-ox adds nothing to the producer side, so
tasks stay ordinary `django.tasks` tasks and moving to another backend is a
settings change and a drain, run in the same order as above with the roles
reversed.

One behaviour does not travel, and it is worth deciding about on the way in
rather than on the way out. Enqueueing inside `transaction.atomic()` ties the
task to that transaction, so it disappears on rollback. A broker-based backend
cannot do this: the enqueue leaves your process the moment you call it. Code
that depends on a rollback removing a task will behave differently once the
queue lives in a broker, and it will do so quietly.

If you want to keep that option open, wrap enqueues in
`transaction.on_commit()`, the way a broker-based backend requires. django-ox
runs correctly either way, and the task is enqueued after the commit instead of
inside it. You give up the guarantee and keep the portability.

If you would rather have the guarantee, take it, and write the dependency down
somewhere the next person will find it.

---

# Source: https://oxpull.com/django-ox/choosing/

# Choosing a task backend

This page compares django-ox with the task queues a Django team is most likely
to shortlist: django-tasks-db, huey, Celery, django-q2, dramatiq and
procrastinate. Every cell about another project comes from that project's own
documentation, source or issue tracker, with the link in the footnotes and the
date it was read. Where a project's pages do not say, the cell says so rather
than guessing.

The comparison is about fit, not ranking. A team that already runs RabbitMQ and
needs to stop running tasks has a different answer from a team that wants one
fewer service. The last section says where django-ox is not the right fit.

## The table

| | django-ox | django-tasks-db | huey | Celery | django-q2 | dramatiq | procrastinate |
| --- | --- | --- | --- | --- | --- | --- | --- |
| `django.tasks` backend (Django 6) | Yes, native | Yes, native [^tdb-readme] | No. Issue #870 closed 2025-11-14 [^huey-870] | No. Issue #10062 closed 2026-02-03 [^celery-10062] | Pull request #315 open since 2026-02-04, last updated 2026-05-15 [^q2-315] | Not documented [^dq-guide] | Not documented; has its own Django integration [^proc-index] |
| Broker needed | None. The queue is a table in your database | None. Django ORM [^tdb-readme] | Redis, SQLite, PostgreSQL, file or memory storage [^huey-guide] | RabbitMQ, Redis or SQS (stable); Zookeeper, Kafka, Pub/Sub (experimental). SQL databases are result backends only [^celery-brokers]. Issue #5149, PostgreSQL as a broker, open since 2018-10-25 with 95 upvotes [^celery-5149] | Redis (default), IronMQ, SQS, MongoDB or Django ORM [^q2-brokers] | RabbitMQ or Redis [^dq-guide] | None. PostgreSQL is the queue [^proc-index] |
| Transactional enqueue | Yes. Enqueue is an INSERT on your connection; it commits or rolls back with your data | Not documented [^tdb-readme] | Not documented [^huey-guide] | No. Django's own docs name a background task as the case for `on_commit()` [^dj-oncommit] | Not documented [^q2-brokers] | No. Enqueue is a broker send [^dq-guide] | Not documented [^proc-django] |
| Retries and backoff | Exponential backoff, `MAX_ATTEMPTS`, `BACKOFF_INITIAL`, `BACKOFF_MAX`; every attempt's traceback kept | No retry option in the README or the worker flags [^tdb-readme] [^tdb-worker] | `retries`, `retry_delay`, `retry_backoff` [^huey-guide] | `autoretry_for`, `retry_backoff`, `retry_backoff_max` (default 600 s), `retry_jitter` [^celery-tasks] | `max_attempts` (default 0, meaning unlimited) and a `retry` interval; backoff not documented [^q2-configure] | Exponential backoff; `max_retries` default 20, `min_backoff` 15 s, `max_backoff` 7 days [^dq-guide] | Retry strategy per task [^proc-index] |
| Recurring schedules | Cron in settings; every worker dispatches, no scheduler process | None. Issue #259 open since 2026-08-10 [^tasks-259] | `periodic_task(crontab(...))` [^huey-guide] | `celery beat`, a separate process; "ensure only a single scheduler is running" [^celery-beat] | `Schedule` model, editable in admin; cron via croniter [^q2-schedules] | None built in; APScheduler recommended [^dq-cookbook] | `@app.periodic` [^proc-index] |
| Priorities | -100 to 100 | Yes [^tdb-backend] | Yes; on Redis needs 5.0+ and `PriorityRedisHuey` [^huey-guide] | 0 to 255 on RabbitMQ and Redis [^celery-calling] | Not documented [^q2-configure] | Per actor; lower number runs first [^dq-guide] | Yes [^proc-index] |
| Time limit on a running task | `TASK_TIMEOUT`, per queue in `TASK_TIMEOUTS`: `TaskTimeout` raised inside the task at the deadline, and a worker recycle when the thread does not stop. Two caveats leave the grace backstop as the whole enforcement: an interpreter with no facility for raising an exception in another thread, and a thread a coverage tool or debugger is watching. See [Production](production.md#task-timeouts) | Not documented in the README or the worker flags [^tdb-readme] [^tdb-worker] | `timeout` per task or per call; "a `TaskTimeout` is raised and returned to the caller via the result handle" [^huey-guide] | `soft_time_limit` raises `SoftTimeLimitExceeded` inside the task; `time_limit` terminates the process running it, which is replaced. "Time limits don't currently work on platforms that don't support the `SIGUSR1` signal" [^celery-workers] | `timeout`, default `None`: "the number of seconds a worker is allowed to spend on a task before it's terminated" [^q2-configure] | `time_limit` per actor, default 10 minutes, raises `TimeLimitExceeded`. "Time limits are best-effort. They cannot cancel system calls or any function that doesn't currently hold the GIL under CPython" [^dq-guide] | Not documented on the page checked [^proc-index] |
| Worker dies mid-task | Lease renewed every `LOCK_TIMEOUT / 3`; a task whose lease goes stale for `LOCK_TIMEOUT` is put back to READY, or marked LOST when attempts are spent. See [Production](production.md#the-reaper) | Issue #5, open since 2024-06-11: the task 'remains marked as "PROCESSING", and thus is never picked up for re-processing nor marked as completed / failed' [^tdb-5] | "tasks that are mid-execution are lost and will not be retried automatically" [^huey-guide] | `acks_late` re-delivers; the worker still acknowledges "if the child process executing the task is terminated" [^celery-tasks] | Issue #327, open since 2026-05-05: worker death not reported to the monitor, `MAX_ATTEMPTS` ignored [^q2-327] | Not stated on the pages checked [^dq-guide] [^dq-cookbook] | Heartbeat every 10 s; jobs stay in `doing` until a `retry_stalled_jobs` periodic task you define picks them up [^proc-stalled] |
| Health and metrics | `ox_health` command, `django_ox.stats`, a Prometheus endpoint (`/ox/metrics`), structured log events, admin retry and discard | Issue #44, container healthchecks, open since 2026-06-08 [^tdb-44] | Signals [^huey-guide] | Flower, a separate process, with Prometheus integration [^flower] | `qmonitor`, `qinfo`, `Stat` [^q2-monitor] | Prometheus middleware; not in the default middleware list [^dq-prom] | Statistics via events [^proc-index] |
| Databases | PostgreSQL, SQLite and MySQL 8 tested in CI; MariaDB 10.6+ untested | Any Django database [^tdb-readme] | Redis, SQLite, PostgreSQL, file, memory [^huey-guide] | Broker, not a database [^celery-brokers] | Any Django database through the ORM broker [^q2-brokers] | Broker, not a database [^dq-guide] | PostgreSQL 13+ [^proc-index] |
| Licence | BSD 3-Clause | BSD 3-Clause [^tdb-repo] | MIT [^huey-repo] | BSD 3-Clause [^celery-license] | MIT [^q2-pyproject] | LGPL 3.0 [^dq-repo] | MIT [^proc-repo] |

Async tasks: django-ox sets `supports_async_task`, so `async def` tasks
enqueue and run. Celery's most-upvoted open issue is #6552, "Support async
function", open since 2020-12-19 with 98 upvotes [^celery-6552].

## When not to use django-ox

- **You need to stop one chosen task while it runs.** django-ox bounds every
  attempt with `TASK_TIMEOUT`, discards a queued task and retries a failed
  one, but has no call that interrupts a particular running task on demand.
  Celery can revoke and terminate a running task, from Flower or the control
  API [^flower].
- **The queue must live on a different database from your models.** Tasks are
  stored on the default database. Multi-database routing is outside the
  current scope, and a separate queue database would also give up the
  transactional enqueue.
- **Throughput beyond what one database comfortably serves.** The
  [benchmarks](benchmarks.md) page gives measured numbers with the method.
  If your workload is above them, a broker-based queue is the right tool, and
  the cost is the second datastore.
- **Chains, groups and chords.** Not in django-ox. Batches are in
  [Oxpull Pro](pro.md), a paid add-on that is not on sale yet; chains and
  workflows are on the Pro roadmap, undated.
- **CPU-bound tasks in one process.** Worker concurrency is a thread pool.
  Run `ox_worker --processes N --concurrency 1` for N interpreters, or pick
  a queue with a process pool.

## Maintenance

Rows are re-checked each release. If a cell is out of date, open an issue with
the link that shows it, and it will be corrected in the next release.

[^tdb-readme]: https://github.com/RealOrangeOne/django-tasks-db README, checked 2026-08-23.
[^tdb-worker]: https://github.com/RealOrangeOne/django-tasks-db/blob/master/django_tasks_db/management/commands/db_worker.py, `add_arguments`, checked 2026-08-23.
[^tdb-backend]: https://github.com/RealOrangeOne/django-tasks-db/blob/master/django_tasks_db/backend.py, `supports_priority = True`, checked 2026-08-23.
[^tdb-5]: https://github.com/RealOrangeOne/django-tasks-db/issues/5, open, checked 2026-08-23.
[^tdb-44]: https://github.com/RealOrangeOne/django-tasks-db/issues/44, open, checked 2026-08-23.
[^tdb-repo]: https://github.com/RealOrangeOne/django-tasks-db, licence field, checked 2026-08-23.
[^tasks-259]: https://github.com/RealOrangeOne/django-tasks/issues/259, open, checked 2026-08-23.
[^huey-guide]: https://huey.readthedocs.io/en/latest/guide.html, checked 2026-08-23.
[^huey-870]: https://github.com/coleifer/huey/issues/870, closed, checked 2026-08-23.
[^huey-repo]: https://github.com/coleifer/huey, licence field, checked 2026-08-23.
[^celery-tasks]: https://docs.celeryq.dev/en/stable/userguide/tasks.html, checked 2026-08-23.
[^celery-brokers]: https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/index.html, checked 2026-08-23.
[^celery-workers]: https://docs.celeryq.dev/en/stable/userguide/workers.html, Time Limits, checked 2026-08-23.
[^celery-beat]: https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html, checked 2026-08-23.
[^celery-calling]: https://docs.celeryq.dev/en/stable/userguide/calling.html, checked 2026-08-23.
[^celery-5149]: https://github.com/celery/celery/issues/5149, open, checked 2026-08-23.
[^celery-6552]: https://github.com/celery/celery/issues/6552, open, checked 2026-08-23.
[^celery-10062]: https://github.com/celery/celery/issues/10062, closed, checked 2026-08-23.
[^celery-license]: https://github.com/celery/celery/blob/main/LICENSE, checked 2026-08-23.
[^flower]: https://flower.readthedocs.io/en/latest/features.html, checked 2026-08-23.
[^dj-oncommit]: https://docs.djangoproject.com/en/6.0/topics/db/transactions/#performing-actions-after-commit, checked 2026-08-23.
[^q2-brokers]: https://django-q2.readthedocs.io/en/master/brokers.html, checked 2026-08-23.
[^q2-configure]: https://django-q2.readthedocs.io/en/master/configure.html, checked 2026-08-23.
[^q2-schedules]: https://django-q2.readthedocs.io/en/master/schedules.html, checked 2026-08-23.
[^q2-monitor]: https://django-q2.readthedocs.io/en/master/monitor.html, checked 2026-08-23.
[^q2-315]: https://github.com/django-q2/django-q2/pull/315, open, checked 2026-08-23.
[^q2-327]: https://github.com/django-q2/django-q2/issues/327, open, checked 2026-08-23.
[^q2-pyproject]: https://github.com/django-q2/django-q2/blob/master/pyproject.toml, checked 2026-08-23.
[^dq-guide]: https://dramatiq.io/guide.html, checked 2026-08-23.
[^dq-cookbook]: https://dramatiq.io/cookbook.html, checked 2026-08-23.
[^dq-prom]: https://github.com/Bogdanp/dramatiq/blob/master/dramatiq/middleware/__init__.py, `default_middleware`, and `middleware/prometheus.py`, checked 2026-08-23.
[^dq-repo]: https://github.com/Bogdanp/dramatiq, licence field, checked 2026-08-23.
[^proc-index]: https://procrastinate.readthedocs.io/en/stable/, checked 2026-08-23.
[^proc-django]: https://procrastinate.readthedocs.io/en/stable/howto/django/basic_usage.html, checked 2026-08-23.
[^proc-stalled]: https://procrastinate.readthedocs.io/en/stable/howto/production/retry_stalled_jobs.html, checked 2026-08-23.
[^proc-repo]: https://github.com/procrastinate-org/procrastinate, licence field, checked 2026-08-23.

---

# Source: https://oxpull.com/django-ox/production/

# Production

The worker is a plain foreground process: `manage.py ox_worker`, run
under whatever supervises your other processes. It treats a lost database
connection as fatal rather than retrying blind, and relies on the
supervisor to restart it. Run it under `Restart=always` (as in the unit
below). This page covers systemd,
scaling, shutdown, the reaper, and monitoring.

## Running under systemd

```ini
# /etc/systemd/system/ox-worker.service
[Unit]
Description=django-ox worker
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=exec
User=app
Group=app
WorkingDirectory=/srv/myproject
Environment=DJANGO_SETTINGS_MODULE=myproject.settings
ExecStart=/srv/myproject/.venv/bin/python manage.py ox_worker --processes 2 --concurrency 4
Restart=always
RestartSec=5

# systemd sends SIGTERM on stop; the worker drains in-flight tasks and
# exits 0. Give the drain at least as long as your longest task before
# systemd escalates to SIGKILL. KillMode=mixed sends that SIGTERM to the
# supervisor alone, which forwards it once; the default of sending it to
# every process in the group would reach each worker twice, and a second
# signal is the force-exit.
KillSignal=SIGTERM
KillMode=mixed
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target
```

```
sudo systemctl enable --now ox-worker
journalctl -u ox-worker -f
```

There are two ways to put several worker processes on one host. The unit
above uses `--processes 2`: one unit, one supervisor, two workers, and one
place to set the flags. The other is a template unit (`ox-worker@.service`
with the same `[Service]` body and `--processes 1`), started as
`ox-worker@1`, `ox-worker@2`, and so on, which makes each worker its own
unit with its own journal entry and restart counter.

Use `--processes` unless you need to stop, restart or give flags to one
worker at a time. A supervisor that restarts a dead worker in a second, with
one command to edit, is the common case. The template unit is the right
shape when the workers differ, for instance one unit per queue with its own
`--lock-timeout`, and for that a queue flag per unit says more than a
process count.

## Running in containers

The worker is a foreground process that exits 0 on SIGTERM, so it needs no
special entrypoint. The setting that matters is the grace period: give the
runtime longer than your slowest task before it escalates to SIGKILL.

```yaml
services:
  worker:
    image: myapp:latest
    command: python manage.py ox_worker --processes 2 --concurrency 4
    stop_grace_period: 5m
    restart: unless-stopped
    depends_on:
      - db
    healthcheck:
      test: ["CMD", "python", "manage.py", "ox_health"]
      interval: 60s
      timeout: 15s
      start_period: 30s
```

Docker's default grace period is 10 seconds, which will kill a worker mid-task
and leave the reaper to clean up. `stop_grace_period` is the container
equivalent of `TimeoutStopSec`. On Kubernetes it is
`terminationGracePeriodSeconds` on the pod spec.

`--processes` inside one container, or one worker per container with the
replica count doing the scaling, both work. The runtime restarts a container,
the supervisor restarts a worker process, and each takes about a second. One
container per worker keeps the runtime's own health and restart accounting
per worker, which is worth having on an orchestrator; `--processes` keeps the
number of containers down on a single host.

With no flags, `ox_health` checks that the database answers, which is what a
per-container probe should test. Queue-wide checks belong in fleet alerting
rather than in a probe: see
[which check goes where](monitoring.md#health-checks-ox_health), and the
liveness probe example there for queues with steady traffic.

Run migrations before rolling workers, as an init container or a job, not from
the worker itself. Several workers starting at once would race the same
migration.

Roll every process before using a status the old version cannot read. 0.3.0
adds DISCARDED: a 0.2.1 process that reads a discarded row raises
`ValueError` from `get_result()` and `refresh()`, and its `ox_prune` cannot
delete the row. Migrate, finish the rollout, then discard. A rollback to
0.2.1 with discarded rows present keeps the crash until those rows are
deleted by hand (`DELETE FROM django_ox_oxtask WHERE status = 'DISCARDED'`);
reversing the migration does not remove them.

## Graceful shutdown

On SIGTERM or SIGINT the worker:

1. Stops claiming new tasks immediately.
2. Waits for in-flight tasks to finish, however long they take.
3. Closes its database connections and exits with code 0.

A second signal during the drain forces an immediate exit, code 130. Whatever
was running is abandoned mid-flight. The reaper on a surviving worker reclaims
it later, and it counts as a failed attempt.

One other exit code exists. A worker exits 75 when it recycles itself after
a task thread that its timeout could not stop; see
[Task timeouts](#task-timeouts). A process manager on `Restart=always` or
`Restart=on-failure` restarts it either way.

This maps directly onto rolling deploys: send SIGTERM, wait, start the new
version. The only tuning point is the process manager's kill escalation
(`TimeoutStopSec` above) relative to your longest task.

With `--processes` above 1, the signal goes to the supervisor, and SIGHUP
counts as well as SIGTERM and SIGINT. The sequence is:

1. First signal: the supervisor forwards SIGTERM to every worker process
   and waits for each to drain. It exits 0 when all of them did, otherwise
   with the first non-zero code. A worker process cannot act on a signal
   until it has installed its handler, which is after Django is imported,
   so a stop that lands in that window kills it outright. It had claimed
   no work, so the supervisor logs `worker_process_stopped_early` and
   still exits 0. A restart, or a deploy that rolls twice, is not a
   failure to report to the process manager.
2. Second signal: forwarded again, which is the force-exit on each worker.
   A worker that cannot act on it (stopped, stuck in a C call) gets five
   seconds, then SIGKILL, logged as `supervisor_killed_workers` at ERROR.
3. A third signal sends the SIGKILL at once.

Send the signal to the supervisor only. A worker that also receives the
terminal's copy of a Ctrl-C has seen two signals. That is why each worker
runs in its own process group, and why the systemd unit above sets
`KillMode=mixed`.

A worker whose supervisor dies without signalling it (SIGKILL, an OOM kill)
does not run on as an orphan. On Linux the kernel sends it SIGTERM the
moment the supervisor exits (`PR_SET_PDEATHSIG`), so it drains through its
ordinary signal path. Everywhere else, and on Linux in the window before
that flag is set, the worker notices within one poll interval that its
parent pid has changed, logs `worker_orphaned` at WARNING, drains and
exits.

## Scaling out

Run as many workers as you need, on as many hosts as you need, pointed at
the same database. No coordinator, no leader election. Two things make
concurrent workers safe:

- **Claiming is atomic.** On PostgreSQL, a claim is one
  `UPDATE ... WHERE id = (SELECT ... FOR UPDATE SKIP LOCKED) RETURNING`
  statement, so workers never block each other on the head of the queue.
  On other databases with `SKIP LOCKED` support (MySQL 8+), the worker
  uses `SELECT ... FOR UPDATE SKIP LOCKED` in a short transaction. Databases without it, SQLite included, fall back to an optimistic
compare-and-set UPDATE. That is atomic everywhere, but under contention workers
can retry against each other for the head of the queue.
- **Recurring schedules need no dedicated node.** Every worker dispatches;
  a unique constraint guarantees each tick fires once. See
  [Recurring tasks](recurring-tasks.md#many-workers-one-tick).

Workers can also be split by queue: run
`ox_worker --queues emails --concurrency 8` next to
`ox_worker --queues default` to isolate slow or bursty workloads.

### Threads and processes

`--concurrency N` is a thread pool inside one process. That fits the
common Django task profile: email, HTTP calls to third parties, ORM work.
For CPU-bound tasks the GIL makes threads the wrong tool; use
`--processes N` there, with `--concurrency 1`, and the same command on one
host gives you N interpreters.

```
python manage.py ox_worker --processes 4 --concurrency 1
```

Each process is a complete worker with its own database connections, lease
renewal and reaper, and its own worker id with the slot number on the end, so
`worker_ids` on a task row says which process ran it. Nothing is shared
across the processes except the database: the supervisor starts each one as a
fresh interpreter running `ox_worker --processes 1` with the same flags, so a
worker under the supervisor is the same code as a worker started by hand. The
supervisor itself never opens a database connection.

The children start the way the supervisor was started. `python
/srv/app/manage.py ox_worker --processes 2` from any working directory runs
`/srv/app/manage.py` again for each child; `django-admin` or `python -m
django` runs `python -m django` with `DJANGO_SETTINGS_MODULE` set in the
child's environment. `--settings` and `--pythonpath` are passed on, and the
children inherit the supervisor's working directory.

A worker process that exits, whatever the cause and whatever the code, is a
death: a crash, a kill, and a clean exit 0 all count the same, because a
worker is meant to run until told to stop. The supervisor restarts the slot
and logs `worker_process_restarted` with the exit code at WARNING. Its
in-flight tasks go through the ordinary lease path: the reaper on a
surviving process takes them back after `LOCK_TIMEOUT`. The restart policy
is per slot:

- The first restart comes after one second. Each further death within 60
  seconds of the slot's last start doubles the delay (1, 2, 4, 8, 16, 30
  seconds, capped at 30). A slot that has run for 60 seconds starts the
  sequence over at one second.
- More than five deaths of one slot inside one minute stops the supervisor:
  it logs `supervisor_restart_cap` at ERROR with the slot index, drains the
  other workers, and exits 1 whatever the children's own exit codes were, so
  a unit on `Restart=on-failure` restarts it too. A worker that cannot stay
  up hands the fault to the process manager and its restart policy rather
  than logging a restart a second forever.
- Because the count is per slot, every worker dying at once (a database
  restart, a deploy that changes a connection string) is one restart each,
  not a trip. Six workers that all die in the same second all come back a
  second later.

Two things `--processes` does not do. It does not run on Windows, where
there are no POSIX signals to forward; run one `ox_worker` per process there.
And it does not replace a process manager: the supervisor is a foreground
process that expects to be restarted itself, like the single worker.

## The lease

A worker that claims a task takes a lease on it: the row records who holds
it, when the lock was last refreshed, and a lease number that goes up by one
every time the task changes hands. Three things follow from that number, and
they are worth understanding together because they are what makes recovery
safe.

**The worker keeps its own lease alive.** While a task is executing, its
worker refreshes the lock timestamp on the rows it is running, one statement
per interval however many are in flight, and it keeps doing so through a
graceful drain. So a task that takes an hour does not look abandoned after
five minutes. The reaper reclaims work from workers that stopped reporting,
not from tasks that are merely slow.

**A finish write only lands while the lease still holds.** When a worker
records success, failure or a retry, the UPDATE carries the lease number it
was given at claim time. If the task was taken off it in the meantime, that
number no longer matches and the write is dropped rather than applied, and no
completion is signalled for it. This is arithmetic rather than timing: no
pause is long enough to get around it, so a task that finished cannot be put
back on the queue by a straggler.

**Timestamps come from the database.** With `USE_TZ` on, the lock time is
written by the database server and compared against the database server's
clock, so two hosts with drifting clocks do not produce false reclaims. If you
run workers on more than one host, that is the setting which gives them one
clock, and it is Django's default.

With `USE_TZ` off the worker's clock is used instead. A database's own clock
does not always match what these columns hold there: SQLite's is UTC while the
columns carry naive local time, and reading one against the other would make
`ox_prune --older-than` treat rows that finished seconds ago as hours old.
Under that setting, keep `TIME_ZONE` and the timezone your workers run in the
same, which is what Django assumes of it anyway.

### Task timeouts

`TASK_TIMEOUT` bounds how long one attempt may run. It is off by default.
With it set, the worker raises `django_ox.exceptions.TaskTimeout` inside the
task when the deadline passes:

- **A sync task** gets the exception on its own thread, at the next line of
  Python it executes. `finally` blocks run, an open `transaction.atomic()`
  rolls back, and the thread returns to the pool. The worker then drops the
  thread's database connections, since the exception may have landed inside
  the driver with a statement in flight, and records the outcome on a fresh
  one. The task may catch `TaskTimeout` to clean up and then re-raise it.
  Inside the task the exception is bare: `str(exc)` is empty and
  `exc.timeout` is `None`, and the worker fills both in when it records the
  attempt. A task that catches it and returns is allowed, and the attempt
  is recorded as whatever the task went on to do, provided it returns or
  raises within `TASK_TIMEOUT_GRACE`; one still running then is treated as
  a thread that did not stop, below. An exception raised while the task
  unwinds from `TaskTimeout` (a cleanup that fails, say) is recorded as the
  timeout, with that exception in the traceback. `raise ... from None`
  breaks that chain, and the attempt is then recorded as the exception it
  names, with no `task_timed_out` event.
- **An async task** is cancelled inside its event loop at the deadline. The
  coroutine sees `asyncio.CancelledError` at the `await` it was on, as any
  cancelled coroutine does, and should let it propagate; the worker records
  the attempt with `TaskTimeout`. `except TaskTimeout` inside an async task
  never fires, because nothing can raise another class at a running
  coroutine's `await`.

The attempt is recorded as failed with a `TaskTimeout` error that names the
timeout. The attempt was consumed at claim time, so the retry rule is the
ordinary one: back to READY on the backoff while attempts remain, FAILED
when they are spent. The worker logs `task_timed_out` at WARNING, then the
usual `task_retrying` or `task_failed`. `TaskTimeout` subclasses
`TimeoutError`, so code written for one treats it as one.

A long loop can check the clock instead of being interrupted between two
steps. `django_ox.deadline()` returns the attempt's deadline as a `datetime`,
and `django_ox.remaining()` the seconds left; both return `None` when no
timeout applies.

```python
import django_ox
from django.tasks import task


@task
def export(report_id):
    for chunk in chunks_of(report_id):
        left = django_ox.remaining()
        if left is not None and left < 5:
            return {"paused_at": chunk.offset}
        write(chunk)
    return {"done": True}
```

**A thread that does not stop.** The exception is delivered when the thread
next executes Python, so under a pool of CPU-bound threads it can lag the
deadline by a few multiples of the interpreter's 5 ms switch interval. A
thread blocked in a C call stays blocked until the call returns: a socket
read with no timeout, `time.sleep()`, a lock, a long statement waiting on the
database. The exception lands when the call returns, and if that is within
the grace the attempt is an ordinary timeout. `TASK_TIMEOUT_GRACE` (default
30 seconds) is how long the worker waits for the thread after the deadline.
A task that caught `TaskTimeout` and is still running then looks the same
from outside, and is treated the same way. If the thread is still running at
the grace, the worker:

1. Records the attempt as failed, with a `TaskTimeout` whose message says
   the thread did not stop within the grace, and moves the lease number in
   the same write, the
   way the reaper does when it takes a row off a worker that went quiet.
   The outcome the thread eventually reports is refused by that number.
2. Logs `task_stuck` at ERROR and `worker_recycling` at WARNING.
3. Stops claiming, drains its other in-flight tasks, and exits with code 75
   (`EX_TEMPFAIL`). The stuck thread dies with the process.

Under `--processes` (the unit above) the supervisor restarts the slot after
one second, logs `worker_process_recycled`, and does not count the exit
against the restart cap; systemd sees nothing. A single-process worker
under systemd comes back on `Restart=always`, and on `Restart=on-failure`,
since 75 is non-zero. A container runtime on `restart: unless-stopped` does
the same.

Between the stuck record and the process exit the task may run twice: its
retry is claimable the moment the record lands, and the stuck thread keeps
executing until the worker's other in-flight tasks have drained. The thread
cannot write its outcome to the row, but its side effects are real. That is
the at-least-once contract every task already lives under: write tasks to be
safe to run twice.

Put timeouts on sockets and HTTP clients where you can. A task that returns
to Python regularly is one the soft timeout stops cleanly; the recycle is the
backstop.

Set per-queue values where one number does not fit:

```python
"QUEUES": ["default", "exports", "webhooks"],
"OPTIONS": {
    "TASK_TIMEOUT": 60,
    "TASK_TIMEOUTS": {"exports": 3600, "webhooks": 10},
    "TASK_TIMEOUT_GRACE": 30,
},
```

A queue in `TASK_TIMEOUTS` uses its own value; `None` there exempts the
queue from the global limit. Every queue named there must be in `QUEUES`
(`django_ox.E005` otherwise), unless `QUEUES` is `[]`. A timeout longer than
`LOCK_TIMEOUT` is fine: the lease is renewed for as long as the task runs.

Timeouts use CPython's own facility for raising an exception in another
thread, which every supported Python has. On an interpreter without it, the
worker logs `timeouts_backstop_only` once at startup and enforces timeouts
by the grace backstop alone.

**Under a coverage tool or a debugger.** A tool that watches a thread runs a
callback between one bytecode and the next, and those callbacks hold locks
of their own. An exception raised into such a thread can land inside one,
leaving a lock held with nobody to release it. So the worker does not raise
into a watched thread. It asks on the task's own thread as it registers the
attempt, and two things count:

- a trace function on that thread (`sys.settrace`), which is what
  `coverage run` installs before Python 3.14, and what `pdb` and most
  debuggers install;
- a registered `sys.monitoring` tool with events enabled, which is what
  `coverage run` uses from Python 3.14 on.

A profile hook (`sys.setprofile`) is not consulted, so a sampling profiler
that installs one leaves timeouts alone.

For a **sync task** on a watched thread, `TaskTimeout` is not raised and the
grace backstop is the whole enforcement:

- A task that returns before `TASK_TIMEOUT` plus `TASK_TIMEOUT_GRACE` is
  recorded as whatever it did, however long it ran. There is no
  `task_timed_out` event, and nothing else says a deadline passed. With the
  default 30-second grace this is the common case.
- A task still running then is recorded as failed with `TaskTimeout`, and
  the worker recycles: `task_stuck` at ERROR, `worker_recycling` at WARNING,
  exit code 75. Every timeout that reaches the backstop costs a worker
  restart.

An **async task** is not affected. It is cancelled inside its event loop at
the deadline, watched or not.

The worker logs `timeouts_backstop_only` once, on the first attempt it
registers under the tool rather than at startup, with `reason=tracing_tool`
and `tracer` naming the mechanism. That line is the check: if it is in the
log, this is what is happening; if it is not, timeouts are being raised
inside tasks as usual.

A tool that starts watching a thread after that thread's attempt was
registered is not seen until the next attempt, since a thread's trace hook
cannot be read from outside it.

A test suite that measures coverage over its own task bodies sees this
shape. To run an attempt unwatched, take the calling thread's trace hook off
for the length of the call and put it back afterwards; a `sys.monitoring`
tool cannot be taken off a single thread.

## The reaper

Workers still die: OOM kills, node failures, `kill -9`. A dead worker stops
refreshing its lock, and the reaper picks the task up. It runs inside every
worker, on an interval derived from the lock timeout.

A RUNNING task whose lock has not been refreshed for `LOCK_TIMEOUT` (default
300 seconds, per-worker override `--lock-timeout`) is taken back, and what
happens next depends on whether the task has attempts left:

- **Attempts remaining.** The task goes back to READY and the lease number
  goes up, so the old worker cannot write to it again. This is the ordinary
  case, and it is a guess the system already absorbs: at-least-once execution
  means the task may run twice, which is why task bodies must be idempotent.
- **No attempts remaining.** The row is marked LOST. LOST means what it says:
  the worker holding this task stopped reporting and nobody observed how the
  attempt ended. The reaper does not record a failure, because it did not see
  one. It has watched a lock go quiet, and that is all it writes down.

The attempt was already consumed when the task was claimed, so a
crash-looping task cannot retry forever; it stops after `MAX_ATTEMPTS` like
any other task.

The reclaim is a compare-and-set on the lease number, so a reaper running
late cannot stomp a task that finished or was already reclaimed.

### What LOST looks like from the outside

`django.tasks` has four result statuses and django-ox does not add a fifth to
them. A LOST task reads as `FAILED` through `get_result()`, and
`result.errors` ends with a `TaskAbandoned` record whose text says the lease
was lost and the outcome was never observed. `is_finished` is true, so code
that waits for a result terminates instead of waiting for a worker that is
not coming back.

The row keeps the distinction the four statuses cannot carry. Its status
column is LOST rather than FAILED, `queue_stats()` reports it in its own
`lost` column, and `ox_prune --include-failed` treats it like a failed row
for retention. Like a failed row it can be retried or discarded, from the
admin or with `django_ox.actions`; see
[Retrying and discarding](monitoring.md#retrying-and-discarding).

One case is worth knowing about before it surprises you. If the worker
holding a LOST task was starved rather than dead, and it comes back and
records a success, the row becomes SUCCESSFUL and a caller reading it twice
sees `FAILED` and then `SUCCESSFUL`. Only that one execution can do this, and
only while the row is still LOST. It is the honest cost of giving a
four-valued API an answer for a task whose outcome nobody saw, and the
alternative, reporting it as still running forever, hangs every caller that
waits on it.

It takes two things at once: the task's attempts spent, and its lease allowed
to lapse. Renewal holds the lease for as long as the worker is answering, so
what reaches this state is a worker that went unresponsive for longer than
`LOCK_TIMEOUT` and then came back, which is the case you asked the reaper to
act on in the first place. If your callers cannot tolerate seeing it, raise
`LOCK_TIMEOUT` until a merely slow worker is never reclaimed; the cost is that
a genuinely dead one takes that much longer to notice.

### Tuning LOCK_TIMEOUT

Set `LOCK_TIMEOUT` above the longest gap you expect between a worker's lease
renewals, not above your longest task. Renewal runs every `LOCK_TIMEOUT / 3`
seconds, so two consecutive renewals can be missed before anything is
reclaimed. The value is really a statement about how long a worker may be
unresponsive before you want its work handed to somebody else: too low and a
paused or overloaded worker loses tasks it was going to finish, too high and
recovery after a real crash is slow.

Watch for `task_lease_lost` in the logs. It records an attempt whose result
was discarded because the lease had already been reclaimed, and a steady
trickle of it means the timeout is short relative to how long your workers
go unresponsive.

**Tasks must be idempotent.** Execution is at-least-once by design: a task
is retried both when it raises and when its worker dies mid-run. Write
task bodies so that running twice is harmless (upserts, idempotency keys,
"already sent?" checks).

## PostgreSQL, MySQL or SQLite

All three run the full worker suite in CI. Guidance:

- **PostgreSQL** is the production recommendation. It gets the
  single-statement `SKIP LOCKED` claim path, and it handles many workers
  and high write concurrency the way you would expect.
- **MySQL 8** claims with `SELECT ... FOR UPDATE SKIP LOCKED` in a short
  transaction and runs the full suite in CI on the oldest and newest Python
  and Django corners.
- **SQLite** is fine for development, tests, and small single-host
  deployments in the same situations where SQLite is fine as your Django
  database at all. Claiming uses the compare-and-set path and remains
  correct with multiple workers, but SQLite's single-writer nature makes
  many busy workers on one file a poor fit.

The queue lives in your default database, inside your existing backup and
migration story. That is the point: one system of record, one thing to
operate.

## Monitoring

Monitoring has a [dedicated page](monitoring.md). The operational
summary:

- **The table is the queue.** `django_ox.stats` exposes queue depth,
  backlog age, throughput and failure rate as plain functions. Backlog
  depth and backlog age are the two numbers worth alerting on.
- **`manage.py ox_health`** turns thresholds on those numbers into an
  exit code, for cron alerting and container probes.
- **The Prometheus endpoint.** Mounting `django_ox.urls` serves the same
  numbers as gauges at `GET /ox/metrics`.
- **Logs.** The worker logs to the `django_ox` logger: lifecycle at
  INFO, retries and reaper reclaims at WARNING, terminal failures and
  unhandled worker errors at ERROR, each with stable extra keys for JSON
  log handlers. Under systemd this lands in the journal.
- **Per-task forensics.** Each row keeps its attempts count, the id of
  every worker that ran it, timestamps for enqueue/start/finish, and the
  full traceback of every failed attempt.

## Pruning on a timer

Finished rows accumulate; prune them on a schedule sized to how long you
need results and tracebacks to stay queryable. With systemd:

```ini
# /etc/systemd/system/ox-prune.service
[Unit]
Description=Prune finished django-ox tasks

[Service]
Type=oneshot
User=app
WorkingDirectory=/srv/myproject
Environment=DJANGO_SETTINGS_MODULE=myproject.settings
ExecStart=/srv/myproject/.venv/bin/python manage.py ox_prune --older-than 7d
```

```ini
# /etc/systemd/system/ox-prune.timer
[Unit]
Description=Daily django-ox prune

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
```

Or from cron, or as a recurring task pointed at a small wrapper task of
your own. Flag reference on the
[Configuration](configuration.md#ox_prune) page. FAILED rows are kept by
default so tracebacks survive until you have looked at them; add
`--include-failed` once that is not needed.

---

# Source: https://oxpull.com/django-ox/monitoring/

# Monitoring

The queue and the result store are one database table. That means metrics are
just queries, with no agent or exporter process to run. There are four ways in:

- **`django_ox.stats`**, plain functions returning queue metrics.
- **`manage.py ox_health`**, the same numbers as an exit code, for cron
  alerting and container probes.
- **A Prometheus endpoint**, the same numbers as gauges, rendered by a view
  you mount where your scraper can reach it.
- **Structured log events** on the `django_ox` logger, with stable extra
  keys for JSON log handlers.

## Queue statistics

`django_ox.stats` is a small module of read-only functions. Each one is a single
ORM query over the task table. No extra state, no signals. Safe to call from a
request, a shell or a metrics collector, on every tested database.

```python
from datetime import timedelta

from django_ox import stats

stats.queue_stats()
# [QueueStats(queue_name="default", ready=3, running=1, failed=0, successful=214),
#  QueueStats(queue_name="emails", ready=0, running=0, failed=2, successful=560)]

stats.ready_count()  # tasks eligible to run right now
stats.oldest_ready_age()  # timedelta, or None when nothing waits
stats.throughput(timedelta(minutes=5))  # terminal outcomes per minute
stats.failure_rate(timedelta(minutes=5))  # 0.0 to 1.0, or None
stats.last_claim_age()  # time since a worker last claimed
```

| Function | Returns | Semantics |
| --- | --- | --- |
| `queue_stats()` | `list[QueueStats]` | Raw row counts per queue and status (`ready`, `running`, `failed`, `successful`, `lost`, `discarded`), one entry per queue with any rows. The `ready` column counts every READY row, including tasks deferred to a future `run_after`. `lost` counts tasks whose worker stopped reporting with no attempts left; see [the reaper](production.md#the-reaper). `discarded` counts tasks an operator closed without running; see [Retrying and discarding](#retrying-and-discarding). |
| `ready_count()` | `int` | READY tasks eligible to run now, mirroring the worker's dequeue predicate: deferred tasks do not count until `run_after` passes. This is the backlog number. |
| `oldest_ready_age()` | `timedelta \| None` | Age of the oldest task waiting to run, measured from when it became eligible (`run_after` when set, `enqueued_at` otherwise), so a task deferred by a week does not read as a week of backlog. |
| `throughput(window)` | `float` | Tasks reaching a terminal state (SUCCESSFUL or FAILED) per minute over the trailing window (default 5 minutes). |
| `failure_rate(window)` | `float \| None` | Fraction of terminal outcomes in the window that FAILED, or `None` when nothing finished. Retries still pending are not outcomes and do not count. |
| `last_claim_age()` | `timedelta \| None` | Time since any worker last claimed a task, or `None` if none ever was. This is claim activity, not a heartbeat: idle workers over an empty queue record nothing. |

Every function except `queue_stats()` accepts a `queue_name` keyword to
scope the metric to one queue.

**Alert on two numbers: backlog depth (`ready_count`) and backlog age
(`oldest_ready_age`).** Neither works alone. Depth looks fine while one poisoned
task starves the queue. Age looks fine during a flood of fresh work.

## Health checks: ox_health

`ox_health` turns thresholds on those metrics into an exit code. Zero when every
enabled check passes. Non-zero with a one-line reason on stderr when one fails.
With no flags, it checks only that the database answers.

```
python manage.py ox_health --max-backlog 1000 --max-age 600
```

| Flag | Default | Meaning |
| --- | --- | --- |
| `--queue` | all queues | Restrict the checks to one queue. |
| `--max-backlog` | off | Fail when more than this many READY tasks are eligible to run. Deferred tasks do not count. |
| `--max-age` | off | Fail when the oldest waiting task has waited longer than this many seconds since becoming eligible. |
| `--worker-timeout` | off | Fail when no worker has claimed a task within this many seconds, or no claim was ever recorded. |

On success it prints the measured values, which is useful in cron mail
and probe logs:

```
OK: backlog=3 oldest_age=12s last_claim_age=2s
```

Which check goes where:

- **`--max-backlog` and `--max-age` measure the whole queue.** Put them in
  fleet-level alerting, from cron or a monitoring agent. Do not put them in a
  per-worker probe: a shared backlog would fail every worker's probe and restart
  healthy workers without shifting the backlog.
- **`--worker-timeout` is the closest thing to a liveness check here.** Claiming
  is the only trace a worker leaves, so it works on queues with steady traffic
  and will false-alarm on ones that are legitimately idle.
- **For bursty queues, prefer `--max-age`.** It only fires when work exists and
  is not being picked up.

As a Kubernetes liveness probe on the worker container, for a queue with
steady traffic:

```yaml
livenessProbe:
  exec:
    command:
      ["python", "manage.py", "ox_health", "--worker-timeout", "300"]
  periodSeconds: 60
  timeoutSeconds: 10
  failureThreshold: 3
```

From cron, for alerting on the queue itself:

```
*/5 * * * * cd /srv/myproject && .venv/bin/python manage.py ox_health \
    --max-backlog 1000 --max-age 600 || /usr/local/bin/page-someone
```

## Prometheus

`django_ox.metrics` renders the stats functions in the Prometheus text
format, from the standard library alone. Mount the view and point a scrape
job at it:

```python
# urls.py
from django.urls import include, path

urlpatterns = [
    path("ox/", include("django_ox.urls")),  # GET /ox/metrics
]
```

```yaml
# prometheus.yml
scrape_configs:
  - job_name: django_ox
    metrics_path: /ox/metrics
    static_configs:
      - targets: ["app.internal:8000"]
```

The response is `text/plain; version=0.0.4`. A scraper that sends
`Accept: application/openmetrics-text` gets the OpenMetrics form of the
same text, and `HEAD` is answered for load-balancer checks. Each scrape is
five aggregate queries over the task table, however many queues there are.

**The endpoint has no authentication of its own.** The numbers are not
secret, but the queue names and the shape of your traffic are yours to keep,
so put the route behind the project's policy before it goes near the public
side of a load balancer. Two one-line ways:

```python
from django.contrib.auth.decorators import login_required
from django_ox.views import metrics

path("ox/metrics", login_required(metrics))  # session auth, for a human
```

```python
from django.http import HttpResponseForbidden


def from_prometheus(view):
    def guard(request, *args, **kwargs):
        if request.META["REMOTE_ADDR"] not in {"10.0.0.12"}:
            return HttpResponseForbidden()
        return view(request, *args, **kwargs)

    return guard


path("ox/metrics", from_prometheus(metrics))  # the scraper's address only
```

A network policy that only admits the scraper to the path does the same
job without code.

Every metric is a gauge, with one sample per queue that has any row:

| Metric | Labels | Value |
| --- | --- | --- |
| `django_ox_tasks` | `queue`, `status` | Rows by status, one of `ready`, `running`, `failed`, `successful`, `lost`, `discarded`. The same numbers as `queue_stats()`, so `ready` includes deferred tasks. |
| `django_ox_ready_tasks` | `queue` | `ready_count()`: READY tasks eligible to run now. The backlog number. |
| `django_ox_oldest_ready_age_seconds` | `queue` | `oldest_ready_age()` in seconds. Absent when nothing waits. |
| `django_ox_last_claim_age_seconds` | `queue` | `last_claim_age()` in seconds. Absent until a worker has claimed on that queue. |
| `django_ox_throughput_per_minute` | `queue` | `throughput()` over the default five-minute window. |
| `django_ox_failure_rate` | `queue` | `failure_rate()` over the same window, 0 to 1. Absent when nothing finished. |

There are no counters. The table is pruned, so a monotonic count of finished
tasks cannot be read from it; throughput and the failure rate are
trailing-window readings instead, and `rate()` in PromQL is not the tool
for them. Alert on `django_ox_ready_tasks` and
`django_ox_oldest_ready_age_seconds` the same way as on the functions.

The metric names and label names above are public API from the release that
ships them; see [API stability](stability.md). The help text is not.

### With an existing registry

A project that already runs `prometheus_client` (on its own or through
django-prometheus) can register the same numbers with its registry instead
of mounting a second endpoint. `prometheus_client` is not a dependency of
django-ox; `collector()` imports it when called and raises `ImportError`
when it is missing.

```python
from prometheus_client import REGISTRY

from django_ox import metrics

REGISTRY.register(metrics.collector())
```

### OpenTelemetry

django-ox does not ship an OpenTelemetry exporter. The stats functions fit
an observable gauge callback, so a project that already has an OTel meter
can read the queue through it in a few lines. This is the whole recipe;
nothing in django-ox imports `opentelemetry`.

```python
from opentelemetry import metrics as otel
from opentelemetry.metrics import Observation

from django_ox import stats


def observe_backlog(options):
    for row in stats.queue_stats():
        yield Observation(row.ready, {"queue": row.queue_name})


meter = otel.get_meter("django_ox")
meter.create_observable_gauge("django_ox.ready_tasks", callbacks=[observe_backlog])
```

The same shape reads `oldest_ready_age()` or `failure_rate()` per queue.

## Log events

The worker logs through the standard library logger named `django_ox`. No
logging dependency, no imposed format. Configure handlers and formatters in
`LOGGING` as usual.

Lifecycle events carry an `extra` dictionary with stable keys, so a JSON
formatter that serialises record attributes gets consistent fields to index.
The message text is not part of the contract. The keys are.

| Event | Level | When |
| --- | --- | --- |
| `worker_started` | INFO | The run loop starts. |
| `task_claimed` | DEBUG | A task was claimed from the queue. |
| `task_started` | DEBUG | Execution of an attempt begins. |
| `task_succeeded` | INFO | The task reached SUCCESSFUL. |
| `task_timed_out` | WARNING | An attempt ran past its `TASK_TIMEOUT` and is recorded as failed; a `task_retrying` or `task_failed` record follows. It counts timeouts recorded as failures, not deadlines that passed: a task that catches `TaskTimeout` and returns produces no event, and neither does a timeout on a worker logging `timeouts_backstop_only`, where the attempt ends in `task_stuck` or in whatever the task went on to do. |
| `task_stuck` | ERROR | A timed-out attempt's thread did not stop within `TASK_TIMEOUT_GRACE`. The attempt is recorded as failed and the worker is recycling. On a worker logging `timeouts_backstop_only` nothing is raised inside the task, so this is the ordinary end of a timeout there rather than a pathological one. |
| `worker_recycling` | WARNING | The worker stopped claiming after a stuck thread; it drains its other tasks and exits with code 75. One follows every `task_stuck`, so on a worker logging `timeouts_backstop_only` every timeout that reaches the backstop costs a worker restart. |
| `timeouts_backstop_only` | WARNING | Once per worker: `TaskTimeout` is not raised inside a running sync task, because the interpreter cannot raise an exception inside another thread (`reason=interpreter`, logged at startup) or a coverage tool or debugger is watching the worker's threads (`reason=tracing_tool`, logged on the first attempt registered under it). `TASK_TIMEOUT_GRACE` is the whole enforcement while it stands. See [Task timeouts](production.md#task-timeouts). |
| `task_retrying` | WARNING | An attempt failed with retries remaining. |
| `task_failed` | ERROR | The task reached FAILED, out of attempts. |
| `task_reclaimed` | WARNING | The reaper took a task back from a worker that stopped refreshing its lock. |
| `task_lease_lost` | WARNING | A worker finished an attempt whose lease had already been reclaimed, so its write was dropped and no result was signalled. |
| `lease_renew_failed` | WARNING | A lease renewal statement failed. The worker keeps going and tries again on the next interval. |
| `schedule_dispatched` | INFO | A recurring tick enqueued its task. |
| `worker_error` | ERROR | The execution wrapper itself raised (an internal worker error, not a task failure). |
| `worker_draining` | INFO | Shutdown began with tasks still in flight. |
| `worker_stopped` | INFO | The run loop exited. |
| `supervisor_started` | INFO | `ox_worker --processes N` started its worker processes. |
| `worker_process_restarted` | WARNING | A worker process exited on its own and is being restarted. |
| `worker_process_recycled` | WARNING | A worker process exited with code 75 after a stuck task thread and is being restarted. Not counted against the restart cap. |
| `worker_process_stopped_early` | INFO | A stop signal reached a worker process before it had finished starting, so it died on the signal rather than draining. It had claimed no work, and the supervisor does not count it as a failure. |
| `supervisor_restart_cap` | ERROR | More than five deaths of one slot in a minute; the supervisor is stopping with exit code 1. |
| `supervisor_killed_workers` | ERROR | Worker processes still running five seconds after the second stop signal were sent SIGKILL. |
| `supervisor_stopped` | INFO | Every worker process has exited. |
| `worker_orphaned` | WARNING | A worker process found its supervisor gone and is draining. |

| Key | Present on | Meaning |
| --- | --- | --- |
| `event` | all events | The event name from the table above. |
| `worker_id` | all worker events | Unique id of the worker emitting the record. With `--processes`, the slot number is the last part of the id. |
| `task_id` | task events | The task's UUID, as a string. |
| `task_path` | task events | Dotted path of the task function. |
| `queue` | task events | Queue name. |
| `attempt` | task events | Attempts consumed so far, including the current one. |
| `duration_ms` | `task_succeeded`, `task_retrying`, `task_failed`, `task_timed_out`, `task_stuck`, `task_lease_lost` | Wall-clock duration of the attempt, in milliseconds. |
| `timeout_s` | `task_timed_out`, `task_stuck` | The timeout that applied, in seconds. |
| `grace_s` | `task_stuck`, `timeouts_backstop_only` | `TASK_TIMEOUT_GRACE`, in seconds. |
| `reason` | `timeouts_backstop_only` | Why the backstop is the whole enforcement: `interpreter` or `tracing_tool`. |
| `tracer` | `timeouts_backstop_only` with `reason=tracing_tool` | How the worker's threads are being watched: `sys.settrace` when a trace function is installed, which does not say which tool installed it, or `sys.monitoring (NAME)` for a registered tool, which names itself. |
| `exception` | `task_retrying`, `task_failed` | Exception class name of the failure. |
| `status` | `task_reclaimed` | Status after reclaim: `READY` (requeued) or `LOST` (out of attempts). |
| `dropped_status` | `task_lease_lost` | Status the dropped write would have set: `SUCCESSFUL`, `FAILED` or `READY`. |
| `schedule` | `schedule_dispatched` | Schedule name from `SCHEDULES`. |
| `queues`, `concurrency` | `worker_started` | The worker's configuration. |
| `pending` | `worker_draining` | In-flight tasks at shutdown. |
| `processes` | `supervisor_started` | Worker processes the supervisor runs. |
| `worker_index`, `exit_code` | `worker_process_restarted`, `worker_process_recycled`, `supervisor_restart_cap` | Which slot exited and how. A negative code is the signal that killed it. |
| `delay` | `worker_process_restarted`, `worker_process_recycled` | Seconds until the slot is started again. |
| `task_id`, `exit_code` | `worker_recycling` | The stuck task that started the recycle, and the code the worker will exit with, 75. |
| `restarts` | `supervisor_restart_cap` | Deaths of that slot inside the window. |
| `worker_indexes` | `supervisor_killed_workers` | The slots that were killed. |
| `parent_pid` | `worker_orphaned` | The supervisor pid the worker was started under. |
| `exit_code` | `supervisor_stopped` | The code the supervisor exits with. |

`task_claimed` and `task_started` are DEBUG because they fire once per
attempt; run `ox_worker -v 2` (or set the logger to DEBUG) when you want
them. Everything a dashboard usually wants survives at INFO.

`task_lease_lost` should be rare. It means a worker went unresponsive long
enough for the reaper to take its task away, and the worker's own result was
dropped when it finally finished, because the row no longer belonged to it.
Treat a steady trickle as a signal that `LOCK_TIMEOUT` is short relative to
how long your workers stall, rather than as noise; the
[Production](production.md#tuning-lock_timeout) page covers the tuning.

`throughput()` and `failure_rate()` count SUCCESSFUL and FAILED rows only. A
LOST task is not an outcome, so it is in neither number; read the `lost`
column from `queue_stats()` for it.

## Monitoring recipes

- **Alerting.** Alert on `ready_count` and `oldest_ready_age` (via
  `ox_health` thresholds or the functions directly), and on
  `failure_rate` rising above your normal baseline. Throughput is better
  as a dashboard line than an alert: its healthy value depends entirely
  on offered load.
- **Prometheus.** Mount `django_ox.urls` and scrape `/ox/metrics`, or
  register `django_ox.metrics.collector()` with a registry you already run.
  Both are covered [above](#prometheus).
- **journald.** Under systemd, WARNING and above maps onto journal
  priorities, so `journalctl -u ox-worker -p warning` shows exactly
  retries, reclaims and failures. Pair it with `ox_health` in a timer for
  active checks.
- **Poisoned-task triage.** When `failure_rate` spikes, the rows have the
  forensics: filter FAILED rows and read `errors` (per-attempt
  tracebacks), `attempts` and `worker_ids` to see what died where. The
  admin page below shows the same fields, and the two actions close the
  loop once the cause is fixed.

## Retrying and discarding

Two operator actions live in `django_ox.actions`. Each is one
compare-and-set UPDATE on the row's status and lease number, so it either
moves the row from the state it read or does nothing and says so. Neither
touches a RUNNING row: that row belongs to the worker holding its lease,
and only the reaper takes a lease away.

```python
from django_ox import actions

actions.retry(result.id)  # True if the row was requeued
actions.discard(result.id)  # True if the row was closed
```

| Function | Accepts | Does |
| --- | --- | --- |
| `retry(result_id)` | FAILED, LOST | Sets the row back to READY for one more attempt, clears `run_after` so it is eligible at once, and raises `max_attempts` to `attempts + 1`. The count, `worker_ids` and every per-attempt traceback stay as they were, so the record still says what happened before. The lease number goes up, so a LOST row's last worker, if it is still alive somewhere, writes nothing over the retry. |
| `discard(result_id)` | READY, FAILED, LOST | Marks the row DISCARDED. A READY task that is discarded never runs; a discarded FAILED or LOST task is not retried. The attempt records stay. |

`retry_many(selection)` and `discard_many(selection)` make the same move
for a queryset or a list of ids. They run one conditional UPDATE per
thousand rows inside one transaction and return `(changed, skipped)`. The
admin actions use them, so a select-across of a hundred thousand rows is a
hundred statements, and either all of it lands or none does.

The actions write the table directly and send no `django.tasks` signal: a
discard finishes the result without `task_finished`, and a retry requeues
it without `task_enqueued`.

Both single-row functions return `False` for any other state, for an id
that is not in the table, and for a malformed id. `RUNNING` and `SUCCESSFUL` rows are never
matched. A retry that races a second retry of the same row, or a discard
that races a worker's claim, resolves to exactly one winner: the UPDATE
pins the lease number it read, and the loser matches zero rows.

A retried task is one more attempt, not a fresh set. If the new attempt
fails, the row is FAILED again with one more traceback, and can be retried
again. A task retried while its worker is still missing gets the same
treatment as any at-least-once task: make the body idempotent.

DISCARDED is the sixth value in the row's status column and reads as
`FAILED` through `django.tasks`, which has four statuses, so `is_finished`
is true and callers waiting on the result return. `queue_stats()` reports
it in its own `discarded` column, and `ox_prune` deletes discarded rows
with successful ones.

### The admin page

When `django.contrib.admin` is installed, django-ox registers the task
table with it. Nothing is added to a project without the admin. The
change list shows id, task path, queue, status, attempts, and the enqueue
and finish times, filters on status and queue, and searches by id and
path. The detail page is read-only and lays out every attempt's
traceback. The two actions, **Retry selected tasks** and **Discard
selected tasks**, call `retry_many` and `discard_many` on the selection and
report how many moved and how many were skipped for being in a state the
action does not accept.

The admin does not add, edit or delete rows. A hand-edited status would
bypass the lease, and a delete could take a row from under a running
worker; `ox_prune` is the way rows leave the table. The actions need the
`change_oxtask` permission; viewing needs `view_oxtask`.

---

# Source: https://oxpull.com/django-ox/benchmarks/

# Benchmarks

django-ox 0.1.0 against django-tasks-db 0.12.0 (the other database
backend for the Tasks API) on identical no-op workloads, PostgreSQL 16.
Full methodology, raw JSON with every sample, and per-process logs are in
the `benchmarks/` directory of the repository. Every run is reported;
nothing was discarded.

## Results

Final 3-run matrix, runs interleaved between backends, all runs shown,
measured on django-ox 0.1.0.

| Metric | django-ox (r1 / r2 / r3) | django-tasks-db (r1 / r2 / r3) |
| --- | --- | --- |
| Enqueue throughput (tasks/sec, higher better) | 657 / 528 / 557 | 447 / 450 / 454 |
| Enqueue latency in `transaction.atomic()`, p50 ms | 0.53 / 0.43 / 0.55 | 0.55 / 0.54 / 0.43 |
| End-to-end, 2,000 tasks, concurrency 1 (tasks/sec) | 90.1 / 88.1 / 90.1 | 69.2 / 69.9 / 69.6 |
| End-to-end, 2,000 tasks, concurrency 4 (tasks/sec) | 393 / 373 / 397 | 366 / 373 / 376 |

Reading: django-ox wins enqueue throughput and concurrency-1 end-to-end
in all three runs, and edges or ties concurrency 4. In-transaction enqueue
latency is a tie at roughly half a millisecond for both; run-to-run drift
on the machine is larger than the difference between the backends.

A diagnostic cell at a non-default `--interval 0.1` produced the same throughput
as the defaults: 87.8, 86.6 and 88.5 tasks/sec. So the poll interval does not
bound throughput. With tasks in flight, the worker wakes on task completion
rather than on the polling clock.

## Where the numbers come from

Two properties of the worker's claim path drive the end-to-end results:

- **On PostgreSQL, claiming a task is a single statement**:
  `UPDATE ... WHERE id = (SELECT ... FOR UPDATE SKIP LOCKED) RETURNING`,
  with all per-attempt bookkeeping folded into the same UPDATE. A full
  claim-and-execute cycle costs 2 SQL statements per task. On a network
  link where every round trip costs real milliseconds, statement count
  matters even more than it does on this localhost setup.
- **The worker wakes on task completion.** When executor slots are busy,
  the run loop waits on the in-flight futures rather than sleeping the
  poll interval, so `--interval` only governs how often an idle worker
  checks for new work.

Both properties above are proven by published cells:
the single-statement claim by the concurrency-1 rows, the completion wake-up by
the diagnostic cell. A regression in either shows up as a changed number rather
than a changed claim.

## How to read these numbers

- **No-op task bodies.** The tasks do nothing, so these numbers measure
  framework overhead only. Real tasks do work; with realistic bodies the
  differences here shrink as a fraction of total runtime. If your tasks
  average even 100 ms, both backends will feel identical.
- **Single machine, one day.** An Apple M1 Max running both the workers
  and the database; no controlled thermal or background-load environment.
  The numbers are indicative of relative behavior, not absolute claims.
- **Localhost database.** PostgreSQL 16 in Docker on the same machine,
  sub-millisecond round trips. Real deployments have network latency
  between app and database, which changes end-to-end numbers materially
  and increases the weight of per-task statement count.
- **The concurrency-4 comparison is imperfect by construction.**
  django-tasks-db's worker has no concurrency option, so its
  "concurrency 4" is four separate processes: four interpreters without a
  shared GIL, four Django boots inside the timed window. django-ox's is
  four threads in one process. This is the fairest available mapping, though
  still an approximation. (It also means django-ox wins that cell
  while sharing one interpreter.)
- **Small N.** 2,000 tasks and 500 latency samples separate the backends
  here but say nothing about p99+ tails or sustained load. Sustained load
  and worker-failure behavior are covered separately by the
  [soak and chaos run](#reliability-under-load) below.

## Reliability under load

Speed is not the product claim; the durability construction is. A
separate soak and chaos harness ran django-ox 0.1.0 for 21.5 minutes of
sustained mixed load on PostgreSQL 16: 37,802 tasks across three
scenarios, including 9 minutes in which a random worker was SIGKILLed
every 20 to 45 seconds (16 kills total, 39 interrupted claims). Every
task reached a terminal state, every interrupted claim was reclaimed and
re-executed inside the documented bound, retry counts stayed bounded on
every row, and zero tasks were lost or double-completed. Latency under
kill-chaos was within noise of the undisturbed baseline (p50 0.217 s vs
0.211 s).

The full report, including the harness design, every assertion, and the
caveats, is in
[`benchmarks/SOAK-2026-08-16.md`](https://github.com/oxpull/django-ox/blob/main/benchmarks/SOAK-2026-08-16.md).

## What to take from this

The shipped worker beats the reference implementation on every measured
cell or ties it, and holds its documented guarantees under sustained load
and repeated worker kills. But throughput on no-op tasks is not why you
choose django-ox. The claim is the durability construction: transactional
enqueue, at-least-once execution with a reaper, bounded retries with
per-attempt tracebacks. The benchmark exists to show that choosing those
semantics costs you nothing on worker performance.

---

# Source: https://oxpull.com/django-ox/stability/

# API stability

This page states what counts as django-ox's public API, how versions
change, and which Python and Django versions are supported. It is a
promise about compatibility, so you can pin `django-ox` with confidence.

## Public API

These are the supported surfaces. Changes to them are versioned and
announced in the [changelog](changelog.md). This page, not a module's
`__all__`, is the statement of what is public; today the two agree.

- **The backend path** `django_ox.backend.OxBackend`, referenced as a
  string in the `TASKS` setting, `QUEUES` beside `OPTIONS`, and every
  `OPTIONS` key it reads: `MAX_ATTEMPTS`, `LOCK_TIMEOUT`,
  `BACKOFF_INITIAL`, `BACKOFF_MAX`, `TASK_TIMEOUT`, `TASK_TIMEOUTS`,
  `TASK_TIMEOUT_GRACE`, and `SCHEDULES` (with its documented
  per-schedule keys).
- **The management commands** and their flags: `ox_worker`, `ox_prune`,
  `ox_health`. `ox_worker`'s exit codes: 0 after a drain, 130 on a forced
  exit, 75 when the worker recycles itself after a stuck task thread.
  Under `--processes`, the supervisor exits 0 when every worker drained
  or recycled, 1 when a slot hit the restart cap, and otherwise with the
  first other non-zero worker code.
- **The timeout helpers** `django_ox.deadline()` and `django_ox.remaining()`,
  callable from inside a task.
- **The metrics module** `django_ox.stats`: `queue_stats`, `ready_count`,
  `oldest_ready_age`, `throughput`, `failure_rate`, `last_claim_age`, the
  `QueueStats` dataclass, and `DEFAULT_WINDOW`, the trailing window the
  rate functions default to.
- **The Prometheus surface**: `django_ox.metrics.render_prometheus`,
  `render_openmetrics` and `collector`, the view `django_ox.views.metrics`,
  the `django_ox.urls` module with its `metrics` route name, and the metric
  names and label names listed on the [Monitoring](monitoring.md#prometheus)
  page. `METRIC_NAMES` is that list in code; `CONTENT_TYPE_PROMETHEUS` and
  `CONTENT_TYPE_OPENMETRICS` are the content types the view serves. A
  scraped name is a contract with every dashboard that reads it, so a
  rename is a breaking change. Help text is not part of the contract.
- **The actions module** `django_ox.actions`: `retry` and `discard`, their
  accepted states, and their return values; `retry_many` and
  `discard_many`, the selections they accept and their `(changed, skipped)`
  return. `RETRYABLE_STATUSES` and `DISCARDABLE_STATUSES` are those
  accepted states in code; `UPDATE_CHUNK_SIZE` is exported for reading and
  its value may change. The admin page that calls
  them is a convenience over this module; its layout is not a contract,
  the two action names are.
- **The bulk module** `django_ox.bulk`: `enqueue_many(task, calls)`, its
  `(args, kwargs)` call shape, the input-order return and the
  all-or-nothing write. `INSERT_CHUNK_SIZE` is exported for reading; its
  value may change.
- **The exceptions** `django_ox.exceptions.TaskAbandoned`, recorded against
  tasks whose worker stopped reporting with no attempts left (it records the
  lost lease, not a cause of failure), and `django_ox.exceptions.TaskTimeout`,
  a `TimeoutError` raised inside a task that ran past its `TASK_TIMEOUT` and
  recorded against the attempt.
- **The structured-log contract**: the event names and stable `extra` keys
  documented on the [Monitoring](monitoring.md) page.
- **The database schema** of `OxTask` and `OxScheduleTick`, evolved only
  through shipped migrations.
- **`django_ox.__version__`.**

The producer-side API is `django.tasks` itself (`@task`, `.enqueue()`,
`get_result()`); django-ox adds nothing there and follows Django's
contract.

### Not public

Everything else is an implementation detail and may change in any release
without notice. That covers the `django_ox.worker.Worker` internals, the cron
parser (`django_ox.cron`), the row-to-dataclass conversion
(`django_ox.results`), the schedule loader (`django_ox.schedules`), the
supervisor behind `--processes` (`django_ox.supervisor`) and the hidden
`--worker-index` flag it starts each child with, and any name starting
with an underscore. The exact SQL a claim emits and the
model's non-schema helper methods are not part of the contract.

## Versioning

django-ox follows [Semantic Versioning](https://semver.org/). Before 1.0,
the pre-1.0 rule applies:

- **0.x minor releases may contain breaking changes.** Any
  break to a public surface above is called out in the changelog under a
  `Changed` or `Removed` heading, with the migration step.
- **0.x.y patch releases are bug fixes only** and never break a public
  surface.

Pin accordingly: `django-ox~=0.3.1` accepts patch releases only;
`django-ox>=0.3,<0.4` accepts the current minor line.

Once 1.0 ships, breaking changes to the public API will require a major
version bump, in the usual SemVer way.

## Deprecation policy

When a public surface is going to be removed or changed incompatibly, and
a compatible path exists, it is deprecated before removal rather than
dropped outright:

- The deprecation is documented in the changelog and, where it can be,
  surfaced at runtime (a `DeprecationWarning` or a `manage.py check`
  message).
- A deprecated surface keeps working for **at least one full minor release**
  (pre-1.0) or one major release (post-1.0) before it is removed.

Security fixes are exempt. A surface that cannot be kept without leaving a
vulnerability open may change in a patch release. That is documented in the
changelog, and in a security advisory where relevant.

## Supported Python and Django

Each django-ox release is tested against the matrix below in CI, on SQLite
and PostgreSQL 16 across the grid and MySQL 8 on the oldest and newest
corners; these are the supported combinations.

| | Django 6.0 | Django 6.1 |
| --- | --- | --- |
| **Python 3.12** | tested | tested |
| **Python 3.13** | tested | tested |
| **Python 3.14** | tested | tested |

The support floor tracks Django's own: when a Python or Django version
reaches end of life upstream, a later django-ox minor release may drop it,
announced in the changelog. Databases: PostgreSQL, SQLite and MySQL 8 are
tested in CI. MariaDB 10.6+ uses the same claim path, since Django's own
floor guarantees `SELECT ... FOR UPDATE SKIP LOCKED` there, but it is not
part of the tested matrix.

---

# Source: https://oxpull.com/django-ox/pro/

# Pro

Everything documented on this site is free, open source (BSD 3-Clause), and
stays that way. The durable queue, transactional enqueue, retries, reaper,
graceful drain, priorities, deferred tasks, recurring tasks and pruning are
the free tier, permanently. Nothing that works today moves behind the paid
tier.

**Oxpull Pro** is a paid add-on for two problems that show up once a queue is
carrying real volume. Both are built and tested. It is not on sale yet: the
purchase and delivery path is still being set up, and the waitlist below is
how to hear when it opens.

## What Pro adds

- **Unique tasks.** Deduplicate at enqueue time, so the same job cannot be
  queued twice. The lock is written in the same transaction as the task, so
  the two commit or roll back together, and a lock whose task died is
  released rather than stranded.
- **Batches.** Enqueue a group, read its progress as a count, and fire a
  callback once every member has settled. Completion is computed by querying
  the task rows rather than by counting signals, so a worker dying mid-task
  cannot strand a batch: the reconciler picks it up on the next tick.

Both run on the databases the free tier tests in CI: SQLite, PostgreSQL and
MySQL 8. MariaDB 10.6+ takes the same claim path but is not part of the tested
matrix. Batches are tested to 100,000 members in a single batch.

## What Pro is not

Workflows and chains are on the roadmap, undated. Rate limiting, a web
dashboard and encrypted payloads are not in Pro and are not dated. Metrics
stay free: the stats API and the health command are in the open source
package and remain there.

## Delivery

Pro will install from a private package index using credentials issued per
company. There is no licence key and no runtime check. A licence check is one
more thing of ours that can break your production, so we did not build one.
The credential controls access to the index rather than to code you have
already installed, so if it lapses, what is deployed keeps running.

## Pricing

Planned at **$399 per year, per company**, flat. One licence to cover a whole
organisation and every environment, with a seven-day money-back period.

## Waitlist

If Pro would earn its keep in your deployment, join the waitlist and say which
of the two features matters to you. That ordering decides what gets built
after these.

[Join the Pro waitlist](https://oxpull.com/#waitlist){ .md-button }

---

# Source: https://oxpull.com/django-ox/agents/

# For AI assistants

Exact steps for setting up django-ox in an existing Django project, the facts
to get right while doing it, and how to prove it works. Machine-readable
copies: [llms.txt](llms.txt) (facts and links) and
[llms-full.txt](llms-full.txt) (every page of this site in one file).
Context7 library id: `/oxpull/django-ox`.

## Set up django-ox in this project

Requires Python 3.12+ and Django 6.0+. Check before installing:

```
python -c "import django, sys; print(django.__version__, sys.version.split()[0])"
```

Install:

```
pip install django-ox
```

or, with uv:

```
uv add django-ox
```

Edit `settings.py`:

```python
INSTALLED_APPS = [
    # ...
    "django_ox",
]

TASKS = {
    "default": {
        "BACKEND": "django_ox.backend.OxBackend",
    }
}
```

Create the table:

```
python manage.py migrate django_ox
```

Verify. The expected output is the second line:

```
python manage.py ox_health
OK: backlog=0 oldest_age=none last_claim_age=none
```

`manage.py check` also runs the django-ox system checks, so a bad schedule
or timeout option fails here, as `django_ox.E002` to `E005`, before anything
deploys.

Start a worker in its own process, next to the web server, under the same
supervisor:

```
python manage.py ox_worker
```

Settings with every option named, for when the defaults need changing:

```python
TASKS = {
    "default": {
        "BACKEND": "django_ox.backend.OxBackend",
        "QUEUES": ["default"],  # [] allows any queue name
        "OPTIONS": {
            "MAX_ATTEMPTS": 3,  # claims per task before FAILED
            "LOCK_TIMEOUT": 300,  # seconds a worker may stop renewing its lease
            "BACKOFF_INITIAL": 5,  # first retry delay, seconds; doubles each attempt
            "BACKOFF_MAX": 600,  # retry delay ceiling, seconds
            "TASK_TIMEOUT": None,  # seconds one attempt may run; None is no limit
            "TASK_TIMEOUTS": {},  # per-queue values, {"queue": seconds}
            "TASK_TIMEOUT_GRACE": 30,  # seconds a timed-out thread gets to stop
            "SCHEDULES": {},  # recurring tasks, see Recurring tasks
        },
    }
}
```

## Facts to get right

- `QUEUES` sits beside `OPTIONS`, not inside it. Inside `OPTIONS` it is
  ignored without warning; the symptom is `InvalidTask: Queue 'X' is not
  valid for backend.`
- Tasks are plain `django.tasks` tasks. `from django.tasks import task`,
  decorate with `@task`, call `.enqueue(...)`. Nothing is imported from
  `django_ox` in task code.
- The worker imports a task by its dotted path, so the module must be
  importable in the worker process and the worker runs the same code as the
  producer; nothing is registered and there is no autodiscovery. `async def`
  tasks run.
- Priority and deferral are Django API: `task.using(priority=N)` with N from
  -100 to 100, higher first, and `task.using(run_after=...)` with a timedelta
  or datetime.
- Tasks run only while `ox_worker` is running. It is a separate process.
- SIGTERM and SIGINT both drain and exit 0; a second signal forces an
  immediate exit with code 130.
- `enqueue()` is one INSERT on the default connection. Inside
  `transaction.atomic()` the task is visible to workers only after commit and
  is gone on rollback. Do not add `transaction.on_commit()` around it.
- Many calls of one task go through `django_ox.bulk.enqueue_many(task,
  [(args, kwargs), ...])`: one INSERT per 1,000 rows, one transaction, results
  in input order. Set queue, priority and `run_after` once with `.using(...)`.
- Execution is at-least-once. Write tasks to be safe to run twice: guard on
  state already in the database, not on a flag in memory.
- The task function runs outside any transaction. Open
  `transaction.atomic()` inside the task when it needs `select_for_update()`.
- An attempt is consumed at claim time, so a worker dying mid-run uses one.
  Retry delay after attempt n is `BACKOFF_INITIAL * 2 ** (n - 1)`, capped at
  `BACKOFF_MAX`.
- `LOCK_TIMEOUT` bounds an unresponsive worker, not task length. The lease is
  renewed every `LOCK_TIMEOUT / 3` seconds while the task runs.
- The worker polls; `--interval` (default 1.0 s) is the idle sleep, so a task
  starts within one interval of its commit. There is no LISTEN/NOTIFY.
- `TASK_TIMEOUT` bounds one attempt. At the deadline `TaskTimeout` is raised
  inside the task on its own thread (an async task is cancelled) and the
  attempt is recorded as failed and retried on the backoff. A thread that
  has not stopped `TASK_TIMEOUT_GRACE` seconds later is recorded as failed
  and the worker exits 75 so its supervisor restarts it. Per-queue values go
  in `TASK_TIMEOUTS`; there is no per-task value. `django_ox.remaining()`
  reads the seconds left from inside a task. On a thread a coverage tool or
  a debugger is watching (a `sys.settrace` hook, or a `sys.monitoring` tool
  with events enabled) nothing is raised inside a sync task: the worker logs
  `timeouts_backstop_only`, a task that returns within `TASK_TIMEOUT_GRACE`
  is recorded as whatever it did, and one still running then is recorded as
  failed and recycles the worker. An async task is cancelled at the deadline
  either way.
- Claiming: one `UPDATE ... SKIP LOCKED ... RETURNING` statement on
  PostgreSQL; `SELECT ... FOR UPDATE SKIP LOCKED` on MySQL 8+; an atomic
  compare-and-set UPDATE on SQLite and other databases without `SKIP LOCKED`.
- `--concurrency N` is a thread pool in one process. `--processes N` runs N
  such workers under one supervisor; a worker process that dies is restarted
  after one second, doubling to 30 s, and more than five deaths of one slot
  in a minute stops the supervisor with exit 1. CPU-bound work wants
  `--processes N --concurrency 1`.
- Recurring tasks go in `OPTIONS["SCHEDULES"]`. Every worker dispatches them;
  there is no scheduler process to start.
- `ox_prune --older-than 7d` deletes finished rows; FAILED rows stay unless
  `--include-failed`. READY and RUNNING rows are never deleted.
- `path("ox/", include("django_ox.urls"))` mounts `GET /ox/metrics`, the
  queue stats as Prometheus gauges. It has no authentication of its own;
  wrap it with `login_required` or restrict it by network.
- Run `migrate` before rolling workers, not from the worker.
- `django_ox.actions.retry(result_id)` requeues a FAILED or LOST task for
  one more attempt. `django_ox.actions.discard(result_id)` closes a READY,
  FAILED or LOST task without running it. Neither touches a RUNNING task.
  With `django.contrib.admin` installed, the task table appears in the admin
  with the same two actions.
- A particular running task cannot be interrupted on demand; `TASK_TIMEOUT`
  bounds every attempt. Tasks live on the default database.
- In tests use `django.tasks.backends.immediate.ImmediateBackend` or
  `django.tasks.backends.dummy.DummyBackend` for `TASKS`.
- Batches and unique tasks are in [Oxpull Pro](pro.md), a paid add-on that
  is not on sale yet. `django_ox.stats` and `ox_health` are in django-ox.

## How to verify it works

Define a task in any installed app:

```python
# myapp/tasks.py
from django.tasks import task


@task
def add(a, b):
    return a + b
```

Enqueue one from `python manage.py shell`:

```python
>>> from myapp.tasks import add
>>> result = add.enqueue(1, 2)
>>> result.status
TaskResultStatus.READY
```

Start `python manage.py ox_worker` in another terminal. The worker logs
`Worker <id> starting: queues=['default'] concurrency=1 poll=1.0s schedules=0`
to stderr, then `Task id=<id> path=myapp.tasks.add succeeded in <n>ms`.
With `DEBUG = True`, Django's own `Task id=... state=RUNNING` and
`state=SUCCESSFUL` lines appear between them.

Back in the shell:

```python
>>> result.refresh()
>>> result.status
TaskResultStatus.SUCCESSFUL
>>> result.return_value
3
```

`python manage.py ox_health --worker-timeout 60` now exits 0 and reports a
recent `last_claim_age`. Stop the worker with Ctrl-C; it drains and exits 0.

## Links

- [llms.txt](llms.txt): the facts above with links, in the llms.txt shape.
- [llms-full.txt](llms-full.txt): the whole site in one file.
- [Configuration](configuration.md), [Production](production.md),
  [Monitoring](monitoring.md), [Common patterns](patterns.md).
- Context7: `/oxpull/django-ox`. Source:
  [github.com/oxpull/django-ox](https://github.com/oxpull/django-ox).

---

# Source: https://oxpull.com/django-ox/changelog/

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.3.1] - 2026-08-24

### Fixed

- `ox_worker --processes N` exits 0 when a stop signal arrives while a
  worker process is still starting up. A worker cannot act on a signal
  until it has installed its handler, and that is after Django has been
  imported. A signal landing before then killed the worker outright, and
  the supervisor reported that worker's 143 as its own exit code. A unit
  on `Restart=on-failure` reads that as a fault and starts the service
  again. The worker had claimed no work, so there is nothing to report:
  the supervisor now logs `worker_process_stopped_early` and exits 0.

### Changed

- A worker enforces `TASK_TIMEOUT` on a sync task through the grace backstop
  alone while a coverage tool or a debugger is watching the thread the
  attempt runs on. Two things count: a trace function (`sys.settrace`, which
  `coverage run` installs before Python 3.14, and which `pdb` and most
  debuggers install), and a registered `sys.monitoring` tool with events
  enabled (which `coverage run` uses from Python 3.14 on). Nothing is raised
  inside the task. A task that returns within `TASK_TIMEOUT` plus
  `TASK_TIMEOUT_GRACE` is recorded as whatever it did, however long it ran,
  with no `task_timed_out` event; one still running then is recorded as
  failed and recycles the worker. An async task is cancelled at its deadline
  either way, and a profile hook (`sys.setprofile`) is not consulted.
- The worker logs `timeouts_backstop_only` once under such a tool, on the
  first attempt it registers rather than at startup. That event now carries
  `reason` (`interpreter` or `tracing_tool`) and, for a tool, `tracer`:
  `sys.settrace`, or `sys.monitoring (NAME)`.

## [0.3.0] - 2026-08-24

### Upgrading

- Run `python manage.py migrate django_ox`, and roll every process, web and
  worker, to 0.3.0 before the first discard. This release adds the
  DISCARDED status: a 0.2.1 process that reads a DISCARDED row raises
  `ValueError` from `get_result()` and `refresh()`, and 0.2.1's `ox_prune`
  cannot delete such rows. Rolling back with DISCARDED rows present keeps
  that crash until the rows are removed by hand
  (`DELETE FROM django_ox_oxtask WHERE status = 'DISCARDED'`); reversing
  the migration does not remove them.

### Added

- `TASK_TIMEOUT`, `TASK_TIMEOUTS` and `TASK_TIMEOUT_GRACE` backend options, a
  limit on how long one attempt may run. At the deadline the worker raises
  `django_ox.exceptions.TaskTimeout` inside the task, on the task's own
  thread, so `finally` blocks run and an open `transaction.atomic()` rolls
  back; an async task is cancelled inside its event loop instead. The attempt
  is recorded as failed with the `TaskTimeout` error and retried on the usual
  backoff, or marked FAILED when attempts are spent. A thread that has not
  stopped `TASK_TIMEOUT_GRACE` seconds later (default 30) is treated as
  stuck: the worker records the attempt as failed, moves the lease number so
  the thread can write nothing to the row, stops claiming, drains its other
  tasks and exits with code 75, which `--processes` restarts without counting
  it against the restart cap. `TASK_TIMEOUTS` maps a queue name to its own
  value, and a key that is not in `QUEUES` fails `manage.py check` as
  `django_ox.E005`; a bad value fails it as `django_ox.E004`. Off by default.
  `django_ox.deadline()` and `django_ox.remaining()` read the attempt's
  deadline from inside a task. `TaskTimeout` subclasses `TimeoutError`. New
  log events: `task_timed_out`, `task_stuck`, `worker_recycling`,
  `worker_process_recycled` and `timeouts_backstop_only`.
- `ox_worker --processes N`. Above 1, the command supervises N copies of
  itself, each a full worker with its own database connections, lease
  renewal, reaper and `--concurrency` thread pool, so `--processes 2
  --concurrency 4` runs eight tasks at once. Every worker id ends in its slot
  number. SIGTERM, SIGINT or SIGHUP to the supervisor is forwarded once and
  the supervisor exits 0 when every worker drained, or with the first
  non-zero code; a second signal is the force-exit, and a worker that has
  not exited five seconds later is SIGKILLed. A worker process that dies,
  by any exit code, is restarted with `worker_process_restarted` at
  WARNING, after one second and then with a doubling delay up to 30
  seconds; more than five deaths of one slot in a minute stops the
  supervisor with exit code 1 and `supervisor_restart_cap` at ERROR. A
  worker whose supervisor dies drains and exits (`worker_orphaned`). The
  children are started the way the supervisor was, `manage.py` by absolute
  path or `python -m django`, with `--settings` and `--pythonpath` passed
  on, so the command works from any working directory. `--processes 1`, the
  default, is the worker as before. POSIX only.
- A Prometheus endpoint. `path("ox/", include("django_ox.urls"))` mounts
  `GET /ox/metrics`, which renders the `django_ox.stats` numbers as gauges in
  the Prometheus text format (OpenMetrics on request), from the standard
  library alone. The metric names are `django_ox_tasks{queue,status}`,
  `django_ox_ready_tasks`, `django_ox_oldest_ready_age_seconds`,
  `django_ox_last_claim_age_seconds`, `django_ox_throughput_per_minute` and
  `django_ox_failure_rate`, and they are public API from this release. The
  view has no authentication of its own. `django_ox.metrics.collector()`
  returns a collector for a `prometheus_client` registry when that package is
  installed; it is not a dependency.
- `django_ox.actions.retry(result_id)` and `django_ox.actions.discard(result_id)`.
  A retry puts a FAILED or LOST task back to READY for one more attempt, keeping
  its attempt count, worker ids and every traceback, and clearing the backoff so
  it is eligible at once. A discard closes a READY, FAILED or LOST task without
  running it. Each is one compare-and-set on the row's status and lease number,
  so two retries of one row requeue it once, a discard that races a claim loses
  to it, and a LOST row's missing worker cannot write over its retry. Neither
  touches a RUNNING task. `retry_many(selection)` and `discard_many(selection)`
  take a queryset or a list of ids and make the same move in one conditional
  UPDATE per thousand rows inside one transaction, returning
  `(changed, skipped)`.
- The task table in the Django admin, registered only when
  `django.contrib.admin` is installed: a list with status and queue filters and
  search by id or path, a read-only detail page with every attempt's traceback,
  and **Retry selected tasks** and **Discard selected tasks** actions that
  report how many rows moved and how many were skipped. The actions call
  `retry_many` and `discard_many`, so a select-across of any size is a few
  statements in one transaction. The admin does not add, edit or delete rows.
- `django_ox.bulk.enqueue_many(task, calls)`, the bulk form of `enqueue()`.
  `calls` is a list of `(args, kwargs)` pairs; the rows are written with one
  INSERT per 1,000 inside one transaction and the `TaskResult` list comes back
  in input order. The task is validated and every argument serialised before
  the first write, so a rejected call inserts nothing. Each row is built by the
  same code as `enqueue()`, so workers see no difference.
- `OxTask.Status.DISCARDED`, a sixth value in django-ox's own status column. It
  reads as `FAILED` through `django.tasks` and `is_finished` is true for it.
  `queue_stats()` reports it in a `discarded` column, and `ox_prune` deletes
  discarded rows with successful ones.

### Fixed

- `manage.py check` runs the django-ox checks, `django_ox.E001` to `E005`,
  in a project that imports `django.tasks` nowhere else. Django registers its
  tasks check when that module is first imported, and a project without the
  admin or a task module on its import path reached `check` without it, so
  every django-ox check passed silently. The worker's own startup check was
  unaffected.

### Changed

- **A migration ships with this release.** Run `python manage.py migrate
  django_ox` when you upgrade. It adds the new status choice.
- `QueueStats` has a sixth field, `discarded`, keyword-defaulted like `lost`.
- A queued task can now be discarded and a failed or lost one retried, and
  every attempt can be bounded with `TASK_TIMEOUT`; interrupting one chosen
  running task on demand stays outside scope.
- A worker that is already draining, because it is recycling, treats the
  operator's first signal as the drain it is doing rather than as the
  force-exit; the second signal is still the force-exit.

## [0.2.1] - 2026-08-20

### Fixed

- A task that succeeded after its lease was lost no longer keeps the reaper's
  lost-lease record in `errors`. That record says the outcome was never
  observed, and the success write is that observation, so anything reading
  `result.errors` was handed an exception nobody raised on a task that worked.
  A failure resolving the same way already dropped it, and the two now agree. A task that is still LOST keeps the record: it is
  the only thing on the row that says why the result reads as failed.

### Added

- `tools/check_release.py --dist` opens the built wheel and sdist and checks
  that each one carries every migration in the source tree, along with the
  licence and the package modules. A packaging rule that stops shipping a
  migration leaves a distribution that imports and passes its tests, and fails
  on somebody's upgrade against a column that is not there. The release
  workflow runs it after the build.

## [0.2.0] - 2026-08-20

### Fixed

- A worker whose task had been taken back by the reaper could still write its
  own outcome over the row, so a task that had already finished could be moved
  back to READY and run a second time after its result had been reported. Every
  claim now stamps the row with a lease number, and every finish write carries
  that number in its WHERE clause, so a write from a worker that no longer
  holds the task matches nothing and is dropped instead of applied. No
  completion is signalled for a dropped write.
- The reaper no longer records a failure it did not observe. When a lock aged
  out with no attempts left it wrote FAILED and invented a `TaskAbandoned`
  exception to explain it, on no evidence beyond a clock. It now records the
  task as LOST, which says the worker stopped reporting and the outcome was
  never seen, and nothing more.
- Lock timestamps are written and compared using the database server's clock
  rather than each worker's own, so two hosts with drifting clocks no longer
  produce false reclaims. This applies when `USE_TZ` is on. With `USE_TZ` off
  the worker's clock is used instead, because the database's clock does not
  always match what these columns hold: SQLite's is UTC while the columns carry
  naive local time, and reading one against the other would make `ox_prune
  --older-than` treat rows that finished seconds ago as hours old.
- On databases without `SELECT ... FOR UPDATE SKIP LOCKED`, which includes
  SQLite, a claim read its row back in a second statement and could come away
  holding a lease granted to a different worker, if the reaper reclaimed the
  row in the gap between the two. The read is now pinned to the lease the claim
  was granted, so a worker that lost the row inside that gap comes back with
  nothing rather than with someone else's lease.

### Added

- **Lease renewal.** A worker refreshes the lock on the tasks it is running,
  one statement per interval however many are in flight, and keeps doing so
  through a graceful drain. A long task on a healthy worker is no longer
  reclaimed while it is still running. `LOCK_TIMEOUT` now bounds how long a
  worker may go unresponsive, not how long a task may take. The renewal
  interval is `LOCK_TIMEOUT / 3`, overridable as `renew_interval` when
  embedding `Worker` directly.
- `OxTask.Status.LOST`, a fifth value in django-ox's own status column. It
  reads as `FAILED` through `django.tasks`, which has four statuses and gets no
  fifth from us, and `is_finished` is true for it, so callers waiting on a
  result still terminate. The row keeps the distinction: `queue_stats()`
  reports a `lost` column and `ox_prune --include-failed` covers it. If the
  worker holding a LOST task comes back and records a real outcome, that
  outcome replaces LOST; only that one execution can.
- `task_lease_lost` and `lease_renew_failed`, two WARNING log events. Both are
  documented on the Monitoring page.

### Changed

- **A migration ships with this release.** Run `python manage.py migrate
  django_ox` when you upgrade. It adds the `lease_epoch` column and the new
  status choice.
- `task_reclaimed` now reports `status` as `READY` or `LOST`, where it
  previously reported `READY` or `FAILED`.
- `QueueStats` has a fifth field, `lost`. It is keyword-defaulted, so existing
  code that constructs one keeps working.

## [0.1.2] - 2026-08-18

The worker, the public API and the database schema are unchanged. This
release updates the project description that appears on the package page,
and the documentation that ships with it.

### Changed

- README now leads with what the backend removes from a deployment: the
  queue lives in the database the application already runs, so there is no
  broker to provision, secure, upgrade or back up. The transactional
  guarantee follows it rather than opening.

### Added

- Migration guidance now covers moving *away* from django-ox as well as to
  it: which behaviour carries over to a broker-backed backend, which does
  not, and how to keep the option open.
- Worked examples for routing a queue to its own worker, choosing a lock
  timeout for long tasks, overriding a schedule's queue and priority,
  verifying that a schedule is live, and running the worker in containers.
- `context7.json`, so documentation indexers read the project description,
  the supported versions and the setup steps rather than inferring them.

## [0.1.1] - 2026-08-17

The worker, the public API and the database schema are unchanged. This
release updates the packaging metadata and the project description that
appears on the package page.

### Changed

- Packaging metadata now carries a `Documentation` URL, so the
  documentation site is linked directly from the package page.
- README now carries release and CI status badges, a link to the
  documentation site, and a scope statement: what the core covers, what is
  deliberately outside it, and which features belong to the commercial
  tier.

## [0.1.0] - 2026-08-16

Initial release.

### Added

- `OxBackend`, a database-backed backend for Django's Tasks framework
  (`django.tasks`, Django 6.0+). Tasks are stored in the application database;
  no broker required.
- Transactional enqueue: `enqueue()` is a single INSERT on the caller's
  connection, so a task enqueued inside `transaction.atomic()` commits or
  rolls back with the business data.
- `ox_worker` management command: claims tasks with
  `SELECT ... FOR UPDATE SKIP LOCKED` where supported (PostgreSQL, MySQL 8+)
  and an atomic compare-and-set UPDATE elsewhere (including SQLite).
  Configurable via `--backend`, `--queues`, `--concurrency` (thread pool),
  `--interval`, and `--lock-timeout`.
- Retries with exponential backoff (`MAX_ATTEMPTS`, `BACKOFF_INITIAL`,
  `BACKOFF_MAX`), keeping the full traceback of every attempt.
- Reaper: tasks whose worker died are returned to the queue after
  `LOCK_TIMEOUT` and count as a failed attempt.
- Graceful drain: on SIGTERM/SIGINT the worker stops claiming, finishes
  in-flight tasks, then exits; a second signal forces an immediate exit.
- Priorities (-100 to 100, higher first) and deferred tasks (`run_after`),
  with the corresponding `supports_*` flags declared on the backend.
- Result store: `get_result()`, `refresh()`, and the async variants, with
  status, return value, and per-attempt errors readable from the database.
- `ox_prune` management command: batched deletion of finished task rows
  (`--older-than`, `--include-failed`, `--batch-size`, `--dry-run`).
- `django_ox.stats`: read-only queue metrics as plain ORM queries, on
  both supported databases: per-queue status counts, backlog depth and
  age, throughput and failure rate over a trailing window, and time
  since the last task claim.
- `ox_health` management command: exits non-zero with a one-line reason
  when the database is unreachable or a `--max-backlog`, `--max-age` or
  `--worker-timeout` threshold is breached; built for cron alerting and
  container probes.
- Structured logging: worker lifecycle events (claim, start, success,
  retry, failure, reclaim, dispatch, shutdown) log to the `django_ox`
  logger with stable extra keys (`event`, `task_id`, `queue`, `attempt`,
  `duration_ms`, ...) for JSON log handlers.
- Recurring tasks: cron schedules declared in the `TASKS` setting
  (`SCHEDULES` option), dispatched by the workers themselves; a unique
  constraint on (schedule, tick) makes each tick fire exactly once across
  any number of workers. Five-field cron syntax plus `@hourly`-style
  shortcuts; misconfigured schedules fail at startup and in
  `manage.py check`. On recovery after downtime, only the latest missed
  tick fires.
- System check `django_ox.E003`: a schedule name defined on more than
  one backend is rejected, at worker startup and in `manage.py check`,
  because the tick log is keyed by schedule name alone and shared names
  would let the backends suppress each other's ticks.
- Strict cron validation: expressions that can never fire and step values
  larger than a field's range (such as `*/61` in the minute field) are
  rejected at parse time rather than misfiring silently. Schedule
  dispatch is robust to clock skew between workers: a tick row dated in
  the future cannot suppress ticks that are due.

### Security

- A stored `task_path` must resolve to a `django.tasks` Task (a function
  registered with `@task`). A row naming any other importable callable is
  rejected as an un-runnable task instead of being executed, so the
  worker never invokes an arbitrary dotted path pulled from the table.
  `SECURITY.md` documents the full trust model, the JSON-only
  serialization, and the guidance to keep secrets out of task arguments.
- An API stability and deprecation policy (`docs/stability.md`) covers
  the public API surface, the pre-1.0 SemVer rule, the deprecation
  window, and the supported Python and Django matrix.

[0.3.1]: https://github.com/oxpull/django-ox/compare/v0.3.0...v0.3.1
[0.3.0]: https://github.com/oxpull/django-ox/compare/v0.2.1...v0.3.0
[0.2.1]: https://github.com/oxpull/django-ox/releases/tag/v0.2.1
[0.2.0]: https://github.com/oxpull/django-ox/releases/tag/v0.2.0
[0.1.2]: https://github.com/oxpull/django-ox/releases/tag/v0.1.2
[0.1.1]: https://github.com/oxpull/django-ox/releases/tag/v0.1.1
[0.1.0]: https://github.com/oxpull/django-ox/releases/tag/v0.1.0