Your customer paid. Then the worker died.
Payment capture, receipts, invoices and customer webhooks run in a background process your users never see. When a deploy or an out-of-memory kill stops that process halfway, the work is gone and nobody is told. django-ox keeps that work in your database and hands it to another worker.
Picture your store at 17:42 on a Friday. A customer's card is charged. The worker about to mark the order paid and send the receipt dies mid-task, because a deploy just restarted its container.
The web request that started all this returned 200 ten seconds earlier. The customer's bank shows the charge. Your store shows "payment pending". No alert fires, because nothing raised an error. On Monday, support opens an email from someone who paid for something you never shipped.
The half of your product nobody watches
Every Django company runs work after the response has gone out. Capturing the payment after checkout. Sending the confirmation. Raising the invoice. Firing the webhook that tells a customer's system an order exists. Provisioning the account so a new user can log in. Building the export a customer asked for this morning.
Each of those is a task. A task runs in a worker: a process that picks jobs off a queue and runs them. Your web servers sit behind a load balancer with health checks. Your workers get one line in a Procfile.
How a worker dies
Workers die on ordinary days. A deploy restarts every container. The kernel's out-of-memory killer picks the biggest process, and a worker building a large export is the biggest process. A container crashes on a bad config. The cloud reclaims the node. The autoscaler scales in. None of this pages anyone, because the process was supposed to stop.
The task doesn't know that. It was halfway through. If it was marked running in a table, it stays marked running. If it was popped off a queue, it left with the process. Either way, nothing comes back for it unless something was built to come back for it.
What it costs, in the companies' own words
Mailgun has written the half-finished version up twice. In September 2013, during a spike in messages, its monitoring scripts began killing and restarting slow delivery nodes. The postmortem says some messages "were sent but not marked as delivered in our system and were retried after the process restarted". A small number of customers received duplicates. A year later its hosting provider rebooted cloud instances, which triggered a fault in Mailgun's load balancers. That postmortem lists "Approximately 100,000 duplicate emails sent" among the effects.
On Christmas Day 2021, Santander UK made 75,000 payments twice, £130 million in total. The bank called it a "scheduling issue", AFP and Fortune reported. It said it would work with other banks over the coming days to recover the duplicated transactions. The money came from the bank's own reserves.
GitLab's 2017 postmortem is the quiet version. An engineer removed the primary database's directory by accident. The regular backups turned out to have been failing on a pg_dump version mismatch. The failure emails had been rejected by the receiving mail server. GitLab wrote that it was "never aware of the backups failing, until it was too late". GitLab's own estimate of the loss: "roughly 5,000 projects, 5,000 comments and 700 new user accounts".
GitHub's October 2018 outage shows what recovery looks like. After a 43-second network partition and a database failover, GitHub deliberately paused webhook delivery and Pages builds. Its analysis counted over five million webhook events and 80 thousand Pages builds queued. When processing resumed, about 200,000 webhook payloads had outlived an internal TTL and were dropped.
GitLab's infrastructure team put the mechanism itself in writing in 2020. A worker pod killed for exceeding its memory limit gets no chance to finish or requeue its jobs. Interrupted jobs wait for a recovery process that runs hourly by default. That "could be a long time to wait", the note says, "with a customer waiting on the UI to update and getting impatient". It's an engineering note, and it describes the exact shape of the problem.
You're smaller than a bank or a mail provider. The pattern is close to the same. Background work that should run once runs twice, or not when it should. Someone cleans up after the fact. At your scale, picture a receipt that never arrived and a chargeback the month after. Or a new user who gave up at "setting up your workspace" because provisioning stopped halfway.
Tasks survive a dead worker
django-ox is a free, BSD-licensed worker for Django's Tasks framework. The queue is your existing database: PostgreSQL, MySQL or SQLite. There's nothing else to run.
Two things follow. First, the task is a row in the same database as the order. You enqueue inside transaction.atomic(), and the task commits or rolls back with your data. No on_commit() hook, and no window where the order exists and the task doesn't.
Second, each worker holds a lease on the task it's running. If the worker dies, the lease expires and a reaper hands the task to another worker. The default lease is 300 seconds. Our kill test used 15.
That test ran five trials of 2,000 tasks, with 20 worker kills and replacements in each. It compared django-ox 1.3.0 with django-tasks-db 0.13.0 on PostgreSQL 16. Every django-ox trial finished 2,000 of 2,000 tasks. django-tasks-db, the default most Django teams start with, left 13 to 19 tasks per trial stuck in RUNNING.
Execution is at-least-once. Across those 10,000 tasks the trials recorded 7 repeated effects. Mailgun's 2013 sentence describes the same mechanism: sent, not marked as delivered, retried. So write tasks that are safe to run twice. Check the order's state before you capture, or hand your payment provider an idempotency key.
Retries work the same way. In a retry test, each of 20 tasks raised an exception on its first attempt. django-ox retried them, and 20 of 20 succeeded on attempt two in every run. django-tasks-db marked all 20 FAILED after the first exception.
The Django admin shows every task, and you can inspect, retry or discard it from there. Recurring schedules run without a scheduler process. Health checks and metrics are built in. The optional oxpull app, for Django 6.0 and later, adds unique tasks, batches, rate limiting and workflows, all free. django-ox supports Django 5.2 LTS through the django-tasks backport, plus 6.0 and 6.1, on Python 3.12 or newer.
Speed comes second, and it's there too. With pooled or persistent connections and matched settings, django-ox ran 1.4 to 1.7 times the tasks per second of django-tasks-db in our benchmark.
Try it on your own worker
pip install django-ox
Then read the benchmarks, kill test included.