Building a Background Job Pipeline That Holds Up

A GEDCOM file holds 800 people. The import finishes clean. The database ends up with 975. Nothing crashed, no job failed, no red anywhere. A third-party service blipped, the job retried exactly as configured, and the second attempt rewrote everything the first attempt had already saved. Good retry logic caused that duplicate. That’s the failure you inherit the moment you move real work off the request path, and it’s why a background job pipeline needs more than a queue.

A Queue Defers Work. It Does Not Make It Reliable.

Large imports don’t belong in an HTTP request. A GEDCOM file, a CSV, a data migration — any of those can carry hundreds of thousands of rows. So you put the work on a queue. Everybody does.

Here’s the part nobody mentions. The naive version is worse than the synchronous version in one specific way. When a synchronous handler blows up, you return a 500 and somebody sees it. A background job that blows up returns nothing to anyone. It fails in the dark.

Three failures bite you. The third-party API goes down and nothing retries. Something retries and half your data lands twice. Or the process dies mid-job on a deploy. Every one of those is a configuration problem in your background job pipeline, not a queue problem.

🚀 Complete JavaScript Guide (Beginner + Advanced)

🚀 NodeJS – The Complete Guide (MVC, REST APIs, GraphQL, Deno)

Why I put the queue on Postgres

I used BullMQ here. Version 6 added a Postgres backend, and the app already runs Postgres. That’s one less service to run, back up, monitor, and pay for. Job pickup uses LISTEN/NOTIFY, so it’s event driven rather than a poll on a timer. Redis still wins on raw throughput at tens of thousands of jobs a minute, and it has years more production hardening behind it. One limitation worth knowing: you can’t enqueue a job inside your own transaction yet.

Retries and Backoff Are Configuration, Not Defaults

BullMQ defaults to one attempt. Not three. One. If you never pass an options object, you don’t have retries. You have a queue that gives up immediately, which is most of the way to not having a queue at all.

I asked Claude Code for exponential backoff and told it to state the resulting delay sequence. That second part matters. A config that reads fine and produces a two-hour final retry is a thing that happens. Mine landed around 1, 2, 4, and 8 seconds, with a total window under a minute. That fits the failure I’m defending against: a service that blinks and comes back.

await importQueue.add(
  'import-gedcom',
  { uploadId },
  { attempts: 5, backoff: { type: 'exponential', delay: 1000 } }
);

Notice where that config lives. It sits on the job, not the worker. Different job types want different retry profiles. A place-name lookup should retry hard. A job that charges someone’s card should not. Backoff also buys resilience with detection time — a job failing for a real reason now takes much longer to land somewhere you’ll notice.

Idempotency Comes Before Retries in a Background Job Pipeline

Those retries introduced the bug from the intro. The service went down partway through an import. The job backed off, recovered, and started over. It didn’t resume — it restarted. Everything the first attempt wrote was still sitting there, and the second attempt wrote it all again.

These are non-idempotent writes. The more you harden the queue, the more attempts you add, and the more often your handler runs twice.

I asked for the menu of fixes before picking one: a unique constraint with upsert, a transaction around the import, a job-level idempotency key, or delete-and-redo. I took the constraint and the transaction together. The transaction gives me all-or-nothing per attempt. The constraint is there because I don’t trust that transaction to cover every path forever. Someone will add a write outside it eventually, and I’d rather that fail loudly at the database than quietly duplicate a family.

await prisma.$transaction(async (tx) => {
  for (const person of people) {
    await tx.person.upsert({
      where: { treeId_externalId: { treeId, externalId: person.externalId } },
      create: { ...person, treeId },
      update: { ...person },
    });
  }
});

Same test, second time around: kill the service mid-import, bring it back, and the file lands with 800 members.

Plan for the Failures Nobody Is Watching

Not every failure deserves a retry

A corrupt file burned five attempts and 16 seconds producing five identical parse errors, plus a worker slot a real job could have used. Parse and validation errors will never succeed on attempt two. BullMQ ships UnrecoverableError for exactly this, and the job fails immediately without touching the remaining attempts. It replaces the job.discard() method removed in version 6. Keep retry as the default and make unrecoverable the explicit exception. Misclassify a transient error the other way and you throw away a job that would have worked.

Failed jobs need somewhere to land

Failed jobs sit in a failed set that nobody reads, and default retention eventually drops those rows. Then you have no record the work was ever requested. BullMQ doesn’t ship a dead letter queue, so I built one. Exhausted jobs move to a separate queue with the original payload and context, retention is set explicitly on both, and a small check script prints the job ID, the upload path, and the error that killed it. The main queue cleans up aggressively. The DLQ keeps everything, because the whole point is that a human reads it later. I’ll be straight about the limit: that’s a dead letter queue, not a dead letter system. No alerts, no replay, no retry.

Shutdown is where jobs quietly vanish

This last one loses the most jobs. A deploy sends SIGTERM, the worker dies mid-job, and the job doesn’t fail and doesn’t retry. It sits waiting on a worker that no longer exists, and nothing alerts, because the queue still thinks it’s running. Await worker.close() on SIGTERM and SIGINT. It stops accepting new jobs, waits for the current one, and hands unfinished work back as stalled so another worker picks it up.

for (const signal of ['SIGTERM', 'SIGINT']) {
  process.on(signal, async () => {
    await worker.close(); // await matters - this is the whole fix
    process.exit(0);
  });
}

My handlers already existed. They just didn’t wait, which is the same outcome with extra steps. And in Docker, your stop grace period has to outlast that handler. Mine defaulted to 10 seconds while the import takes 40, so I moved it to 60.

Key Takeaways

• A queue is a deferral feature, not a reliability feature, and everything that makes it reliable is something you configure on top of it.
• BullMQ retries default to a single attempt, so a queue without an explicit options object gives up the first time anything goes wrong.
• Idempotent writes have to come before retries, because retrying a non-idempotent handler is how an 800-person file turns into 975 records.
• Parse and validation errors should fail immediately through UnrecoverableError, while transient errors keep the default retry behavior.
• Find the job in your app that writes to the database and run it twice on purpose with the same input; if the second run changes anything, you have an idempotency bug waiting for the day you add retries.

Conclusion

Moving slow work into the background solves one problem and quietly creates several others. Deferral is all a queue gives you out of the box. Retries, backoff, idempotent writes, unrecoverable errors, a dead letter queue, and a graceful shutdown are the parts you add yourself. Build your background job pipeline in that order, with idempotency before retries, or you’ll teach loud failures to fail silently.