Module 17 / Event-driven data and streaming

Module 17 · lesson 6 of 8 · Read 8 min

Batch processing

Large jobs over all the data, and when they beat streams.

On the first of every month, Tiffin pays 60,000 restaurants for the previous month's orders, minus commission, minus refunds, plus adjustments. Every number has to be right to the rupee, and finance needs to explain each one if a restaurant disputes it.

You could build this as a stream, keeping a running total per restaurant as events arrive. But think about what can go wrong in a month: a late refund, a processor bug that double counted a weekend, an event dropped as "too late". A running total carries every mistake forward. A job that reads the whole month of finished data once, after the month is over, has none of those problems. That's batch processing, and it's older, simpler and often better than streaming.

What a batch job is

A batch job reads a fixed, finished set of input, computes a result, writes it out, and stops.

The word that matters is finished. Last month's orders won't change any more (or if they do, you rerun the job). So there are no watermarks, no late events and no running state to recover. If the job crashes, you fix it and run it again from the start. If the code had a bug, you fix it and run it again. The input is still there.

That rerun property is the big one. A batch job with immutable input and replaceable output can be run as many times as you like, and gives the same answer every time. Streams can get close to that. Batch gets it for free.

You've met small batch jobs already: the nightly scheduled jobs from Module 7. This lesson is about what happens when the data gets too big for one query on one database.