Tiffin's monthly report job splits its work into 200 tasks. 199 of them finish in about a minute. The last one runs for 25 minutes, and the job can't finish until it does. Adding machines changes nothing: the extra machines finish their share early and sit idle.
That's data skew: the work isn't split evenly, because the data isn't. The batch processing lesson mentioned it in one line. This lesson shows how to spot it, and the fixes for joins, aggregations and streams.
Why one task gets everything
Recall the shape of a distributed job: map, shuffle by key, reduce. The shuffle sends every row with the same key to the same task. That's what makes a join or a group-by correct, and it's also the problem. If one key has far more rows than the others, one task gets them all.
Tiffin has 600 million orders. Grouped by brand_id across 200 tasks, an even split would give each task 3 million rows. But one cloud-kitchen brand runs 1,200 outlets and has 8% of all orders: 48 million rows, all on one task.
| Rows | Time at ~50,000 rows/s | |
|---|---|---|
| A typical task | ~3 million | ~1 minute |
| The task holding the big brand | ~48 million | ~16 minutes, more if it spills to disk |
A distributed job is as slow as its slowest task. Skew happens when one key, or a few, hold a large share of the rows, so the task that gets them does most of the work.
Module 14 met the same problem in storage as a hot key. In processing it's worse in one way: the job waits for the hot task, and more memory per task often isn't enough, because the hot task spills to disk and slows down further.
Spotting it
You find skew by comparing the slowest task to the typical one, not by looking at averages.
- Batch (Spark and similar): in a stage's task list, compare the maximum and median duration and shuffle read size. A max 10 times the median is skew.
- Streams (Flink and Kafka consumers): one parallel instance is busy or backpressured while the rest are idle, or consumer lag grows on one Kafka partition only.