Module 20 / Databases in depth

Module 20 · lesson 10 of 10 · Read 8 min

Inside a columnar file

Parquet row groups, encodings, and skipping data you never read.

What you'll learn

  1. Tiffin's data lake
  2. The layout
  3. Encodings: making each column small
  4. Skipping work
  5. What it costs
  6. Where it breaks

The previous lesson said a columnar store keeps each column's values together, so a query reads only the columns it uses. This lesson opens one of those files. Apache Parquet is the format you'll meet most: data lakes on S3, Athena, Spark, BigQuery external tables, DuckDB and most warehouses read it. Knowing how it's laid out explains why one query scans 3 GB and a nearly identical one scans 400 GB.

Tiffin's data lake

Tiffin exports every order to S3 for analytics: 200 million orders a year, 40 columns each, about 400 GB a year as CSV. The finance team asks:

SELECT city, SUM(total_paise)
FROM orders
WHERE order_date BETWEEN DATE '2026-10-12' AND DATE '2026-10-18'
GROUP BY city;

That query touches 3 of the 40 columns and one week of 52. On CSV, the engine must read every byte of every row to find those three fields. On well-laid-out Parquet, it reads a few megabytes. Here's how.