Module 22 / Networking and API essentials

Module 22 · lesson 7 of 7 · Read 9 min

Encoding and schema evolution

JSON, Protobuf, Avro, and changing a format without breaking anyone.

Code is easy to change. You deploy, and the old version is gone. Data isn't like that. An order event written to Kafka last month is still there. A Tiffin app installed 18 months ago on a phone that never updates still calls your API every evening. During a rolling deploy, old and new servers run side by side and read each other's messages.

So every time you change the shape of your data, someone is going to read the new shape with old code, or the old shape with new code. This lesson covers the formats used to send data between programs, and the rules that let you change them without breaking anyone.

Quick recap. Services talk by sending encoded data: JSON over REST, Protobuf over gRPC (see "REST, gRPC or GraphQL?" in this module), or messages on a queue or Kafka topic (see Module 15). Module 2 ("Pagination, errors and versioning") covered versioning a public API. This lesson goes underneath that: the encodings themselves and what changes they survive.

From objects to bytes

Inside a program, an order is an object in memory, with pointers and language-specific types. To send it over a network or save it to disk, you have to turn it into a flat sequence of bytes. That's encoding (also called serialisation). The reader does the reverse, decoding.

Let's encode the same small order three ways:

  • order ID 812
  • status "placed"
  • total 64,000 paise (₹640)
  • note "no onions"