The partition key makes one kind of question cheap: "give me the record with this key". Real apps ask lots of other questions. Tiffin stores orders by customer, but the kitchen tablet wants orders by restaurant, the rider app wants orders by rider, and support wants orders by phone number.
On one machine, you'd add an index (Module 2) and move on. Once the data is split across machines, an index has to be split too, and there are only two basic ways to split it. Each one makes either reads or writes harder.
The setup
Tiffin partitions its orders across 32 partitions by hashing customer_id. That makes the customer's own screens fast: "my orders" and "my current order" each go to one partition.
Now the kitchen tablet needs this, every 5 seconds, for each of 60,000 restaurants:
SELECT id, items, status
FROM orders
WHERE restaurant_id = 4417
AND status IN ('placed', 'accepted');
Restaurant 4417's orders came from thousands of different customers, so they're spread across all 32 partitions. Without help, this query asks every partition. That's 60,000 / 5 = 12,000 tablet queries a second, times 32 partitions, which is 384,000 partition reads a second for a question that returns maybe three rows. We need an index on restaurant_id.
A secondary index lets you find records by a field other than the partition key. In a partitioned system it must be partitioned too, either alongside the data (local) or by the indexed value (global).