Sharding
🧷 Sharding
Why Shard? The Ceiling Replication Doesn't Remove
Replication scales reads (Chapter 5's read preference) and provides failover, but it doesn't scale writes or storage — a 10-node replica set still has exactly one primary accepting writes, and every node still needs enough disk to hold 100% of the data. Sharding distributes the data horizontally: each shard holds a portion of the total dataset, so total storage and total write throughput both grow as shards are added.
The Building Blocks of a Sharded Cluster
| Component | Role |
|---|---|
| Shard | Holds a portion of the data; in production, each shard is itself a full replica set (Chapter 5) for redundancy |
| mongos | The query router applications actually connect to; routes each operation to the correct shard(s) |
| Config servers | Store the cluster's metadata — which shard key ranges live on which shard |
An application talks to mongos exactly as if it were a single mongod — the routing to the correct shard(s) happens transparently underneath.
The Shard Key
The shard key is the field (or fields) chosen when a collection is sharded, and it determines which shard each document lives on. It's chosen once, when sharding a collection, and changing it afterward is a genuinely heavy operation (resharding) — getting it right up front matters far more here than most schema decisions covered so far in this course.
Choosing a Good Shard Key
Three properties make a shard key good:
- High cardinality — many distinct possible values, so documents can actually spread across many shards rather than piling onto just a few.
- Even write distribution — new writes should land across shards roughly evenly, not concentrate on one.
- Query isolation — common queries should ideally be answerable by consulting just one (or a few) shards, not every shard in the cluster.
A Bad Shard Key vs. a Better One
Monotonically Increasing (e.g. _id or a timestamp)
Every new document's key value is higher than the last. All new writes land in the same range — and therefore the same shard — creating a hotspot while other shards sit idle. Sharding on this field barely helps write throughput at all.
{ customer_id: "hashed" }
Hashing scrambles the key's natural order before distributing it, so consecutive writes (even from the same fast-growing range of underlying IDs) land essentially randomly across shards — spreading write load evenly, at the cost of range queries no longer targeting a single shard.
Chunks & the Balancer
Within a sharded collection, data is split into chunks — contiguous ranges of shard key values. As chunks grow past a size threshold, MongoDB splits them further. The balancer is a background process that automatically migrates chunks between shards whenever their distribution becomes uneven — keeping roughly the same number of chunks (and therefore roughly the same amount of data) on every shard, without a human manually moving anything.
Sharding Is the Right Call When
A single replica set's primary genuinely can't hold the working data set in memory, or can't keep up with write throughput, even after the schema and indexing work from earlier chapters.
Sharding Isn't Yet Needed When
The dataset comfortably fits on one well-resourced replica set — sharding adds real operational complexity (choosing a shard key you're stuck with, managing mongos/config servers) that isn't worth paying for before it's actually necessary.
A Query That Benefits From Query Isolation
With customer_id as the shard key, a query filtering by customer_id can be routed directly to the one shard holding that customer's data:
💻 Coding Challenges
Challenge 1: Spot the Hotspot
A team proposes sharding an events collection on created_at (a timestamp) so events can be range-queried by date efficiently. What goes wrong with writes, and why?
Goal: Practice recognizing the monotonically-increasing-key hotspot pattern in a new example, not just the _id case from this chapter.
Challenge 2: Explain Query Isolation
A collection is sharded on tenant_id (a multi-tenant SaaS app, one value per customer organization). Explain why a query filtering by tenant_id is fast, and why a query that omits it is slow.
Goal: Practice explaining scatter-gather versus targeted routing in terms of the shard key actually appearing (or not) in the query filter.
Challenge 3: Shard or Not?
A company's product catalog has 50,000 documents and comfortably runs on a single 3-node replica set with room to spare. Should they shard it now, in anticipation of future growth? Justify your answer using this chapter's decision criteria.
Goal: Practice resisting the urge to shard preemptively, weighing real operational complexity against a need that doesn't yet exist.
If a query's filter doesn't include the shard key, mongos has no way to know which shard holds the matching documents — it has to send the query to every shard and merge the results, a scatter-gather query. This isn't a hard error, just a much slower path than a query that targets one shard directly — and it's easy to end up here by accident if the fields an application actually filters by most often weren't the ones considered when the shard key was chosen. Picking a shard key genuinely means picking it around the queries the application runs most, not just around "spreads writes evenly."
🎯 What's Next
The next chapter is Performance & Security — revisiting index strategy with everything this course has covered since mongodb1-7, the database profiler, authentication/authorization, and connection string security (tying back to the site's Database Security course).