Load Balancers and Caching Strategies

Hook #

The last course gave you the method for designing systems; this one is the catalog of parts you design with — the standard building blocks, each studied deeply enough to use well. We start with the two most ubiquitous: load balancers and caches. You met both earlier (load balancers in the networking course, caches at the hardware level), but here they become design tools with real strategy choices. A load balancer isn't just "spread the traffic" — it's L4 or L7, one of several algorithms, with health checks that decide who's alive. A cache isn't just "make it fast" — it's a strategy (cache-aside, write-through, write-behind, refresh-ahead), an eviction policy (LRU, LFU, TTL), and an invalidation problem that's famously one of the two hard things in computer science. This lesson turns two familiar components into deliberate design decisions, and grounds them in how Rails actually implements caching — so the strategies aren't abstract, they're Rails.cache and Russian doll caching you can reach for.

What you'll be able to do by the end of this lesson #

• Explain load-balancer choices as design decisions: L4 vs L7, the algorithms (round-robin, least-connections, IP-hash), and the role of health checks.
• Distinguish the caching strategies — cache-aside (lazy), write-through, write-behind (write-back), refresh-ahead — and when each fits.
• Explain the cache-invalidation problem and why it's genuinely hard (keeping cached data consistent with the source of truth).
• Map these strategies onto Rails: Rails.cache, fragment caching, Russian-doll caching, and Redis as a backend.

A quick try before we start #

Before reading: your app reads a user's profile constantly and updates it rarely. You add a cache. When a profile is updated, how do you keep the cache from serving the stale old version? (Options: delete the cache entry on write so the next read repopulates it — cache-aside with invalidation; or update the cache at write time — write-through; or set a short TTL and tolerate brief staleness.) There's no free answer — each has a cost, and choosing among them is the caching design decision. Sitting with "how do I not serve stale data?" is the doorway to the whole lesson.

Why this matters here #

Load balancing and caching are in nearly every system you'll design or operate, and using them well — not just placing them on a diagram — is a real skill with real consequences. The caching-strategy choice determines your consistency/performance trade-off: cache-aside is simple and common but has a brief window where reads can miss or serve stale; write-through keeps the cache consistent at the cost of slower writes; write-behind is fast but risks data loss on failure. Picking the wrong one causes either stale-data bugs or performance problems. Cache invalidation is a genuine source of subtle production bugs (the joke that it's one of the two hard problems is earned). And load-balancer choices — L4 vs L7, sticky sessions vs. stateless, which algorithm — determine what routing you can do and how failures are handled. These are daily decisions for anyone building scalable systems, and knowing the strategies (rather than just "add a cache") is what makes those decisions deliberate. The Rails lens makes it immediately usable: these exact strategies are Rails.cache, fragment caching, and Russian-doll caching, which you can apply the moment you understand the patterns.

Within the course, this is the first component deep-dive of the building-blocks catalog, applying the trade-off discipline from Course 10.1 to two specific parts. It deepens what earlier courses introduced: load balancing (Course 7.2's L4/L7) and caching (Course 5.2's cache organization and eviction policies), now as design strategies rather than mechanisms. The eviction policies (LRU/LFU/TTL) are the exact ones from the memory-systems course, reappearing at the application-cache layer — the spiral curriculum making the point that a distributed application cache faces the same "which to evict when full?" problem as a CPU cache, just at a different scale. This sets the pattern for the rest of the course: take a known component, study its strategy space, and learn to choose deliberately.

The engineer's lens #

The core insight is that caching is a set of named strategies each making a different consistency-vs-performance trade-off, and "add a cache" is not a decision until you've chosen the strategy — because the strategy determines how, and how badly, you can serve stale data. Cache-aside (lazy loading): the app checks the cache, and on a miss, loads from the database and populates the cache; writes invalidate (delete) the cache entry. Simple, common, resilient (cache failure just means more DB load), but there's a window where concurrent reads/writes can leave stale or missing entries. Write-through: writes go to the cache and the database synchronously, keeping them consistent, at the cost of slower writes. Write-behind (write-back): writes go to the cache immediately and to the database asynchronously — fast writes, but you can lose data if the cache fails before flushing. Refresh-ahead: proactively refresh popular entries before they expire, hiding latency for hot data. Each is a point on the consistency-vs-latency-vs-durability space, exactly the kind of trade-off Course 10.1 taught you to make explicit — and the choice depends on your read/write ratio and staleness tolerance. This connects to a deep truth about caching: the hard part isn't storing data fast, it's invalidation — keeping the cache consistent with a source of truth that changes. Every caching strategy is really a different answer to "when and how do I deal with the source changing?", and recognizing that "add a cache" is shorthand for "choose a caching strategy and an invalidation approach" is what separates caching that helps from caching that introduces stale-data bugs. When you reach for Rails.cache or Russian-doll caching (which is cache-aside with clever key-based invalidation — the cache key includes a timestamp so a change auto-invalidates), you're choosing a strategy, and knowing the menu makes the choice deliberate.

The second lens is that generic components become concrete — and your designs become credible — when you know how the real implementations actually behave, not just what they do abstractly. "A load balancer distributes traffic" and "a cache makes reads fast" are the abstract versions everyone knows. The valuable knowledge is the concrete behavior: that an L7 load balancer can route by URL path or header (enabling path-based routing and canary deploys) while an L4 one can't because it doesn't parse HTTP; that Redis is single-threaded yet extremely fast because being single-threaded avoids lock contention and context-switch overhead (a surprising insight from the OS course — sometimes not being concurrent is faster); that a health check failing too aggressively causes needless failovers while too slowly extends outages (the failure-detection trade-off from distributed systems). These concrete details are what let you reason about a design's real behavior rather than its cartoon version — "we'll use Redis for the cache, and since it's single-threaded, a single slow command (like a big KEYS scan) blocks everything, so we avoid those in hot paths" is a concrete design consideration that a purely abstract understanding misses. The habit worth building is: whenever you place a component in a design, know how the specific implementation you'd actually use behaves under load and failure. That concreteness — grounded in the mechanisms you've studied across this whole path — is what makes a system design trustworthy rather than hand-wavy, and it's why understanding the fundamentals underneath these building blocks matters even when you're "just" assembling them.

What to focus on in the resources #

• Caching strategies and invalidation (Web Scalability + Alex Xu). Nail the four patterns (cache-aside, write-through, write-behind, refresh-ahead) and when each fits, and take the invalidation problem seriously — it's the hard part. This strategy choice is the design decision.
• Load-balancer decisions (Alex Xu). L4 vs L7 (revisit from Course 7.2 as a design choice), the algorithms, and health checks. Focus on what each enables (L7 → content-based routing) and the failure-detection trade-off in health checks.
• The Rails caching implementation (Rails Guides). Map the abstract strategies to Rails.cache, fragment caching, and Russian-doll caching. Russian-doll caching is especially worth understanding — it's cache-aside with automatic key-based invalidation (the key changes when the data does).
• Skip on first pass: specific load-balancer product configs, exhaustive eviction-algorithm variants (you have LRU/LFU/TTL from Course 5.2), and Redis-cluster internals (next lessons touch distributed caches). Get the caching strategies, the invalidation problem, the LB decisions, and the Rails mapping.

Explain it back #

Explain to a colleague the difference between cache-aside and write-through caching, and how you'd decide between them. A strong answer: cache-aside (lazy) means the app loads from the DB on a cache miss and populates the cache, invalidating (deleting) the entry on writes — simple, resilient to cache failure, but with a window where reads can be stale or miss. Write-through means writes go to both cache and DB synchronously, keeping them consistent, at the cost of slower writes. You'd choose based on the read/write ratio and staleness tolerance: cache-aside for read-heavy data where brief staleness is acceptable (the common case), write-through when the cache must stay consistent with the source and write latency can absorb the cost. Bonus: name why cache invalidation is the genuinely hard part, and how Russian-doll caching sidesteps manual invalidation (the cache key encodes the data's version).

Where this connects #

Backward: Course 7.2's load balancing (L4/L7, algorithms — now as design choices) and Course 5.2's caching (organization and LRU/LFU/TTL eviction — the same policies at the application-cache layer). Cache invalidation connects to the consistency trade-offs from Quarter 9, and Redis-is-fast-because-single-threaded connects to the concurrency/context-switch ideas from Course 6.1. This applies Course 10.1's trade-off discipline to specific components.

Forward: Lesson 2 covers message queues and event-driven architecture (the async building block); lessons 3–5 cover CDNs/storage, search, and the rate-limiting/consistent-hashing/infra-glue layer. The caching strategies here reappear in nearly every case study in Course 10.3 and the caching parts of the URL-shortener capstone. The "know the concrete behavior" habit applies to every building block in this course.