Data Pipelines & Streaming covers: Apache Kafka 1.0 Cookbook, Database Internals. Understand data movement — how data gets from one system to another, how event-driven architectures work, and how streaming systems process data in real-time. This module bridges databases with the broader data infrastructure. Rails already has event-driven patterns: Active Job publishes jobs to a queue (Sidekiq/Redis), Action Cable uses pub/sub for real-time updates, and Active Support Notifications provides an in-process event system. CDC from PostgreSQL extends this to cross-service data flow: when a record changes in your Rails app's database, Debezium captures it from the WAL and publishes it to Kafka, where other services can consume it. This is how you break a monolith without losing data consistency. Understanding data pipelines also helps you design analytics: instead of running expensive aggregate queries on your production database, stream changes to a data warehouse (BigQuery, Redshift) and run analytics there.
This course unlocks once you've finished its prerequisite. Open prerequisite →
Kafka feels like overkill for a Rails app. And honestly, for most Rails apps, it is. If you are processing a hundred webhooks a minute, Sidekiq is fine. If you are syncing data be…
Here is a bug you may have shipped without knowing it:
A Postgres read replica and a Kafka consumer are the same thing.
Work through each item before the checkpoint.
A real Kafka broker (one docker compose up away) and a Ruby producer/consumer pair around an order-event stream: schema-validated envelopes, per-key ordering, consumer groups that…
5 lessons. Read in order; spiral back when you need to. By the end you'll have used the core ideas twice — once on the abstract, once on something you'll meet at work next week.