Framed reading 30 minutes + canonical resource

Shell, Pipes, and Regex

Hook #

You've been using a terminal since the first lesson of this path — typing gcc, make, git status. What you may not have done is programmed the shell itself. The shell is a small programming language with its own variables, control flow, and conditionals, designed around a single operating-system primitive: the process pipeline. That primitive turns out to be one of the most reusable abstractions in computing. The reason grep "error" log.txt | sort | uniq -c | sort -rn works without any program named grep-sort-uniq-sort ever existing is the Unix-shell idea, and this lesson is where it stops being magic.

What you'll be able to do by the end of this lesson #

Write a shell pipeline that solves a real text-processing problem (find the top 10 most frequent IPs in a log file, or list the largest files in a directory tree).
Read shell scripts containing if, for, command substitution $(...), and process substitution <(...) without flipping a coin.
Use grep, sed, and awk for the 80% of text-extraction tasks they cover; know that a real programming language is the right tool for the other 20%.
Write a regex that matches what you intended (and only what you intended) by working it out at regex101.com first.

A quick try before we start #

Before reading on: open a terminal in any directory and type ls | wc -l. Now type ls | grep "\.rb$" | wc -l (substitute another extension if Ruby isn't here). You just composed three programs that were written years apart by different people, none of whom coordinated, and the output is correct. Why does this work? Hold the question.

Why this matters here #

The shell is the glue layer of every project you'll touch professionally. CI pipelines are shell scripts. Build systems shell out. Deployment scripts are shell. When something breaks in a way Stack Overflow doesn't cover, the fix is usually strace, lsof, journalctl, and a pipeline of grep | awk | sort to isolate the signal. Engineers who can compose a pipeline in 30 seconds debug production faster than engineers who can't — not because the pipeline is magic, but because it lets them ask precise questions of large data interactively, without writing a program.

A second reason: the shell is the original example of "small composable tools with a shared interface" — a design idea that shows up later in microservices, in Unix domain sockets, in HTTP, in your favourite functional language's pipe operator. The shell is where that idea was first proven at scale; understanding it makes the later instances feel inevitable instead of arbitrary.

The engineer's lens #

The shell's contract is small: every program reads from stdin, writes to stdout, and writes errors to stderr. A pipe (|) connects one program's stdout to the next program's stdin. That's it. The reason grep "error" log.txt | sort | uniq -c works is that grep produces lines, sort accepts lines, uniq accepts lines, and at no point did anyone write a grep-sort-uniq program; the protocol (text lines on stdout/stdin) is the integration layer.

The implication is that any program that reads lines from stdin and writes lines to stdout automatically composes with every other Unix tool ever written. Your own scripts can join this ecosystem with no extra work — a Ruby script that prints one item per line is a first-class pipeline citizen.

The shell's programming-language layer is where most people get tripped up. A handful of rules carry most of the surprises:

Word splitting. Unquoted variable expansions are split on whitespace and re-tokenised. cp file contains a space. The fix is always cp "$file" dest/. Quote your expansions.
Globbing. * expands to filenames before the command runs. rm *.bak becomes rm a.bak b.bak c.bak before rm ever sees a *. If there are no matches, the glob stays literal in bash (shopt -s nullglob changes this).
Exit codes. Every program exits with a number: 0 is success, anything else is failure. && runs the next thing only on success; || runs it only on failure. set -e makes a script abort on the first failure; set -euo pipefail is the strict version most production scripts open with.
Command substitution. (ls) is almost always wrong (word splitting strikes again); for f in * is right.

What to focus on in MIT Lecture 2 #

Shell variables and quoting — the difference between var", and 'file dest/ example above is the demonstration. Internalise quoting now and avoid 80% of beginner shell bugs.
Functions and scripts — the section on writing a shell function, sourcing vs. executing a script, and the 1, # arguments.
Shell tools — the demonstrations of find, xargs, tee, tldr. xargs especially: it converts stdout into argv, which is the bridge from "tools that produce names" (find) to "tools that take names as args" (rm, cp).
Skip on first pass — the dotfile management section. Lesson 3 covers it.

What to focus on in MIT Lecture 4 (regex section only) #

The character classes — \d, \w, \s, and their inverses \D, \W, \S. The bracket form [a-z], [^0-9]. These cover 80% of what real regexes need.
Quantifiers — *, +, ?, {n,m}. The greedy-vs-lazy distinction (.* vs .*?) trips up most beginners; watch it.
Groups and captures — parentheses (...) group; $1/\1 references the first group. Captures are what makes sed 's/(.*) -> (.*)/\2 came from \1/' possible.
Anchors — ^ (start of line),  matches "wxyzabc" — both of which feel wrong unless you've internalised what no anchor means.

The 20%-case for regex: when you find yourself writing nested lookaheads or (?P<name>...) capture groups, you're probably in territory where a real parser would be cleaner. The regex is doing what you asked; the question is whether what you asked is the right shape.

Explain it back #

In your own words, as if explaining to a colleague who is competent in a high-level language but has only ever used the terminal to run other people's commands:

Why does cat file.txt | grep error | wc -l work — but cat file.txt | grep error | sort | grep -v test | wc -l also works, with no integration code anywhere? Name the contract that makes both pipelines compose.
The classic "for f in $(ls)" anti-pattern fails on filenames with spaces. Why? What does the shell actually do before the loop starts running?
You want to find every line in app.log that contains an IP address, then count how many times each unique IP appears, sorted by count. Write the pipeline. (You'll need grep -oE, sort, uniq -c, and sort -rn.)

A small exercise #

In a real project directory of yours:

Find every file modified in the last 7 days. (Hint: find . -mtime -7.)
Of those, find only the ones in app/. (Combine with grep or find's -path.)
Count how many of those are Ruby files. (Add | grep -c '\.rb$'.)

When the pipeline doesn't return what you expected, add tee /tmp/debug.log between stages to see what each step is actually emitting. That's the shell debugger: tee and head.

Where this connects #

Backward: Course 1.2's amortized analysis is the cost-model lens that applies here too — each pipeline stage is O(n) over the stream, the stream is the data structure, and a long pipeline is just function composition on streams.

Forward: Lesson 2 (Git Internals) leans on what you know now about file paths, content addressing, and how programs read/write trees of bytes. Lesson 3 (Daily Toolkit) covers the rest of the toolchain — vim, debuggers, containers — most of which are shells running other programs in particular ways. Every later course in this path assumes you can read and write a small pipeline without help; production debugging in particular (Course 6.1, 10.x, 11.x) is largely shell fluency applied to logs and live systems.

next · lesson 2 of 10

Git Internals: Objects, Refs, and the DAG →

That's the free preview. Sign in to continue this course.

New here? Make a desk →