Undefined Behavior Deep Dive

Course 3 · Ch 4
Undefined Behavior Deep Dive
Every UB thread this course has flagged since Chapter 2, finally given its full, formal explanation

"Undefined behavior" has appeared in nearly every chapter of this course. This one finally defines it precisely, explains the genuinely counter-intuitive reason it's dangerous, and catalogs every instance flagged so far in one place.

What "Undefined Behavior" Actually Means

The C standard defines three genuinely different categories, and conflating them is a real, common mistake:

  • Unspecified behavior — the standard allows several possible results; the implementation picks one, but doesn't have to document which
  • Implementation-defined behavior — unspecified, but the implementation must document its choice consistently (c3-1's bit-field ordering is exactly this)
  • Undefined behavior — the standard imposes literally no requirements at all. Anything is a conforming result — including things that look nonsensical, like appearing to work perfectly, or genuinely misbehaving in a way that has nothing obviously to do with the actual bug

Why the Compiler Is Allowed to Assume UB Never Happens

This is the real, counter-intuitive payoff. Compilers don't merely fail to detect UB — they actively use its absence as a proven fact during optimization. If code contains a signed overflow, the compiler is standard-sanctioned in assuming it never actually occurs, and can eliminate branches or checks that logically depended on that "impossible" case — producing results that look like a compiler bug, but are the compiler correctly exploiting a guarantee the standard itself grants it. The C community's own phrase for the theoretical extreme of this is that UB could licitly make "demons fly out of your nose" — a real, standard-permitted outcome, however absurd it sounds.

A Consolidated Catalog, By Chapter

  • Signed integer overflowc1-2
  • Out-of-bounds array accessc1-6
  • Dereferencing NULL or a dangling pointerc1-7
  • Buffer overflow via strcpyc1-8
  • Use-after-free, double-freec2-2, c2-3
  • Reading the "wrong" union memberc2-1, in most cases
  • Excessive or negative shiftsc3-1
  • An unprotected data racec3-3

Signed Overflow, Revisited With the Real Mechanism

for (int i = 0; i + 1 > i; i++) { // intended to stop once i reaches INT_MAX }

The programmer intended this loop to terminate once i overflows. But since signed overflow is UB, the compiler is entitled to assume i + 1 > i is always true — overflow "can't happen" — and may optimize this into a genuine infinite loop, silently removing the termination condition the programmer relied on. This is a real, documented class of surprising optimizer behavior, not a hypothetical.

Strict Aliasing

A new rule: the compiler assumes two pointers of different, unrelated types never point at the same memory (with narrow exceptions like char *). Reinterpreting a float * as an int * and dereferencing it violates this — undefined behavior — and permits the compiler to reorder or cache reads through such pointers in ways that produce genuinely wrong results if the assumption turns out to be false. c2-1's union type-punning gotcha is closely related: a union is one of the narrow, standard-sanctioned ways to reinterpret bits in some cases — a raw pointer cast between unrelated types is not, and violates strict aliasing outright.

Rust's Answer

Rust's entire type system and borrow checker exist substantially to make every category on this chapter's own catalog structurally impossible to write in safe code — not detected as a separate check layered on top, but literally inexpressible. rust3-2's own unsafe blocks are the deliberate, opt-in escape hatch — the one place a Rust programmer takes on exactly the same responsibility C always has, by choice, in a clearly marked and narrow scope.

UB categoryCRust
Signed overflowUB — compiler may assume it never happenspanics in debug builds, wraps in release — always defined
Out-of-bounds accessUB — no check at allpanics — checked on every access
Type punningUB via raw pointer casts; unions are narrowly sanctionedrequires an explicit unsafe block
UBSan is a different tool from ASan
-fsanitize=undefined compiles in instrumentation specifically for UB classes like signed overflow and strict-aliasing violations — genuinely distinct from c2-3's AddressSanitizer, which targets memory-safety bugs specifically. Real projects commonly run both together.
UB isn't a rare edge case in obscure code
Several of this chapter's own catalog entries — signed overflow, an off-by-one array access, a forgotten free — are genuinely easy to write by accident in completely ordinary-looking code. Treating UB as something only exotic, unusual programs risk is itself a dangerous assumption.

Coding Challenges

Challenge 1

For each entry in this chapter's consolidated UB catalog, name which specific chapter first introduced it and, in one sentence each, what triggers it.

📄 View solution
Challenge 2

Explain why "the compiler is allowed to assume UB never happens" is a genuinely different, stronger claim than "the compiler doesn't check for UB" — and why that distinction is what makes the i + 1 > i infinite-loop example possible.

📄 View solution
Challenge 3

Explain the difference between unspecified, implementation-defined, and undefined behavior, giving one concrete example of each from this course.

📄 View solution

Chapter 4 Quick Reference

  • Unspecified — one of several allowed results, undocumented; implementation-defined — unspecified, but documented; undefined — no requirements at all
  • The compiler actively assumes UB never happens, and optimizes on that assumption — not merely "fails to catch it"
  • A full UB catalog spans nearly every chapter of this course, from signed overflow to data races
  • Strict aliasing — pointers of unrelated types are assumed never to overlap; violating it is UB
  • -fsanitize=undefined (UBSan) — a distinct tool from ASan, targeting overflow/aliasing-class UB specifically
  • Rust's type system makes this entire catalog structurally inexpressible in safe code — unsafe is the deliberate, narrow opt-out
  • Next chapter: Debugging & Tooling — gdb, valgrind, ASan/UBSan, and static analysis, brought together