Performance

Python Advanced — Performance

Python Advanced

Course 3 · Chapter 7 · Performance

🚀 Performance

Python trades raw speed for developer productivity by design — but "slow" is relative, and most Python programs are never actually CPU-bound in the way that matters. This chapter covers profiling first (finding the real bottleneck before touching anything), common optimization patterns once you've found one, and a brief, honest overview of Cython and PyPy for the cases that genuinely need more.

🔬 Profiling with cProfile

# from the command line, profiling an entire script:
python -m cProfile -s cumulative script.py
        

# or programmatically, profiling just one function:
import cProfile

def slow_function():
    total = sum(n ** 2 for n in range(1_000_000))
    return total

cProfile.run("slow_function()")
        

# output (abbreviated):
#          3 function calls in 0.089 seconds
#   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#        1    0.089    0.089    0.089    0.089 script.py:3(slow_function)
        

ncalls is how many times a function was called; tottime is time spent inside that function alone (excluding calls it made to other functions); cumtime is total time including those nested calls. Sorting by cumtime (as -s cumulative does) quickly surfaces which function is actually responsible for the most total time — the real bottleneck, rather than a guess.

⚠ Never Guess at a Bottleneck — Always Profile First

Intuition about which part of a program is "the slow part" is wrong astonishingly often — the real cost is frequently somewhere unglamorous and easy to overlook, like a function called thousands of times inside a loop rather than one dramatic-looking computation. Optimizing code without profiling first risks spending real effort making an already-fast piece of code marginally faster while the actual bottleneck goes untouched. Profile first, every time — it's often described as "premature optimization is the root of all evil," and the fix is simply: measure, don't assume.

⚡ Common Optimization Patterns

# prefer built-ins (implemented in C) over manual Python loops:
total = sum(numbers)              # fast
total = 0
for n in numbers: total += n   # noticeably slower for large inputs

# use a set for membership tests, not a list (Course 1, Chapter 6):
if item in allowed_set:    # O(1) — fast regardless of size
if item in allowed_list:   # O(n) — scans the whole list every time

# cache expensive repeated calls (Course 2, Chapter 6):
@lru_cache
def expensive_lookup(key):
    ...
        

Most real Python speedups come from reaching for the right tool rather than micro-tuning syntax: sum()/max()/min() and other built-ins run their loop in C, not interpreted Python bytecode; a set's in check is a hash lookup instead of a linear scan; @lru_cache (Course 2, Chapter 6) eliminates redundant work entirely rather than making it faster. Each of these is a genuine "use the right data structure/tool," not a clever trick.

🧊 Cython Overview

# a .pyx file — Python syntax, with optional C-style type declarations:
def sum_squares(int n):
    cdef int i, total = 0
    for i in range(n):
        total += i * i
    return total
        

Cython is a superset of Python that compiles to C — you write mostly ordinary Python, optionally adding static type declarations (cdef int i) in the hottest sections, and Cython compiles those parts down to genuine C-speed code. It's a targeted tool: used for the specific hot loop or numeric kernel a profiler identified as the real bottleneck, not for rewriting an entire application.

⚙️ PyPy Overview

PyPy is an alternative Python interpreter (a drop-in replacement for the standard CPython interpreter) with a Just-In-Time (JIT) compiler — it watches which code paths actually run repeatedly and compiles those to machine code on the fly, often making long-running pure-Python programs several times faster with zero code changes. The trade-off: PyPy's compatibility with C extensions (many popular data-science and numerics libraries) has historically lagged behind CPython, so it's not always a safe drop-in for every project.

Baseline Speed: Go/Rust vs Python

Language	Why it starts faster
Go / Rust	Compiled ahead of time to native machine code, with static types checked before the program ever runs — no interpreter overhead, no runtime type-checking cost.
Python	Interpreted, dynamically typed by default (Course 1, Chapter 2; Chapter 5 of this course) — every operation carries interpreter and type-checking overhead Go/Rust simply don't have.

This gap is real and expected — it's the trade Python makes for its flexibility and development speed. The right response is never "avoid Python for anything performance-sensitive," but rather: profile, optimize the specific hot path (often with the patterns above, or Cython for the truly critical inner loop), and accept that most of a typical program was never the bottleneck to begin with.

cProfile

Finds the real bottleneck — ncalls, tottime, cumtime per function.

Reach for the right tool

Built-ins, sets for membership, @lru_cache — usually bigger wins than micro-tuning.

Cython

Compile hot Python code to C, with optional static type declarations.

PyPy

A JIT-compiled alternative interpreter — often faster, with some C-extension trade-offs.

💻 Coding Challenges

Challenge 1: Profile Two Approaches

Write two functions that each sum the squares of numbers from 0 to 999,999 — one using a manual for loop with +=, one using sum() with a generator expression (Course 2, Chapter 5). Profile both with cProfile.run() and compare their tottime.

Goal: See the built-in-vs-manual-loop performance difference measured directly, not just asserted.

→ Solution

Challenge 2: List vs Set Membership, Timed

Build a list and a set, each containing the numbers 0 to 99,999. Time how long 10,000 in checks take against the list, versus the same 10,000 checks against the set, using time.time().

Goal: Directly measure the O(n) vs O(1) membership-test difference this chapter describes.

→ Solution

Challenge 3: Identify the Bottleneck

Given a function process() that calls slow_helper() 1,000 times in a loop, and slow_helper() itself does a small, fast computation but is simply called very often — write the cProfile.run("process()") call needed to find this, and explain in a comment which output column (ncalls, tottime, or cumtime) would make this specific pattern (many cheap calls, not one expensive one) obvious.

Goal: Practice reading profiler output to distinguish "called too often" from "individually slow," a real and common distinction.

→ Solution

🎯 What's Next

Next chapter: Capstone: Building a Real CLI Tool — combining OOP, type hints, packaging, and testing into one small real tool, closing out Python Advanced.