Principal lecturer: Prof Neil Lawrence
Taken by: MPhil ACS, Part III
Code: L172
Term: Michaelmas
Hours: 16 (8 × 2hr lectures)
When: Tuesdays, 13 October – 1 December 2026
Where: FW26, William Gates Building
Class limit: max. 10 students
Prerequisites: Undergraduate-level probability and statistics (distributions, Bayes theorem), linear algebra (matrix operations, eigenvalues), and basic multivariate calculus. No prior physics or thermodynamics assumed.
Develop the mathematical connections between thermodynamics, information theory, and Bayesian inference, and understand how they provide a toolkit for reasoning about the foundations of intelligent systems.
Entropy appears in three apparently separate traditions — thermodynamics (Boltzmann, Gibbs), information theory (Shannon), and Bayesian inference (Jaynes) — and turns out to be the same mathematical object viewed from different operational assumptions. The operational split is: entropy forbids, probability prescribes. The course covers:
- that theme, via perpetual motion and the human–machine bandwidth gap of The Atomic Human; the Boltzmann distribution and free energy,
- Shannon entropy and its formal equivalence to thermodynamic entropy; the exponential family as the MaxEnt family,
- Maxwell's demon and Landauer's principle: the thermodynamic cost of decision-making,
- information geometry: the Fisher metric, Crooks' thermodynamic length, dually flat geometry, and natural gradient descent,
- multi-information, an entropy game, quantum information,
- probability transport, Schrödinger bridges, and information-theoretic limits on intelligent agency.
Four ten-minute in-class Moodle quizzes sit at the start of lectures 3, 5, 7 and 8.
Each lecture is followed by a formative self-study exercise exploring the session's central concept from thermodynamic, information-theoretic, and Bayesian perspectives.
Equip students to reason rigorously about entropy across thermodynamics, information theory, and Bayesian inference, and to evaluate claims about intelligent systems using information-theoretic constraints.
Four take-home mini-project worksheets (15% each, 60% total), each consisting of a short Python notebook and a written reflection. Four short in-class Moodle quizzes (10% each, 40% total).
Shannon (1948), Bell System Technical Journal; Jaynes (1957), Physical Review; Landauer (1961), IBM Journal; Amari & Nagaoka (2000), Methods of Information Geometry (Chapters 1–3); Crooks (2007), Physical Review Letters. Cover & Thomas (2006) and MacKay (2003) as background. Welling, Lu and Holdijk (2026), Generative AI and Stochastic Thermodynamics (GAIST), is a supplementary monograph: Chapter 3 for free energy, Chapter 5 for the ELBO, Chapters 14 and 22 for the Wasserstein and Schrödinger-bridge geometries. It is not a substitute for Crooks (2007) or Amari.