Lectures

Important1. Introduction & Motivation

• Date: August 25, 2025 (Monday)
• Slides
• Papers mentioned:
  • What is information? by Christoph Adami, 2016
  • Different languages, similar encoding efficiency… by Coupé et al., 2019
  • Efficient compression in color naming… by Zaslavsky et al., 2018
  • Thermodynamics of information by Parrondo et al., 2015

Important2. Probability Theory I

• Date: August, 27 (Wednesday)
• Lecture Notes
• Reference: Gardiner Chapter 2

Important3. Probability Theory II

• Date: September 03, 2025 (Wednesday)
• Lecture Notes
• Reference: Gardiner Ch 2
• Random walk demonstration

Important4. Entropy

• Date: September 08, 2025 (Monday)
• Lecture Notes
• Reference: Cover & Thomas Ch 2
• Shannon’s original paper

Important5. Applications of Entropy I (Fano’s inequality, Statistical Mechanics)

• Date: September 10, 2025 (Wednesday)
• Lecture Notes
• Reference: Cover & Thomas Ch 2 & Parrondo/Horowitz/Sagawa Paper

Important6. Applications of Entropy II (Landauer’s principle, AEP)

• Date: September 15, 2025 (Monday)
• Lecture Notes
• Reference: Parrondo/Horowitz/Sagawa Paper & Cover & Thomas Ch 3

Important7. Applications of Entropy III (AEP, Coding & Compression)

• Date: September 17, 2025 (Wednesday)
• Lecture Notes
• Reference: Cover & Thomas Ch 3, 5

Important8. Applications of Entropy IV (Optimal coding & Phase Transitions)

• Date: September 22, 2025 (Monday)
• Lecture Notes
• Reference: Cover & Thomas Ch 5, Vogel/Saravia/Cortez paper
• Compressing a phase transition code

Important9. Relative Entropy

• Date: September 24, 2025 (Wednesday)
• Lecture Notes
• Reference: Cover & Thomas Ch 2

Important10. Applications of relative entropy

• Date: September 29, 2025 (Monday)
• Lecture Notes
• Reference: Parrondo/Horowitz/Sagawa Paper & Cover & Thomas Ch 5
• Other references: Thermodynamics of information book chapter

Important11. Mutual information, Data Processing Inequality, and Sufficient Statistics

• Date: October 1, 2025 (Wednesday)
• Lecture Notes
• Reference: Cover & Thomas Ch 2

Important12. Channel Coding I

• Date: October 6, 2025 (Monday)
• Lecture Notes
• Reference: Cover & Thomas Ch 7

Important13. Channel Coding II

• Date: October 8, 2025 (Monday)
• Lecture Notes
• Reference: Cover & Thomas Ch 7

Important14. Horse Racing

• Date: October 20, 2025 (Monday)
• Lecture Notes
• Reference: Cover & Thomas Ch 6

Important15. Natural Selection

• Date: October 22, 2025 (Wednesday)
• Lecture Notes
• Reference: Cover & Thomas Ch 6 and Rivoire/Leibler

Important16. Natural Selection (+ Continous Random Variables)

• Date: October 27, 2025 (Monday)
• Lecture Notes
• Reference: Rivoire/Leibler & Cover & Thomas Ch 9

Important17. Differential entropy

• Date: October 29, 2025 (Wednesday)
• Lecture Notes
• Reference: Cover & Thomas Ch 9

Important18. Analog communication

• Date: November 3, 2025 (Monday)
• Lecture Notes
• Reference: Cover & Thomas Ch 9

Important19. Biological Signal Transduction

• Date: November 5, 2025 (Wednesday)
• Lecture Notes
• Reference: Bialek Section 6.4

Important20. Maximum Entropy Estimates

• Date: November 10, 2025 (Monday)
• Lecture Notes
• Reference: Cover & Thomas Ch 12

Important21. Exponential Random Graph Models & Stochastic Processes

• Date: November 12, 2025 (Wednesday)
• Lecture Notes
• References: Fronczak and Cover & Thomas Ch 4

Important22. Entropy Rates of Stochastic Processes

• Date: November 17, 2025 (Monday)
• Lecture Notes
• References: Cover & Thomas Ch 4