Entropy and Information: Concepts and Applications CMPLXSYS 445 Fall 2025, University of Michigan

Syllabus

Course number: CMPLXSYS 445
Course Title: Entropy and Information: Concepts and Applications
Class Time & Location: Mondays & Wednesdays, 10:00-11:30AM; East Hall, Room B844 (Basement)

Instructor: Bhaskar Kumawat (kumawatb@umich.edu)
Office hours location: 4052 Biological Sciences Building
In-person office hours: Tuesdays 10:00 AM - 11:00 AM
Zoom office hours: Thursdays 1:30 PM - 2:30 PM

Course description

Information theory was initially proposed by Claude Shannon as a tool to measure the theoretical limits of communication. Since then, the ideas from information theory have found applications beyond engineering, especially in the natural sciences where we wish to measure the performance limits of complex natural (or built) systems. This course introduces the fundamental ideas in Information Theory—Entropy, Relative Entropy, and Information—and is aimed towards students that wish to apply these principals to systems across the natural sciences. While the question of “What is Information?” is of fundamental importance, we will not be addressing it here. We will instead focus on the types of questions that information theory can (and cannot) answer and see how it can be applied to logical problem solving. We will discuss a few of these applications in detail—in evolution, physics, computation, and finance—and conclude with student led presentations on topics that can utilize information-theoretic reasoning. The course is primarily lecture and assignment based, but will have oppurtunities for active learning through interactive simulations and the final presentation.

Prerequisites

A first course in undergraduate probability theory is strongly recommended (eg. MATH 425). While we will go over the required probability concepts in the first few weeks, previous exposure to the concepts will be provide a smoother transition to the idea of information. Familiarity with any one scientific computing package is also recommended for the computational problems in the assignments (eg. python, julia, MATLAB, mathematica, R).

Assessment

Assignments (50%)

Assignments will be due every Thursday at 11:59 PM (Midnight) via Gradescope. Each assignment will be worth 20 points, and 2 points will be deducted for each day late up to 3 days at which point it will receive a 0. The lowest grade among the assignments will be dropped. All questions should be posted to the Ed discussion board (also linked at the top of the webpage).

Midterm exam (20%)

A take-home midterm exam will be administered during the 24 hour period from Wednesday Oct. 16, 10:00 AM - Thursday Oct. 17, 10:00 AM. There will be no class on Wednesday, Oct. 16. The exam will be open-book/notes, and you will be able to use a scientific computing package if you wish.

Presentation (10%)

Part of the aim of this course is to illustrate the tools of information theory through applications. Since there are too many for me to cover, each of you will be responsible for highlighting one such additional application. During the last two classes, each of you will be responsible for a short (5 minutes max.) presentation on any topic related to information theory. A list of possible/suggested topics are below. A recorded practice presentation will be due Oct. 7 (2.5%, graded complete/incomplete). The following week a subset of your peers will review and provide constructive criticism due Nov. 15 (2.5%, graded complete/incomplete). The presentations will be scored on accuracy and clarity of presentation (5%).

Some topic ideas: Maximum Caliber, Maximum Entropy, Coherence of Biochemical Oscillators, Gene Transcription, Quantum Information, Information Fluctuation Theorems, Information Geometry/Fisher Information, Inference, Kolmogorov Complexity, Tuning a Piano, Language, Community Detection, Transfer Entropy, Machine Learning.

Final Exam (20%)

The final exam will be in-person, on Monday, Dec. 15 4:00 - 6:00 pm. The exam will be open-notes, which in this case means that you can bring and use a) Class notes that you’ve taken yourself, and b) your own solutions to the homework problems. If your copy of the book or notes are on a device such as a tablet, phone, or laptop then you may bring and use that device solely for the purpose of viewing the notes or homework solutions. You should also bring some paper and a pen.

Course Policies

Attendance

The course is primarily in-person and attendance is expected (but not enforced) as you are responsible for all material presented in class. While an abridged, typed version of the lecture notes will be provided, you will be allowed to bring only hand-written notes for the final exam. As such, taking your own handwritten notes during the lectures is essential.

Grading

Final letter grades will be assigned based on a traditional scale with A’s (90% and above), B’s ( 80-89%), C’s (70-79%), D’s (60-69%), and E’s (59% and below).

Recording

Any of the course lectures may be audio/video recorded and made available to other students in this course (but will not be shared outside the class). While these audio/video recordings are meant to record only the instructor, student audio/video may be recorded in the process. If you are concerned about being recorded, please contact the instructor during the first week of class.

Students may not record or distribute any class activity without written permission from the instructor, except as necessary as part of approved accommodations for students with disabilities.

Illness & Internet connectivity

If you are unable to attend class because of an illness (physical or mental), taking care of an ill relative or are experiencing an injury/surgery impacting your ability to participate, please let the instructor know as soon as possible. In these occasional cases, recordings of the lectures or lecture notes will be made available. Please use this link to report an illness to the instructor. If these issues become chronic, you will work with the instructor to find a more permanent solution.

Student Mental Health and Well-Being

The University of Michigan is committed to advancing the mental health and wellbeing of its students. If you or someone you know is feeling overwhelmed, depressed, and/or in need of support, services are available. For help, contact Counseling and Psychological Services (CAPS) at (734) 764-8312 and https://caps.umich.edu during and after hours, on weekends and holidays, or through its counselors physically located in schools on both North and Central Campus. You may also consult University Health Service (UHS) at (734) 764-8320 and https://uhs.umich.edu/mentalhealthsvcs, or for alcohol or drug concerns, see https://wolverinewellness.umich.edu/know-thefacts. For a listing of other mental health resources available on and off campus, visit https://uhs.umich.edu/mentalhealthsvcs.

Accessibility and Accommodations

The University of Michigan recognizes disability as an integral part of diversity and is committed to creating an inclusive and equitable educational environment for students with disabilities. Disability can include: mental health conditions, ADHD, learning disabilities, autism, chronic illness, physical conditions, sensory conditions, and more. If you anticipate or are experiencing barriers based on disability or temporary injuries, Services for Students with Disabilities (SSD) (ssd.umich.edu) is the office that students work with to explore reasonable accommodations, tools, and resources.

If you are already connected with SSD and have approved accommodations, please share your letter through Accommodate as soon as possible so that we can discuss how your accommodations will be implemented in this course. The sooner I know about your disability access-needs, the more equipped I can be to facilitate accommodations. You should reach out to me and/or your Disability Access Coordinator if you have any questions or concerns about your accommodations.
If you have not connected with SSD and anticipate or are experiencing a disability-related barrier, and would like to discuss accommodations and/or resources, please contact SSD by completing their initial information form (https://ssd.umich.edu/)
If you have a temporary medical injury/condition, such as a broken arm, I may be able to assist in minimizing classroom barriers. In situations where additional assistance is needed, you should contact the SSD as noted above.

For more information, call 734-763-3000 or email ssdoffice@umich.edu.

Generative AI

Work generated by AI tools like ChatGPT is a source that may inform your writing, just as books, papers, lecture notes, websites, and the results of Google searches can. As always, it’s your responsibility to assess the reliability of every source you use. In that context, you should be aware that the output of AI tools is constructed to sound plausible, and not checked by any expert. These results are often completely invented.

The use of generative AI tools (e.g. U-M GPT, ChatGPT, Dall-e, etc.) is permitted in this course for the following activities:

Brainstorming and refining your ideas;
Finding sources of information on a topic;
Drafting an outline to organize your thoughts; and
Checking grammar and style.

The use of generative AI tools is not permitted in this course for the following activities:

Impersonating you in a learning context, such as by using the tool to compose peer-reviews, solutions to problems etc.
Writing a draft of an assignment problem.
Writing entire sentences or paragraphs for assignments or presentations.

Academic Misconduct & Plagiarism

The University of Michigan community functions best when its members treat one another with honesty, fairness, respect, and trust. The college promotes the assumption of personal responsibility and integrity, and prohibits all forms of academic dishonesty and misconduct. Thus, homework assignments and exams must reflect your own work. The first plagiarized assignment will be given zero points. Further, acts of academic misconduct will be referred to the LSA Office of the Assistant Dean for Undergraduate Education. Being found responsible for academic misconduct will usually result in a grade sanction, in addition to any sanction from the college. For more information, including examples of behaviors that are considered academic misconduct and potential sanctions, please see lsa.umich.edu/lsa/academics/academic-integrity.html

Title IX Statement

Title IX prohibits discrimination on the basis of sex, which includes sexual misconduct — including harassment, domestic and dating violence, sexual assault, and stalking. We understand that sexual violence can undermine students’ academic success and we encourage anyone dealing with sexual misconduct to talk to someone about their experience, so they can get the support they need. Confidential support and academic advocacy can be found with the Sexual Assault Prevention and Awareness Center (SAPAC) on their 24-hour crisis line, 734.936.3333 and at sapac@umich.edu.

Alleged violations can be non-confidentially reported to the Equity, Civil Rights, and Title IX Office (ECRT) at ecrtoffice@umich.edu.