Welcome to HST 953!

HST.953 Fall 2021

Friday, 9:30 AM - 12:30 PM

E25-117

Instructors: Dr. Marzyeh Ghassemi, Dr. Leo Celi

Course Staff: Dr. Eric Gottlieb, Dr. Ned McCague, Dr. Kennith Paik

TA: Abbas Zeitoun

Syllabus

Overview

HST.953 is a course about the practical considerations for operationalizing machine learning in healthcare settings. We begin the course with a focus on robust, private and fair machine learning (ML) using real retrospective healthcare data. We follow this with experiences in visualization (VIS) that target utility and clinical value. Finally, we explore the intermediate "implementation science" (IMP) tying together how real models might be potentially used through a visual system by practicing clinical staff.

The course will involve three homework assignments (one each on machine learning, visualization and implementation) followed by a course project proposal and presentation.

All students are required to complete human subjects training and submit proof of access for MIMIC-III and the eICU-CRD databases.
All students regardless of their enrollment status are expected to join a project group and contribute to a final project.

HST.953 is not intended to teach graduate machine learning or visualization skills to students, and we expect that students will have some working knowledge of both in order to complete homework assignments and the project.

We recommend the following courses, or some equivalent experience with subject matter in ML, visualization and HCI:

Recommended Courses:
CS Grad ML 6.867
CS Grad ML in Health 6.S897 / HST.956
CS Grad Visualization 6.813

Grading

Weekly Reflections/edX: While everyone might benefit from both edX content and weekly readings, you will be asked at the beginning of the semester to choose between completing weekly reflections or completing the Collaborative Data Science for Healthcare edX course.

The weekly reflections, corresponding to Week 2 - Week 10, will be done as a Canvas discussion, are due before class, and are worth 1 point (1.67% of your grade) per week. This means that reflections are worth a total of 15% of your grade.
Alternatively, you may choose to complete the Collaborative Data Science for Healthcare course throughout the semester to earn that 15% of your grade. The course may be accessed here: https://www.edx.org/course/collaborative-data-science-for-healthcare

Three Problem Sets: Problem sets 1, 2, and 3 are each work 10 points, or 16.67% of your grade. This means problem sets are worth a total of 50% of your grade.

Course Final Project: The submission of the project teams is worth 1 point (1.67% of your grade), the final project presentation is worth 10 points (16.67% of your grade) and the final project write up is worth 10 points (16.67% of your grade).

Plagiarism: Student code submissions may be submitted by the instructors to a plagiarism detection tool for a review of similarity and detection of possible plagiarism. Submissions will be used solely for the purpose of detecting similarity, and are not retained indefinitely on the server; typically results are deleted after 14 days but may be removed sooner. For more information on the tool used, refer to https://theory.stanford.edu/~aiken/moss/.

Extra Credit

Scribing: We have a weekly extra credit of 1 point for Latexing/scribing the lectures for the week, and submitting them to the class for review. A maximum of 5 extra credit points can be earned per student.

Schedule

Week	Date	Lecture	Materials	Assignments
1/ML	Sept 10, 2021	9:30 - 10:10 Dr. Marzyeh Ghassemi "Course Introduction and Overview" 10:20 - 11:20 Dr. Steven Horng and Dr. David Sontag "Strong Partnerships to Create Deployable Models and Health" 11:30 - 11:50 Ned McCague "Project Overview; Why Does This Even Matter?" 12:00 - 12:30 Dr. Leo Celi "Past Experiences with HST 953 and Course Advice"	[Slides] HST 953 - Lecture 1.pdf A Review of Challenges and Opportunities in Machine Learning for Health Ethical Machine Learning in Healthcare Do no harm: a roadmap for responsible machine learning for health care Fast, Structured Clinical Documentation via Contextual Autocomplete Improving Documentation of Presenting Problems in the Emergency Department Using a Domain-Specific Ontology and Machine Learning-Driven User Interfaces Electronic Medical Record Phenotyping Using the Anchor and Learn Framework	Problemset 1 Out
2/ML	Sept 17, 2021	9:40 - 10:40 Dr. Alistair Johnson "Creating Clinical Databases For Machine Learning" 10:40 - 12:30 Dr. Marzyeh Ghassemi - "Project Pitches"	[Slides] Guest Lecture - Alistair Johnson [Lecture] Zoom Readings: MIMIC-III, a freely accessible critical care database Reproducibility in machine learning for health research: Still a ways to go Recurrent Neural Networks for Multivariate Time Series with Missing Values Extra Material: [Video] Deep dive into MIMIC-IV [Code] Deep dive into MIMIC-IV [Video] Going through a study done in MIMIC-IV [Code] Going through a study done in MIMIC-IV	Week 2 Reflection Due
3/ML	Sept 24, 2021	9:40 - 10:40 Dr. Stephanie Hyland - "Representation learning in ML and Health" 11:00 - 12:00 Dr. Andrew Beam "Healthcare Data Horror Stories"	RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism Temporal Pointwise Convolutional Networks for Length of Stay Prediction in the Intensive Care Unit Reading Race: AI Recognises Patient's Racial Identity In Medical Images. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study	Week 3 Reflections Due Group Project Signup
4/VIS	Oct 1, 2021	9:40 - 10:40 Dr. Emma Pierson - "Using Machine Learning to Increase Health Equality" 11:00 - 12:30 Dr. Ziad Obermeyer - "Dissecting Algorithmic Bias"	Predicting A While Hoping For B Algorithmic Bias Playbook An algorithmic approach to reducing unexplained pain disparities in underserved populations Extra Material: Algorithmic Bias Initiative	Problemset 1 Due Week 4 Reflections Due Problemset 2 Out
5/VIS	Oct 8, 2021	9:40 - 10:40 Dr. Fanny Chevalier - "Visualization in Healthcare" 11:00 - 12:30 "Project Check In Time Available"	Data-Driven Healthcare: Challenges and Opportunities for Interactive Visualization CarePre: An Intelligent Clinical Decision Assistance System PhenoLines: Phenotype Comparison Visualizations for Disease Subtyping via Topic Models PROACT: Iterative Design of a Patient-Centered Visualization for Effective Prostate Cancer Health Risk Communication Extra Material: Why Doctors Hate Their Computers	Week 5 Reflections Due
6/VIS	Oct 15, 2021	9:40 - 10:40 Dr. Sonali Parbhoo 11:00 - 11:40 Ned McCague	Biases in electronic health record data due to processes within the healthcare system: retrospective observational study Can You Trust This Prediction? Auditing Pointwise Reliability After Learning Second opinion needed: communicating uncertainty in medical machine learning Pre-emptive learning-to-defer for sequential medical decision-making under uncertainty Communicating uncertainty about facts, numbers and science	Week 6 Reflections Due
7/IMP	Oct 22, 2021	9:40 - 10:40 Dr. Anna Goldenberg - "Lessons Learned for ML for Deployment in Healthcare" 11:00 - 12:00 Anika Heavener - "And So What? Measuring Deployments from Financial, Clinical and Patient Experience."	External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients Forecasting Emergency Department Capacity Constraints for COVID Isolation Beds Feedback Loop and Bias Amplification in Recommender Systems	Week 7 Reflections Due Problemset 2 Due Problemset 3 Out
8/IMP	Oct 29, 2021	9:40 - 10:40 Dr. Muhammad Mamdani 11:00 - 12:00 Dr. Vincent X. Liu	Automated Identification of Adults at Risk for In-Hospital Clinical Deterioration Evaluation of an intervention targeted with predictive analytics to prevent readmissions in an integrated health system: observational study Development of a healthcare system COVID Hotspotting Score in California: an observational study with prospective validation Implementing machine learning in medicine Problems in the deployment of machine-learned models in health care Evaluation of machine learning solutions in medicine	Week 8 Reflections Due
9/IMP	Nov 5, 2021	9:40 - 10: 40 Dr. Tristan Naumann - "NLP for Health Case Study" 11:00 - 12:00 "Mid-Point Project Presentations"	[Slides] Natural Language Processing for Healthcare The false hope of current approaches to explainable artificial intelligence in health care	Week 9 Reflections Due
10/IMP	Nov 12, 2021	9:40 - 10:40 Donny Cheung 11:00 - 12:00 Project Help	Artificial intelligence sepsis prediction algorithm learns to say “I don’t know”	Week 10 Reflections Due Problemset 3 Due
11	Nov 19, 2021	Project Work Week, In-class Help
12	Nov 26, 2021	THANKSGIVING WEEK, Project Work		Final Project Presentations
13	Dec 3, 2021	Final Presentations		Final Project Reports Due

Project Details

Projects and Authorship

A note on collaboration: Research is a collaborative activity and we encourage all students to collaborate and learn from each other. In general, when you put your name on something for research, you must: a) have materially contributed to the work, b) be able to defend the
research, and c) acknowledge the contribution of others. Keep this in mind when working together and submitting material for evaluation.

A note on authorship: As noted, the expectation is that by the end of the course the final project will be sufficiently developed to submit to a peer-reviewed journal. The author order can be a somewhat controversial issue and is left to the project participants to decide. We would strongly encourage you to discuss what the order will be, or what philosophy you will use to decide the order while forming groups. In the case of a dispute during or after the course, the instructors will likely not be able to mediate in any meaningful way. We would also recommend equal authorship (now more common), but the decision is left to each team.

For the clinicians: If you expect a certain level of authorship (first, last, etc.) you should mention this in your project pitch. Keep in mind that this is a two-way street involving both clinicians and data scientists. If a project fails to garner enough interest, it may not be able to be completed as part of the course.

A note on acknowledgement: Papers that result from work done during this course should recognize the contributions of the course in an acknowledgement or in other sections. The suggested language is: "This manuscript was composed by participants in the HST.953 course at the Massachusetts Institute of Technology, Fall 2021.'"

Project Descriptions

Intravenous Insulin Infusion, DKA, Hypoglycemia, and Machine Learning - Zach Taxin
Identifying Acid-Base Status in Patients in the ICU - Nathan Raines and Marcus Foo
Mapping Serum Lactate to Outcomes Across Race-Ethnicities - Brian Bustos
Simulation of Time-Limited Trials for Critically-Ill Patients on Mechanical Ventilation - Anthony O’brien
Relative Hypoglycemia and Mortality in Critically-Ill Patients - Sreekar Mantena
Carbon Dioxide Gap in the ICU: How Can We Evaluate Tissue Perfusion at the Bedside? - Asher Mendelson
AKI Trajectory Across Marginalized ICU Populations - Eric Gottlieb and Periklis Kyriazis
Trends in Sepsis Survival - Traci King and Sunil Nair
Two's a Party, Three's a Crowd: The Three Presser Problem - Emmett Kistler, Gloria Hyunjung and Leo Celi
Predicting COPD from Chest X-rays - Patrick Seastedt

Clinical Collaborators

Bermea, Rene (BIDMC)

Foo, Marcus (BIDMC)

Gottlieb, Eric (MIT)

King, Traci (BIDMC)

Kyriazis, Peri (BIDMC)

Litchman, Tess (BIDMC)

Maley, Jason (MIT)

Mendelson, Asher (Winnipeg)

Mohammed, Sara (HMS)

Nair, Sunil (Pennsylvania)

O'Brien, Anthony (Boston)

Raines, Nate (BIDMC)

Rush, Barret (Winnipeg)

Seastedt, Pat (BIDMC)

Taxin, Zachary (BIDMC)

Ziegler, Jen (Winnipeg)

Data Science Mentorship

We realize that there are many backgrounds engaging with this course, and have a volunteer group of 15 data science volunteers who will be assigned to sets of 2-3 students and can assist with homework and project questions. You will receive your mentor assignment the second week of class.

HST.953 Clinical Data Learning, Visualization, and Deployments