A little about me…
Christopher Teixeira
Principal Data Scientist
The MITRE Corporation
Interests
- Data Analytics
- Applied Statistics
- Operations Research
Education
- MS in Operations Research, George Mason University
- BSc in Mathematics, Worcester Polytechnic Institute
What is machine learning?
Machine Learning is the study of computer algorithms that improve automatically through experience. Applications range from data mining programs that discover general rules in large data sets to information filtering systems that automatically learn users’ interests.
Supervised Learning
Unsupervised Learning
Reinforcement Learning
Machine learning can be pretty amazing
Search and rescue with airborne optical sectioning
Source: Nature.com
Machine learning can be pretty amazing
Using Data to Create Paths out of Homelessness
Source: DataKind.org
Sometimes things can go wrong…
Racial bias in health algorithms
The U.S. health care system uses commercial algorithms to guide health decisions. Obermeyer et al. find evidence of racial bias in one widely used algorithm, such that Black patients assigned the same level of risk by the algorithm are sicker than White patients (see the Perspective by Benjamin). The authors estimated that this racial bias reduces the number of Black patients identified for extra care by more than half. Bias occurs because the algorithm uses health costs as a proxy for health needs. Less money is spent on Black patients who have the same level of need, and the algorithm thus falsely concludes that Black patients are healthier than equally sick White patients. Reformulating the algorithm so that it no longer uses costs as a proxy for needs eliminates the racial bias in predicting who needs extra care.
Credit: The Washington Post; iStock
Sometimes they can be even worse…
Child abuse and neglect fatalities
Illinois Department of Child and Family Services used an algorithm for determining future risk of abuse or neglect.
However, several children were found dead after having received a score of “0”, or very little risk of abuse or neglect.
Credit: Chicago Tribute
Consider the typical lifecycle for a machine learning project
- Each step presents the possibility to introduce errors
- Experience will help prevent the most common issues
- Communication can be the most important skill to avoid common issues
- Some aspects might be out of your control, like how the model is integrated into a larger system
Source: Microsoft
Consider the business case or problem
Data, data, data
- Did you examine the data you collected?
- Did you introduce any bias in how it was cleaned?
- How did you deal with missing data or outliers?
- Did something change in the data pipeline?
- How did you calculate that feature?
Credit: Astera
Verify and validate your model results
- How did you sample your training dataset?
- Did you count for potential bias across time?
- Does your model use data in an appropriate way?
- How adaptable is your model to other similar problems?
- Did you set model performance thresholds before testing?
Implementation and monitoring the model results
- Did your model get inserted into the business process correctly?
- Are staff checking the model results and performance over time?
- How adaptable is your model to similar problems or situations?
- Do the users or people it affects understand and approve?
- Is there a new policy or change in behavior behind how the model is created?
Reach out to discuss!
Types of Machine Learning
Supervised learning is the machine learning task of learning a function that maps an input to an output based on example input-output pairs. 1
Unsupervised learning algorithms infer patterns from a dataset without reference to the known, or labeled, outcomes. 2
Reinforcement learning is concerned with how intelligent agents learn overtime by interacting with their environment and balancing exploration and exploitation. 3
