causes of missing data

Understanding the Causes of Missing Data

By Steven Young
AD

Missing data, a daunting challenge in the realm of data analysis, can significantly impact the accuracy and reliability of results. It occurs when expected data is absent, either partially or entirely. Addressing this issue is crucial for obtaining meaningful insights and making informed decisions. Let's delve into the reasons behind missing data and explore ways to handle it.

Missing data can stem from various reasons, broadly classified into three categories: Missing Completely at Random (MCAR), Missing at Random (MAR), and Not Missing at Random (NMAR). Understanding these categories is key to developing appropriate strategies for handling missing data.

Understanding Missing Data Mechanisms

Missing Completely at Random (MCAR) occurs when the probability of data being missing is independent of both observed and unobserved data. In such cases, data is missing purely by chance, and there's no way to predict which observations will be missing. For instance, a sudden server malfunction could cause data loss randomly throughout a dataset.

AD
5 Causes of Missing Data in Enterprise Systems
5 Causes of Missing Data in Enterprise Systems
Missing Data in Time-Series: Machine Learning Techniques | Towards Data Science
Missing Data in Time-Series: Machine Learning Techniques | Towards Data Science
a flow diagram with the words handling missing data in different colors and shapes, including arrows
a flow diagram with the words handling missing data in different colors and shapes, including arrows

Missing at Random (MAR) happens when the probability of data being missing depends on observed data but not on unobserved data. For example, if survey responses are missing only for individuals who didn't provide their age (an observed variable), the data is considered missing at random. MAR is the most common and least challenging missing data mechanism to handle.

Handling Missing Data: MCAR & MAR

For MCAR data, simple imputation methods like mean, median, or mode imputation can be used. However, these methods assume a symmetric distribution, which may not always hold true. Alternatively, multiple imputation, a more advanced technique, can be employed to generate multiple datasets with imputed values.

Visualize the data: How missing data biases data-driven decisions
Visualize the data: How missing data biases data-driven decisions
Top 5 Mistakes to Avoid in Data Engineering Consulting
Top 5 Mistakes to Avoid in Data Engineering Consulting
Different types of missing data.
Different types of missing data.

For MAR data, similar methods as MCAR can be used, with the advantage that we can incorporate auxiliary variables (variables related to the missing values) into our models. This helps to reduce bias and makes the imputation more accurate.

Handling Missing Data: NMAR

Not Missing at Random (NMAR) is the most complex scenario, where the probability of data being missing depends on the unobserved data. For example, patients who felt worse might be less likely to report their symptoms (unobserved data). Handling NMAR data is challenging, often requiring complex models like selection models or pattern mixture models.

AD

In some cases, the 'delete' method might be the only viable solution for NMAR data. This involves deleting or excluding observations with missing data from the analysis. However, this can lead to a loss of valuable information and may introduce bias if the deleted observations are not random.

Half of UK smaller businesses have lost vital data
Half of UK smaller businesses have lost vital data
the common data collection errors poster
the common data collection errors poster
5 Common Data Analysis Mistakes
5 Common Data Analysis Mistakes
Missing values are one of the biggest hidden risks in data analytics. A dataset may look complete at first glance, but even a small pattern of missing data can affect statistical accuracy, machine learning performance, and business decisions. This is why strategic data cleansing is not just a technical step. It is a critical part of building trustworthy analysis. In KNIME, missing value handling becomes more structured through visual workflows. The process can begin with identification usin... Data Cleansing, Data Analytics, Machine Learning
Missing values are one of the biggest hidden risks in data analytics. A dataset may look complete at first glance, but even a small pattern of missing data can affect statistical accuracy, machine learning performance, and business decisions. This is why strategic data cleansing is not just a technical step. It is a critical part of building trustworthy analysis. In KNIME, missing value handling becomes more structured through visual workflows. The process can begin with identification usin... Data Cleansing, Data Analytics, Machine Learning
Data Loss Prevention Market Size, Share & Trends - 2032
Data Loss Prevention Market Size, Share & Trends - 2032
Common Causes of Data Loss and How to Prevent Them.
Common Causes of Data Loss and How to Prevent Them.

Causes of Missing Data and Prevention

Understanding the causes of missing data can help prevent it. Some common causes include:

  • Incomplete data collection due to errors or omissions during data entry.
  • Missing historical data due to system changes or data migration issues.
  • Data loss due to technical issues like server failures or database corruption.
  • Data privacy regulations leading to missing values or redacted information.

To prevent missing data, regular data audits, robust data collection processes, data backup mechanisms, and adherence to data privacy regulations are crucial.

AD

In the dynamic field of data science, missing data is an inevitable challenge. However, with a solid understanding of its mechanisms and appropriate handling strategies, we can effectively manage and minimize the impact of missing data on our analyses and decisions.

Missing Data Overview: Types, Implications & Handling
Missing Data Overview: Types, Implications & Handling
AD