Data Cleaning

  1. The Data Cleaning Workflow
    1. Phase 1: Data Inspection and Assessment
      1. Initial Data Review
        1. Data Source Evaluation
          1. File Format Assessment
            1. Encoding Detection
            2. Data Type Identification
              1. Automatic Type Inference
                1. Manual Type Verification
                  1. Mixed Type Detection
                  2. Descriptive Statistics Generation
                    1. Numerical Summaries
                      1. Categorical Summaries
                        1. Missing Value Counts
                        2. Data Distribution Analysis
                          1. Distribution Visualization
                            1. Normality Testing
                              1. Outlier Identification
                              2. Metadata Review
                                1. Data Dictionary Analysis
                                  1. Source Documentation
                                    1. Business Context Understanding
                                  2. Phase 2: Issue Identification and Prioritization
                                    1. Systematic Error Detection
                                      1. Automated Quality Checks
                                        1. Rule-Based Validation
                                          1. Statistical Anomaly Detection
                                          2. Problem Documentation
                                            1. Issue Cataloging
                                              1. Severity Assessment
                                                1. Impact Analysis
                                                2. Priority Assignment
                                                  1. Business Impact Ranking
                                                    1. Technical Complexity Assessment
                                                      1. Resource Requirement Estimation
                                                      2. Root Cause Analysis
                                                        1. Source Investigation
                                                          1. Process Analysis
                                                            1. System Evaluation
                                                          2. Phase 3: Cleaning Strategy Development
                                                            1. Cleaning Approach Selection
                                                              1. Method Evaluation
                                                                1. Trade-off Analysis
                                                                  1. Resource Planning
                                                                  2. Validation Strategy Design
                                                                    1. Quality Metrics Definition
                                                                      1. Testing Procedures
                                                                        1. Acceptance Criteria
                                                                        2. Implementation Planning
                                                                          1. Task Sequencing
                                                                            1. Dependency Management
                                                                              1. Timeline Development
                                                                            2. Phase 4: Data Transformation and Correction
                                                                              1. Error Correction Implementation
                                                                                1. Systematic Corrections
                                                                                  1. Manual Interventions
                                                                                    1. Automated Fixes
                                                                                    2. Data Standardization
                                                                                      1. Format Standardization
                                                                                        1. Value Normalization
                                                                                          1. Unit Conversion
                                                                                          2. Structure Optimization
                                                                                            1. Schema Refinement
                                                                                              1. Data Type Conversion
                                                                                                1. Relationship Establishment
                                                                                                2. Quality Enhancement
                                                                                                  1. Enrichment Processes
                                                                                                    1. Validation Implementation
                                                                                                      1. Consistency Enforcement
                                                                                                    2. Phase 5: Verification and Quality Assurance
                                                                                                      1. Cleaning Results Validation
                                                                                                        1. Before-After Comparison
                                                                                                          1. Quality Metric Calculation
                                                                                                            1. Statistical Validation
                                                                                                            2. Change Documentation
                                                                                                              1. Transformation Log
                                                                                                                1. Decision Rationale
                                                                                                                  1. Impact Assessment
                                                                                                                  2. Final Dataset Preparation
                                                                                                                    1. Clean Dataset Generation
                                                                                                                      1. Metadata Update
                                                                                                                        1. Documentation Completion
                                                                                                                        2. Quality Reporting
                                                                                                                          1. Summary Reports
                                                                                                                            1. Stakeholder Communication
                                                                                                                              1. Lessons Learned Documentation
                                                                                                                              2. Reproducibility Assurance
                                                                                                                                1. Script Documentation
                                                                                                                                  1. Process Standardization
                                                                                                                                    1. Version Control

                                                                                                                                Previous

                                                                                                                                2. Core Concepts of Data Quality

                                                                                                                                Go to top

                                                                                                                                Next

                                                                                                                                4. Common Types of Data Quality Issues