Data Cleaning

9.

Tools and Technologies for Data Cleaning

9.1.

Spreadsheet Applications

9.1.1.

Microsoft Excel

9.1.1.1.

Built-in Functions

9.1.1.1.1.

9.1.1.1.2.

9.1.1.1.3.

Lookup Functions

9.1.1.1.4.

Statistical Functions

9.1.1.2.

Data Validation Tools

9.1.1.2.1.

Input Restrictions

9.1.1.2.2.

Custom Validation Rules

9.1.1.2.3.

9.1.1.2.4.

9.1.1.3.

Advanced Features

9.1.1.3.1.

Conditional Formatting

9.1.1.3.2.

9.1.1.3.3.

9.1.1.3.4.

9.1.2.

9.1.2.1.

Native Functions

9.1.2.2.

Add-ons and Extensions

9.1.2.3.

Google Apps Script

9.1.2.4.

Collaboration Features

9.1.3.

LibreOffice Calc

9.1.3.1.

Open Source Alternative

9.1.3.2.

Macro Capabilities

9.1.3.3.

Extension Support

9.2.

Programming Languages and Libraries

9.2.1.

Python Ecosystem

9.2.1.1.

9.2.1.1.1.

9.2.1.1.1.1.

DataFrame Operations

9.2.1.1.1.2.

Data Manipulation

9.2.1.1.1.3.

Missing Data Handling

9.2.1.1.1.4.

Grouping and Aggregation

9.2.1.1.2.

9.2.1.1.2.1.

Array Operations

9.2.1.1.2.2.

Mathematical Functions

9.2.1.1.2.3.

9.2.1.1.2.4.

9.2.1.2.

Specialized Libraries

9.2.1.2.1.

9.2.1.2.1.1.

High-Performance DataFrames

9.2.1.2.1.2.

Lazy Evaluation

9.2.1.2.1.3.

Memory Efficiency

9.2.1.2.2.

9.2.1.2.2.1.

Parallel Computing

9.2.1.2.2.2.

Out-of-Core Processing

9.2.1.2.2.3.

Scalable Analytics

9.2.1.2.3.

9.2.1.2.3.1.

Pandas Acceleration

9.2.1.2.3.2.

Distributed Computing

9.2.1.3.

Text Processing

9.2.1.3.1.

Regular Expressions (re)

9.2.1.3.2.

9.2.1.3.3.

Natural Language Toolkit (NLTK)

9.2.1.3.4.

9.2.1.4.

Data Quality Libraries

9.2.1.4.1.

Great Expectations

9.2.1.4.2.

9.2.1.4.3.

9.2.1.4.4.

9.2.1.5.

Utility Libraries

9.2.1.5.1.

9.2.1.5.2.

9.2.1.5.3.

9.2.1.5.4.

9.2.2.

9.2.2.1.

9.2.2.1.1.

9.2.2.1.1.1.

Data Manipulation Grammar

9.2.2.1.1.2.

Pipe Operations

9.2.2.1.1.3.

Grouping Functions

9.2.2.1.2.

9.2.2.1.2.1.

9.2.2.1.2.2.

Missing Data Tools

9.2.2.1.2.3.

Nested Data Handling

9.2.2.1.3.

9.2.2.1.3.1.

High-Performance Operations

9.2.2.1.3.2.

Memory Efficiency

9.2.2.1.3.3.

Fast Aggregations

9.2.2.2.

String Processing

9.2.2.2.1.

9.2.2.2.2.

9.2.2.2.3.

Regular Expressions

9.2.2.3.

Data Quality Packages

9.2.2.3.1.

9.2.2.3.2.

VIM (Visualization and Imputation of Missing values)

9.2.2.3.3.

mice (Multiple Imputation)

9.2.2.3.4.

9.2.2.4.

Specialized Packages

9.2.2.4.1.

lubridate (Date/Time)

9.2.2.4.2.

forcats (Categorical Data)

9.2.2.4.3.

readr (Data Import)

9.3.

Database Systems and SQL

9.3.1.

SQL Data Manipulation

9.3.1.1.

Data Definition Language (DDL)

9.3.1.2.

Data Manipulation Language (DML)

9.3.1.3.

Data Query Language (DQL)

9.3.1.4.

Data Control Language (DCL)

9.3.2.

Advanced SQL Features

9.3.2.1.

Window Functions

9.3.2.2.

Common Table Expressions (CTEs)

9.3.2.3.

Stored Procedures

9.3.2.4.

User-Defined Functions

9.3.3.

String and Date Functions

9.3.3.1.

Text Processing Functions

9.3.3.2.

Pattern Matching

9.3.3.3.

Date Arithmetic

9.3.3.4.

Format Conversion

9.3.4.

Data Quality Constraints

9.3.4.1.

Primary Key Constraints

9.3.4.2.

Foreign Key Constraints

9.3.4.3.

Unique Constraints

9.3.4.4.

Check Constraints

9.3.4.5.

Not Null Constraints

9.3.5.

Database-Specific Features

9.3.5.1.

9.3.5.1.1.

Advanced Data Types

9.3.5.1.2.

Full-Text Search

9.3.5.1.3.

9.3.5.2.

9.3.5.2.1.

String Functions

9.3.5.2.2.

9.3.5.2.3.

Regular Expressions

9.3.5.3.

9.3.5.3.1.

T-SQL Extensions

9.3.5.3.2.

Data Quality Services

9.3.5.3.3.

Integration Services

9.3.5.4.

9.3.5.4.1.

9.3.5.4.2.

Advanced Analytics

9.3.5.4.3.

9.4.

Specialized Data Cleaning Tools

9.4.1.

Open Source Tools

9.4.1.1.

9.4.1.1.1.

Interactive Data Cleaning

9.4.1.1.2.

Faceting and Filtering

9.4.1.1.3.

Clustering and Reconciliation

9.4.1.1.4.

Expression Language

9.4.1.2.

9.4.1.2.1.

Distributed Processing

9.4.1.2.2.

MLlib for Data Quality

9.4.1.2.3.

Structured Streaming

9.4.1.3.

Talend Open Studio

9.4.1.3.1.

9.4.1.3.2.

Data Integration

9.4.1.3.3.

9.4.2.

Commercial Tools

9.4.2.1.

Trifacta Wrangler

9.4.2.1.1.

Visual Data Preparation

9.4.2.1.2.

Machine Learning Suggestions

9.4.2.1.3.

Collaboration Features

9.4.2.2.

Alteryx Designer

9.4.2.2.1.

Drag-and-Drop Interface

9.4.2.2.2.

Predictive Analytics

9.4.2.2.3.

Spatial Analytics

9.4.2.3.

Informatica Data Quality

9.4.2.3.1.

Enterprise Data Quality

9.4.2.3.2.

9.4.2.3.3.

Data Standardization

9.4.2.4.

IBM InfoSphere QualityStage

9.4.2.4.1.

Data Standardization

9.4.2.4.2.

Matching and Deduplication

9.4.2.4.3.

Data Investigation

9.4.3.

Cloud-Based Solutions

9.4.3.1.

AWS Glue DataBrew

9.4.3.2.

Google Cloud Dataprep

9.4.3.3.

Microsoft Power BI Dataflows

9.4.3.4.

Databricks Data Engineering

9.5.

Workflow and Pipeline Tools

9.5.1.

9.5.1.1.

Workflow Orchestration

9.5.1.2.

Task Dependencies

9.5.1.3.

Monitoring and Alerting

9.5.2.

9.5.2.1.

Modern Workflow Engine

9.5.2.2.

Dynamic Workflows

9.5.2.3.

9.5.3.

9.5.3.1.

Batch Job Pipeline

9.5.3.2.

Dependency Resolution

9.5.3.3.

Failure Recovery

9.5.4.

9.5.4.1.

Data Pipeline Framework

9.5.4.2.

9.5.4.3.

Testing Framework

Previous

8. Techniques for Fixing Structural Errors

Go to top

Next

10. Advanced Data Cleaning Topics