Machine Learning with Apache Spark

3.

Data Preparation and Feature Engineering

3.1.

Loading and Saving Data

3.1.1.

Supported Data Sources

3.1.1.1.

3.1.1.2.

3.1.1.3.

3.1.1.4.

3.1.1.5.

3.1.1.6.

3.1.1.7.

3.1.1.8.

3.1.2.

Reading Data into DataFrames

3.1.2.1.

Data Source API

3.1.2.2.

Schema Inference

3.1.2.3.

Custom Schema Definition

3.1.2.4.

Read Options and Configurations

3.1.3.

Writing DataFrames to Storage

3.1.3.1.

3.1.3.2.

Partitioning Strategies

3.1.3.3.

Compression Options

3.1.4.

Handling Large Datasets

3.1.4.1.

Memory Management

3.1.4.2.

Partitioning Considerations

3.1.4.3.

Performance Optimization

3.1.5.

Data Quality and Validation

3.1.5.1.

Schema Validation

3.1.5.2.

3.1.5.3.

Constraint Checking

3.2.

Data Exploration and Manipulation with Spark SQL

3.2.1.

Querying DataFrames with SQL

3.2.1.1.

Registering Temporary Views

3.2.1.2.

Global Temporary Views

3.2.1.3.

Executing SQL Queries

3.2.1.4.

Complex Query Patterns

3.2.2.

DataFrame API Operations

3.2.2.1.

Selection Operations

3.2.2.1.1.

Selecting Columns

3.2.2.1.2.

Column Expressions

3.2.2.1.3.

Conditional Selection

3.2.2.2.

Filtering Operations

3.2.2.2.1.

Basic Filtering

3.2.2.2.2.

Complex Conditions

3.2.2.2.3.

3.2.2.3.

Grouping and Aggregation

3.2.2.3.1.

GroupBy Operations

3.2.2.3.2.

Aggregation Functions

3.2.2.3.3.

Window Functions

3.2.2.4.

Joining DataFrames

3.2.2.4.1.

3.2.2.4.2.

3.2.2.4.3.

3.2.2.4.4.

Join Optimization

3.2.2.5.

Sorting and Ordering

3.2.2.5.1.

Single Column Sorting

3.2.2.5.2.

Multiple Column Sorting

3.2.2.5.3.

Custom Ordering

3.2.2.6.

Handling Missing Data

3.2.2.6.1.

Detecting Missing Values

3.2.2.6.2.

Dropping Missing Values

3.2.2.6.3.

Filling Missing Values

3.2.2.6.4.

Imputation Strategies

3.2.2.7.

Data Type Operations

3.2.2.7.1.

Casting Data Types

3.2.2.7.2.

Type Conversion

3.2.2.7.3.

Schema Evolution

3.2.2.8.

User-Defined Functions

3.2.2.8.1.

3.2.2.8.2.

Registering UDFs

3.2.2.8.3.

Performance Considerations

3.2.2.8.4.

Vectorized UDFs

3.3.

Feature Engineering with Spark ML

3.3.1.

Feature Extraction

3.3.1.1.

Text Feature Extraction

3.3.1.1.1.

3.3.1.1.2.

3.3.1.1.3.

CountVectorizer

3.3.1.1.4.

3.3.1.1.5.

N-gram Extraction

3.3.1.1.6.

StopWordsRemover

3.3.1.2.

Numerical Feature Extraction

3.3.1.2.1.

Polynomial Features

3.3.1.2.2.

Interaction Features

3.3.2.

Feature Transformation

3.3.2.1.

Categorical Transformations

3.3.2.1.1.

3.3.2.1.2.

3.3.2.1.3.

3.3.2.2.

Vector Operations

3.3.2.2.1.

VectorAssembler

3.3.2.2.2.

3.3.2.2.3.

3.3.2.3.

Scaling Transformations

3.3.2.3.1.

3.3.2.3.2.

3.3.2.3.3.

3.3.2.3.4.

3.3.2.3.5.

3.3.2.4.

3.3.2.4.1.

3.3.2.4.2.

QuantileDiscretizer

3.3.2.5.

Dimensionality Reduction

3.3.2.5.1.

3.3.2.5.2.

3.3.3.

Feature Selection

3.3.3.1.

Statistical Selection

3.3.3.1.1.

3.3.3.1.2.

Univariate Feature Selection

3.3.3.1.3.

Variance Threshold

3.3.3.2.

Model-Based Selection

3.3.3.2.1.

Feature Importance

3.3.3.2.2.

Recursive Feature Elimination

3.3.4.

Advanced Feature Engineering

3.3.4.1.

Handling Categorical Variables

3.3.4.1.1.

High Cardinality Categories

3.3.4.1.2.

Rare Category Handling

3.3.4.1.3.

Target Encoding

3.3.4.2.

Handling Imbalanced Data

3.3.4.2.1.

Sampling Techniques

3.3.4.2.2.

Class Weight Adjustment

3.3.4.2.3.

Synthetic Data Generation

3.3.4.3.

Dealing with Outliers

3.3.4.3.1.

Outlier Detection

3.3.4.3.2.

Outlier Treatment

3.3.4.3.3.

Robust Statistics

3.3.4.4.

Time Series Features

3.3.4.4.1.

Date/Time Extraction

3.3.4.4.2.

3.3.4.4.3.

Rolling Statistics

Previous

2. Core Apache Spark Concepts for Machine Learning

Go to top

Next

4. Building Machine Learning Pipelines