Apache Spark

4.

Spark SQL and Structured APIs

4.1.

Structured API Foundation

4.1.1.

Motivation for Structure

4.1.1.1.

Performance Optimization

4.1.1.2.

Code Generation

4.1.1.3.

4.1.2.

4.1.2.1.

RDD vs DataFrame vs Dataset

4.1.2.2.

Type Safety Considerations

4.1.2.3.

Performance Characteristics

4.2.

4.2.1.

DataFrame Concepts

4.2.1.1.

Tabular Data Abstraction

4.2.1.2.

Schema Definition

4.2.1.3.

Catalyst Integration

4.2.2.

Schema Management

4.2.2.1.

StructType Definition

4.2.2.2.

StructField Configuration

4.2.2.3.

Data Type System

4.2.2.3.1.

Primitive Types

4.2.2.3.2.

4.2.2.3.3.

4.2.3.

DataFrame Creation

4.2.3.1.

4.2.3.2.

From External Sources

4.2.3.3.

From Collections

4.2.3.4.

Schema Inference

4.3.

4.3.1.

Type Safety Features

4.3.1.1.

Compile-Time Type Checking

4.3.1.2.

Runtime Type Safety

4.3.2.

4.3.2.1.

Automatic Encoders

4.3.2.2.

Custom Encoders

4.3.2.3.

Serialization Optimization

4.3.3.

DataFrame-Dataset Relationship

4.3.3.1.

Conversion Methods

4.3.3.2.

API Compatibility

4.3.3.3.

Performance Implications

4.4.

DataFrame and Dataset Operations

4.4.1.

Untyped Transformations

4.4.1.1.

Column Selection

4.4.1.1.1.

select Operations

4.4.1.1.2.

withColumn Operations

4.4.1.1.3.

drop Operations

4.4.1.2.

4.4.1.2.1.

filter Operations

4.4.1.2.2.

where Operations

4.4.1.3.

Grouping and Aggregation

4.4.1.3.1.

groupBy Operations

4.4.1.3.2.

4.4.1.3.3.

pivot Operations

4.4.1.4.

Joining Operations

4.4.1.4.1.

4.4.1.4.2.

4.4.1.4.3.

4.4.1.4.4.

Broadcast Joins

4.4.1.5.

Ordering Operations

4.4.1.5.1.

orderBy Operations

4.4.1.5.2.

sort Operations

4.4.2.

Typed Transformations

4.4.2.1.

Functional Operations

4.4.2.1.1.

4.4.2.1.2.

flatMap Operations

4.4.2.1.3.

filter Operations

4.4.2.2.

Grouping Operations

4.4.2.2.1.

groupByKey Operations

4.4.2.2.2.

mapGroups Operations

4.4.2.2.3.

flatMapGroups Operations

4.4.3.

Action Operations

4.4.3.1.

Data Collection

4.4.3.1.1.

show Operations

4.4.3.1.2.

collect Operations

4.4.3.1.3.

take Operations

4.4.3.2.

Aggregation Actions

4.4.3.2.1.

count Operations

4.4.3.2.2.

reduce Operations

4.4.3.3.

4.4.3.3.1.

write Operations

4.4.3.3.2.

foreach Operations

4.5.

4.5.1.

SQL Query Execution

4.5.1.1.

4.5.1.2.

Query Registration

4.5.1.3.

Result Processing

4.5.2.

View Management

4.5.2.1.

Temporary Views

4.5.2.2.

Global Temporary Views

4.5.2.3.

4.5.3.

User-Defined Functions

4.5.3.1.

UDF Registration

4.5.3.2.

UDF Usage in SQL

4.5.3.3.

Performance Considerations

4.6.

Data Sources and I/O

4.6.1.

Data Source API

4.6.1.1.

DataFrameReader

4.6.1.2.

DataFrameWriter

4.6.1.3.

Options Configuration

4.6.2.

4.6.2.1.

4.6.2.1.1.

Columnar Storage Benefits

4.6.2.1.2.

Schema Evolution

4.6.2.1.3.

Predicate Pushdown

4.6.2.2.

4.6.2.2.1.

Optimization Features

4.6.2.2.2.

Compression Options

4.6.2.3.

4.6.2.3.1.

Schema Inference

4.6.2.3.2.

Nested Data Handling

4.6.2.4.

4.6.2.4.1.

Header Processing

4.6.2.4.2.

4.6.2.5.

4.6.2.5.1.

Schema Registry Integration

4.6.2.5.2.

Evolution Support

4.6.3.

External Systems

4.6.3.1.

4.6.3.1.1.

HDFS Integration

4.6.3.1.2.

S3 Connectivity

4.6.3.1.3.

Local File System

4.6.3.2.

Relational Databases

4.6.3.2.1.

JDBC Connectivity

4.6.3.2.2.

MySQL Integration

4.6.3.2.3.

PostgreSQL Integration

4.6.3.2.4.

SQL Server Integration

4.6.3.3.

4.6.3.3.1.

Cassandra Integration

4.6.3.3.2.

HBase Connectivity

4.6.3.3.3.

MongoDB Support

4.7.

Query Optimization

4.7.1.

Catalyst Optimizer

4.7.1.1.

Logical Plan Optimization

4.7.1.2.

Physical Plan Generation

4.7.1.3.

Rule-Based Optimization

4.7.1.4.

Cost-Based Optimization

4.7.2.

Tungsten Execution Engine

4.7.2.1.

Whole-Stage Code Generation

4.7.2.2.

Vectorized Processing

4.7.2.3.

Memory Management

4.7.2.4.

Binary Processing

Previous

3. Spark Architecture and Execution

Go to top

Next

5. Structured Streaming