Apache Spark

8.

Performance Tuning and Optimization

8.1.

Monitoring and Debugging

8.1.1.

8.1.1.1.

Jobs Tab Analysis

8.1.1.2.

Stages Tab Insights

8.1.1.3.

Storage Tab Monitoring

8.1.1.4.

Environment Tab Review

8.1.1.5.

Executors Tab Metrics

8.1.1.6.

SQL Tab Query Plans

8.1.2.

Logging and Metrics

8.1.2.1.

Log Level Configuration

8.1.2.2.

8.1.2.3.

External Monitoring Integration

8.1.3.

Performance Profiling

8.1.3.1.

8.1.3.2.

Memory Profiling

8.1.3.3.

8.2.

Memory Management Optimization

8.2.1.

Memory Architecture

8.2.1.1.

Execution Memory

8.2.1.2.

8.2.1.3.

Unified Memory Manager

8.2.2.

Garbage Collection Tuning

8.2.2.1.

GC Algorithm Selection

8.2.2.2.

GC Parameter Tuning

8.2.2.3.

Memory Pressure Management

8.2.3.

Memory Overhead Optimization

8.2.3.1.

Off-Heap Storage

8.2.3.2.

Memory Fraction Tuning

8.2.3.3.

Spill Management

8.3.

Serialization Optimization

8.3.1.

Serialization Formats

8.3.1.1.

Java Serialization

8.3.1.2.

Kryo Serialization

8.3.1.3.

Custom Serializers

8.3.2.

Serialization Configuration

8.3.2.1.

Kryo Registration

8.3.2.2.

Buffer Size Tuning

8.3.2.3.

Compression Options

8.4.

Data Layout and Partitioning

8.4.1.

Partitioning Strategies

8.4.1.1.

Hash Partitioning

8.4.1.2.

Range Partitioning

8.4.1.3.

Custom Partitioning

8.4.2.

Partition Management

8.4.2.1.

Optimal Partition Count

8.4.2.2.

Partition Size Guidelines

8.4.2.3.

Repartitioning vs Coalescing

8.4.3.

Data Skew Handling

8.4.3.1.

8.4.3.2.

Salting Techniques

8.4.3.3.

Broadcast Joins

8.5.

Caching and Persistence

8.5.1.

8.5.1.1.

Memory-Only Storage

8.5.1.2.

Memory-and-Disk Storage

8.5.1.3.

Disk-Only Storage

8.5.1.4.

Serialized Storage

8.5.1.5.

Replication Options

8.5.2.

Caching Strategies

8.5.2.1.

8.5.2.2.

8.5.2.3.

Cache Monitoring

8.5.3.

Persistence Best Practices

8.5.3.1.

Checkpoint Usage

8.5.3.2.

Lineage Management

8.6.

Join Optimization

8.6.1.

Join Strategies

8.6.1.1.

Broadcast Hash Join

8.6.1.2.

Shuffle Hash Join

8.6.1.3.

Sort-Merge Join

8.6.1.4.

8.6.2.

8.6.2.1.

Broadcast Hints

8.6.2.2.

8.6.2.3.

8.6.3.

Skewed Join Handling

8.6.3.1.

8.6.3.2.

Adaptive Query Execution

8.7.

Common Performance Issues

8.7.1.

Data Skew Problems

8.7.1.1.

Identification Methods

8.7.1.2.

Mitigation Strategies

8.7.2.

Shuffle Optimization

8.7.2.1.

Shuffle Partitions Tuning

8.7.2.2.

Shuffle Service Configuration

8.7.3.

Small Files Problem

8.7.3.1.

File Consolidation

8.7.3.2.

Compaction Strategies

8.7.4.

Resource Contention

8.7.4.1.

CPU Bottlenecks

8.7.4.2.

Memory Bottlenecks

8.7.4.3.

I/O Bottlenecks

8.7.5.

Task Scheduling Issues

8.7.5.1.

Straggler Tasks

8.7.5.2.

8.7.5.3.

Dynamic Allocation

Previous

7. Graph Processing with GraphX

Go to top

Back to Start

1. Introduction to Apache Spark