Apache Spark

  1. Spark SQL and Structured APIs
    1. Structured API Foundation
      1. Motivation for Structure
        1. Performance Optimization
          1. Code Generation
            1. Query Planning
            2. API Comparison
              1. RDD vs DataFrame vs Dataset
                1. Type Safety Considerations
                  1. Performance Characteristics
                2. DataFrame API
                  1. DataFrame Concepts
                    1. Tabular Data Abstraction
                      1. Schema Definition
                        1. Catalyst Integration
                        2. Schema Management
                          1. StructType Definition
                            1. StructField Configuration
                              1. Data Type System
                                1. Primitive Types
                                  1. Complex Types
                                    1. Null Handling
                                  2. DataFrame Creation
                                    1. From RDDs
                                      1. From External Sources
                                        1. From Collections
                                          1. Schema Inference
                                        2. Dataset API
                                          1. Type Safety Features
                                            1. Compile-Time Type Checking
                                              1. Runtime Type Safety
                                              2. Encoder System
                                                1. Automatic Encoders
                                                  1. Custom Encoders
                                                    1. Serialization Optimization
                                                    2. DataFrame-Dataset Relationship
                                                      1. Conversion Methods
                                                        1. API Compatibility
                                                          1. Performance Implications
                                                        2. DataFrame and Dataset Operations
                                                          1. Untyped Transformations
                                                            1. Column Selection
                                                              1. select Operations
                                                                1. withColumn Operations
                                                                  1. drop Operations
                                                                  2. Row Filtering
                                                                    1. filter Operations
                                                                      1. where Operations
                                                                      2. Grouping and Aggregation
                                                                        1. groupBy Operations
                                                                          1. agg Operations
                                                                            1. pivot Operations
                                                                            2. Joining Operations
                                                                              1. Inner Joins
                                                                                1. Outer Joins
                                                                                  1. Cross Joins
                                                                                    1. Broadcast Joins
                                                                                    2. Ordering Operations
                                                                                      1. orderBy Operations
                                                                                        1. sort Operations
                                                                                      2. Typed Transformations
                                                                                        1. Functional Operations
                                                                                          1. map Operations
                                                                                            1. flatMap Operations
                                                                                              1. filter Operations
                                                                                              2. Grouping Operations
                                                                                                1. groupByKey Operations
                                                                                                  1. mapGroups Operations
                                                                                                    1. flatMapGroups Operations
                                                                                                  2. Action Operations
                                                                                                    1. Data Collection
                                                                                                      1. show Operations
                                                                                                        1. collect Operations
                                                                                                          1. take Operations
                                                                                                          2. Aggregation Actions
                                                                                                            1. count Operations
                                                                                                              1. reduce Operations
                                                                                                              2. Output Actions
                                                                                                                1. write Operations
                                                                                                                  1. foreach Operations
                                                                                                              3. SQL Interface
                                                                                                                1. SQL Query Execution
                                                                                                                  1. SQL Context
                                                                                                                    1. Query Registration
                                                                                                                      1. Result Processing
                                                                                                                      2. View Management
                                                                                                                        1. Temporary Views
                                                                                                                          1. Global Temporary Views
                                                                                                                            1. View Lifecycle
                                                                                                                            2. User-Defined Functions
                                                                                                                              1. UDF Registration
                                                                                                                                1. UDF Usage in SQL
                                                                                                                                  1. Performance Considerations
                                                                                                                                2. Data Sources and I/O
                                                                                                                                  1. Data Source API
                                                                                                                                    1. DataFrameReader
                                                                                                                                      1. DataFrameWriter
                                                                                                                                        1. Options Configuration
                                                                                                                                        2. File Formats
                                                                                                                                          1. Parquet
                                                                                                                                            1. Columnar Storage Benefits
                                                                                                                                              1. Schema Evolution
                                                                                                                                                1. Predicate Pushdown
                                                                                                                                                2. ORC
                                                                                                                                                  1. Optimization Features
                                                                                                                                                    1. Compression Options
                                                                                                                                                    2. JSON
                                                                                                                                                      1. Schema Inference
                                                                                                                                                        1. Nested Data Handling
                                                                                                                                                        2. CSV
                                                                                                                                                          1. Header Processing
                                                                                                                                                            1. Type Inference
                                                                                                                                                            2. Avro
                                                                                                                                                              1. Schema Registry Integration
                                                                                                                                                                1. Evolution Support
                                                                                                                                                              2. External Systems
                                                                                                                                                                1. File Systems
                                                                                                                                                                  1. HDFS Integration
                                                                                                                                                                    1. S3 Connectivity
                                                                                                                                                                      1. Local File System
                                                                                                                                                                      2. Relational Databases
                                                                                                                                                                        1. JDBC Connectivity
                                                                                                                                                                          1. MySQL Integration
                                                                                                                                                                            1. PostgreSQL Integration
                                                                                                                                                                              1. SQL Server Integration
                                                                                                                                                                              2. NoSQL Systems
                                                                                                                                                                                1. Cassandra Integration
                                                                                                                                                                                  1. HBase Connectivity
                                                                                                                                                                                    1. MongoDB Support
                                                                                                                                                                                2. Query Optimization
                                                                                                                                                                                  1. Catalyst Optimizer
                                                                                                                                                                                    1. Logical Plan Optimization
                                                                                                                                                                                      1. Physical Plan Generation
                                                                                                                                                                                        1. Rule-Based Optimization
                                                                                                                                                                                          1. Cost-Based Optimization
                                                                                                                                                                                          2. Tungsten Execution Engine
                                                                                                                                                                                            1. Whole-Stage Code Generation
                                                                                                                                                                                              1. Vectorized Processing
                                                                                                                                                                                                1. Memory Management
                                                                                                                                                                                                  1. Binary Processing

                                                                                                                                                                                              Previous

                                                                                                                                                                                              3. Spark Architecture and Execution

                                                                                                                                                                                              Go to top

                                                                                                                                                                                              Next

                                                                                                                                                                                              5. Structured Streaming