Real-Time Analytics and Stream Processing

  1. Stream Processing Frameworks and Technologies
    1. Apache Spark Streaming
      1. Discretized Streams
        1. Micro-Batch Model
          1. RDD-Based Processing
            1. Batch Interval Configuration
            2. Fault Tolerance in D-Streams
              1. RDD Lineage
                1. Checkpoint Recovery
              2. Structured Streaming
                1. Unified Batch and Streaming API
                  1. DataFrame and Dataset APIs
                    1. Catalyst Optimizer
                    2. Event Time and Watermark Support
                      1. Built-in Time Handling
                        1. Late Data Management
                      2. Spark Ecosystem Integration
                        1. DataFrames and SQL
                          1. Spark SQL Integration
                            1. Catalog Support
                            2. Machine Learning Integration
                              1. MLlib Streaming
                                1. Model Serving
                            3. Apache Kafka Ecosystem
                              1. Kafka as Distributed Log
                                1. Partitioning and Replication
                                  1. Topic Partitioning
                                    1. Replica Management
                                    2. Log Retention and Compaction
                                      1. Time-Based Retention
                                        1. Key-Based Compaction
                                      2. Kafka Connect for Integration
                                        1. Source and Sink Connectors
                                          1. Database Connectors
                                            1. File System Connectors
                                            2. Integrating with External Systems
                                              1. Schema Registry Integration
                                                1. Transformation Capabilities
                                              2. Kafka Streams for Stream Processing
                                                1. Stream Transformations and Aggregations
                                                  1. KStream and KTable APIs
                                                    1. Stateful Operations
                                                    2. State Stores and Fault Tolerance
                                                      1. Local State Stores
                                                        1. Changelog Topics
                                                      2. ksqlDB for Streaming SQL
                                                        1. Declarative Stream Processing
                                                          1. SQL-Based Stream Processing
                                                            1. Materialized Views
                                                            2. Real-Time Querying
                                                              1. Interactive Queries
                                                                1. Push and Pull Queries
                                                            3. Additional Frameworks
                                                              1. Apache Storm
                                                                1. Topology-Based Processing
                                                                  1. DAG Processing Model
                                                                    1. Real-Time Guarantees
                                                                    2. Spouts and Bolts
                                                                      1. Data Source Components
                                                                        1. Processing Logic Components
                                                                      2. Apache Samza
                                                                        1. Partitioned State Management
                                                                          1. Local State Storage
                                                                            1. Fault Tolerance
                                                                            2. Kafka Integration
                                                                              1. Native Kafka Support
                                                                                1. Stream Partitioning
                                                                              2. Cloud-Native Solutions
                                                                                1. Google Cloud Dataflow
                                                                                  1. Apache Beam Runtime
                                                                                    1. Unified Batch and Stream Model
                                                                                    2. Amazon Kinesis
                                                                                      1. Shard-Based Scaling
                                                                                        1. Kinesis Data Analytics
                                                                                        2. Azure Stream Analytics
                                                                                          1. SQL-Based Processing
                                                                                            1. IoT Integration

                                                                                      Previous

                                                                                      4. Core Components of Streaming Pipelines

                                                                                      Go to top

                                                                                      Next

                                                                                      6. Data Formats and Serialization