Data Engineering

  1. Stream Processing and Real-Time Data
    1. Stream Processing Fundamentals
      1. Event Time vs. Processing Time
        1. Timestamp Handling
          1. Late Data Management
            1. Clock Synchronization Issues
            2. Windowing Concepts
              1. Tumbling Windows
                1. Sliding Windows
                  1. Session Windows
                    1. Custom Window Functions
                    2. Watermarks and Late Data
                      1. Watermark Generation Strategies
                        1. Late Data Handling Policies
                          1. Out-of-Order Event Processing
                          2. State Management
                            1. Stateful vs. Stateless Processing
                              1. State Backends
                                1. Checkpointing Mechanisms
                              2. Processing Guarantees
                                1. At-Least-Once Processing
                                  1. Duplicate Handling Strategies
                                    1. Idempotent Operations
                                    2. Exactly-Once Processing
                                      1. Transactional Processing
                                        1. Deduplication Techniques
                                        2. At-Most-Once Processing
                                          1. Fire-and-Forget Patterns
                                            1. Performance vs. Reliability Trade-offs
                                          2. Apache Kafka Platform
                                            1. Kafka Architecture
                                              1. Broker Cluster Management
                                                1. Topic Organization
                                                  1. Partition Distribution
                                                    1. Replication and Fault Tolerance
                                                    2. Producer Configuration
                                                      1. Message Serialization
                                                        1. Partitioning Strategies
                                                          1. Acknowledgment Settings
                                                            1. Retry and Error Handling
                                                            2. Consumer Implementation
                                                              1. Consumer Groups
                                                                1. Offset Management
                                                                  1. Partition Assignment
                                                                    1. Consumer Rebalancing
                                                                    2. Kafka Connect Framework
                                                                      1. Source Connectors
                                                                        1. Sink Connectors
                                                                          1. Connector Configuration
                                                                            1. Schema Registry Integration
                                                                          2. Stream Processing Engines
                                                                            1. Apache Spark Streaming
                                                                              1. DStream Processing Model
                                                                                1. Micro-Batch Architecture
                                                                                  1. Integration with Spark Core
                                                                                  2. Kafka Streams
                                                                                    1. Stream Processing Library
                                                                                      1. Topology Definition
                                                                                        1. Local State Stores
                                                                                          1. Interactive Queries
                                                                                          2. Engine Comparison and Selection
                                                                                            1. Latency Requirements
                                                                                              1. Throughput Capabilities
                                                                                                1. Fault Tolerance Features
                                                                                                  1. Operational Complexity

                                                                                              Previous

                                                                                              7. Batch Data Processing Systems

                                                                                              Go to top

                                                                                              Next

                                                                                              9. Data Pipeline Architecture and Orchestration