Data Lakes and Lakehouses

  1. Core Technologies Enabling Data Lakehouses
    1. Open Table Formats
      1. Table Format Fundamentals
        1. Metadata Management
          1. Transaction Log Maintenance
            1. Schema Evolution Handling
              1. Time Travel Capabilities
              2. Apache Iceberg
                1. Architecture Overview
                  1. Metadata Structure
                    1. Snapshot Management
                      1. Partition Evolution
                        1. Hidden Partitioning
                          1. Schema Evolution Features
                          2. Apache Hudi
                            1. Copy-on-Write Tables
                              1. Merge-on-Read Tables
                                1. Timeline Management
                                  1. Incremental Processing
                                    1. Upsert Operations
                                    2. Delta Lake
                                      1. Transaction Log Design
                                        1. ACID Guarantees
                                          1. Time Travel Queries
                                            1. Schema Enforcement
                                              1. Data Versioning
                                                1. Streaming Integration
                                              2. Metadata and Catalog Systems
                                                1. Metastore Architecture
                                                  1. Centralized Metadata Repository
                                                    1. Table Registration
                                                      1. Partition Management
                                                        1. Statistics Collection
                                                        2. Data Discovery Mechanisms
                                                          1. Search and Browse Capabilities
                                                            1. Data Profiling
                                                              1. Usage Analytics
                                                                1. Recommendation Systems
                                                                2. Data Lineage Tracking
                                                                  1. Source-to-Target Mapping
                                                                    1. Transformation Documentation
                                                                      1. Impact Analysis
                                                                        1. Dependency Tracking
                                                                        2. Catalog Implementation Options
                                                                          1. Unity Catalog
                                                                            1. Multi-Cloud Support
                                                                              1. Fine-Grained Access Control
                                                                                1. Data Governance Features
                                                                                2. AWS Glue Data Catalog
                                                                                  1. Serverless Architecture
                                                                                    1. Crawler Automation
                                                                                      1. Integration Capabilities
                                                                                      2. Apache Hive Metastore
                                                                                        1. Open Source Foundation
                                                                                          1. Hadoop Ecosystem Integration
                                                                                            1. Custom Extensions
                                                                                        2. Query Engines and Processing Frameworks
                                                                                          1. Apache Spark
                                                                                            1. Core Engine Architecture
                                                                                              1. Batch Processing Capabilities
                                                                                                1. Streaming Processing Features
                                                                                                  1. SQL Interface
                                                                                                    1. DataFrame API
                                                                                                      1. Machine Learning Libraries
                                                                                                      2. Trino Query Engine
                                                                                                        1. Distributed Architecture
                                                                                                          1. Federated Query Capabilities
                                                                                                            1. Connector Ecosystem
                                                                                                              1. Performance Optimizations
                                                                                                              2. Specialized Analytics Engines
                                                                                                                1. Dremio
                                                                                                                  1. Data Virtualization
                                                                                                                    1. Self-Service Analytics
                                                                                                                      1. Query Acceleration
                                                                                                                      2. Starburst
                                                                                                                        1. Enterprise Features
                                                                                                                          1. Security Enhancements
                                                                                                                            1. Performance Optimizations

                                                                                                                      Previous

                                                                                                                      4. Data Lakehouse: The Unified Architecture

                                                                                                                      Go to top

                                                                                                                      Next

                                                                                                                      6. Implementation and Management Strategies