DOCSEARCHbyAlgolia
Submit Docs

Lam Tran Data Engineering Blog

website·Indexed 30 days ago

https://lam-tran.dev/

Technical blog by data engineer Lam Tran, featuring articles on big data, data engineering, and machine learning. Topics include Spark catalyst optimizer, structured streaming watermarks, Kafka architecture, Snowflake, Databricks, and database indexing, as well as explorations into algorithms, linear algebra, and computer vision.

Tokens
32.4K
Snippets
98
Records
182
Agent score
50%

What's inside Lam Tran

  1. Overview of db-schemachange for Database Change Management

    db-schemachange is a lightweight Python-based tool used to manage database objects across multiple platforms, including Databricks, Snowflake, MySQL, Postgres, SQL Server, and Oracle. It implements an Imperative-style approach to Database Change Management (DCM), similar to the Flyway database migration tool. By integrating db-schemachange with version control and CI/CD pipelines, developers can automate the approval and deployment of database changes, enabling Data DevOps practices.

  2. Overview of db-schemachange for Database Change Management

    db-schemachange is a lightweight Python-based tool used to manage database objects across multiple platforms, including Databricks, Snowflake, MySQL, Postgres, SQL Server, and Oracle. It implements an Imperative-style approach to Database Change Management (DCM), inspired by the Flyway migration tool. By integrating db-schemachange with a version control system and CI/CD pipelines, teams can apply modern software delivery practices to database deployments, enabling Data DevOps.

  3. Overview of db-schemachange for Database Change Management

    db-schemachange is a lightweight Python-based tool used to manage database objects across multiple platforms, including Databricks, Snowflake, MySQL, Postgres, SQL Server, and Oracle. It implements an Imperative-style approach to Database Change Management (DCM), inspired by the Flyway migration tool. By integrating db-schemachange with version control and CI/CD pipelines, developers can apply modern software delivery practices to database deployments, facilitating Data DevOps.

  4. Overview of the Spark Catalyst Optimizer

    The Spark Catalyst optimizer is the core component of Spark SQL designed to optimize structured queries expressed via SQL or the DataFrame/Dataset APIs. Its primary goal is to minimize application execution time and operational costs by transforming high-level queries into an optimized physical execution plan.

  5. Overview of db-schemachange for Database Change Management

    db-schemachange is a lightweight Python-based tool used to manage database objects across multiple platforms, including Databricks, Snowflake, MySQL, Postgres, SQL Server, and Oracle. It implements an Imperative-style approach to Database Change Management (DCM), inspired by the Flyway migration tool. By integrating db-schemachange with a version control system and a CI/CD pipeline, developers can apply modern software delivery practices (Data DevOps) to approve and deploy database changes.

  6. Overview of the Spark Catalyst execution pipeline

    The Spark Catalyst optimizer processes structured queries through four primary phases:

    1. Parsing and Analyzing: Transforms the query into an 'unresolved logical plan'. The Analyzer uses the Catalog to verify if relations exist, check column names/types, and resolve attributes to unique IDs.
    2. Logical Plan Optimization: Applies standard optimization rules to the analyzed logical plan to improve efficiency.
    3. Physical Planning: Converts the optimized logical plan into one or more physical plans (SparkPlan) that specify the actual execution strategy (e.g., choosing between a BroadcastHashJoin or SortMergeJoin).
    4. Code Generation: Compiles the final physical plan into Java bytecode for high-performance execution.

  7. Understand Merge Sort principles

    Merge Sort is a highly efficient algorithm based on the Divide and Conquer principle. It recursively divides the original array into two sub-arrays until each sub-array contains only one element, then merges those sub-arrays back together in a sorted manner.

  8. Understand Multiversion Concurrency Control (MVCC) in storage engines

    Storage engines typically avoid simple row lock mechanisms to maintain high performance in environments with concurrent reads and writes. Instead, they implement row locking through Multiversion Concurrency Control (MVCC), which allows multiple versions of a data item to exist simultaneously to reduce contention.

  9. Select the appropriate statistical chart for data evaluation

    Choose a chart based on the specific goal of your data analysis:

    1. Overall Data Overview:

    • Histogram plot: Use to represent data distribution, where the x-axis is the range of random variable values and the y-axis is the frequency.
    • Box plot: Use to visualize key values such as min, max, median, 25th, 50th, and 75th percentiles, and the Interquartile Range (IQR). Useful for identifying data cut-off points or outliers.
    • Time series plot: Use to represent data changes over time.

    2. Variable Comparison:

    • Bar chart: Use to compare values across multiple variables.
    • Line chart: Use to compare how a single variable changes over time (similar to time series plot).
    • Radar chart: Use to compare multiple groups.

    3. Correlation Analysis:

    • Scatter plot: Use to identify correlations between two or more variables.

  10. Understand Snowflake micro-partitioning architecture

    Snowflake uses micro-partitioning, an automatic partitioning scheme where tables are divided into micro-partitions during data ingestion based on natural sort order and file size. Unlike traditional data lakes, no manual partition keys are required.

    Key characteristics include:

    • Size: Each micro-partition contains 50 MB to 500 MB of uncompressed data (~16 MB compressed).
    • Format: Data is stored in a columnar format, with Snowflake selecting the most efficient compression algorithm based on data type and statistics.
    • Metadata: Snowflake stores the value range for each column, the number of distinct values, and other properties for each micro-partition to enable efficient query pruning and cost calculation by the optimizer.

  11. Manage state using Redux for optimized re-renders

    Redux provides a 'single source of truth' by maintaining the application state in a single store. State is read-only and can only be changed by emitting actions processed by pure functions called reducers. This architecture prevents unnecessary re-renders because only the components specifically subscribed to the changed state (via the store) will re-render.

  12. Understand how MVCC works in InnoDB

    Multi-Version Concurrency Control (MVCC) in InnoDB allows high concurrency by taking snapshots of data. This ensures a transaction sees a consistent view of the data regardless of other concurrent changes.

    Key mechanisms include:

    • Transaction IDs: InnoDB assigns a transaction ID when a session starts reading data.
    • Undo Logs: Changes to a record are written to an undo log. A rollback pointer in the record points to this log to allow data reversion.
    • Read Consistency: When a session reads a record, InnoDB compares the record's transaction ID with the session's data view. If the record is invisible (e.g., modified by an uncommitted transaction), InnoDB applies the undo logs to the data view until a readable version of the record is reconstructed.
    • Durability: Undo logs are copied to redo logs to ensure data recovery during system failures.