Data Engineering

Python pathway

Programming Foundations / Practice rooms / Track curriculum and enrollment

1. Ingestion & Parsing

   Every pipeline starts by reading messy bytes off disk and turning them into clean records. Parse delimited text and JSON by hand, shape rows into dictionaries, infer and coerce types, and survive malformed input without crashing the job. These are the unglamorous primitives every loader is built from.

   • Delimited Text: 5 lessons
   • Rows as Records: 5 lessons
   • Type Inference & Coercion: 5 lessons
   • JSON and Nested Data: 5 lessons
   • Surviving Bad Data: 5 lessons

2. Cleaning & Validation

   Raw records are full of duplicates, nulls, and values that violate the rules your warehouse assumes. Deduplicate, handle missing data, enforce constraints, and emit a data-quality report, the gate every batch passes through before it is trusted downstream.

   • Deduplication: 5 lessons
   • Missing Values: 5 lessons
   • Constraints: 5 lessons
   • Cleaning Transforms: 5 lessons
   • Data-Quality Report: 5 lessons

3. Transformations & Joins

   The heart of every pipeline: reshape records and combine datasets. Group and aggregate, pivot, then build the three join algorithms every query engine relies on, nested-loop, hash, and sort-merge, plus the set operations and window functions that SQL gives you for free.

   • Grouping & Aggregation: 5 lessons
   • Reshaping: 5 lessons
   • Joins: 5 lessons
   • Sort-Merge & Set Operations: 5 lessons
   • Window Functions: 5 lessons

4. Big-Data Algorithms

   When the data does not fit in memory, you reach for clever algorithms instead of more RAM. Sort data larger than memory with external merge sort, test membership with a Bloom filter, estimate distinct counts the way a warehouse does, and track frequencies with a count-min sketch. These are the structures behind every system that claims to scale.

   • External Merge Sort: 5 lessons
   • Bloom Filters: 5 lessons
   • Cardinality Estimation: 5 lessons
   • Count-Min Sketch: 5 lessons
   • Streaming Summaries: 5 lessons

5. File Formats & Encoding

   Why is Parquet so much smaller and faster than CSV? Because it stores data by column and squeezes each column with the right encoding. Build the columnar layout, run-length and dictionary and delta encodings, variable-length integers, checksums, and the zone maps that let a query skip whole blocks of data unread.

   • Columnar Layout: 5 lessons
   • Run-Length Encoding: 5 lessons
   • Dictionary Encoding: 5 lessons
   • Delta & Varint: 5 lessons
   • Blocks & Integrity: 5 lessons

6. MapReduce & Partitioning

   The model that made big data parallel: map each record to key-value pairs, shuffle pairs to the machine that owns their key, and reduce each group. Build the map, shuffle, and reduce stages, the classic word count and inverted index, and the hash and range partitioning that spreads work across workers, plus how to spot the skew that ruins it.

   • The Map Stage: 5 lessons
   • The Shuffle: 5 lessons
   • The Reduce Stage: 5 lessons
   • Full MapReduce Jobs: 5 lessons
   • Partitioning & Skew: 5 lessons

7. Stream Processing

   Unbounded data never stops arriving, so you compute over windows of time instead of whole datasets, and you cope with events that show up late and out of order. Build event-time handling, tumbling, sliding, and session windows, the watermarks that decide when a window is done, and the incremental aggregation that keeps fixed state as the stream flows.

   • Event Time: 5 lessons
   • Tumbling Windows: 5 lessons
   • Sliding & Session Windows: 5 lessons
   • Watermarks & Late Data: 5 lessons
   • Incremental Aggregation: 5 lessons

8. Pipelines & Orchestration

   A pipeline is a directed acyclic graph of tasks, and an orchestrator decides what runs when. Build the DAG, order tasks by topological sort so dependencies run first, group independent tasks into parallel stages, retry flaky tasks with exponential backoff, make runs idempotent so reprocessing is safe, and schedule and backfill. This is a small Airflow.

   • The DAG: 5 lessons
   • Execution Order: 5 lessons
   • Retries & Backoff: 5 lessons
   • Idempotency: 5 lessons
   • Scheduling: 5 lessons

9. A Mini Query Engine

   Every warehouse turns a query into a plan of operators and runs them. Build the relational operators (select, project, sort, group, aggregate), compose them into a query executor, add an optimizer that pushes filters down and prunes columns, then handle change data capture and slowly-changing dimensions, the patterns that keep a warehouse in sync with its sources.

   • Relational Operators: 5 lessons
   • Aggregation Operators: 5 lessons
   • The Query Executor: 5 lessons
   • The Optimizer: 5 lessons
   • Change Data Capture: 5 lessons

10. Capstone: A Mini Warehouse ETL

    Assemble everything into one pipeline: ingest raw CSV into typed records, drop the bad rows, enrich each fact with a dimension lookup and derive new fields, aggregate into the metrics a business actually asks for, load them into a warehouse table with idempotent upserts and change capture, and orchestrate the stages as a DAG. This is a small data warehouse, end to end.

    • The Ingest Stage: 5 lessons
    • The Transform Stage: 5 lessons
    • The Aggregate Stage: 5 lessons
    • The Load Stage: 5 lessons
    • Orchestrating the Pipeline: 5 lessons