Python practice in Data Engineering

Browse the rooms before signing in. Opening a room requires an account and follows your existing access. Practice does not issue certificates.

• Drop the Dupes

  A retry replayed the same records into the load. Remove exact duplicates, keeping the first of each.

  dedup. Free room.

• Log Line Split

  The ingestion job choked on a raw log dump. Split it into clean, non-blank lines before anything else runs.

  parsing. Free room.

• Parse the Params

  A tracking pixel encodes its payload as k=v&k=v. Parse the query string into a dict, where a repeated key takes its last value.

  parsing. Free room.

• Tally Events

  The dashboard needs a count of each event type from today's batch. Group and count by the event column.

  aggregation. Free room.

• Completeness Audit

  A quality gate needs the completeness of the key column before the batch is allowed through.

  data-quality. Account access required.

• Match the Keys

  An inner join only keeps rows whose key exists on both sides. Return the sorted set of keys present in BOTH input lists, the heart of a hash join.

  joins. Account access required.

• Revenue Roll-Up

  Finance wants revenue grouped by region. Sum the value column within each group.

  aggregation. Account access required.

• Table Loader

  A partner sent a CSV with a header row. Load it into record dicts the rest of the pipeline can use.

  parsing, records. Account access required.

• Delta-Encode the Column

  Columnar stores shrink monotonic columns by keeping the first value, then the gaps. Delta-encode a column so a run of slowly-changing values compresses to small numbers.

  columnar. Account access required.

• Sessionize the Clicks

  Group a user's click timestamps into sessions, starting a new one whenever there is a gap over five seconds.

  streaming. Account access required.

• Squeeze the Column

  A sorted status column repeats endlessly. Run-length encode it into (value, count) runs to shrink the file.

  encoding. Account access required.

• Stitch the Streams

  Two feeds share an id. Join them with a hash join so each left row picks up its matching right fields.

  joins. Account access required.

• Encode the Categories

  A low-cardinality country column wastes space. Dictionary-encode it into integer codes plus a value-to-id map.

  encoding. Account access required.

• Hold the Watermark

  A streaming job decides when a window is complete by tracking a watermark: the latest event time minus an allowance for stragglers. Compute the current watermark.

  streaming. Account access required.

• Schedule the DAG

  The orchestrator needs a run order for the pipeline tasks. Topologically sort the dependency graph (or report a cycle).

  orchestration. Account access required.

• Windowed Counts

  A metrics job buckets events into one-minute tumbling windows. Count events per window by their timestamp.

  streaming. Account access required.

• Capture the Changes

  Compare last night's snapshot with tonight's and classify every change into inserts, updates, and deletes for the incremental load.

  cdc. Account access required.

• Merge the Runs

  External sort writes sorted runs to disk, then merges them. Implement the two-way merge that walks both sorted runs in lockstep into one sorted output.

  external-sort. Account access required.

• Run the Warehouse

  The nightly job runs the whole ETL: ingest the CSV, drop bad and unmatched rows, enrich with the product dimension, and return revenue by category.

  pipeline. Account access required.

• Sort the Spill

  The dataset is too big for memory. Sort it with external merge sort, never sorting more than the chunk size at once.

  sorting. Account access required.