warming up your workspace

Python Foundations through Data Engineering

Never written code before? Start here. You will learn the absolute basics of Python, output, variables, types, decisions, loops, and functions, using records, batches, pipelines, and throughput as your playground. By the end you are ready for Project 1.

First words

  1. Your first program: the pipeline is online
  2. Printing numbers: how many records arrived?
  3. Comments: annotate your pipeline
  4. Strings: naming your table
  5. Arithmetic: what is the throughput?

Remembering things

  1. Variables: store the record count
  2. Reassignment: a new batch arrives
  3. Three types: int, float, and str
  4. Conversion: parse a field from a CSV row
  5. Good names: writing a record card

Making choices

  1. Booleans: is this row valid?
  2. if: let only good rows through
  3. if/else: accept or drop the row
  4. elif: classify rows into partitions
  5. and / or / not: is the batch ready to load?

Doing it again

  1. while: count down the batch window
  2. for + range: scan each row in a batch
  3. Accumulation: total bytes across a batch
  4. break: stop the pipeline on a bad row
  5. Nested loops: scan partitions and rows

Many values and reuse

  1. Lists: store the batch sizes
  2. Loop a list: find the largest batch
  3. Your first function: label the pipeline
  4. Parameters and return: rows per second
  5. Capstone: mini pipeline report

Explore the full field