warming up your workspace

Query API versus storage engine, the split that reshaped data infrastructure

For decades a database was one monolith: the thing that stored your data and the thing that answered questions about it were the same locked box, and to switch query engines you migrated everything. The defining move of the 2020s, and the one that reached its conclusion in 2026 as Apache Iceberg became the default table format across Snowflake, Databricks, and DuckDB alike, was to break that box in two. Storage became a pile of open files anyone can read; the query engine became a swappable layer on top. Understand the split by building both halves, and keeping them honestly separate.

The one idea

A database does two jobs that have nothing to do with each other. One is physical: lay bytes on disk, lay them out well, hand back the ones asked for. The other is logical: parse a query, decide what to fetch, combine it into an answer. Fuse them and you are married to one vendor. Separate them, define a clean interface between "what data do you need" and "here are the bytes", and you can point three different query engines at the same files, or upgrade the engine without touching a byte of storage. The interface is where all the leverage lives, because a smart storage engine can do a lot before the query layer ever sees a row.

Build the storage engine

Store data by column, not by row. All the countries together, all the spends together. This is the layout that makes analytics fast, and it makes two optimizations possible that the query layer can request but the engine performs.

class ColumnStore:
    def __init__(self):
        self.columns = {}          # name -> list of values, laid out column-wise
        self.reads = 0
    def scan(self, needed, predicate_col=None, predicate=None):
        n = len(next(iter(self.columns.values())))
        # predicate pushdown: filter rows before reading the data columns
        if predicate_col is not None:
            keep = [i for i in range(n)
                    if (self._touch() or True) and predicate(self.columns[predicate_col][i])]
        else:
            keep = list(range(n))
        # column pruning: only ever read the columns the query asked for
        return {c: [self.columns[c][i] for i in keep] for c in needed}

The two ideas doing the work:

  • Column pruning. The query wants user and spend, so the engine never touches country's data column or the big notes column. In a row store, every row drags all its columns along; in a column store, unused columns cost nothing.
  • Predicate pushdown. The filter country == 'IN' is evaluated inside the storage engine, on just the country column, so the data columns are read only for surviving rows. The work happens next to the bytes instead of after they have been shipped up to the query layer.

Build the query API on top

The query layer knows nothing about column layout. It composes intent, then hands a plan down through the interface.

class Query:
    def select(self, *cols): self._cols = cols; return self
    def where(self, col, pred): self._pc, self._p = col, pred; return self
    def run(self): return self.store.scan(self._cols, self._pc, self._p)

res = Query(store).select("user", "spend").where("country", lambda c: c == "IN").run()

Proof: the engine reads a fraction of the table

result: {'user': ['u0', 'u3', 'u6'], 'spend': [0, 30, 60]}
cells read by engine: 15  (of 36 total in the table)

The table has 36 cells. Answering the query touched 15, the filter column plus two data columns for only the matching rows. The notes column, half the table's bulk, was never read. And because the query API and the storage engine talk through one narrow scan interface, the same store served a completely different second query with no change to storage at all. That is the whole architecture in miniature: the engine does the pruning and pushdown, the query layer just says what it wants.

Three details that matter in the real thing:

  • Real formats store column statistics per file chunk, min and max values, so the engine can skip entire files that cannot match a predicate without reading them at all. Pushdown scales from rows to files to whole partitions.
  • The interface is a contract, not a suggestion. Iceberg and Delta Lake standardize it as a table format: a spec for how files, schemas, and snapshots are described, so any compliant engine can read and write the same table safely, even concurrently.
  • Separating compute from storage is also what makes cloud data warehouses elastic: storage sits cheap in object storage, and you spin query engines up and down independently based on load.

Where this shows up

This split is the architecture of every modern data platform. Parquet is the columnar file format; Iceberg and Delta are the table formats defining the interface; Spark, Trino, DuckDB, Snowflake, and a dozen others are interchangeable query engines reading the same tables. The 2026 consolidation onto open formats happened precisely because once storage is an open standard, the query engine becomes a competitive, swappable commodity, which is exactly what you just built at small scale.

If you want to build the file format, the statistics-based skipping, and a query planner that pushes work down to storage, that is the system the data-engineering track on IWTLP constructs from the bytes up.

Sources