What moving a Dockerized pipeline off my laptop taught me about containers, networking, and hidden assumptions.

The post I Deployed My Data Pipeline to AWS. Then Everything That Was “Local” Broke. appeared first on Towards Data Science.

A production account of scaling an enterprise integration pipeline from 500 to 8,000 events per second, and the two correctness guarantees the throughput work was never allowed to trade away.

The post How to Scale an Integration Pipeline Without Breaking Correctness appeared first on Towards Data Science.

Building my first dbt models and learning what “analysis-ready” data actually means

The post I Thought Loading Data Was the Finish Line. It Was the Starting Point. appeared first on Towards Data Science.