Diagram of a Data Pipeline


When I worked as a data infrastructure engineer, I noticed a disconnect between the work my team did and the designers, front-end engineers, and business teams' understanding of it. The vocabulary surrounding cloud-based data storage is often nebulous. On one hand, as engineers, we talk about "the stack," a linear, multilayered model for describing how network technologies interact with each other. On the other, we also talk about "the cloud" and the various services that we use to structure and manage information contained within it (while acknowledging that "the cloud" is not a monolithic entity, but a massive network of data centers). To get us all on the same page, I made a diagram to communicate what overhauling our current pipeline looked like, accompanied by a glossary.