Mechanics of Data Pipelines

06/13/2017 - 12:20 to 13:00
long talk (40 min)

Session abstract: 

This talk focused the topic on how to model data pipelines as retroactive, immutable data structures. It covers the topic of how do you build a data pipelines for a growing organization where different teams depend on each others data and need to be able to re-process data when errors occur upstream. I draw comparisons between the microservice architectures for both stream and batch processings and provide some guiding principals towards building resiliant systems based on experience scaling out infrastructure at SoundCloud.