Modern data processing environments resemble factory lines, transforming raw data to valuable data products. The lean principles that have successfully transformed manufacturing are equally applicable to data processing, and are well aligned with the new trend known as DataOps. In this presentation, we will explain how applying lean and DataOps principles can be implemented as technical data processing solutions and processes in order to eliminate waste and improve data innovation speed. We will go through how to eliminate the following types of waste in data processing systems:
* Cognitive waste - unclear source of truth, dependency sprawl, duplication, ambiguity.
* Operational waste - overhead for deployment, upgrades, and incident recovery.
* Delivery waste - friction and delay in development, testing, and deployment.
* Product waste - misalignment to business value, detach from use cases, push driven development, vanity quality assurance.
We will primarily focus on technical solutions, but some of the waste mentioned requires organisational refactoring to eliminate.