Thursday, August 22
3:00 pm - 3:40 pm
Three or Four Shades of Blues: Success and Failure Adopting Workflow Management Tools
Hard Coded Stage
About the event
Apache Airflow, Apache Nifi, and AWS Step Functions are tools designed to enable declarative programming workflows. They take existing code and systems and allow users to stitch them together, manage failures and conditional logic, and execution on a schedule from a simple programming interface. These tools are famous for tasks like cron jobs, ETL and machine learning pipelines. While these tools are simple on the surface, adopting them reveals many limitations. This talk covers the successes and failures of utilizing these three projects in succession on a small team. We evaluate each project on their:
* User experience
* Programming Model
* Deployment Topology
* Scalability characteristics
We discuss where each project fails along these dimensions and why we settled on AWS Step Functions for these workloads. We conclude by discussing how these workflow systems can be improved and promising projects on the horizon.
Jowanza Joseph is a Software Engineer at Pluralsight. Jowanza's work focuses on distributed data processing and messaging systems. Previous to Pluralsight, Jowanza worked at Adobe, Edge by Ascential and Zagg. Jowanza contributes to several open source projects and blogs about big data and productivity.