Data Pipelines with Apache Airflow on Cloud Composer with Rodrigo Silveira (DDM)

About the talk

Whether you call it an ETL, a workflow, or a data pipeline, you do not want to build such a thing by hand. Instead, a framework such as Apache Airflow can make you far more productive by doing a lot of the heavy lifting for you. Google Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow.

This presentation will walk through what data pipelines are and what problems they solve, how frameworks can help, and how Google's Cloud Composer can take your operations to the next level.

Finally, we will cover some pitfalls to avoid, pro tips, and best practices to help you make the most out of Cloud Composer.

-------------------------------
About the speaker

Rodrigo Silveira is a former software engineer at DDM, now converted into a machine learning engineer. His focus is on building performant predictive models and deploying them into production.