jm + airflow   7

boundary-layer
Declarative Airflow Workflows in YAML, from Etsy
airflow  python  batch  cron  etl 
15 days ago by jm
Best practices with Airflow
interesting presentation describing how to architect Airflow ETL setups; see also https://gtoonstra.github.io/etl-with-airflow/principles.html
etl  airflow  batch  architecture  systems  ops 
october 2016 by jm
Airflow/AMI/ASG nightly-packaging workflow
Some tantalising discussion on twitter of an Airflow + AMI + ASG workflow for ML packaging:

'We build models using Airflow. We deploy new models as AMIs where each AMI is model + scoring code. The AMI is hence a version of code + model at a point in time : #immutable_infrastructure. It's natural for Airflow to build & deploy the model+code with each Airflow DAG Run corresponding to a versioned AMI. if there's a problem, we can simply roll back to the previous AMI & identify the problematic model building Dag run. Since we use ASGs, Airflow can execute a rolling deploy of new AMIs. We could also have it do a validation & ASG rollback of the AMI if validation fails. Airflow is being used for reliable Model build+validation+deployment.'
ml  packaging  airflow  asg  ami  deployment  ops  infrastructure  rollback 
september 2016 by jm

Copy this bookmark:



description:


tags: