jm + workflow   7

Argo Workflows & Pipelines
Nice new workflow system built on Kubernetes and Docker
k8s  kubernetes  docker  containers  workflow  pipelines  architecture  batch  nightly-jobs  ops 
21 days ago by jm
Git team workflows: merge or rebase?
Well-written description of the pros and cons. I'm a rebaser, fwiw.

(via Darrell)
via:darrell  git  merging  rebasing  history  git-log  coding  workflow  dev  teams  collaboration  github 
june 2015 by jm
Airflow
Airbnb's workflow management system; works off a DAG defined in Python code (ugh). Nice UI though, but I think Pinboard's take is neater
airbnb  open-source  python  workflow  jobs  cron  scheduling  batch 
june 2015 by jm
Luigi
A really excellent-looking workflow/orchestration engine for Hadoop, Pig, Hive, Redshift and other ETL jobs, featuring inter-job dependencies, cron-like scheduling, and failure handling. Open source, from Spotify
workflow  orchestration  scheduling  cron  spotify  open-source  luigi  redshift  pig  hive  hadoop  emr  jobs  make  dependencies 
july 2014 by jm
Factual/drake
a simple-to-use, extensible, text-based data workflow tool that organizes command execution around data and its dependencies. Data processing steps are defined along with their inputs and outputs and Drake automatically resolves their dependencies. [...] Drake is similar to GNU Make, but designed especially for data workflow management. It has HDFS [and S3] support, allows multiple inputs and outputs, and includes a host of features designed to help you bring sanity to your otherwise chaotic data processing workflows.


Via Nelson. Looks interesting, although I'd like to see more features around retries, single-executor locking, parallelism, alerting/metrics, and unattended cron-like operation -- those are always the hard part when I wind up coding up a data pump.
make  data  data-pump  drake  via:nelson  pipelines  workflow 
november 2013 by jm

Copy this bookmark:



description:


tags: