jm + pipelines   9

Nextflow - A DSL for parallel and scalable computational pipelines
Data-driven computational pipelines

Nextflow enables scalable and reproducible scientific workflows using software containers. It allows the adaptation of pipelines written in the most common scripting languages.

Its fluent DSL simplifies the implementation and the deployment of complex parallel and reactive workflows on clouds and clusters.


GPLv3 licensed, open source
computation  workflows  pipelines  batch  docker  ops  open-source 
11 days ago by jm
Pipeline Development Tools
"pipelint.sh" -- command line Jenkins pipeline linting
pipelines  jenkins  ci  coding  dev 
24 days ago by jm
Blue Ocean
new Jenkins UX. looks great
jenkins  tests  ui  ux  pipelines  testing 
may 2016 by jm
Jenkins 2.0
built-in support for CI/CD deployment pipelines, driven from a checked-in DSL file. great stuff, very glad to see them going this direction. (via Eric)
via:eric  jenkins  ci  cd  deployment  pipelines  testing  automation  build 
march 2016 by jm
The Totally Managed Analytics Pipeline: Segment, Lambda, and Dynamo
notable mainly for the details of Terraform support for Lambda: that's a significant improvement to Lambda's production-readiness
aws  pipelines  data  streaming  lambda  dynamodb  analytics  terraform  ops 
october 2015 by jm
This Canadian Artist Halted Pipeline Development by Copyrighting His Land as a Work of Art
One of the really important pieces on my land was this white-picket fence. The picket fence is probably 100 yards or less, within 100 yards of where they wanted to build this pipeline. I [plan to] extend it 8 feet every year for the rest of my life and I've been doing that for 25 years. It got me thinking, where does this piece end? Does it end at the actual structure of the fence or the things growing around it, growing through it, that are part of the photography, the documentation of it? I realized at that point that [the fence], and the other sculptures and pieces and incursions and conceptual works, were actually integral to that piece of land and to my practice.

I had not intended for it to be a political piece, it was just a piece, an idea the follow-through of which at some point became poetic, you go, "Wait a minute the fence actually stopped them!" But the fence doesn't actually enclose anything. It's just a straight line. And it's marking something that's actually unmarkable, which is time. And one day it'll be gone, as will I. The land will be changed--but it was just this crazy irony that kicked into play when I was standing there with those oil negotiators.
copyright  art  pipelines  canada  politics  oil  land  conceptual-art  ip 
november 2014 by jm
Collection Pipeline
a nice summarisation of the state of pipe/stream-oriented collection operations in various languages, from Martin Fowler
martin-fowler  patterns  coding  ruby  clojure  streams  pipelines  pipes  unix  lambda  fp  java  languages 
july 2014 by jm
Continuous Delivery with ETL Systems [video]
Lonely Planet and Dr Foster Intelligence both make heavy use of ETL in their products, and both organisations have applied the principles of Continuous Delivery to their delivery process.
Some of the Continuous Delivery norms need to be adapted in the context of ETL, and some interesting patterns emerge, such as running Continuous Integration against data, as well as code.
etl  video  presentations  lonely-planet  dr-foster-intelligence  continuous-delivery  deployment  pipelines 
march 2014 by jm
Factual/drake
a simple-to-use, extensible, text-based data workflow tool that organizes command execution around data and its dependencies. Data processing steps are defined along with their inputs and outputs and Drake automatically resolves their dependencies. [...] Drake is similar to GNU Make, but designed especially for data workflow management. It has HDFS [and S3] support, allows multiple inputs and outputs, and includes a host of features designed to help you bring sanity to your otherwise chaotic data processing workflows.


Via Nelson. Looks interesting, although I'd like to see more features around retries, single-executor locking, parallelism, alerting/metrics, and unattended cron-like operation -- those are always the hard part when I wind up coding up a data pump.
make  data  data-pump  drake  via:nelson  pipelines  workflow 
november 2013 by jm

Copy this bookmark:



description:


tags: