we built our own image building tool, Makisu, a solution that allows for more flexible, faster container image building at scale. Specifically, Makisu:

requires no elevated privileges, making the build process portable.

uses a distributed layer cache to improve performance across a build cluster.

provides flexible layer generation, preventing unnecessary files in images.

is Docker-compatible, supporting multi-stage builds and common build commands.

Uses:

Web-scale configuration management of all Linux/Unix systems;
Application deployment;
Immutable systems build definition;
Maintaining stateful services such as database and messaging platforms;
Automating one-off tasks & processes;
Deployment and management of the undercloud.

Features:

Python 3 DSL;
Declarative resource model with imperative capabilities;
Type / Provider plugin seperation;
Implicit ordering (with handler notification);
Formalized “Plan” vs “Apply” evaluation stages;
Early validation prior to runtime;
Programatically scoped variables;
Strong object-orientation

Jib takes advantage of layering in Docker images and integrates with your build system to optimize Java container image builds in the following ways:

Simple - Jib is implemented in Java and runs as part of your Maven or Gradle build. You do not need to maintain a Dockerfile, run a Docker daemon, or even worry about creating a fat JAR with all its dependencies. Since Jib tightly integrates with your Java build, it has access to all the necessary information to package your application. Any variations in your Java build are automatically picked up during subsequent container builds.

Fast - Jib takes advantage of image layering and registry caching to achieve fast, incremental builds. It reads your build config, organizes your application into distinct layers (dependencies, resources, classes) and only rebuilds and pushes the layers that have changed. When iterating quickly on a project, Jib can save valuable time on each build by only pushing your changed layers to the registry instead of your whole application.

Reproducible - Jib supports building container images declaratively from your Maven and Gradle build metadata, and as such can be configured to create reproducible build images as long as your inputs remain the same.

Using the monorepo as our requirements specification, we began exploring alternative approaches to achieving the same benefits. What are the core problems that a monorepo approach strives to solve? Can we develop a solution that works within the confines of a traditional binary integration world, where code is shared? Our approach, while still experimental, can be distilled into three key features:

Publisher feedback — provide the owner of shared code fast feedback as to which of their consumers they just broke, both direct and transitive. Also, allow teams to block releases based on downstream breakages. Currently, our engineering culture puts sole responsibility on consumers to resolve these issues. By giving library owners feedback on the impact they have to the rest of Netflix, we expect them to take on additional responsibility.

Managed source — provide consumers with a means to safely increment library versions automatically as new versions are released. Since we are already testing each new library release against all downstreams, why not bump consumer versions and accelerate version adoption, safely.

Distributed refactoring — provide owners of shared code a means to quickly find and globally refactor consumers of their API. We have started by issuing pull requests en masse to all Git repositories containing a consumer of a particular Java API. We’ve run some early experiments and expect to invest more in this area going forward.

backward compatible replacement for Dockerfile. Yes, you can take any Dockerfile, rename it to Rockerfile and use rocker build instead of docker build. ... Rocker aims to solve the following use cases, which are painful with plain Docker:

Mount reusable volumes on build stage, so dependency management tools may use cache between builds.
Share ssh keys with build (for pulling private repos, etc.), while not leaving them in the resulting image.
Build and run application in different images, be able to easily pass an artifact from one image to another, ideally have this logic in a single Dockerfile.
Tag/Push images right from Dockerfiles.
Pass variables from shell build command so they can be substituted to a Dockerfile.
And more. These are the most critical issues that were blocking our adoption of Docker at Grammarly.

The most challenging part is caching. While implementing those features seems to be not a big deal, it's not trivial to do that just by utilising Docker’s image cache (the one that docker build does). Actually, it is the main reason why those features are still not in Docker. With Rocker we achieve this by introducing a set of trade-offs. Search this page for "trade-off" to find out more details.

Concourse is a CI system composed of simple tools and ideas. It can express entire pipelines, integrating with arbitrary resources, or it can be used to execute one-off builds, either locally or in another CI system.

The try server runs a similar configuration to the continuous integration server, except that it is triggered not on commits but on "try job request", in order to test code pre-commit.

To prove how easy [MITM attacking Mavencentral JARs] is to do, I wrote dilettante, a man-in-the-middle proxy that intercepts JARs from maven central and injects malicious code into them. Proxying HTTP traffic through dilettante will backdoor any JARs downloaded from maven central. The backdoored version will retain their functionality, but display a nice message to the user when they use the library.

Manages migrations for your Cassandra data stores. Pillar grew from a desire to automatically manage Cassandra schema as code. Managing schema as code enables automated build and deployment, a foundational practice for an organization striving to achieve Continuous Delivery.

Pillar is to Cassandra what Rails ActiveRecord migrations or Play Evolutions are to relational databases with one key difference: Pillar is completely independent from any application development framework.

Amazon, Etsy, Google and Facebook are all primarily software development shops which command enormous amounts of resources. They are, to use Christopher Little’s metaphor, unicorns. How can the rest of us adopt continuous delivery? That’s the subject of my talk, which describes four case studies of organizations that adopted continuous delivery, with varying degrees of success.

One of my favourites – partly because it’s embedded software, not a website – is the story of HP’s LaserJet Firmware team, who re-architected their software around the principles of continuous delivery. People always want to know the business case for continuous delivery: the FutureSmart team provide one in the book they wrote that discusses how they did it.

The book introduces “Infrastructure as Code,” test-driven development, Chef, and cucumber-chef, and then proceeds to a simple example using Chef to provision a shared Linux server. The recipes for the server are developed test-first, demonstrating both the technique and the workflow.