Makisu
10 weeks ago by jm
Uber’s Fast, Reliable Docker Image Builder for Apache Mesos and Kubernetes.
makisu
docker
containers
ops
build
mesos
kubernetes
building
we built our own image building tool, Makisu, a solution that allows for more flexible, faster container image building at scale. Specifically, Makisu:
requires no elevated privileges, making the build process portable.
uses a distributed layer cache to improve performance across a build cluster.
provides flexible layer generation, preventing unnecessary files in images.
is Docker-compatible, supporting multi-stage builds and common build commands.
10 weeks ago by jm
OpsMop
november 2018 by jm
'a next-generation, no-compromise automation system'.
opsmop
ops
configuration-management
deployment
build
Uses:
Web-scale configuration management of all Linux/Unix systems;
Application deployment;
Immutable systems build definition;
Maintaining stateful services such as database and messaging platforms;
Automating one-off tasks & processes;
Deployment and management of the undercloud.
Features:
Python 3 DSL;
Declarative resource model with imperative capabilities;
Type / Provider plugin seperation;
Implicit ordering (with handler notification);
Formalized “Plan” vs “Apply” evaluation stages;
Early validation prior to runtime;
Programatically scoped variables;
Strong object-orientation
november 2018 by jm
spotify/dockerfile-maven: A set of Maven tools for dealing with Dockerfiles
august 2018 by jm
'a Maven plugin and extension which help to seamlessly integrate Docker with Maven. The design goals are:
Don't try to do anything fancy. Dockerfiles are how you build Docker projects; that's what this plugin uses. They are mandatory.
Make the Docker build process integrate with the Maven build process. If you bind the default phases, when you type mvn package, you get a Docker image. When you type mvn deploy, your image gets pushed.
Make the goals remember what you are doing. You can type mvn dockerfile:build and later mvn dockerfile:tag and later mvn dockerfile:push without problems. This also eliminates the need for something like mvn dockerfile:build -DalsoPush; instead you can just say mvn dockerfile:build dockerfile:push.
Integrate with the Maven build reactor. You can depend on the Docker image of one project in another project, and Maven will build the projects in the correct order. This is useful when you want to run integration tests involving multiple services.'
Looks very nice and well-run -- shame it's Maven instead of Gradle...
java
docker
maven
build
coding
packaging
Don't try to do anything fancy. Dockerfiles are how you build Docker projects; that's what this plugin uses. They are mandatory.
Make the Docker build process integrate with the Maven build process. If you bind the default phases, when you type mvn package, you get a Docker image. When you type mvn deploy, your image gets pushed.
Make the goals remember what you are doing. You can type mvn dockerfile:build and later mvn dockerfile:tag and later mvn dockerfile:push without problems. This also eliminates the need for something like mvn dockerfile:build -DalsoPush; instead you can just say mvn dockerfile:build dockerfile:push.
Integrate with the Maven build reactor. You can depend on the Docker image of one project in another project, and Maven will build the projects in the correct order. This is useful when you want to run integration tests involving multiple services.'
Looks very nice and well-run -- shame it's Maven instead of Gradle...
august 2018 by jm
Google Cloud Platform Blog: Introducing Jib
july 2018 by jm
'build Java Docker images better':
build
google
java
docker
maven
gradle
coding
builds
jars
fat-jars
packaging
Jib takes advantage of layering in Docker images and integrates with your build system to optimize Java container image builds in the following ways:
Simple - Jib is implemented in Java and runs as part of your Maven or Gradle build. You do not need to maintain a Dockerfile, run a Docker daemon, or even worry about creating a fat JAR with all its dependencies. Since Jib tightly integrates with your Java build, it has access to all the necessary information to package your application. Any variations in your Java build are automatically picked up during subsequent container builds.
Fast - Jib takes advantage of image layering and registry caching to achieve fast, incremental builds. It reads your build config, organizes your application into distinct layers (dependencies, resources, classes) and only rebuilds and pushes the layers that have changed. When iterating quickly on a project, Jib can save valuable time on each build by only pushing your changed layers to the registry instead of your whole application.
Reproducible - Jib supports building container images declaratively from your Maven and Gradle build metadata, and as such can be configured to create reproducible build images as long as your inputs remain the same.
july 2018 by jm
AWS CodeBuild Plugin - Jenkins - Jenkins Wiki
december 2017 by jm
Trigger AWS CodeBuild jobs as build steps for a Jenkins project. :thinking_face_emoji:
jenkins
hacks
aws
codebuild
build
coding
ci
december 2017 by jm
GTK+ switches build from Autotools to Meson
august 2017 by jm
'The main change is that now GTK+ takes about ⅓ of the time to build
compared to the Autotools build, with likely bigger wins on older/less
powerful hardware; the Visual Studio support on Windows should be at
least a couple of orders of magnitude easier (shout out to Fan
Chun-wei for having spent so, so many hours ensuring that we could
even build on Windows with Visual Studio and MSVC); and maintaining
the build system should be equally easier for everyone on any platform
we currently support.'
Looking at http://mesonbuild.com/ it appears to be Python-based and
AL2-licensed open source.
On the downside, though, the Meson file is basically a Python script,
which is something I'm really not fond of :( more details at http://taint.org/2011/02/18/001527a.html .
meson
build
coding
dev
autotools
gtk+
python
compared to the Autotools build, with likely bigger wins on older/less
powerful hardware; the Visual Studio support on Windows should be at
least a couple of orders of magnitude easier (shout out to Fan
Chun-wei for having spent so, so many hours ensuring that we could
even build on Windows with Visual Studio and MSVC); and maintaining
the build system should be equally easier for everyone on any platform
we currently support.'
Looking at http://mesonbuild.com/ it appears to be Python-based and
AL2-licensed open source.
On the downside, though, the Meson file is basically a Python script,
which is something I'm really not fond of :( more details at http://taint.org/2011/02/18/001527a.html .
august 2017 by jm
Towards true continuous integration – Netflix TechBlog – Medium
may 2017 by jm
Netflix discuss how they handle the eternal dependency-management problem which arises with lots of microservices:
What I find interesting is that Amazon dealt effectively with the first two many years ago, in the form of their "Brazil" build system, and Google do the latter (with Refaster?). It would be amazing to see such a system released into an open source form, but maybe it's just too heavyweight for anyone other than a giant software company on the scale of a Google, Netflix or Amazon.
brazil
amazon
build
microservices
dependencies
coding
monorepo
netflix
google
refaster
Using the monorepo as our requirements specification, we began exploring alternative approaches to achieving the same benefits. What are the core problems that a monorepo approach strives to solve? Can we develop a solution that works within the confines of a traditional binary integration world, where code is shared? Our approach, while still experimental, can be distilled into three key features:
Publisher feedback — provide the owner of shared code fast feedback as to which of their consumers they just broke, both direct and transitive. Also, allow teams to block releases based on downstream breakages. Currently, our engineering culture puts sole responsibility on consumers to resolve these issues. By giving library owners feedback on the impact they have to the rest of Netflix, we expect them to take on additional responsibility.
Managed source — provide consumers with a means to safely increment library versions automatically as new versions are released. Since we are already testing each new library release against all downstreams, why not bump consumer versions and accelerate version adoption, safely.
Distributed refactoring — provide owners of shared code a means to quickly find and globally refactor consumers of their API. We have started by issuing pull requests en masse to all Git repositories containing a consumer of a particular Java API. We’ve run some early experiments and expect to invest more in this area going forward.
What I find interesting is that Amazon dealt effectively with the first two many years ago, in the form of their "Brazil" build system, and Google do the latter (with Refaster?). It would be amazing to see such a system released into an open source form, but maybe it's just too heavyweight for anyone other than a giant software company on the scale of a Google, Netflix or Amazon.
may 2017 by jm
Instead of containerization, give me strong config & deployment primitives
april 2017 by jm
Reasonable list of things Docker does badly at the moment, and a call to fix them. I still think Docker/rkt are a solid approach, if not 100% there yet though
docker
containers
complaining
whinge
networking
swarm
deployment
architecture
build
packaging
april 2017 by jm
grammarly/rocker
docker
rocker
build
containers
dockerfiles
may 2016 by jm
backward compatible replacement for Dockerfile. Yes, you can take any Dockerfile, rename it to Rockerfile and use rocker build instead of docker build. ... Rocker aims to solve the following use cases, which are painful with plain Docker:
Mount reusable volumes on build stage, so dependency management tools may use cache between builds.
Share ssh keys with build (for pulling private repos, etc.), while not leaving them in the resulting image.
Build and run application in different images, be able to easily pass an artifact from one image to another, ideally have this logic in a single Dockerfile.
Tag/Push images right from Dockerfiles.
Pass variables from shell build command so they can be substituted to a Dockerfile.
And more. These are the most critical issues that were blocking our adoption of Docker at Grammarly.
The most challenging part is caching. While implementing those features seems to be not a big deal, it's not trivial to do that just by utilising Docker’s image cache (the one that docker build does). Actually, it is the main reason why those features are still not in Docker. With Rocker we achieve this by introducing a set of trade-offs. Search this page for "trade-off" to find out more details.
may 2016 by jm
fiunchinho/dockerize-me
may 2016 by jm
'Tired of copy/pasting Dockerfiles around? Not sure about best practices for Dockerfiles or Docker entry points? This tool lets you Dockerize your applications using best practices to define your Dockerfile and Docker entry point files.'
The best practices in question are defined here: https://github.com/docker-library/official-images#review-guidelines
docker
dockerfile
images
build
best-practices
alpine
containers
The best practices in question are defined here: https://github.com/docker-library/official-images#review-guidelines
may 2016 by jm
CD at LMAX: Testing into Production and Back Again
may 2016 by jm
Chock-full of excellent build/test ideas from LMAX's Continuous Delivery setup. Lots of good ideas to steal
testing
lmax
build
test
continuous-delivery
dev
may 2016 by jm
The Challenges of Container Configuration // Speaker Deck
may 2016 by jm
Some good advice on Docker metadata/config from Gareth Rushgrove
docker
metadata
configuration
build
devops
dev
containers
slidfes
may 2016 by jm
Jenkins 2.0
march 2016 by jm
built-in support for CI/CD deployment pipelines, driven from a checked-in DSL file. great stuff, very glad to see them going this direction. (via Eric)
via:eric
jenkins
ci
cd
deployment
pipelines
testing
automation
build
march 2016 by jm
Google is Maven Central's New Best Friend
october 2015 by jm
google now mirroring Maven Central.
google
maven
maven-central
jars
hosting
java
packages
build
october 2015 by jm
remind101/conveyor
october 2015 by jm
'A fast build system for Docker images', open source, in Go, hooks into Github
build
ci
docker
github
go
october 2015 by jm
Declaratively Provision Docker Images Using Nix
may 2015 by jm
I really wish Docker/CoreOS would look at copying some of the deterministic-build ideas from Nix; see also http://gregoryszorc.com/blog/2014/10/13/deterministic-and-minimal-docker-images/
build
packaging
docker
nix
nix-docker
deterministic-builds
nixos
apollo
brazil
may 2015 by jm
Understanding the Docker Cache for Faster Builds
may 2015 by jm
good advice. see also the Best Practices official doc at https://docs.docker.com/articles/dockerfile_best-practices/
docker
build
packaging
cache
best-practices
tips
may 2015 by jm
How to change Gradle cache location
may 2015 by jm
$GRADLE_USER_HOME, basically -- it may also be possible to set from the Gradle script itself too
gradle
build
caching
environment
unix
cache
may 2015 by jm
Build in the Cloud: Accessing Source Code
march 2015 by jm
Google reinvented ClearCase
clearcase
google
source-code
coding
scm
build
march 2015 by jm
[Nix-dev] Pulling a programs source code from a git repo
march 2015 by jm
Nix supports building from git sha. excellent
nix
packaging
build
dev
ci
march 2015 by jm
Managing private Nix packages outside the Nixpkgs tree
march 2015 by jm
Useful for private-repo Nix usage
nix
packaging
deployment
ops
build
ci
nixpkgs
march 2015 by jm
Stu Hood and Brian Degenhardt, Scala at Twitter, SF Scala @Twitter 20150217
march 2015 by jm
'Stu Hood and Brian Degenhardt talk about the history of Scala at Twitter, from inception until today, covering 2.10 migration, the original Alex Payne’s presentation from way back, pants, and more. The first five years of Scala at Twitter and the years ahead!'
Very positive indeed on the monorepo concept.
monorepo
talks
scala
sfscala
stu-hood
twitter
pants
history
repos
build
projects
compilation
gradle
maven
sbt
Very positive indeed on the monorepo concept.
march 2015 by jm
Concourse
ci
concourse-ci
build
deployment
continuous-integration
continuous-deployment
devops
march 2015 by jm
Concourse is a CI system composed of simple tools and ideas. It can express entire pipelines, integrating with arbitrary resources, or it can be used to execute one-off builds, either locally or in another CI system.
march 2015 by jm
Try Server
march 2015 by jm
Good terminology for this concept:
See also https://wiki.mozilla.org/ReleaseEngineering/TryServer for the Moz take on it.
build
ci
integration
try-server
jenkins
buildbot
chromium
development
The try server runs a similar configuration to the continuous integration server, except that it is triggered not on commits but on "try job request", in order to test code pre-commit.
See also https://wiki.mozilla.org/ReleaseEngineering/TryServer for the Moz take on it.
march 2015 by jm
Enjoy Bintray and use it as pain-free gateway to Maven Central
february 2015 by jm
ahh, interesting! This looks much easier (via JBaruch)
via:jbaruch
build
bintray
java
release
oss
february 2015 by jm
On-Demand Jenkins Slaves With Amazon EC2
august 2014 by jm
This is very likely where we'll be going for our acceptance tests in Swrve
testing
jenkins
ec2
spot-instances
scalability
auto-scaling
ops
build
august 2014 by jm
How to take over the computer of any JVM developer
jars
dependencies
java
build
clojure
security
mitm
http
proxies
backdoors
scala
maven
gradle
july 2014 by jm
To prove how easy [MITM attacking Mavencentral JARs] is to do, I wrote dilettante, a man-in-the-middle proxy that intercepts JARs from maven central and injects malicious code into them. Proxying HTTP traffic through dilettante will backdoor any JARs downloaded from maven central. The backdoored version will retain their functionality, but display a nice message to the user when they use the library.
july 2014 by jm
Pillar
migrations
database
ops
pillar
cassandra
activerecord
scala
continuous-delivery
automation
build
june 2014 by jm
Manages migrations for your Cassandra data stores. Pillar grew from a desire to automatically manage Cassandra schema as code. Managing schema as code enables automated build and deployment, a foundational practice for an organization striving to achieve Continuous Delivery.
Pillar is to Cassandra what Rails ActiveRecord migrations or Play Evolutions are to relational databases with one key difference: Pillar is completely independent from any application development framework.
june 2014 by jm
Videos from the Continuous Delivery track at QCon SF 2012
may 2013 by jm
Think we'll be watching some of these in work soon -- Jez Humble's talk (the last one) in particular looks good:
continuous-integration
continuous-delivery
build
release
process
dev
deployment
videos
qcon
towatch
hp
Amazon, Etsy, Google and Facebook are all primarily software development shops which command enormous amounts of resources. They are, to use Christopher Little’s metaphor, unicorns. How can the rest of us adopt continuous delivery? That’s the subject of my talk, which describes four case studies of organizations that adopted continuous delivery, with varying degrees of success.
One of my favourites – partly because it’s embedded software, not a website – is the story of HP’s LaserJet Firmware team, who re-architected their software around the principles of continuous delivery. People always want to know the business case for continuous delivery: the FutureSmart team provide one in the book they wrote that discusses how they did it.
may 2013 by jm
Test-Driven Infrastructure with Chef
march 2013 by jm
Interesting idea.
tdd
chef
server
provisioning
build
deploy
linux
coding
ops
sysadmin
The book introduces “Infrastructure as Code,” test-driven development, Chef, and cucumber-chef, and then proceeds to a simple example using Chef to provision a shared Linux server. The recipes for the server are developed test-first, demonstrating both the technique and the workflow.
march 2013 by jm
Why djb redo won't be the Git of build systems
january 2011 by jm
A counter-argument: "so, redo, from a conceptual point of view, has a really good and simple approach (very djb-y), and I'm sure it's an excellent tool for new projects, but for existing projects that already use make in a non-recursive fashion, it would a maintenance PITA. And that's why I conclude that redo in its current conceptual state will never be the Git of build systems. make is still more flexible, and even though it has its flaws, it's still good enough for most people, and also a de-facto standard."
redo
build
djb
building
make
compilation
from delicious
january 2011 by jm
The things make got right (and how to make it better)
january 2011 by jm
jgc provides a good demonstration of how a general-purpose programming language tends to make a crappy DSL -- specifically Rakefiles
dsl
build
make
coding
jgc
languages
configuration
makefiles
rake
ruby
from delicious
january 2011 by jm
good Hacker News thread on djb's "redo"
january 2011 by jm
YA make-replacement build system. the thread is better than the linked article, btw
hacker-news
via:fanf
make
build
djb
redo
compilation
building
coding
open-source
from delicious
january 2011 by jm
Kohsuke "Hudson" Kawaguchi leaving Sun
april 2010 by jm
and starting a new company around Hudson
hudson
ci
sun
oracle
acquisitions
startups
build
from delicious
april 2010 by jm
related tags
acquisitions ⊕ activerecord ⊕ alpine ⊕ amazon ⊕ android ⊕ apollo ⊕ architecture ⊕ auto-scaling ⊕ automation ⊕ autotools ⊕ aws ⊕ backdoors ⊕ bazel ⊕ best-practices ⊕ bintray ⊕ brazil ⊕ build ⊖ buildbot ⊕ building ⊕ builds ⊕ cache ⊕ caching ⊕ cassandra ⊕ cd ⊕ chef ⊕ chromium ⊕ ci ⊕ clearcase ⊕ clojure ⊕ codebuild ⊕ coding ⊕ compilation ⊕ complaining ⊕ concourse-ci ⊕ configuration ⊕ configuration-management ⊕ containers ⊕ continuous-delivery ⊕ continuous-deployment ⊕ continuous-integration ⊕ database ⊕ dependencies ⊕ deploy ⊕ deployment ⊕ deterministic-builds ⊕ dev ⊕ development ⊕ devops ⊕ djb ⊕ docker ⊕ dockerfile ⊕ dockerfiles ⊕ dsl ⊕ ec2 ⊕ environment ⊕ facebook ⊕ fat-jars ⊕ github ⊕ go ⊕ google ⊕ gradle ⊕ gtk+ ⊕ hacker-news ⊕ hacks ⊕ history ⊕ hosting ⊕ hp ⊕ http ⊕ hudson ⊕ images ⊕ integration ⊕ ios ⊕ iphone ⊕ jars ⊕ java ⊕ jenkins ⊕ jgc ⊕ kubernetes ⊕ languages ⊕ libraries ⊕ linux ⊕ lmax ⊕ make ⊕ makefiles ⊕ makisu ⊕ maven ⊕ maven-central ⊕ meson ⊕ mesos ⊕ metadata ⊕ microservices ⊕ migrations ⊕ mitm ⊕ monorepo ⊕ netflix ⊕ networking ⊕ nix ⊕ nix-docker ⊕ nixos ⊕ nixpkgs ⊕ open-source ⊕ ops ⊕ opsmop ⊕ oracle ⊕ oss ⊕ packages ⊕ packaging ⊕ pants ⊕ pillar ⊕ pipelines ⊕ process ⊕ projects ⊕ provisioning ⊕ proxies ⊕ python ⊕ qcon ⊕ rake ⊕ redo ⊕ refaster ⊕ release ⊕ repos ⊕ rocker ⊕ ruby ⊕ sbt ⊕ scala ⊕ scalability ⊕ scm ⊕ security ⊕ server ⊕ sfscala ⊕ slidfes ⊕ source-code ⊕ spot-instances ⊕ startups ⊕ stu-hood ⊕ sun ⊕ swarm ⊕ sysadmin ⊕ talks ⊕ tdd ⊕ test ⊕ testing ⊕ tips ⊕ towatch ⊕ try-server ⊕ twitter ⊕ unix ⊕ via:eric ⊕ via:fanf ⊕ via:jbaruch ⊕ videos ⊕ whinge ⊕Copy this bookmark: