jm + containers   50

Kubernetes Best Practices // Speaker Deck
A lot of these are general Docker/containerisation best practices, too.

(via Devops Weekly)
k8s  kubernetes  devops  ops  containers  docker  best-practices  tips  packaging 
july 2017 by jm
Instead of containerization, give me strong config & deployment primitives
Reasonable list of things Docker does badly at the moment, and a call to fix them. I still think Docker/rkt are a solid approach, if not 100% there yet though
docker  containers  complaining  whinge  networking  swarm  deployment  architecture  build  packaging 
april 2017 by jm
Capturing all the flags in BSidesSF CTF by pwning Kubernetes/Google Cloud
good exploration of the issues with running a CTF challenge (or any other secure infrastructure!) atop Kubernetes and a cloud platform like GCE
gce  google-cloud  kubernetes  security  docker  containers  gke  ctf  hacking  exploits 
april 2017 by jm
ctop
Top for containers (ie Docker)
docker  containers  top  ops  go  monitoring  cpu 
march 2017 by jm
pachyderm
'Containerized Data Analytics':
There are two bold new ideas in Pachyderm:

Containers as the core processing primitive
Version Control for data

These ideas lead directly to a system that's much more powerful, flexible and easy to use.

To process data, you simply create a containerized program which reads and writes to the local filesystem. You can use any tools you want because it's all just going in a container! Pachyderm will take your container and inject data into it. We'll then automatically replicate your container, showing each copy a different chunk of data. With this technique, Pachyderm can scale any code you write to process up to petabytes of data (Example: distributed grep).

Pachyderm also version controls all data using a commit-based distributed filesystem (PFS), similar to what git does with code. Version control for data has far reaching consequences in a distributed filesystem. You get the full history of your data, can track changes and diffs, collaborate with teammates, and if anything goes wrong you can revert the entire cluster with one click!

Version control is also very synergistic with our containerized processing engine. Pachyderm understands how your data changes and thus, as new data is ingested, can run your workload on the diff of the data rather than the whole thing. This means that there's no difference between a batched job and a streaming job, the same code will work for both!
analytics  data  containers  golang  pachyderm  tools  data-science  docker  version-control 
february 2017 by jm
How-to Debug a Running Docker Container from a Separate Container
arguably this shouldn't be required -- building containers without /bin/sh, strace, gdb etc. is just silly
strace  docker  ops  debugging  containers 
february 2017 by jm
Measuring Docker IO overhead - Percona Database Performance Blog
See also https://www.percona.com/blog/2016/02/05/measuring-docker-cpu-network-overhead/ for the CPU/Network equivalent. The good news is that nowadays it's virtually 0 when the correct settings are used
docker  percona  overhead  mysql  deployment  performance  ops  containers 
november 2016 by jm
J1 2015 "Debugging Java Apps in Containers: No Heavy Welding Gear Required"
Some good slides with tips on running java apps in production in Docker
java  docker  ops  containers 
september 2016 by jm
Skyliner
Coda Hale's new gig on how they're using Docker, AWS, etc. I like this: "Use containers. Not too much. Mostly for packaging."
docker  aws  packaging  ops  devops  containers  skyliner 
september 2016 by jm
Some thoughts on operating containers
R.I.Pienaar talks about the conventions he uses when containerising; looks like a decent approach.
ops  containers  docker  ripienaar  packaging 
june 2016 by jm
grammarly/rocker
backward compatible replacement for Dockerfile. Yes, you can take any Dockerfile, rename it to Rockerfile and use rocker build instead of docker build. ... Rocker aims to solve the following use cases, which are painful with plain Docker:

Mount reusable volumes on build stage, so dependency management tools may use cache between builds.
Share ssh keys with build (for pulling private repos, etc.), while not leaving them in the resulting image.
Build and run application in different images, be able to easily pass an artifact from one image to another, ideally have this logic in a single Dockerfile.
Tag/Push images right from Dockerfiles.
Pass variables from shell build command so they can be substituted to a Dockerfile.
And more. These are the most critical issues that were blocking our adoption of Docker at Grammarly.

The most challenging part is caching. While implementing those features seems to be not a big deal, it's not trivial to do that just by utilising Docker’s image cache (the one that docker build does). Actually, it is the main reason why those features are still not in Docker. With Rocker we achieve this by introducing a set of trade-offs. Search this page for "trade-off" to find out more details.
docker  rocker  build  containers  dockerfiles 
may 2016 by jm
fiunchinho/dockerize-me
'Tired of copy/pasting Dockerfiles around? Not sure about best practices for Dockerfiles or Docker entry points? This tool lets you Dockerize your applications using best practices to define your Dockerfile and Docker entry point files.'

The best practices in question are defined here: https://github.com/docker-library/official-images#review-guidelines
docker  dockerfile  images  build  best-practices  alpine  containers 
may 2016 by jm
GitLab Container Registry
GitLab continue to out-innovate Github, which is just wanking around with breaking the UI these days
gitlab  github  git  ci  cd  containers  docker  deployment  coding 
may 2016 by jm
The Challenges of Container Configuration // Speaker Deck
Some good advice on Docker metadata/config from Gareth Rushgrove
docker  metadata  configuration  build  devops  dev  containers  slidfes 
may 2016 by jm
A Decade Of Container Control At Google
The big thing that can be gleaned from the latest paper out of Google on its container controllers is that the shift from bare metal to containers is a profound one – something that may not be obvious to everyone seeking containers as a better way – and we think cheaper way – of doing server virtualization and driving up server utilization higher. Everything becomes application-centric rather than machine-centric, which is the nirvana that IT shops have been searching for. The workload schedulers, cluster managers, and container controllers work together to get the right capacity to the application when it needs it, whether it is a latency-sensitive job or a batch job that has some slack in it, and all that the site recovery engineers and developers care about is how the application is performing and they can easily see that because all of the APIs and metrics coming out of them collect data at the application level, not on a per-machine basis. To do this means adopting containers, period. There is no bare metal at Google, and let that be a lesson to HPC shops or other hyperscalers or cloud builders that think they need to run in bare metal mode.
google  containers  kubernetes  borg  bare-metal  ops 
april 2016 by jm
About Microservices, Containers and their Underestimated Impact on Network Performance
shock horror, Docker-SDN layers have terrible performance. Still pretty lousy perf impacts from basic Docker containerization, presumably without "--net=host" (which is apparently vital)
docker  performance  network  containers  sdn  ops  networking  microservices 
january 2016 by jm
Amazon EC2 Container Registry
hooray, Docker registry here at last
ecs  docker  registry  ops  containers  aws 
december 2015 by jm
Why We Chose Kubernetes Over ECS
3 months ago when we, at nanit.com, came to evaluate which Docker orchestration framework to use, we gave ECS the first priority. We were already familiar with AWS services, and since we already had our whole infrastructure there, it was the default choice. After testing the service for a while we had the feeling it was not mature enough and missing some key features we needed (more on that later), so we went to test another orchestration framework: Kubernetes. We were glad to discover that Kubernetes is far more comprehensive and had almost all the features we required. For us, Kubernetes won ECS on ECS’s home court, which is AWS.
kubernetes  ecs  docker  containers  aws  ec2  ops 
december 2015 by jm
Anatomy of a Modern Production Stack
Interesting post, but I think it falls into a common trap for the xoogler or ex-Amazonian -- assuming that all the BigCo mod cons are required to operate, when some are luxuries than can be skipped for a few years to get some real products built
architecture  ops  stack  docker  containerization  deployment  containers  rkt  coreos  prod  monitoring  xooglers 
september 2015 by jm
Why Docker is Not Yet Succeeding Widely in Production
Spot-on points which Docker needs to address. It's still production-ready, and _should_ be used there, it just has significant rough edges...
docker  containers  devops  deployment  releases  linux  ops 
july 2015 by jm
Google Cloud Platform announces new Container Registry
Yay. Sensible Docker registry pricing at last. Given the high prices, rough edges and slow performance of the other registry offerings, I'm quite happy to see this.
Google Container Registry helps make it easy for you to store your container images in a private and encrypted registry, built on Cloud Platform. Pricing for storing images in Container Registry is simple: you only pay Google Cloud Storage costs. Pushing images is free, and pulling Docker images within a Google Cloud Platform region is free (Cloud Storage egress cost when outside of a region).

Container Registry is now ready for production use:

* Encrypted and Authenticated - Your container images are encrypted at rest, and access is authenticated using Cloud Platform OAuth and transmitted over SSL
* Fast - Container Registry is fast and can handle the demands of your application, because it is built on Cloud Storage and Cloud Networking.
* Simple - If you’re using Docker, just tag your image with a gcr.io tag and push it to the registry to get started.  Manage your images in the Google Developers Console.
* Local - If your cluster runs in Asia or Europe, you can now store your images in ASIA or EU specific repositories using asia.gcr.io and eu.gcr.io tags.
docker  registry  google  gcp  containers  cloud-storage  ops  deployment 
june 2015 by jm
Eric Brewer interview on Kubernetes
What is the relationship between Kubernetes, Borg and Omega (the two internal resource-orchestration systems Google has built)?

I would say, kind of by definition, there’s no shared code but there are shared people.

You can think of Kubernetes — especially some of the elements around pods and labels — as being lessons learned from Borg and Omega that are, frankly, significantly better in Kubernetes. There are things that are going to end up being the same as Borg — like the way we use IP addresses is very similar — but other things, like labels, are actually much better than what we did internally.

I would say that’s a lesson we learned the hard way.
google  architecture  kubernetes  docker  containers  borg  omega  deployment  ops 
may 2015 by jm
Kubernetes compared to Borg
'Here are four Kubernetes features that came from our experiences with Borg.'
google  ops  kubernetes  borg  containers  docker  networking 
april 2015 by jm
Cluster-Based Architectures Using Docker and Amazon EC2 Container Service
In this post, we’re going to take a deeper dive into the architectural concepts underlying cluster computing using container management frameworks such as ECS. We will show how these frameworks effectively abstract the low-level resources such as CPU, memory, and storage, allowing for highly efficient usage of the nodes in a compute cluster. Building on some of the concepts detailed in the earlier posts, we will discover why containers are such a good fit for this type of abstraction, and how the Amazon EC2 Container Service fits into the larger ecosystem of cluster management frameworks.
docker  aws  ecs  ec2  ops  hosting  containers  mesos  clusters 
april 2015 by jm
Microservices and elastic resource pools with Amazon EC2 Container Service
interesting approach to working around ECS' shortcomings -- bit specific to Hailo's microservices arch and IPC mechanism though.

aside: I like their version numbering scheme: ISO-8601, YYYYMMDDHHMMSS. keep it simple!
versioning  microservices  hailo  aws  ec2  ecs  docker  containers  scheduling  allocation  deployment  provisioning  qos 
april 2015 by jm
EC2 Container Service Hands On
Sounds like a good start, but this isn't great:
There is no native integration with Autoscaling or ELBs.
ec2  containers  docker  ecs  ops 
december 2014 by jm
CoreOS is building a container runtime, Rocket
Whoa, trouble at mill in Dockerland!
When Docker was first introduced to us in early 2013, the idea of a “standard container” was striking and immediately attractive: a simple component, a composable unit, that could be used in a variety of systems. The Docker repository included a manifesto of what a standard container should be. This was a rally cry to the industry, and we quickly followed. Brandon Philips, co-founder/CTO of CoreOS, became a top Docker contributor, and now serves on the Docker governance board. CoreOS is one of the most widely used platforms for Docker containers, and ships releases to the community hours after they happen upstream. We thought Docker would become a simple unit that we can all agree on.

Unfortunately, a simple re-usable component is not how things are playing out. Docker now is building tools for launching cloud servers, systems for clustering, and a wide range of functions: building images, running images, uploading, downloading, and eventually even overlay networking, all compiled into one monolithic binary running primarily as root on your server. The standard container manifesto was removed. We should stop talking about Docker containers, and start talking about the Docker Platform. It is not becoming the simple composable building block we had envisioned.
coreos  docker  linux  containers  open-source  politics  rocket 
december 2014 by jm
veggiemonk/awesome-docker
A curated list of Docker resources.
linux  sysadmin  docker  ops  devops  containers  hosting 
november 2014 by jm
"Linux Containers And The Future Cloud" [slides]
by Rami Rosen -- extremely detailed presentation into the state of Linux containers, LXC, Docker, namespaces, cgroups, and checkpoint/restore in userspace (via lusis)
lsx  docker  criu  namespaces  cgroups  linux  via:lusis  ops  containers  rami-rosen  presentations 
october 2014 by jm
AWS Elastic Beanstalk for Docker
This is pretty amazing. nice work, Beanstalk team. not sure how well it integrates with the rest of AWS though
aws  amazon  docker  ec2  beanstalk  ops  containers  linux 
april 2014 by jm
Is Google building a hulking floating data center in SF Bay?
Looks pretty persuasive, especially considering they hold a patent on the design
google  data-centers  bay-area  ships  containers  shipping  sea  wave-power  treasure-island 
october 2013 by jm
Dockerize
A tool to manage inter-container dependencies so that continuous delivery with Jenkins and Docker is feasible. Looks very helpful
docker  provisioning  vms  containers  dockerize  jenkins  continuous-delivery  continuous-integration 
october 2013 by jm
Mesosphere · Docker on Mesos
This is cool. Deploy Docker container images onto a Mesos cluster: key point, in the description of the Redis example: 'there’s no need to install Redis or its supporting libraries on your Mesos hosts.'
mesos  docker  deployment  ops  images  virtualization  containers  linux 
september 2013 by jm
Docker: Git for deployment
Docker is to deployment as Git is to development.

Developers are able to leverage Git's performance and flexibility when building applications. Git encourages experiments and doesn't punish you when things go wrong: start your experiments in a branch, if things fall down, just git rebase or git reset. It's easy to start a branch and fast to push it.

Docker encourages experimentation for operations. Containers start quickly. Building images is a snap. Using another images as a base image is easy. Deploying whole images is fast, and last but not least, it's not painful to rollback.

Fast + flexible = deployments are about to become a lot more enjoyable.
docker  deployment  sysadmin  ops  devops  vms  vagrant  virtualization  containers  linux  git 
august 2013 by jm
Docker
'the Linux container engine'. I totally misunderstood what Docker was -- this is cool.
Heterogeneous payloads: Any combination of binaries, libraries, configuration files, scripts, virtualenvs, jars, gems, tarballs, you name it. No more juggling between domain-specific tools. Docker can deploy and run them all.

Any server: Docker can run on any x64 machine with a modern linux kernel - whether it's a laptop, a bare metal server or a VM. This makes it perfect for multi-cloud deployments.

Isolation: Docker isolates processes from each other and from the underlying host, using lightweight containers.

Repeatability: Because each container is isolated in its own filesystem, they behave the same regardless of where, when, and alongside what they run.
lxc  containers  virtualization  cloud  ops  linux  docker  deployment 
july 2013 by jm
C++ B-Tree
a new C++ template library from Google which implements an in-memory B-Tree container type, suitable for use as a drop-in replacement for std::map, set, multimap and multiset. Lower memory use, and reportedly faster due to better cache-friendliness
c++  google  data-structures  containers  b-trees  stl  map  set  open-source 
february 2013 by jm

related tags

allocation  alpine  amazon  analytics  architecture  aufs  aws  b-trees  bare-metal  bay-area  beanstalk  best-practices  borg  bryan-cantrill  btrfs  bugs  build  c++  cd  cgroups  ci  cloud  cloud-storage  clusters  coding  complaining  configuration  containerization  containers  continuous-delivery  continuous-integration  coreos  cpu  criu  ctf  data  data-centers  data-pipelines  data-science  data-structures  debugging  deployment  dev  devops  docker  dockerfile  dockerfiles  dockerize  dumb-init  ec2  ecs  exploits  filesystems  flavour-of-the-month  gce  gcp  git  github  gitlab  gke  go  golang  google  google-cloud  hacking  hailo  hosting  images  infrastructure  init  java  jenkins  joyent  k8s  kubernetes  linux  lsx  luigi  lxc  map  mesos  metadata  microservices  monitoring  mysql  namespaces  network  networking  omega  open-source  ops  overhead  overlayfs  pachyderm  packaging  percona  performance  politics  presentations  prod  production  provisioning  qos  rami-rosen  reference  registry  releases  reviews  ripienaar  rkt  rocker  rocket  routing  scheduling  sdn  sea  security  set  shipping  ships  shopify  signals  skyliner  slidfes  stack  stl  strace  swarm  sysadmin  tips  tools  top  treasure-island  unikernels  unix  vagrant  version-control  versioning  via:fanf  via:lusis  virtualization  vms  wave-power  whinge  workflow  xooglers  yelp 

Copy this bookmark:



description:


tags: