jm + microservices   24

Towards true continuous integration – Netflix TechBlog – Medium
Netflix discuss how they handle the eternal dependency-management problem which arises with lots of microservices:
Using the monorepo as our requirements specification, we began exploring alternative approaches to achieving the same benefits. What are the core problems that a monorepo approach strives to solve? Can we develop a solution that works within the confines of a traditional binary integration world, where code is shared? Our approach, while still experimental, can be distilled into three key features:

Publisher feedback — provide the owner of shared code fast feedback as to which of their consumers they just broke, both direct and transitive. Also, allow teams to block releases based on downstream breakages. Currently, our engineering culture puts sole responsibility on consumers to resolve these issues. By giving library owners feedback on the impact they have to the rest of Netflix, we expect them to take on additional responsibility.

Managed source — provide consumers with a means to safely increment library versions automatically as new versions are released. Since we are already testing each new library release against all downstreams, why not bump consumer versions and accelerate version adoption, safely.

Distributed refactoring — provide owners of shared code a means to quickly find and globally refactor consumers of their API. We have started by issuing pull requests en masse to all Git repositories containing a consumer of a particular Java API. We’ve run some early experiments and expect to invest more in this area going forward.

What I find interesting is that Amazon dealt effectively with the first two many years ago, in the form of their "Brazil" build system, and Google do the latter (with Refaster?). It would be amazing to see such a system released into an open source form, but maybe it's just too heavyweight for anyone other than a giant software company on the scale of a Google, Netflix or Amazon.
brazil  amazon  build  microservices  dependencies  coding  monorepo  netflix  google  refaster 
22 days ago by jm
on Martin Fowler
mcfunley: 'I think at least 50% of my career has been either contributing to or unwinding one [Martin] Fowler-inspired disaster or another.'

See also: continuous deployment, polyglot programming, microservices

Relevant meme:
funny  quotes  architecture  architecture-astronauts  martin-fowler  cargo-cults  coding  design-patterns  enterprise  continuous-deployment  cd  polyglot-programming  microservices  experts 
25 days ago by jm
Service discovery at Stripe
Writeup of their Consul-based service discovery system, a bit similar to smartstack. Good description of the production problems that they saw with Consul too, and also they figured out that strong consistency isn't actually what you want in a service discovery system ;)

HN comments are good too:
consul  api  microservices  service-discovery  dns  load-balancing  l7  tcp  distcomp  smartstack  stripe  cap-theorem  scalability 
november 2016 by jm
Camille Fournier's excellent rant on microservices
I haven’t even gotten into the fact that your microservices are an inter-dependent environment, as much as you may wish otherwise, and one service acting up can cause operational problems for the whole team. Maybe if you have Netflix-scale operational hardening that’s not a problem. Do you? Really? Is that the best place to spend your focus and money right now, all so teams can throw shit against the wall to see if it sticks?
Don’t sell people fantasies. This is not the reality for a mid-sized tech team working in microservices. There are enough valuable components to building out such a system without the fantastical claims of self-organizing teams who build cool hack projects in 2 week sprints that change the business. Microservices don’t make organizational problems disappear due to self-organization. They allow for some additional degrees of team and process independence and force very explicit decoupling, in exchange, there is overall system complexity and overall system coordination overhead. I personally think that’s enough value, especially when you are coming from a monolith that is failing to scale, but this model is not a panacea.
microservices  rants  camille-fournier  architecture  decoupling  dependencies 
july 2016 by jm
3 Reasons AWS Lambda Is Not Ready for Prime Time
This totally matches my own preconceptions ;)
When we at Datawire tried to actually use Lambda for a real-world HTTP-based microservice [...], we found some uncool things that make Lambda not yet ready for the world we live in:

Lambda is a building block, not a tool;
Lambda is not well documented;
Lambda is terrible at error handling

Lung skips these uncool things, which makes sense because they’d make the tutorial collapse under its own weight, but you can’t skip them if you want to work in the real world. (Note that if you’re using Lambda for event handling within the AWS world, your life will be easier. But the really interesting case in the microservice world is Lambda and HTTP.)
aws  lambda  microservices  datawire  http  api-gateway  apis  https  python  ops 
may 2016 by jm
Should create a separate Hystrix Thread pool for each remote call?
Excellent advice on capacity planning and queueing theory, in the context of Hystrix. Should I use a single thread pool for all dependency callouts, or independent thread pools for each one?
threadpools  pooling  hystrix  capacity  queue-theory  queueing  queues  failure  resilience  soa  microservices 
may 2016 by jm
A microservice saviour appears!
In order to prevent such a terrible tragedy from occurring ever again during
our lifetimes, `` has been created to provide all the functionality
of `left-pad` AND the overhead of a TLS handshake and an HTTP request.
Less code is better code, leave the heavy lifting to ``, The String
humor  javascript  jokes  npm  packages  left-pad  strings  microservices  http 
march 2016 by jm
Dropwizard for Go, basically:
a distributed programming toolkit for building microservices in large organizations. We solve common problems in distributed systems, so you can focus on your business logic.
microservices  go  golang  http  libraries  open-source  rpc  circuit-breakers 
january 2016 by jm
About Microservices, Containers and their Underestimated Impact on Network Performance
shock horror, Docker-SDN layers have terrible performance. Still pretty lousy perf impacts from basic Docker containerization, presumably without "--net=host" (which is apparently vital)
docker  performance  network  containers  sdn  ops  networking  microservices 
january 2016 by jm
How Facebook avoids failures
Great paper from Ben Maurer of Facebook in ACM Queue.
A "move-fast" mentality does not have to be at odds with reliability. To make these philosophies compatible, Facebook's infrastructure provides safety valves.

This is full of interesting techniques.

* Rapidly deployed configuration changes: Make everybody use a common configuration system; Statically validate configuration changes; Run a canary; Hold on to good configurations; Make it easy to revert.

* Hard dependencies on core services: Cache data from core services. Provide hardened APIs. Run fire drills.

* Increased latency and resource exhaustion: Controlled Delay (based on the anti-bufferbloat CoDel algorithm -- this is really cool); Adaptive LIFO (last-in, first-out) for queue busting; Concurrency Control (essentially a form of circuit breaker).

* Tools that Help Diagnose Failures: High-Density Dashboards with Cubism (horizon charts); What just changed?

* Learning from Failure: the DERP (!) methodology,
ben-maurer  facebook  reliability  algorithms  codel  circuit-breakers  derp  failure  ops  cubism  horizon-charts  charts  dependencies  soa  microservices  uptime  deployment  configuration  change-management 
november 2015 by jm
Baker Street
client-side 'service discovery and routing system for microservices' -- another Smartstack, then
python  router  smartstack  baker-street  microservices  service-discovery  routing  load-balancing  http 
october 2015 by jm
A new HTTP client library for Android and Java, with a lot of nice features:
HTTP/2 and SPDY support allows all requests to the same host to share a socket.

Connection pooling reduces request latency (if SPDY isn’t available).

Transparent GZIP shrinks download sizes.

Response caching avoids the network completely for repeat requests.

OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems. If your service has multiple IP addresses OkHttp will attempt alternate addresses if the first connect fails. This is necessary for IPv4+IPv6 and for services hosted in redundant data centers. OkHttp initiates new connections with modern TLS features (SNI, ALPN), and falls back to TLS 1.0 if the handshake fails.

Using OkHttp is easy. Its 2.0 API is designed with fluent builders and immutability. It supports both synchronous blocking calls and async calls with callbacks.
android  http  java  libraries  okhttp  http2  spdy  microservices  jdk 
july 2015 by jm
Hystrix-style Circuit Breakers and Bulkheads for Ruby/Rails, from Shopify
circuit-breaker  bulkhead  patterns  architecture  microservices  shopify  rails  ruby  networking  reliability  fallback  fail-fast 
june 2015 by jm
Patterns for building a resilient and scalable microservices platform on AWS
Some good details from Boyan Dimitrov at Hailo, on their orchestration, deployment, provisioning infra they've built
deployment  ops  devops  hailo  microservices  platform  patterns  slides 
may 2015 by jm
'Microservice AntiPatterns'
presentation from last week's Craft Conference in Budapest; Tammer Saleh of Pivotal with a few antipatterns observed in dealing with microservices.
microservices  soa  architecture  design  coding  software  presentations  slides  tammer-saleh  pivotal  craft 
april 2015 by jm
Microservices and elastic resource pools with Amazon EC2 Container Service
interesting approach to working around ECS' shortcomings -- bit specific to Hailo's microservices arch and IPC mechanism though.

aside: I like their version numbering scheme: ISO-8601, YYYYMMDDHHMMSS. keep it simple!
versioning  microservices  hailo  aws  ec2  ecs  docker  containers  scheduling  allocation  deployment  provisioning  qos 
april 2015 by jm
A Journey into Microservices | Hailo Tech Blog
Excellent three-parter from Hailo, describing their RabbitMQ+Go-based microservices architecture. Very impressive!
hailo  go  microservices  rabbitmq  amqp  architecture  blogs 
march 2015 by jm
Scaling Micro-Services in Go - HighLoad++ 2014 // Speaker Deck
good talk from Hailo's Matt Heath, doing nice stuff with Go and a well-supported microservices architecture
microservices  presentation  go  architecture  hailo  presentations 
november 2014 by jm
The "sidecar" pattern
Ha, great name. We use this (in the form of Smartstack).
For what it is worth, we faced a similar challenge in earlier services (mostly due to existing C/C++ applications) and we created what was called a "sidecar".  By sidecar, what I mean is a second process on each node/instance that did Cloud Service Fabric operations on behalf of the main process (the side-managed process).  Unfortunately those sidecars all went off and created one-offs for their particular service.  In this post, I'll describe a more general sidecar that doesn't force users to have these one-offs.

Sidenote:  For those not familiar with sidecars, think of the motorcycle sidecar below.  Snoopy would be the main process with Woodstock being the sidecar process.  The main work on the instance would be the motorcycle (say serving your users' REST requests).  The operational control is the sidecar (say serving health checks and management plane requests of the operational platform).
netflix  sidecars  architecture  patterns  smartstack  netflixoss  microservices  soa 
august 2014 by jm
Richard Clayton - Failing at Microservices
Solid warts-and-all confessional blogpost about a team failing to implement a microservices architecture. I'd put most of the blame on insufficient infrastructure to support them (at a code level), inter-personal team problems, and inexperience with large-scale complex multi-service production deployment and the work it was going to require
microservices  devops  collaboration  architecture  fail  team  deployment  soa 
august 2014 by jm
Microservices - Not a free lunch! - High Scalability
Some good reasons not to adopt microservices blindly. Testability and distributed-systems complexity are my biggest fears
microservices  soa  devops  architecture  testing  distcomp 
august 2014 by jm
Microservices and nanoservices
A great reaction to Martin Fowler's "microservices" coinage, from Arnon Rotem-Gal-Oz:

'I guess it is easier to use a new name (Microservices) rather than say that this is what SOA actually meant'; 'these are the very principles of SOA before vendors pushed the [ESB] in the middle.'

Others have also chosen to define microservices slightly differently, as a service written in 10-100 LOC. Arnon's reaction:

“Nanoservice is an antipattern where a service is too fine-grained. A nanoservice is a service whose overhead (communications, maintenance, and so on) outweighs its utility.”

Having dealt with maintaining an over-fine-grained SOA stack in Amazon, I can only agree with this definition; it's easy to make things too fine-grained and create a raft of distributed-computing bugs and deployment/management complexity where there is no need to do so.
architecture  antipatterns  nanoservices  microservices  soa  services  design  esb 
march 2014 by jm
The Microservice Declaration of Independence
"Microservices" seems to be yet another term for SOA; small, decoupled, independently-deployed services, with well-defined public HTTP APIs. Pretty much all the services I've worked on over the past few years have been built in this style. Still, let's keep an eye on this concept anyway.

Another definition seems to be a more FP-style one: -- where the "microservice" does one narrowly-defined thing, and that alone.
microservices  soa  architecture  handwaving  http  services  web  deployment 
march 2014 by jm

related tags

algorithms  allocation  amazon  amqp  android  antipatterns  api  api-gateway  apis  architecture  architecture-astronauts  aws  baker-street  ben-maurer  blogs  brazil  build  bulkhead  camille-fournier  cap-theorem  capacity  cargo-cults  cd  change-management  charts  circuit-breaker  circuit-breakers  codel  coding  collaboration  configuration  consul  containers  continuous-deployment  craft  cubism  datawire  decoupling  dependencies  deployment  derp  design  design-patterns  devops  distcomp  distributed-systems  dns  docker  ec2  ecs  enterprise  esb  experts  facebook  fail  fail-fast  failure  fallback  funny  go  golang  google  hailo  handwaving  horizon-charts  http  http2  https  humor  hystrix  java  javascript  jdk  jokes  l7  lambda  left-pad  libraries  load-balancing  martin-fowler  microservices  monorepo  nanoservices  netflix  netflixoss  network  networking  npm  okhttp  open-source  ops  packages  patterns  performance  pivotal  platform  polyglot-programming  pooling  presentation  presentations  provisioning  python  qos  queue-theory  queueing  queues  quotes  rabbitmq  rails  rants  refaster  reliability  resilience  router  routing  rpc  ruby  scalability  scheduling  sdn  service-discovery  services  shopify  sidecars  slides  smartstack  soa  software  spdy  strings  stripe  tammer-saleh  tcp  team  testing  threadpools  uptime  versioning  web 

Copy this bookmark: