jm + dependencies   16

Developer Experience Lessons Operating a Serverless-like Platform at Netflix
Very interesting writeup on how Netflix are finding operating a serverless scripting system; they offer scriptability in their backend and it's used heavily by devs to provide features. Lots of having to reinvent the wheel on packaging, deployment, versioning, and test/staging infrastructure
serverless  dependencies  packaging  deployment  versioning  devex  netflix  developer-experience  dev  testing  staging  scripting 
july 2017 by jm
Towards true continuous integration – Netflix TechBlog – Medium
Netflix discuss how they handle the eternal dependency-management problem which arises with lots of microservices:
Using the monorepo as our requirements specification, we began exploring alternative approaches to achieving the same benefits. What are the core problems that a monorepo approach strives to solve? Can we develop a solution that works within the confines of a traditional binary integration world, where code is shared? Our approach, while still experimental, can be distilled into three key features:

Publisher feedback — provide the owner of shared code fast feedback as to which of their consumers they just broke, both direct and transitive. Also, allow teams to block releases based on downstream breakages. Currently, our engineering culture puts sole responsibility on consumers to resolve these issues. By giving library owners feedback on the impact they have to the rest of Netflix, we expect them to take on additional responsibility.

Managed source — provide consumers with a means to safely increment library versions automatically as new versions are released. Since we are already testing each new library release against all downstreams, why not bump consumer versions and accelerate version adoption, safely.

Distributed refactoring — provide owners of shared code a means to quickly find and globally refactor consumers of their API. We have started by issuing pull requests en masse to all Git repositories containing a consumer of a particular Java API. We’ve run some early experiments and expect to invest more in this area going forward.


What I find interesting is that Amazon dealt effectively with the first two many years ago, in the form of their "Brazil" build system, and Google do the latter (with Refaster?). It would be amazing to see such a system released into an open source form, but maybe it's just too heavyweight for anyone other than a giant software company on the scale of a Google, Netflix or Amazon.
brazil  amazon  build  microservices  dependencies  coding  monorepo  netflix  google  refaster 
may 2017 by jm
Camille Fournier's excellent rant on microservices
I haven’t even gotten into the fact that your microservices are an inter-dependent environment, as much as you may wish otherwise, and one service acting up can cause operational problems for the whole team. Maybe if you have Netflix-scale operational hardening that’s not a problem. Do you? Really? Is that the best place to spend your focus and money right now, all so teams can throw shit against the wall to see if it sticks?
Don’t sell people fantasies. This is not the reality for a mid-sized tech team working in microservices. There are enough valuable components to building out such a system without the fantastical claims of self-organizing teams who build cool hack projects in 2 week sprints that change the business. Microservices don’t make organizational problems disappear due to self-organization. They allow for some additional degrees of team and process independence and force very explicit decoupling, in exchange, there is overall system complexity and overall system coordination overhead. I personally think that’s enough value, especially when you are coming from a monolith that is failing to scale, but this model is not a panacea.
microservices  rants  camille-fournier  architecture  decoupling  dependencies 
july 2016 by jm
JitPack
Publish JVM and Android libraries direct from github -- it'll build and package a lib on the fly, caching them via CDN
build  github  java  maven  gradle  dependencies  packaging  libraries 
april 2016 by jm
Javascript libraries and tools should bundle their code
If you have a million npm dependencies, distribute them in the dist package; aka. omnibus packages for JS
packaging  omnibus  npm  webpack  rollup  dependencies  coding  javascript 
march 2016 by jm
How Facebook avoids failures
Great paper from Ben Maurer of Facebook in ACM Queue.
A "move-fast" mentality does not have to be at odds with reliability. To make these philosophies compatible, Facebook's infrastructure provides safety valves.


This is full of interesting techniques.

* Rapidly deployed configuration changes: Make everybody use a common configuration system; Statically validate configuration changes; Run a canary; Hold on to good configurations; Make it easy to revert.

* Hard dependencies on core services: Cache data from core services. Provide hardened APIs. Run fire drills.

* Increased latency and resource exhaustion: Controlled Delay (based on the anti-bufferbloat CoDel algorithm -- this is really cool); Adaptive LIFO (last-in, first-out) for queue busting; Concurrency Control (essentially a form of circuit breaker).

* Tools that Help Diagnose Failures: High-Density Dashboards with Cubism (horizon charts); What just changed?

* Learning from Failure: the DERP (!) methodology,
ben-maurer  facebook  reliability  algorithms  codel  circuit-breakers  derp  failure  ops  cubism  horizon-charts  charts  dependencies  soa  microservices  uptime  deployment  configuration  change-management 
november 2015 by jm
Preventing Dependency Chain Attacks in Maven
using a whitelist of allowed dependency JARs and their SHAs
security  whitelisting  dependencies  coding  jar  maven  java  jvm 
august 2015 by jm
Advantages of Monolithic Version Control
another Dan Luu post -- good summary of the monorepo's upside
monorepo  git  mercurial  versioning  source-control  coding  dependencies 
august 2015 by jm
On Ruby
The horrors of monkey-patching:
I call out the Honeybadger gem specifically because was the most recent time I'd been bit by a seemingly good thing promoted in the community: monkey patching third party code. Now I don't fault Honeybadger for making their product this way. It provides their customers with direct business value: "just require 'honeybadger' and you're done!" I don't agree with this sort of practice. [....]

I distrust everything [in Ruby] but a small set of libraries I've personally vetted or are authored by people I respect. Why is this important? Without a certain level of scrutiny you will introduce odd and hard to reproduce bugs. This is especially important because Ruby offers you absolutely zero guarantee whatever the state your program is when a given method is dispatched. Constants are not constants. Methods can be redefined at run time. Someone could have written a time sensitive monkey patch to randomly undefined methods from anything in ObjectSpace because they can. This example is so horribly bad that no one should every do, but the programming language allows this. Much worse, this code be arbitrarily inject by some transitive dependency (do you even know what yours are?).
ruby  monkey-patching  coding  reliability  bugs  dependencies  libraries  honeybadger  sinatra 
april 2015 by jm
How to take over the computer of any JVM developer
To prove how easy [MITM attacking Mavencentral JARs] is to do, I wrote dilettante, a man-in-the-middle proxy that intercepts JARs from maven central and injects malicious code into them. Proxying HTTP traffic through dilettante will backdoor any JARs downloaded from maven central. The backdoored version will retain their functionality, but display a nice message to the user when they use the library.
jars  dependencies  java  build  clojure  security  mitm  http  proxies  backdoors  scala  maven  gradle 
july 2014 by jm
Luigi
A really excellent-looking workflow/orchestration engine for Hadoop, Pig, Hive, Redshift and other ETL jobs, featuring inter-job dependencies, cron-like scheduling, and failure handling. Open source, from Spotify
workflow  orchestration  scheduling  cron  spotify  open-source  luigi  redshift  pig  hive  hadoop  emr  jobs  make  dependencies 
july 2014 by jm
Database Migrations Done Right
The rule is simple. You should never tie database migrations to application deploys or vice versa. By minimising dependencies you enable faster, easier and cleaner deployments.


A solid description of why this is a good idea, from an ex-Guardian dev.
migrations  database  sql  mysql  postgres  deployment  ops  dependencies  loose-coupling 
may 2014 by jm
Dan Kaminsky on Heartbleed
When I said that we expected better of OpenSSL, it’s not merely that there’s some sense that security-driven code should be of higher quality.  (OpenSSL is legendary for being considered a mess, internally.)  It’s that the number of systems that depend on it, and then expose that dependency to the outside world, are considerable.  This is security’s largest contributed dependency, but it’s not necessarily the software ecosystem’s largest dependency.  Many, maybe even more systems depend on web servers like Apache, nginx, and IIS.  We fear vulnerabilities significantly more in libz than libbz2 than libxz, because more servers will decompress untrusted gzip over bzip2 over xz.  Vulnerabilities are not always in obvious places – people underestimate just how exposed things like libxml and libcurl and libjpeg are.  And as HD Moore showed me some time ago, the embedded space is its own universe of pain, with 90’s bugs covering entire countries.

If we accept that a software dependency becomes Critical Infrastructure at some level of economic dependency, the game becomes identifying those dependencies, and delivering direct technical and even financial support.  What are the one million most important lines of code that are reachable by attackers, and least covered by defenders?  (The browsers, for example, are very reachable by attackers but actually defended pretty zealously – FFMPEG public is not FFMPEG in Chrome.)

Note that not all code, even in the same project, is equally exposed.    It’s tempting to say it’s a needle in a haystack.  But I promise you this:  Anybody patches Linux/net/ipv4/tcp_input.c (which handles inbound network for Linux), a hundred alerts are fired and many of them are not to individuals anyone would call friendly.  One guy, one night, patched OpenSSL.  Not enough defenders noticed, and it took Neel Mehta to do something.
development  openssl  heartbleed  ssl  security  dan-kaminsky  infrastructure  libraries  open-source  dependencies 
april 2014 by jm

Copy this bookmark:



description:


tags: