"18 million visitors in March, a 50% increase from the previous month, with very little IT infrastructure"

"Each vertical team knows their component well but nobody understands the interactions of all the components. The two solutions are 1) well-defined and well-documented interfaces between components, be they hardware or software, and 2) and very experienced, highly-skilled engineer(s) on the team focusing on understanding inter-component interaction and overall system operation, especially in fault modes."

Michael Nygard outlines 8 rules for dealing with complex systems: Embrace Plurality, Contextualize Downstream, Beware Grandiosity, Decentralize, Isolate Failure Domains, Data Outlives Applications, Applications Outlive Integrations, Increase Discoverability.

Derived from the classic textbook Structure and Interpretation of Computer Programs ("SICP") -- s/Scheme/Python

Notable comment: "The logical conclusion is that if a system cannot be 100% automated, then it may be best to deliberately automate less than is technically possible. If human operators are required to routinely solve less than critical problems, they will be better equipped to solve the very rare critical problems."

Highly recommended talk. Keynote Address at the 25th Large Installation System Administration Conference (LISA '11), by Ben Rockwood, Joyent.

Highly Scalable Blog

The Netflix Tech Blog

"There are rules how Scala code is compiled to JVM-bytecode. Because of potential name clashes the generated code is not always intuitive to understand but if the rules are known it is possible to get access to the compiled Scala code within Java."

"Difference between PUT and POST in REST. Essentially, PUTs are idempotent whereas POSTs aren’t."

Scala library for working with Apache Zookeeper. It uses Scalaz (6.x) extensively, in particular Scalaz Promises to eliminate structuring your code as a series of callbacks (i.e.: inversion of control). The result is a much cleaner way to work with ZooKeeper.

Henceforth, a single compute-intensive application will need to harness different kinds of cores, in immense numbers, to get its job done.

The free lunch is over. Now welcome to the hardware jungle

ZooKeeper client wrapper and rich ZooKeeper framework - from Netflix

If you are into unix of any kind (or open source or cloud actually), you should watch this one. Gets really good 30 minutes in.

@patio11

"Goal of the project is to provide more standard and easy way to make dependency injection in Scala projects consuming power of the Scala language. With Scaldi you can define your application modules in pure Scala without any annotations or XML."

"In this paper, we explored the general security risks associated with virtual server images from the public catalogs of cloud service providers. We investigated in detail the security problems of public images that are available on the Amazon EC2 service. Our findings demonstrate that both users and providers of public AMIs may be vulnerable to security risks such as unauthorized access, malware infections, and the loss of sensitive information. The Amazon Web Services Security Team has acknowledged our findings, and has already taken steps to address the security risks we have identified."

Here are some of the key principles that informed our thinking as we set out to make the API more resilient.

1. A failure in a service dependency should not break the user experience for members
2. The API should automatically take corrective action when one of its service dependencies fails
3. The API should be able to show us what’s happening right now, in addition to what was happening 15-30 minutes ago, yesterday, last week, etc.

Regarding Scala's binary compatibility going forward in 2.9.x +

There is a small number of major design decisions that make Kafka different from most other messaging systems:

1. Kafka is designed for persistent messages as the common case

2. Throughput rather than features are the primary design constraint

3. State about what has been consumed is maintained as part of the consumer not the server

4. Kafka is explicitly distributed. It is assumed that producers, brokers, and consumers are all spread over multiple machines.

several of Google's core libraries that we rely on in our Java-based projects: collections, caching, primitives support, concurrency libraries, common annotations, string processing, I/O, and so forth.

(Tim Fox is primary author of https://github.com/purplefox/vert.x )

"leverage the global DNS system to efficiently and accurately route your users to the right global location"

The next generation polyglot asynchronous application framework

(At heart of one of our new designs; using ZooKeeper to elect the writer)

High Scalability writeup from 2008

Adrian Cockcroft (Netflix)

Comprehensive primer on the internal operations of WebKit and Gecko

In this paper, we describe ZooKeeper, a service for coordinating processes of distributed applications. Since ZooKeeper is part of critical infrastructure, ZooKeeper aims to provide a simple and high performance kernel for building more complex coordination primitives at the client. It incorporates elements from group messaging, shared registers, and distributed lock services in a replicated, centralized service. The interface exposed by ZooKeeper has the wait-free aspects of shared registers with an event-driven mechanism similar to cache invalidations of distributed file systems to provide a simple, yet powerful coordination service.

The ZooKeeper interface enables a high-performance service implementation. In addition to the wait-free property, ZooKeeper provides a per client guarantee of FIFO execution of requests and linearizability for all requests that change the ZooKeeper state. These design decisions enable the implementation of a high performance processing pipeline with read requests being satisfied by local servers. We show for the target workloads, 2:1 to 100:1 read to write ratio, that ZooKeeper can handle tens to hundreds of thousands of transactions per second. This performance allows ZooKeeper to be used extensively by client applications.

Excellent article introducing main differences between SQLAlchemy and the Django ORM

"The goal of this post (and future posts on this topic) is to help the reader develop a basic toolkit they could use to reason about distributed systems."

"The use of software switching in the hypervisor has some good points but, in my view they are heavily outweighed by the bad."

Discusses how the Typesafe Stack, and Akka in particular, approaches shared memory in concurrent applications.

Stuart Charlton - @svrc

William Vambenepe - @vambenepe

Nice list of papers and topics