Membase, surprise answer. In general it sounds like they had a pretty crazy time -- rebuilding the plane in flight even more than usual. "This had us on our toes and working 24 hours a day. I think at one point we were up for around 60-plus hours straight, never leaving the computer. We had to scale out web servers using DNS load balancing, we had to get multiple HAProxies, break tables off MySQL to their own databases, transparently shard tables, and more. This was all being done on demand, live, and usually in the middle of the night. We were very lucky that most of our layers were scalable with little or no major modifications needed. Helping us along the way was our very detailed custom server monitoring tools which allowed us to keep a very close eye on load, memory, and even provided real time usage stats on the game which helped with capacity planning. We eventually ended up with easy to launch "clusters" of our app that included NGINX, HAProxy, and Goliath servers all of which independent of everything else and when launched, increased our capacity by a constant. At this point our drawings per second were in the thousands, and traffic that looked huge a week ago was just a small bump on the current graphs."

MySQL as a storage backend -- basically an InnoDB store

HBase and Haystack; all data LZO-compressed; very interesting approach to testing -- they 'shadow the real production workload into the test cluster to test before going into production'. This catches a 'high percentage' of issues before production. nice

'The past decade has seen a revolution in data processing. MapReduce, Hadoop, and related technologies have made it possible to store and process data at scales previously unthinkable. Unfortunately, these data processing technologies are not realtime systems, nor are they meant to be. There's no hack that will turn Hadoop into a realtime system; realtime data processing has a fundamentally different set of requirements than batch processing.

However, realtime data processing at massive scale is becoming more and more of a requirement for businesses. The lack of a "Hadoop of realtime" has become the biggest hole in the data processing ecosystem. Storm fills that hole.'

via Jeff Barr's announcement of the Elasticache launch. from 2008, but a better taxonomy than I've seen elsewhere

great slides, via HN. recommends a canonical Hadoop long-term store and a quick, realtime, separate datastore for "not yet processed by Hadoop" data

Scribe logs events, "ptail" (parallel tail presumably) tails logs from Scribe stores, Puma batch-aggregates, writes to HBase.  Java and Thrift on the backend, PHP in front

'platform for event-driven, scalable, and fault-tolerant architectures on the JVM' .. Actor-based, 'let-it-crash', Apache-licensed, Java and Scala APIs, remote Actors, transactional memory -- looks quite nice

"benchmarks disprove common wisdom" shocker

classic presentation from Paul Tyma of Mailinator regarding the java.nio (event-driven, non-threaded) vs java.io (threaded) model of server concurrency, backing up the scalability of threads on modern JVMs

PDFs, docs, links

from Twitter. looks interesting

great post on the hazards of programming in an async framework, and how damn hard it is. good comments thread too (via jzawodny)

good scaling advice from Linden Labs' Ian Wilkes (who doesn't seem to have a blog, sadly)

looks great

'You have to balance
Python's strengths with its weaknesses: your engineers may be more
productive using Python, but if they have to work around more
platform-level performance/scaling limitations as volume increases, do
you come out ahead? etc.'

lots of gory details on FB's innards via Dare Obasanjo

more on the new async HTTP server from FriendFeed/Facebook, in Python. looks lovely

'an open source version of the scalable, non-blocking web server and tools that power FriendFeed. The FriendFeed application is written using a web framework that looks a bit like web.py or Google's webapp, but with additional tools and optimizations to take advantage of the underlying non-blocking (epoll) infrastructure.'