This is what work looks like now. It’s been this way for so long that most American workers don’t realize that for most of the 20th century, the broad consensus among American business leaders was that working people more than 40 hours a week was stupid, wasteful, dangerous and expensive — and the most telling sign of dangerously incompetent management to boot.

The most essential thing to know about the 40-hour work-week is that, while it was the unions that pushed it, business leaders ultimately went along with it because their own data convinced them this was a solid, hard-nosed business decision.

In fact, research shows that knowledge workers actually have fewer good hours in a day than manual laborers do — on average, about six hours, as opposed to eight.

And finally: these death marches take a longer-term productivity toll as well. Once the crisis has passed and that 60-hour-a-week team gets to go back to its regular 40, it can take several more weeks before the burnout begins to lift enough for them to resume their typical productivity level. So, for a while, you’ll get significantly less than a full 40 out of them.

We set up a small group of people to take every silly idea, every cool trick, everything interesting that existed in any kind of working state somewhere in the game and put them into a single prototype level. ... They all worked together on this one small level for a month while the rest of us basically did nothing. When they were done, we all played it. It was great. It was Die Hard meets Evil Dead. It was the vision. It was going to be our game.

We looked at hundreds of resumes and interviewed a lot of promising applicants, but no one we looked at had enough of the qualities we wanted for us to seriously consider them the overall godlike “game designer” that we were told we needed. In the end, we came to the conclusion that this ideal person didn’t actually exist. Instead, we would create our own ideal by combining the strengths of a cross section of the company, putting them together in a group we called the “Cabal.”

The Cabal consisted only of people that had actual shipping components in the game; there were no dedicated designers. Every member of the Cabal was someone with the responsibility of actually doing the work that their design specified, or at least had the ability to do it if need be. ... Internally, once the success of the Cabal process was obvious, mini-Cabals were formed to come up with answers to a variety of design problems. These mini-Cabals would typically include people most effected by the decision, as well as try to include people completely outside the problem being addressed in order to keep a fresh perspective on things.

Write down everything. Brainstorming is fine during the meetings, but unless it’s all written down, your best ideas will be forgotten within days. The goal is to end up with a document that captures as much as is reasonable about your game, and more importantly answers questions about what people need to work on.

Brandon Ballinger, working on a machine learning startup

I worked on speech recognition and ads at Google, both of which have machine learning at their core. Here are some important lessons I learned about how to operationalize a machine learning system "in the wild":

Make your success metric user happiness. Traditional accuracy measures like precision, square error, ROC, etc. don't capture what you really care about--how users react to your model. For example, if you're running an ad system, your metrics should be revenue per pageview and click through rate. It's completely possible to have a model with a reduced error rate which lowers revenue, due to Simpson's paradox and a host of other reasons.

A drop in core metrics should generate a page. You should treat a model whose metrics nosedive the same as a sever that goes down or a database with bad data. Whoever is oncall should try to diagnose the problem quickly, pulling in others if necessary to fix the problem.

A/B test every model launch. You should always be able to run two models in parallel, showing different models to different users and comparing user behavior. That's the only way to really know you're doing the right thing.

Beware of ugly duckling effects. Machine learning systems tend to become more accurate when they can learn from their own mistakes. That means new models are at an inherent disadvantage: your historical training data contains mistakes from previous models, but not the new one you just trained. That means your brand spankin' new algorithm may look like an ugly duckling initially, but turn into a beautiful swan when you allow it to decide the results that users actually see. You can partially counteract ugly duckling effects by running the new model on a small percentage of traffic, and then ramping that percentage up over time.

Make algorithms robust to noisy features. Techniques like L1 regularization let your machine learning algorithm prune features that contribute relatively little to prediction accuracy. This lets you nicely separate your team: some people focus on the algorithm, and others write features that feed into it. The people inventing the features can just "throw them into the pot" and let the algorithm figure out what's good and bad. (Similarly, you should try to pick an algorithm with reasonable convergence guarantees.)

You can partially de-couple the infrastructure and algorithm teams. Fundamentally, most machine learning algorithms accumulate a distributed hash table of statistics, and then combine those statistics into a score. Developing the distributed hash table is a different task than developing the algorithms that do the accumulation, and can be done by separate sub-teams. However, it's still important that those people work closely together. For example, some algorithms will oscillate if there's a delay between writes and those writes becoming available, which is done in many "eventual consistency" systems like Dynamo. So this is only a partial de-coupling and you still need people on the team to be the "bridge" between infrastructure and algorithms.

Choose online vs. batch carefully. An online system learns in real time, reacting to new user behavior minutes after it happens. But it comes at a huge cost: an online system takes 2-3x as long to develop and maintain, and it's much more sensitive to transient changes. For example, if you get a burst of spam, that noise will immediately be incorporated into your model and start degrading the user experience. Likewise if a machine runs out of memory, the network becomes disconnected, a particular machine is slow (introducing feature "skew"), one of the inputs into the model starts generating garbage, etc.

Version everything. A trained model depends on having stable identifiers; if you change an identifier, the model instantly goes stale. For example, let's say you include the user's language ("en-us") as a feature. Then somebody submits a one-line change to use underscores rather than hyphens ("en_us"). That effectively forces the model to instantly "forget" everything it learned about "en-us", making every language look the same. So version any changes to code/data that generates identifiers in your model.

You need to crunch data at 100x-1000x realtime. If your model is trained on a year of historical data, and your learning process is 10x realtime, then it will take over a month to test out a new feature. Write everything as a Map-Reduce or Storm topology so that you can scale when your data gets big enough.