Netflix' library to implement "edge services" -- ie. a front end to their API, web servers, and streaming servers. Some interesting features: dynamic filtering using Groovy scripts; Hystrix for software load balancing, fault tolerance, and error handling for originated HTTP requests; fine-grained service metrics; Archaius for configuration; and canary requests to detect overload risks. Pretty complex though

on the mechanical-sympathy mailing list. Some really interesting discussion on handling insane quantities of TCP connections using low volumes of hardware:

This talk has some good points and I think the subject is really interesting.  I would take the suggested approach with serious caution.  For starters the Linux kernel is nowhere near as bad as it made out.  Last year I worked with a client and we scaled a single server to 1 million concurrent connections with async programming in Java and some sensible kernel tuning.  I've heard they have since taken this to over 5 million concurrent connections.

BTW Open Onload is an open source implementation.  Writing a network stack is a serious undertaking.  In a previous life I wrote a network probe and had to reassemble TCP streams and kept getting tripped up by edge cases.  It is a great exercise in data structures and lock-free programming.  If you need very high-end performance I'd talk to the Solarflare or Mellanox guys before writing my own.

There are some errors and omissions in this talk.  For example, his range of ephemeral ports is not quite right, and atomic operations are only 15 cycles on Sandy Bridge when hitting local cache.  A big issue for me is when he defined C10M he did not mention the TIME_WAIT issue with closing connections.  Creating and destroying 1 million connections per second is a major issue.  A protocol like HTTP is very broken in that the server closes the socket and therefore has to retain the TCB until the specified timeout occurs to ensure no older packet is delivered to a new socket connection.

Written by Google, this library is a flexible, efficient, and powerful Java client library for accessing any resource on the web via HTTP. It features a pluggable HTTP transport abstraction that allows any low-level library to be used, such as java.net.HttpURLConnection, Apache HTTP Client, or URL Fetch on Google App Engine. It also features efficient JSON and XML data models for parsing and serialization of HTTP response and request content. The JSON and XML libraries are also fully pluggable, including support for Jackson and Android's GSON libraries for JSON.

Not quite as simple an API as Python's requests, sadly, but still an improvement on the verbose Apache HttpComponent API. Good support for unit testing via a built-in mock-response class. Still in beta

Twitter's version of the "canary"/"tracer" request concept

includes "429 Too Many Requests", for rate limits

At Railscamp X it became clear there is a gap in the current HTTP specification. There are many ways for a developer to screw up their implementation, but no code to share the nature of the error with the end user. We humbly suggest the following status codes are included in the HTTP spec in the 7XX range.

Includes such useful status codes as "724 - This line should be unreachable".

'an elegant and simple HTTP library for Python, built for human beings.' 'Requests is an Apache2 Licensed HTTP library, written in Python, for human beings. Python’s standard urllib2 module provides most of the HTTP capabilities you need, but the API is thoroughly broken. It was built for a different time — and a different web. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks. Requests takes all of the work out of Python HTTP/1.1 — making your integration with web services seamless. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, powered by urllib3, which is embedded within Requests.'

'a HTTP client mock library for Python, 100% inspired on ruby's FakeWeb [ https://github.com/chrisk/fakeweb ].' 'HTTPretty monkey patches Python's socket core module, reimplementing the HTTP protocol by mocking requests and responses.'

Troy Hunt, an Aussie software architect working on a .Net security product called ASafaWeb, does a great job extensively deconstructing Tesco's appalling website security on their shopping site. In the process, he gets this wonderful tweet from their customer-care account:

"@troyhunt Let me assure you that all customer passwords are stored securely & in line with industry standards across online retailers."

As he says, this is a clear demonstration that Tesco is in the first stage of the four stages of competence -- "unconscious incompetence": "The individual does not understand or know how to do something and does not necessarily recognise the deficit." ( http://en.wikipedia.org/wiki/Four_stages_of_competence )

by Erik Onnen of Urban Airship. very good presentation on the current state of the art in large-scale low-latency service operation using the JVM on Linux. Lots of good details on async vs sync, HTTPS/TLS/TCP tuning, etc.

'a Java framework for developing ops-friendly, high-performance, RESTful web services. Developed by Yammer to power their JVM-based backend services, Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, lightweight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality HTTP+JSON web service in the shortest time possible.' From Coda Hale/Yammer; includes Guava, Jetty, Jersey, Jackson, Metrics, slf4j. Pretty good baseline to start any new Java service with....

enabling TCP Slow Start on the HTTP server-side decreased internet round-trip page load time by 21% in this case; comments suggest an "ip route" command can also work

'a tool for tunneling SSH through HTTP proxies'. handy

'Starting with Chrome 13, we'll have HTTPS pins for most Google properties. This means that certificate chains for, say, https://www.google.com, must include a whitelisted public key. It's a fatal error otherwise.' good anti-MITM protection

nicely detailed summary

'a reverse proxy, load balancer and HTTPS front-end for Web server(s). Pound was developed to enable distributing the load among several Web-servers and to allow for a convenient SSL wrapper for those Web servers that do not offer it natively. Pound is distributed under the GPL'

'allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads.' actively maintained, Windows and UNIX

Sane suggestions for good HTTP APIs

from one of GitHub's designers, good tips on how the URL UI needs to work these days

'steps towards the glory of REST'. 3 steps, namely: "Level 0: the swamp of POX", "Level 1: Resources", "Level 2: HTTP verbs", and "Level 3: hypermedia controls". +1

'a web service of some sort (or any other way) to pull a current time zone settings for a (US) city. For the parts of the country that don't follow the Daylight Saving Time and basically jump timezones when everyone else is switching summer/winter time... I don't fancy creating own database of the places that don't follow DST. Is there a way to pull this data on demand?' earthtools.org seems the closest thing

free Twitter proxy for access to the Twitter API without requiring OAuth, perfect for stupid read-only stuff like my filter-tweets script (via Padraig)

'We recognize that only through your apps and hacks, can SoundCloud fully realize its potential as the audio platform.'

a clever HTTP session-management trick (via Tony Finch)

nifty; Apache-licensed distributed, RESTful, JSON-over-HTTP, schemaless search server with multi-tenancy

C10K microbenchmarking fun in Javascript (via:simonw)

how the Great Firewall of Oz breaks so much more than the web browser

I'm late to the party, but this sounds lovely

more on the new async HTTP server from FriendFeed/Facebook, in Python. looks lovely

'an open source version of the scalable, non-blocking web server and tools that power FriendFeed. The FriendFeed application is written using a web framework that looks a bit like web.py or Google's webapp, but with additional tools and optimizations to take advantage of the underlying non-blocking (epoll) infrastructure.'

good wrap-up from Anil Dash on "the new push"