jm + http   74

Jauter
This Java library can route paths to targets and create paths from targets and params (reverse routing). This library is tiny, without additional dependencies, and is intended for use together with an HTTP server side library. If you want to use with Netty, see netty-router.
java  jauter  scala  request-routing  http  netty  open-source 
2 days ago by jm
JCDecaux Developer API
web service API for Dublin Bikes data (and other similar bikesharing services run by JCD):
Two kinds of data are delivered by the platform:

Static data provides stable information like station position, number of bike stands, payment terminal availability, etc.
Dynamic data provides station state, number of available bikes, number of free bike stands, etc.
Static data can be downloaded manually in file format or accessed through the API. Dynamic data are refreshed every minute and can be accessed only through the API.


Ruby API: https://github.com/oisin/bikes
jcdecaux  bikesharing  dublin  dublin-bikes  api  web-services  http  json  open-data 
14 days ago by jm
Axel
A nice curl/wget replacement which supports multi-TCP-connection downloads of HTTP/FTP resources. packaged for most Linux variants and OSX via brew
axel  curl  wget  via:johnke  downloading  tcp  http  ftp  ubuntu  debian  unix  linux 
5 weeks ago by jm
Comcast Wi-Fi serving self-promotional ads via JavaScript injection | Ars Technica
Comcast is adding data into the broadband packet stream. In 2007, it was packets serving up disconnection commands. Today, Comcast is inserting JavaScript that is serving up advertisements, according to [Robb] Topolski, who reviewed Singel's data. "It's the duty of the service provider to pull packets without treating them or modifying them or injecting stuff or forging packets. None of that should be in the province of the service provider," he said. "Imagine every Web page with a Comcast bug in the lower righthand corner. It's the antithesis of what a service provider is supposed to do. We want Internet access, not another version of cable TV."


The company appears to be called Front Porch: http://arstechnica.com/tech-policy/2014/09/meet-the-tech-company-performing-ad-injections-for-big-cable/
comcast  ads  injection  security  javascript  http  network-neutrality  isps 
6 weeks ago by jm
UK piracy police arrest man suspected of running proxy server (Wired UK)
The site, Immunicity.org, offers a proxy server and a proxy autoconfiguration file (PAC) to tell browsers to access various blocked sites (PirateBay, KickassTorrents et al) via the proxy.
The Police Intellectual Property Crime Unit has arrested a 20-year-old man in Nottingham on suspicion of copyright infringement for running a proxy server providing access to other sites subject to legal blocking orders.


Is operating a proxy server illegal? Interesting. Seems unlikely that this will go to court though.

(Via TJ McIntyre)
immunicity  via:tjmcintyre  police  uk  piracy  proxies  http  pac  pipcu  copyright 
11 weeks ago by jm
How to take over the computer of any JVM developer
To prove how easy [MITM attacking Mavencentral JARs] is to do, I wrote dilettante, a man-in-the-middle proxy that intercepts JARs from maven central and injects malicious code into them. Proxying HTTP traffic through dilettante will backdoor any JARs downloaded from maven central. The backdoored version will retain their functionality, but display a nice message to the user when they use the library.
jars  dependencies  java  build  clojure  security  mitm  http  proxies  backdoors  scala  maven  gradle 
12 weeks ago by jm
REST Commander: Scalable Web Server Management and Monitoring
We dynamically monitor and manage a large and rapidly growing number of web servers deployed on our infrastructure and systems. However, existing tools present major challenges when making REST/SOAP calls with server-specific requests to a large number of web servers, and then performing aggregated analysis on the responses. We therefore developed REST Commander, a parallel asynchronous HTTP client as a service to monitor and manage web servers. REST Commander on a single server can send requests to thousands of servers with response aggregation in a matter of seconds. And yes, it is open-sourced at http://www.restcommander.com.

Feature highlights:

Click-to-run with zero installation;
Generic HTTP request template supporting variable-based replacement for sending server-specific requests;
Ability to send the same request to different servers, different requests to different servers, and different requests to the same server;
Maximum concurrency control (throttling) to accommodate server capacity;
Commander itself is also “as a service”: with its powerful REST API, you can define ad-hoc target servers, an HTTP request template, variable replacement, and a regular expression all in a single call. In addition, intuitive step-by-step wizards help you achieve the same functionality through a GUI.
rest  http  clients  load-testing  ebay  soap  async  testing  monitoring 
12 weeks ago by jm
Netflix/ribbon
a client side IPC library that is battle-tested in cloud. It provides the following features:

Load balancing;
Fault tolerance;
Multiple protocol (HTTP, TCP, UDP) support in an asynchronous and reactive model;
Caching and batching.

I like the integration of Eureka and Hystrix in particular, although I would really like to read more about Eureka's approach to availability during network partitions and CAP.

https://groups.google.com/d/msg/eureka_netflix/LXKWoD14RFY/-5nElGl1OQ0J has some interesting discussion on the topic. It actually sounds like the Eureka approach is more correct than using ZK: 'Eureka is available. ZooKeeper, while tolerant against single node failures, doesn't react well to long partitioning events. For us, it's vastly more important that we maintain an available registry than a necessary consistent registry. If us-east-1d sees 23 nodes, and us-east-1c sees 22 nodes for a little bit, that's OK with us.'

See also http://ispyker.blogspot.ie/2013/12/zookeeper-as-cloud-native-service.html which corroborates this:

I went into one of the instances and quickly did an iptables DROP on all packets coming from the other two instances. This would simulate an availability zone continuing to function, but that zone losing network connectivity to the other availability zones. What I saw was that the two other instances noticed that the first server “going away”, but they continued to function as they still saw a majority (66%). More interestingly the first instance noticed the other two servers “going away” dropping the ensemble availability to 33%. This caused the first server to stop serving requests to clients (not only writes, but also reads). [...]

To me this seems like a concern, as network partitions should be considered an event that should be survived. In this case (with this specific configuration of zookeeper) no new clients in that availability zone would be able to register themselves with consumers within the same availability zone. Adding more zookeeper instances to the ensemble wouldn’t help considering a balanced deployment as in this case the availability would always be majority (66%) and non-majority (33%).
netflix  ribbon  availability  libraries  java  hystrix  eureka  aws  ec2  load-balancing  networking  http  tcp  architecture  clients  ipc 
july 2014 by jm
"The Tail at Scale"
by Jeffrey Dean and Luiz Andre Barroso, Google. A selection of Google's architectural mechanisms used to defeat 99th-percentile latency spikes: hedged requests, tied requests, micro-partitioning, selective replication, latency-induced probation, canary requests.
google  architecture  distcomp  soa  http  partitioning  replication  latency  99th-percentile  canary-requests  hedged-requests 
july 2014 by jm
lookout/ngx_borderpatrol
BorderPatrol is an nginx module to perform authentication and session management at the border of your network. BorderPatrol makes the assumption that you have some set of services that require authentication and a service that hands out tokens to clients to access that service. You may not want those tokens to be sent across the internet, even over SSL, for a variety of reasons. To this end, BorderPatrol maintains a lookup table of session-id to auth token in memcached.
borderpatrol  nginx  modules  authentication  session-management  web-services  http  web  authorization 
june 2014 by jm
spoofing the samsung smart tv internet check
If this kind of bullshit -- a HTTP GET of an XML file from www.samsung.com -- is how the Samsung Smart TV firmware decides if the internet is working or not, I dread to think how crappy the rest of the code is. (At least in Netnote we performed a bunch of bigco-domain DNS lookups before giving up...)
smart-tv  samsung  fail  xml  http  internet  embedded-software  firmware  crap-code 
april 2014 by jm
Haywire
kellabyte's hack in progress -- 'an asynchronous HTTP server framework written in C. The goal of Haywire is to learn how to create a server with a minimal feature set that can handle a high rate of requests and connections with as low of latency and resource usage as possible. Haywire uses the event loop based libuv platform layer that node.js is built on top of (also written in C). libuv abstracts IOCP on Windows and epoll/kqueue/event ports/etc. on Unix systems to provide efficient asynchronous I/O on all supported platforms.'

Outperforms libevent handily, it seems. Apache-licensed.
server  http  asynchronous  libuv  haywire  kellabyte  c  events  open-source  asl2 
april 2014 by jm
Consul
Nice-looking new tool from Hashicorp; service discovery and configuration service, built on Raft for leader election, Serf for gossip-based messaging, and Go. Some features:

* Gossip is performed over both TCP and UDP;

* gossip messages are encrypted symmetrically and therefore secure from eavesdropping, tampering, spoofing and packet corruption (like the incident which brought down S3 for days: http://status.aws.amazon.com/s3-20080720.html );

* exposes both a HTTP interface and (even better) DNS;

* includes explicit support for long-distance WAN operation as well as on LANs.

It all looks very practical and usable. MPL-licensed.

The only potential risk I can see is that expecting to receive config updates from a blocking poll of the HTTP interface needs some good "best practice" docs, to ensure that people don't mishandle the scenario where there is a network partition between your calling code and the Consul server/agent. Without any heartbeating protocol behind the scenes, HTTP is vulnerable to "hung connections" which would result in a config change being silently missed by the client until the connection eventually is timed out, either by the calling code or the client-side kernel. This could potentially take minutes to occur, which in some usage scenarios could be a big, unforeseen problem.
configuration  service-discovery  distcomp  raft  consensus-algorithms  go  mpl  open-source  dns  http  gossip-protocol  hashicorp 
april 2014 by jm
Why no SSL ? — Varnish version 4.0.0 documentation
Poul-Henning Kemp details why Varnish doesn't do SSL -- basically due to the quality and complexity of open-source SSL implementations:
There is no other way we can guarantee that secret krypto-bits do not leak anywhere they should not, than by fencing in the code that deals with them in a child process, so the bulk of varnish never gets anywhere near the certificates, not even during a core-dump.


Now looking pretty smart, post-Heartbleed.
ssl  tls  varnish  open-source  poul-henning-kemp  https  http  proxies  security  coding 
april 2014 by jm
Daring Fireball: Rethinking What We Mean by 'Mobile Web'
We shouldn’t think of “the web” as only what renders in web browsers. We should think of the web as anything transmitted using HTTP and HTTPS. Apps and websites are peers, not competitors. They’re all just clients to the same services.

+1. Finally, a Daring Fireball post I agree with.
daring-fireball  apps  web  http  https  mobile  apple  android  browsers 
april 2014 by jm
Issue 122 - android-query - HTTP 204 Response results in Network Error (-101)
an empty 204 response to a HTTP PUT will trigger this. See also https://code.google.com/p/android/issues/detail?id=24672, '"java.io.IOException: unexpected end of stream" on HttpURLConnection HEAD call'.
http  urlconnection  httpurlconnection  java  android  dalvik  bugs  204  head  get  exceptions 
march 2014 by jm
The Microservice Declaration of Independence
"Microservices" seems to be yet another term for SOA; small, decoupled, independently-deployed services, with well-defined public HTTP APIs. Pretty much all the services I've worked on over the past few years have been built in this style. Still, let's keep an eye on this concept anyway.

Another definition seems to be a more FP-style one: http://www.slideshare.net/michaelneale/microservices-and-functional-programming -- where the "microservice" does one narrowly-defined thing, and that alone.
microservices  soa  architecture  handwaving  http  services  web  deployment 
march 2014 by jm
htcat/htcat
a utility to perform parallel, pipelined execution of a single HTTP GET. htcat is intended for the purpose of incantations like: htcat https://host.net/file.tar.gz | tar -zx

It is tuned (and only really useful) for faster interconnects: [....] 109MB/s on a gigabit network, between an AWS EC2 instance and S3. This represents 91% use of the theoretical maximum of gigabit (119.2 MiB/s).
go  cli  http  file-transfer  ops  tools 
march 2014 by jm
Good explanation of exponential backoff
I've often had to explain this key feature verbosely, and it's hard to do without handwaving. Great to have a solid, well-explained URL to point to
exponential-backoff  backoff  retries  reliability  web-services  http  networking  internet  coding  design 
march 2014 by jm
"Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" [PDF]
Google paper describing the infrastructure they've built for cross-service request tracing (ie. "tracer requests"). Features: low code changes required (since they've built it into the internal protobuf libs), low performance impact, sampling, deployment across the ~entire production fleet, output visibility in minutes, and has been live in production for over 2 years. Excellent read
dapper  tracing  http  services  soa  google  papers  request-tracing  tracers  protobuf  devops 
march 2014 by jm
Traffic Graph – Google Transparency Report
this is cool. Google are exposing an aggregated 'all services' hit count time-series graph, broken down by country, as part of their Transparency Report pages
transparency  filtering  web  google  http  graphs  monitoring  syria 
february 2014 by jm
Video Processing at Dropbox
On-the-fly video transcoding during live streaming. They've done a great job of this!
At the beginning of the development of this feature, we entertained the idea to simply pre-transcode all the videos in Dropbox to all possible target devices. Soon enough we realized that this simple approach would be too expensive at our scale, so we decided to build a system that allows us to trigger a transcoding process only upon user request and cache the results for subsequent fetches. This on-demand approach: adapts to heterogeneous devices and network conditions, is relatively cheap (everything is relative at our scale), guarantees low latency startup time.
ffmpeg  dropbox  streaming  video  cdn  ec2  hls  http  mp4  nginx  haproxy  aws  h264 
february 2014 by jm
CJEU in #Svensson says that in general it is OK to hyperlink to protected works without permission
IPKat says 'this morning the Court of Justice of the European Union issued its keenly awaited decision in Case C-466/12 Svensson [...]: The owner of a website may, without the authorisation of the copyright holders, redirect internet users, via hyperlinks, to protected works available on a freely accessible basis on another site. This is so even if the internet users who click on the link have the impression that the work is appearing on the site that contains the link.'

This is potentially big news. Not so much for the torrent-site scenario, but for the NNI/NLI linking-to-newspaper-stories scenario.
ip  svensson  cjeu  eu  law  linking  hyperlinks  pirate-bay  internet  web  links  http  copyright 
february 2014 by jm
Home · linkedin/rest.li Wiki
Rest.li is a REST+JSON framework for building robust, scalable service architectures using dynamic discovery and simple asynchronous APIs. Rest.li fills a niche for building RESTful service architectures at scale, offering a developer workflow for defining data and REST APIs that promotes uniform interfaces, consistent data modeling, type-safety, and compatibility checked API evolution.


The new underlying comms layer for Voldemort, it seems.
voldemort  d2  rest.li  linkedin  json  rest  http  api  frameworks  java 
february 2014 by jm
Sky parental controls break many JQuery-using websites
An 11 hour outage caused by a false positive in Sky's anti-phishing filter; all sites using the code.jquery.com CDN for JQuery would have seen errors.
Sky still appears to be blocking code.jquery.com and all files served via the site, and more worryingly is that if you try to report the incorrect category, once signing in on the Sky website you an error page. We suspect the site was blocked due to being linked to by a properly malicious website, i.e. code.jquery.com and some javascript files were being used on a dodgy website and every domain mentioned was subsequently added to a block list.


(via Tony Finch)
via:fanf  sky  filtering  internet  uk  anti-phishing  phish  jquery  javascript  http  web  fps  false-positives 
january 2014 by jm
UK porn filter blocks game update that contained 'sex' in URL
Staggeringly inept. The UK national porn filter blocks based on a regexp match of the URL against /.*sex.*/i -- the good old "Scunthorpe problem". Better, it returns a 404 response. This is also a good demonstration of how web filtering has unintended side effects, breaking third-party software updates with its false positives.
The update to online strategy game League of Legends was disrupted by the internet filter because the software attempted to access files that accidentally include the word “sex” in the middle of their file names. The block resulted in the update failing with “file not found” errors, which are usually created by missing files or broken updates on the part of the developers.
uk  porn  filtering  guardian  regular-expressions  false-positives  scunthorpe  http  web  league-of-legends  sex 
january 2014 by jm
Chartbeat's Lessons learned tuning TCP and Nginx in EC2
a good writeup of basic sysctl tuning for an internet-facing HTTP proxy fleet running in EC2. Nothing groundbreaking here, but it's well-written
nginx  amazon  ec2  tcp  ip  tuning  sysctl  linux  c10k  ssl  http 
january 2014 by jm
websocketd
'like inetd, but for WebSockets' -- 'a small command line tool that will wrap an existing command line interface program, and allow it to be accessed via a WebSocket. It provides a quick mechanism for allowing web-applications to interact with existing command line tools.'

Awesome idea. BSD-licensed. (Via Mike Loukides)
websockets  cli  server  tools  unix  inetd  web  http  open-source 
december 2013 by jm
wrk
a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue.  An optional LuaJIT script can perform HTTP request generation, response processing, and custom reporting.


Written in C, ASL2 licensed.
wrk  benchmarking  http  performance  testing  lua  load-testing  load-generation 
december 2013 by jm
Make The Web Fast - The HAR Show: Capturing and Analyzing performance data with HTTP Archive format — Google Developers
Wow, I didn't know about this. Great idea.
Need a flexible format to record, export, and analyze network performance data? Well, that's exactly what the HTTP Archive format (HAR) is designed to do! Even better, did you know that Chrome DevTools supports it? In this episode we'll take a deep dive into the format (as you'll see, its very simple), and explore the many different ways it can help you capture and analyze your sites performance. Join Ilya Grigorik and Peter Lubbers to find out how to capture HAR network traces in Chrome, visualize the data via an online tool, share the reports with your clients and coworkers, automate the logging and capture of HAR data for your build scripts, and even adapt it to server-side analysis use cases
capturing  logging  performance  http  debugging  trace  capture  har  archives  protocols  recording 
december 2013 by jm
High Performance Browser Networking
slides from Ilya Grigorik's tutorial on the topic at O'Reilly's Velocity conference. lots of good data and tips for internet protocol optimization
slides  presentations  ilya-grigorik  performance  http  https  tcp  tutorials  networking  internet 
november 2013 by jm
Pushing to 100,000 API clients simultaneously
This looks really nice -- it's quite similar to something I was hacking on a while back. Only problem is that it's AGPL-licensed...

'Pushpin makes it easy to create HTTP long-polling and streaming services using any web stack as the backend. It’s compatible with any framework, whether Django, Rails, ASP, or even PHP. Pushpin works as a reverse proxy, sitting in front of your server application and managing all of the open client connections.'
pushpin  opensource  agpl  http  long-polling  reverse-proxy  architecture  callbacks 
november 2013 by jm
NettoSphere demo
'This article will use NettoSphere, a framework build on top of the popular Netty Framework and Atmosphere with support of WebSockets, Server Side Events and Long-Polling. NettoSphere allows [async JVM framework] Atmosphere's applications to run on top of the Netty Framework.'
atmosphere  netty  async  java  scala  websockets  sse  long-polling  http  demos  games 
november 2013 by jm
"High Performance Browser Networking", by Ilya Grigorik, read online for free
Wow, this looks excellent. A must-read for people working on systems with high-volume, low-latency phone-to-server communications -- and free!
How prepared are you to build fast and efficient web applications? This eloquent book provides what every web developer should know about the network, from fundamental limitations that affect performance to major innovations for building even more powerful browser applications—including HTTP 2.0 and XHR improvements, Server-Sent Events (SSE), WebSocket, and WebRTC.

Author Ilya Grigorik, a web performance engineer at Google, demonstrates performance optimization best practices for TCP, UDP, and TLS protocols, and explains unique wireless and mobile network optimization requirements. You’ll then dive into performance characteristics of technologies such as HTTP 2.0, client-side network scripting with XHR, real-time streaming with SSE and WebSocket, and P2P communication with WebRTC.

Deliver optimal TCP, UDP, and TLS performance;
Optimize network performance over 3G/4G mobile networks;
Develop fast and energy-efficient mobile applications;
Address bottlenecks in HTTP 1.x and other browser protocols;
Plan for and deliver the best HTTP 2.0 performance;
Enable efficient real-time streaming in the browser;
Create efficient peer-to-peer videoconferencing and low-latency applications with real-time WebRTC transports


Via Eoin Brazil.
book  browser  networking  performance  phones  mobile  3g  4g  hsdpa  http  udp  tls  ssl  latency  webrtc  websockets  ebooks  via:eoin-brazil  google  http2  sse  xhr  ilya-grigorik 
october 2013 by jm
GCHQ report on 'MULLENIZE' program to 'stain' anonymous electronic traffic
By modifying the User-Agent: header string, each HTTP transaction is "stained" to allow tracking. huh
gchq  nsa  snooping  sniffing  surveillance  user-agent  http  browsers  leaks 
october 2013 by jm
Attacking Tor: how the NSA targets users' online anonymity
As part of the Turmoil system, the NSA places secret servers, codenamed Quantum, at key places on the internet backbone. This placement ensures that they can react faster than other websites can. By exploiting that speed difference, these servers can impersonate a visited website to the target before the legitimate website can respond, thereby tricking the target's browser to visit a Foxacid server.


whoa, I missed this before.
nsa  gchq  packet-injection  attacks  security  backbone  http  latency 
october 2013 by jm
Snowizard
'a Java port of Twitter's Snowflake thrift service presented as an HTTP-based Dropwizard service'.
an HTTP-based service for generating unique ID numbers at high scale with some simple guarantees. supports returning ID numbers as: JSON and JSONP; Google's Protocol Buffers; Plain text.

At GE, we were more interested in the uncoordinated aspects of Snowflake than its throughput requirements, so HTTP was fine for our needs. We also exposed the core of Snowflake as an embeddable module so it can be directly integrated into our applications. We don't have the guarantees that the Snowflake-Zookeeper integration was providing, but that was also acceptable to us. In places where we really needed high throughput, we leveraged the snowizard-core embeddable module directly.


Odd OSS license, though -- BSDish?
java  open-source  ids  soa  services  snowflake  http 
august 2013 by jm
Ivan Ristić: Defending against the BREACH attack
One interesting response to this HTTPS compression-based MITM attack:
The award for least-intrusive and entirely painless mitigation proposal goes to Paul Querna who, on the httpd-dev mailing list, proposed to use the HTTP chunked encoding to randomize response length. Chunked encoding is a HTTP feature that is typically used when the size of the response body is not known in advance; only the size of the next chunk is known. Because chunks carry some additional information, they affect the size of the response, but not the content. By forcing more chunks than necessary, for example, you can increase the length of the response. To the attacker, who can see only the size of the response body, but not anything else, the chunks are invisible. (Assuming they're not sent in individual TCP packets or TLS records, of course.) This mitigation technique is very easy to implement at the web server level, which makes it the least expensive option. There is only a question about its effectiveness. No one has done the maths yet, but most seem to agree that response length randomization slows down the attacker, but does not prevent the attack entirely. But, if the attack can be slowed down significantly, perhaps it will be as good as prevented.
mitm  attacks  hacking  security  compression  http  https  protocols  tls  ssl  tcp  chunked-encoding  apache 
august 2013 by jm
Improved HTTPS Performance with Early SSL Termination
This is a neat hack. Since SSL/TLS connection establishment requires lots of consecutive round trips before the connection is ready, by performing that closer to the user and reusing an existing region-to-region connection behind the scenes, the overall latency is greatly improved. Works for HTTP as well
http  https  ssl  architecture  aws  ec2  performance  latency  internet  round-trip  nginx  tls 
july 2013 by jm
Traditional AQM is not enough!
Jim Gettys on modern web design, HTTP, buffering, and FIFO queues in the network.
Web surfing is putting impulses of packets, without congestion avoidance, into FIFO queues where they do severe collateral damage to anything sharing the link (including itself!). So today’s web behavior incurs huge collateral damage on itself, data centers, the edge of the network, and in particular any application that hopes to have real time behavior. How do we solve this problem?


tl;dr: fq_codel. Now I want it!
buffering  networking  internet  web  http  protocols  tcp  bufferbloat  jim-gettys  codel  fq_codel 
july 2013 by jm
Announcing Zuul: Edge Service in the Cloud
Netflix' library to implement "edge services" -- ie. a front end to their API, web servers, and streaming servers. Some interesting features: dynamic filtering using Groovy scripts; Hystrix for software load balancing, fault tolerance, and error handling for originated HTTP requests; fine-grained service metrics; Archaius for configuration; and canary requests to detect overload risks. Pretty complex though
edge-services  api  netflix  zuul  archaius  canary-requests  http  groovy  hystrix  load-balancing  fault-tolerance  error-handling  configuration 
june 2013 by jm
Martin Thompson, Luke "Snabb Switch" Gorrie etc. review the C10M presentation from Schmoocon
on the mechanical-sympathy mailing list. Some really interesting discussion on handling insane quantities of TCP connections using low volumes of hardware:
This talk has some good points and I think the subject is really interesting.  I would take the suggested approach with serious caution.  For starters the Linux kernel is nowhere near as bad as it made out.  Last year I worked with a client and we scaled a single server to 1 million concurrent connections with async programming in Java and some sensible kernel tuning.  I've heard they have since taken this to over 5 million concurrent connections.

BTW Open Onload is an open source implementation.  Writing a network stack is a serious undertaking.  In a previous life I wrote a network probe and had to reassemble TCP streams and kept getting tripped up by edge cases.  It is a great exercise in data structures and lock-free programming.  If you need very high-end performance I'd talk to the Solarflare or Mellanox guys before writing my own.

There are some errors and omissions in this talk.  For example, his range of ephemeral ports is not quite right, and atomic operations are only 15 cycles on Sandy Bridge when hitting local cache.  A big issue for me is when he defined C10M he did not mention the TIME_WAIT issue with closing connections.  Creating and destroying 1 million connections per second is a major issue.  A protocol like HTTP is very broken in that the server closes the socket and therefore has to retain the TCB until the specified timeout occurs to ensure no older packet is delivered to a new socket connection.
mechanical-sympathy  hardware  scaling  c10m  tcp  http  scalability  snabb-switch  martin-thompson 
may 2013 by jm
google-http-java-client
Written by Google, this library is a flexible, efficient, and powerful Java client library for accessing any resource on the web via HTTP. It features a pluggable HTTP transport abstraction that allows any low-level library to be used, such as java.net.HttpURLConnection, Apache HTTP Client, or URL Fetch on Google App Engine. It also features efficient JSON and XML data models for parsing and serialization of HTTP response and request content. The JSON and XML libraries are also fully pluggable, including support for Jackson and Android's GSON libraries for JSON.


Not quite as simple an API as Python's requests, sadly, but still an improvement on the verbose Apache HttpComponent API. Good support for unit testing via a built-in mock-response class. Still in beta
google  beta  software  http  libraries  json  xml  transports  protocols 
april 2013 by jm
RFC 6585 - Additional HTTP Status Codes
includes "429 Too Many Requests", for rate limits
api  rfc  http  reference  standards  web  rest 
march 2013 by jm
PUBLIC joho / 7XX-rfc
At Railscamp X it became clear there is a gap in the current HTTP specification. There are many ways for a developer to screw up their implementation, but no code to share the nature of the error with the end user. We humbly suggest the following status codes are included in the HTTP spec in the 7XX range.


Includes such useful status codes as "724 - This line should be unreachable".
http  standards  humour  funny  jokes 
january 2013 by jm
Requests: HTTP for Humans
'an elegant and simple HTTP library for Python, built for human beings.' 'Requests is an Apache2 Licensed HTTP library, written in Python, for human beings. Python’s standard urllib2 module provides most of the HTTP capabilities you need, but the API is thoroughly broken. It was built for a different time — and a different web. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks. Requests takes all of the work out of Python HTTP/1.1 — making your integration with web services seamless. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, powered by urllib3, which is embedded within Requests.'
python  http  urllib  libraries  requests  via:mikeste 
january 2013 by jm
HTTPretty
'a HTTP client mock library for Python, 100% inspired on ruby's FakeWeb [ https://github.com/chrisk/fakeweb ].' 'HTTPretty monkey patches Python's socket core module, reimplementing the HTTP protocol by mocking requests and responses.'
mocking  testing  http  python  ruby  unit-tests  tests  monkey-patching 
january 2013 by jm
Lessons in website security anti-patterns by Tesco
Troy Hunt, an Aussie software architect working on a .Net security product called ASafaWeb, does a great job extensively deconstructing Tesco's appalling website security on their shopping site. In the process, he gets this wonderful tweet from their customer-care account:

"@troyhunt Let me assure you that all customer passwords are stored securely & in line with industry standards across online retailers."

As he says, this is a clear demonstration that Tesco is in the first stage of the four stages of competence -- "unconscious incompetence": "The individual does not understand or know how to do something and does not necessarily recognise the deficit." ( http://en.wikipedia.org/wiki/Four_stages_of_competence )
tesco  security  passwords  web  http  https  ssl  funny  dot-net  shopping  uk  customer-care 
july 2012 by jm
High performance network programming on the JVM, OSCON 2012
by Erik Onnen of Urban Airship. very good presentation on the current state of the art in large-scale low-latency service operation using the JVM on Linux. Lots of good details on async vs sync, HTTPS/TLS/TCP tuning, etc.
http  https  scaling  jvm  async  sync  oscon  presentations  tcp 
july 2012 by jm
Dropwizard
'a Java framework for developing ops-friendly, high-performance, RESTful web services. Developed by Yammer to power their JVM-based backend services, Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, lightweight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality HTTP+JSON web service in the shortest time possible.' From Coda Hale/Yammer; includes Guava, Jetty, Jersey, Jackson, Metrics, slf4j. Pretty good baseline to start any new Java service with....
framework  http  java  rest  web  jersey  guava  jackson  jetty  json  web-services  yammer 
may 2012 by jm
Why upgrading your Linux Kernel will make your customers much happier
enabling TCP Slow Start on the HTTP server-side decreased internet round-trip page load time by 21% in this case; comments suggest an "ip route" command can also work
tcp  performance  linux  network  web  http  rtt  slow-start  via:jacob 
march 2012 by jm
corkscrew
'a tool for tunneling SSH through HTTP proxies'. handy
ssh  http  proxies  software  linux  tunneling  isps 
august 2011 by jm
Chrome to get HTTPS public key pinning
'Starting with Chrome 13, we'll have HTTPS pins for most Google properties. This means that certificate chains for, say, https://www.google.com, must include a whitelisted public key. It's a fatal error otherwise.' good anti-MITM protection
https  ssl  http  web  security  mitm  sniffing  chrome 
may 2011 by jm
Pound
'a reverse proxy, load balancer and HTTPS front-end for Web server(s). Pound was developed to enable distributing the load among several Web-servers and to allow for a convenient SSL wrapper for those Web servers that do not offer it natively. Pound is distributed under the GPL'
https  ssl  http  proxy  web  pound  reverse-proxy 
april 2011 by jm
HTTrack
'allows you to download a World Wide Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the "mirrored" website in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads.' actively maintained, Windows and UNIX
web  opensource  http  download  mirror  from delicious
april 2011 by jm
on URL Design
from one of GitHub's designers, good tips on how the URL UI needs to work these days
github  urls  design  ui  usability  webdev  webdesign  http  from delicious
december 2010 by jm
Richardson Maturity Model
'steps towards the glory of REST'. 3 steps, namely: "Level 0: the swamp of POX", "Level 1: Resources", "Level 2: HTTP verbs", and "Level 3: hypermedia controls". +1
rest  leonard-richardson  api  design  coding  martin-fowler  restful  web-services  http  from delicious
november 2010 by jm
Web service - current time zone for a city? - Stack Overflow
'a web service of some sort (or any other way) to pull a current time zone settings for a (US) city. For the parts of the country that don't follow the Daylight Saving Time and basically jump timezones when everyone else is switching summer/winter time... I don't fancy creating own database of the places that don't follow DST. Is there a way to pull this data on demand?' earthtools.org seems the closest thing
dst  daylight-savings  local  timezones  iso8601  dates  times  web-services  http  from delicious
october 2010 by jm
SuperTweet.Net
free Twitter proxy for access to the Twitter API without requiring OAuth, perfect for stupid read-only stuff like my filter-tweets script (via Padraig)
via:pixelbeat  api  oauth  proxy  twitter  web  http  curl  supertweet  from delicious
september 2010 by jm
SoundCloud Developers Manifesto
'We recognize that only through your apps and hacks, can SoundCloud fully realize its potential as the audio platform.'
apps  hacks  soundcloud  mp3  music  hosting  files  json  rest  oauth  apis  http  from delicious
may 2010 by jm
FastMail and sessions
a clever HTTP session-management trick (via Tony Finch)
via:fanf  web  http  sessions  cookies  fastmail  from delicious
march 2010 by jm
ElasticSearch
nifty; Apache-licensed distributed, RESTful, JSON-over-HTTP, schemaless search server with multi-tenancy
search  distributed  rest  json  apache  elasticsearch  http  from delicious
february 2010 by jm
mnot’s Weblog: HTTP + Politics = ?
how the Great Firewall of Oz breaks so much more than the web browser
http  web  politics  australia  internet  proxies  filtering  from delicious
december 2009 by jm
Node.js
I'm late to the party, but this sounds lovely
javascript  server  http  web  comet  closures  node.js  event  from delicious
november 2009 by jm
The technology behind Tornado, FriendFeed's web server
more on the new async HTTP server from FriendFeed/Facebook, in Python. looks lovely
async  http  epoll  python  comet  long-poll  facebook  scaling  scalability  web  friendfeed  tornado  opensource  from delicious
september 2009 by jm
Tornado Web Server
'an open source version of the scalable, non-blocking web server and tools that power FriendFeed. The FriendFeed application is written using a web framework that looks a bit like web.py or Google's webapp, but with additional tools and optimizations to take advantage of the underlying non-blocking (epoll) infrastructure.'
epoll  open-source  python  http  scalability  facebook  scaling  web  from delicious
september 2009 by jm

related tags

3g  4g  99th-percentile  ads  agpl  amazon  android  anti-phishing  apache  api  apis  apple  apps  archaius  architecture  archive  archives  asl2  async  asynchronous  atmosphere  atom  attacks  australia  authentication  authorization  availability  aws  axel  backbone  backdoors  backoff  benchmarking  beta  bikesharing  book  borderpatrol  browser  browsers  bufferbloat  buffering  bugs  build  c  c10k  c10m  callbacks  canary-requests  capture  captures  capturing  cdn  chrome  chunked-encoding  cjeu  cli  clients  clojure  closures  codel  coding  comcast  comet  compression  configuration  consensus-algorithms  conversion  cookies  copyright  crap-code  curl  customer-care  d2  dalvik  dapper  daring-fireball  dates  daylight-savings  debian  debugging  demos  dependencies  deployment  design  devops  distcomp  distributed  distributed-systems  dns  dot-net  download  downloading  dropbox  dst  dublin  dublin-bikes  ebay  ebooks  ec2  edge-services  elasticsearch  embedded-software  epoll  error-handling  eu  eureka  event  events  exceptions  exponential-backoff  facebook  fail  false-positives  fastmail  fault-tolerance  feeds  ffmpeg  file-transfer  files  filtering  firmware  fps  fq_codel  framework  frameworks  friendfeed  ftp  funny  games  gchq  get  github  go  google  gossip-protocol  gradle  graphs  groovy  guardian  guava  h264  hacking  hacks  handwaving  haproxy  har  hardware  hashicorp  haywire  head  hedged-requests  hls  hosting  hsdpa  http  http-push  http2  https  httpurlconnection  humour  hyperlinks  hystrix  ids  ilya-grigorik  immunicity  inetd  infrastructure  injection  internet  ip  ipc  iso8601  isps  jackson  jars  jauter  java  javascript  jcdecaux  jersey  jetty  jim-gettys  jmeter  jmx  jokes  jquery  json  jvm  kellabyte  latency  law  league-of-legends  leaks  leonard-richardson  libraries  libuv  linkedin  linking  links  linux  live  load-balancing  load-generation  load-testing  local  logging  long-poll  long-polling  lua  martin-fowler  martin-thompson  maven  mechanical-sympathy  messaging  microservices  mirror  mitm  mobile  mocking  modules  monitoring  monkey-patching  mp3  mp4  mpl  music  netflix  netty  network  network-neutrality  networking  nginx  node.js  nsa  oauth  open-data  open-source  opensource  ops  oscon  pac  packet-injection  packets  papers  partitioning  passwords  performance  phish  phones  ping  pipcu  piracy  pirate-bay  police  politics  porn  poul-henning-kemp  pound  presentations  production  protobuf  protocols  proxies  proxy  pubsub  pubsubhubbub  pushpin  python  quora  raft  realtime  recording  reference  regular-expressions  reliability  replication  request-routing  request-tracing  requests  rest  rest.li  restful  retries  reverse-proxy  rfc  ribbon  round-trip  rtt  ruby  samsung  scala  scalability  scaling  scunthorpe  search  security  server  service-discovery  services  session-management  sessions  sex  shopping  sky  slides  slow-start  smart-tv  snabb-switch  sniffing  snooping  snowflake  soa  soap  software  soundcloud  speed  sse  ssh  ssl  stack  standards  streaming  supertweet  surveillance  svensson  sync  syria  sysctl  tcp  tesco  testing  tests  times  timezones  tls  tools  tornado  trace  tracer-requests  tracers  tracing  transparency  transports  tuning  tunneling  tutorials  twitter  ubuntu  udp  ui  uk  unit-tests  unix  urlconnection  urllib  urls  usability  user-agent  varnish  via:eoin-brazil  via:fanf  via:jacob  via:johnke  via:mikeste  via:pixelbeat  via:tjmcintyre  video  voldemort  web  web-services  webdesign  webdev  webrtc  websites  websockets  wget  wrk  xhr  xml  yammer  zipkin  zuul 

Copy this bookmark:



description:


tags: