jm + http   123

Unbundling Pokémon Go
tl;dr: on Android, it's a Unity app, talking HTTPS to the backend, using protobuf over HTTP. Interesting notes about the use of certificate pinning and how they should be doing that
https  http  protobuf  pokemon-go  pokemon  apps  android  reversing 
10 weeks ago by jm
3 Reasons AWS Lambda Is Not Ready for Prime Time
This totally matches my own preconceptions ;)
When we at Datawire tried to actually use Lambda for a real-world HTTP-based microservice [...], we found some uncool things that make Lambda not yet ready for the world we live in:

Lambda is a building block, not a tool;
Lambda is not well documented;
Lambda is terrible at error handling

Lung skips these uncool things, which makes sense because they’d make the tutorial collapse under its own weight, but you can’t skip them if you want to work in the real world. (Note that if you’re using Lambda for event handling within the AWS world, your life will be easier. But the really interesting case in the microservice world is Lambda and HTTP.)
aws  lambda  microservices  datawire  http  api-gateway  apis  https  python  ops 
may 2016 by jm
CNBC "How Secure Is Your Password" tester form is a spectacular security shitshow
It not only runs over HTTP, it also sends your password to a bunch of third-party ad trackers. omgwtfbbqfail
fail  wtf  funny  cnbc  clowns  inept  security  passwords  http  ad-trackers 
march 2016 by jm
How we implemented the video player in Mail.Ru Cloud
We’ve recently added video streaming service to Mail.Ru Cloud. Development started with contemplating the new feature as an all-purpose “Swiss Army knife” that would both play files of any format and work on any device with the Cloud available. Video content uploaded to the Cloud mostly falls into one of the two categories: “movies/series” and “users’ videos”. The latter are the videos that users shoot with their phones and cameras, and these videos are most versatile in terms of formats and codecs. For many reasons, it is often a problem to watch these videos on other end-user devices without prior normalization: a required codec is missing, or the file size is too big to download, or whatever.


Mainly around using HLS (HTTP Live Streaming).
hls  http  streaming  video  audio  mail.ru  players  codecs 
march 2016 by jm
left-pad.io
A microservice saviour appears!
In order to prevent such a terrible tragedy from occurring ever again during
our lifetimes, `left-pad.io` has been created to provide all the functionality
of `left-pad` AND the overhead of a TLS handshake and an HTTP request.
Less code is better code, leave the heavy lifting to `left-pad.io`, The String
Experts™.
humor  javascript  jokes  npm  packages  left-pad  strings  microservices  http 
march 2016 by jm
Neutrino Software Load Balancer
eBay's software LB, supporting URL matching, comparable to haproxy, built using Netty and Scala. Used in their QA infrastructure it seems
netty  scala  ebay  load-balancing  load-balancers  url  http  architecture 
february 2016 by jm
go-kit
Dropwizard for Go, basically:
a distributed programming toolkit for building microservices in large organizations. We solve common problems in distributed systems, so you can focus on your business logic.
microservices  go  golang  http  libraries  open-source  rpc  circuit-breakers 
january 2016 by jm
Nchan
Nchan is a scalable, flexible pub/sub server for the modern web, built as a module for the Nginx web server. It can be configured as a standalone server, or as a shim between your application and tens, thousands, or millions of live subscribers. It can buffer messages in memory, on-disk, or via Redis. All connections are handled asynchronously and distributed among any number of worker processes. It can also scale to many nginx server instances with Redis. Messages are published to channels with HTTP POST requests or websockets, and subscribed also through websockets, long-polling, EventSource (SSE), old-fashioned interval polling, and more. Each subscriber can listen to up to 255 channels per connection, and can be optionally authenticated via a custom application url. An events meta channel is also available for debugging.


Also now supports HTTP/2. This used to be called the Nginx HTTP Push Module, and I used it with great results in that form. This is the way to do HTTP push in all its forms....
nginx  pubsub  websockets  sse  http  http-push  http2  redis  long-polling  nchan 
january 2016 by jm
BBC Digital Media Distribution: How we improved throughput by 4x
Replacing varnish with nginx. Nice deep-dive blog post covering kernel innards
nginx  performance  varnish  web  http  bbc  ops 
january 2016 by jm
WebSockets, caution required!
This, so much.
There are very valid technical reasons many of the biggest sites on the Internet have not adopted them. Twitter use HTTP/2 + polling, Facebook and Gmail use Long Polling. Saying WebSockets are the only way and the way of the future, is wrongheaded. HTTP/2 may end up winning this battle due to the huge amount of WebSocket connections web browsers allow, and HTTP/3 may unify the protocols
http  realtime  websockets  long-polling  http2  protocols  transport  web  internet 
january 2016 by jm
raskasa/metrics-okhttp
Metrics integration for OkHttp. looks quite nice
okhttp  java  clients  http  metrics  dropwizard 
december 2015 by jm
AWS Api Gateway for Fun and Profit
good worked-through example of an API Gateway rewriting system
api-gateway  aws  api  http  services  ops  alerting  alarming  opsgenie  signalfx 
december 2015 by jm
Introducing Netty-HTTP from Cask
netty-http library solves [Netty usability issues] by using JAX-RS annotations to build a HTTP path routing layer on top of netty. In addition, the library implements a guava service to manage the HTTP service. netty-http allows users of the library to just focus on writing the business logic in HTTP handlers without having to worry about the complexities of path routing or learning netty pipeline internals to build the HTTP service.

We've written something very similar, although I didn't even bother supporting JAX-RS annotations -- just a simple code-level DSL.
jax-rs  netty  http  cask  java  services  coding 
december 2015 by jm
Google Cloud Platform HTTP/HTTPS Load Balancing
GCE's LB product is pretty nice -- HTTP/2 support, and a built-in URL mapping feature (presumably based on how Google approach that problem internally, I understand they take that approach). I'm hoping AWS are taking notes for the next generation of ELB, if that ever happens
elb  gce  google  load-balancing  http  https  spdy  http2  urls  request-routing  ops  architecture  cloud 
october 2015 by jm
fabio
fast, modern, zero-conf load balancing HTTP(S) router managed by consul; serves 15k reqs/sec, in Go, from eBay
load-balancing  consul  http  https  routing  ebay  go  open-source  fabio 
october 2015 by jm
How Netty is used at Layer
pretty conventional HTTP/1.1, WebSockets and HTTP/2 front-end services with modern Netty practices
netty  http  api-services  coding  java  servers 
october 2015 by jm
Designing the Spotify perimeter
How Spotify use nginx as a frontline for their sites and services
scaling  spotify  nginx  ops  architecture  ssl  tls  http  frontline  security 
october 2015 by jm
Baker Street
client-side 'service discovery and routing system for microservices' -- another Smartstack, then
python  router  smartstack  baker-street  microservices  service-discovery  routing  load-balancing  http 
october 2015 by jm
muxy
a proxy that mucks with your system and application context, operating at Layers 4 and 7, allowing you to simulate common failure scenarios from the perspective of an application under test; such as an API or a web application. If you are building a distributed system, Muxy can help you test your resilience and fault tolerance patterns.
proxy  distributed  testing  web  http  fault-tolerance  failure  injection  tcp  delay  resilience  error-handling 
september 2015 by jm
traefik
Træfɪk is a modern HTTP reverse proxy and load balancer made to deploy microservices with ease. It supports several backends (Docker , Mesos/Marathon, Consul, Etcd, Rest API, file...) to manage its configuration automatically and dynamically.


Hot-reloading is notably much easier than with nginx/haproxy.
proxy  http  proxying  reverse-proxy  traefik  go  ops 
september 2015 by jm
httpry
a specialized packet sniffer designed for displaying and logging HTTP traffic. It is not intended to perform analysis itself, but to capture, parse, and log the traffic for later analysis. It can be run in real-time displaying the traffic as it is parsed, or as a daemon process that logs to an output file. It is written to be as lightweight and flexible as possible, so that it can be easily adaptable to different applications.


via Eoin Brazil
via:eoinbrazil  httpry  http  networking  tools  ops  testing  tcpdump  tracing 
september 2015 by jm
httpbin(1): HTTP Client Testing Service
Testing an HTTP Library can become difficult sometimes. RequestBin is fantastic for testing POST requests, but doesn't let you control the response. This exists to cover all kinds of HTTP scenarios. Additional endpoints are being considered.
http  httpbin  networking  testing  web  coding  hacks 
september 2015 by jm
Diffy: Testing services without writing tests
Play requests against 2 versions of a service. A fair bit more complex than simply replaying logged requests, which took 10 lines of a shell script last time I did it
http  testing  thrift  automation  twitter  diffy  diff  soa  tests 
september 2015 by jm
toxy
toxy is a fully programmatic and hackable HTTP proxy to simulate server failure scenarios and unexpected network conditions. It was mainly designed for fuzzing/evil testing purposes, when toxy becomes particularly useful to cover fault tolerance and resiliency capabilities of a system, especially in service-oriented architectures, where toxy may act as intermediate proxy among services.

toxy allows you to plug in poisons, optionally filtered by rules, which essentially can intercept and alter the HTTP flow as you need, performing multiple evil actions in the middle of that process, such as limiting the bandwidth, delaying TCP packets, injecting network jitter latency or replying with a custom error or status code.
toxy  proxies  proxy  http  mitm  node.js  soa  network  failures  latency  slowdown  jitter  bandwidth  tcp 
august 2015 by jm
rwasa
our full-featured, high performance, scalable web server designed to compete with the likes of nginx. It has been built from the ground-up with no external library dependencies entirely in x86_64 assembly language, and is the result of many years' experience with high volume web environments. In addition to all of the common things you'd expect a modern web server to do, we also include assembly language function hooks ready-made to facilitate Rapid Web Application Server (in Assembler) development.
assembly  http  performance  https  ssl  x86_64  web  ops  rwasa  tls 
august 2015 by jm
OkHttp
A new HTTP client library for Android and Java, with a lot of nice features:
HTTP/2 and SPDY support allows all requests to the same host to share a socket.

Connection pooling reduces request latency (if SPDY isn’t available).

Transparent GZIP shrinks download sizes.

Response caching avoids the network completely for repeat requests.

OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems. If your service has multiple IP addresses OkHttp will attempt alternate addresses if the first connect fails. This is necessary for IPv4+IPv6 and for services hosted in redundant data centers. OkHttp initiates new connections with modern TLS features (SNI, ALPN), and falls back to TLS 1.0 if the handshake fails.

Using OkHttp is easy. Its 2.0 API is designed with fluent builders and immutability. It supports both synchronous blocking calls and async calls with callbacks.
android  http  java  libraries  okhttp  http2  spdy  microservices  jdk 
july 2015 by jm
Patrick Shuff - Building A Billion User Load Balancer - SCALE 13x - YouTube
'Want to learn how Facebook scales their load balancing infrastructure to support more than 1.3 billion users? We will be revealing the technologies and methods we use to route and balance Facebook's traffic. The Traffic team at Facebook has built several systems for managing and balancing our site traffic, including both a DNS load balancer and a software load balancer capable of handling several protocols. This talk will focus on these technologies and how they have helped improve user performance, manage capacity, and increase reliability.'

Can't find the standalone slides, unfortunately.
facebook  video  talks  lbs  load-balancing  http  https  scalability  scale  linux 
june 2015 by jm
Improving testing by using real traffic from production
Gor, a very nice-looking tool to log and replay HTTP traffic, specifically designed to "tee" live traffic from production to staging for pre-release testing
gor  performance  testing  http  tcp  packet-capture  tests  staging  tee 
june 2015 by jm
Automated Nginx Reverse Proxy for Docker
Nice hack. An automated nginx reverse proxy which regenerates as the Docker containers update
nginx  reverse-proxy  proxies  web  http  ops  docker 
june 2015 by jm
Apple to switch APNS protocol to HTTP/2
This is great news -- the current protocol is a binary, proprietary horrorshow, particularly around error reporting. Available "later this year" in production, and Pushy plan to support it.
http2  apns  pushy  apple  push-notifications  protocols  http 
june 2015 by jm
HTTP/2 is here, let's optimize! - Velocity SC 2015 - Google Slides
Changes which server-side developers will need to start considering as HTTP/2 rolls out. Remove domain sharding; stop concatenating resources; stop inlining resources; use server push.
http2  http  protocols  streaming  internet  web  dns  performance 
june 2015 by jm
murbul comments on The security issue of Blockchain.info's Android Wallet is not about system's entropy. It's their own BUGs on PRNG again!
I was in the middle of writing a breakdown of what went wrong, but you've beat me to it.
Basically, they have a LinuxSecureRandom class that's supposed to override the standard SecureRandom. This class reads from /dev/urandom and should provide cryptographically secure random values.
They also seed the generator using SecureRandom#setSeed with data pulled from random.org. With their custom SecureRandom, this is safe because it mixes the entropy using XOR, so even if the random.org data is dodgy it won't reduce security. It's just an added bonus.
BUT! On some devices under some circumstances, the LinuxSecureRandom class doesn't get registered. This is likely because /dev/urandom doesn't exist or can't be accessed for some reason. Instead of screaming bloody murder like any sensible implementation would, they just ignore that and fall back to using the standard SecureRandom.
If the above happens, there's a problem because the default implementation of SecureRandom#setSeed doesn't mix. If you set the seed, it replaces the entropy entirely. So now the entropy is coming solely from random.org.
And the final mistake: They were using HTTP instead of HTTPS to make the webservice call to random.org. On Jan 4, random.org started enforcing HTTPS and returning a 301 Permanently Moved error for HTTP - see https://www.random.org/news/. So since that date, the entropy has actually been the error message (turned into bytes) instead of the expected 256-bit number. Using that seed, SecureRandom will generate the private key for address 1Bn9ReEocMG1WEW1qYjuDrdFzEFFDCq43F 100% of the time. Ouch. This is around the time that address first appears, so the timeline matches.
I haven't had a thorough look at what they've replaced it with in the latest version, but initial impressions are that it's not ideal. Not disastrous, but not good.


Always check return values; always check HTTP status codes.
bugs  android  fail  securerandom  random  prng  blockchain.info  bitcoin  http  randomness  entropy  error-checking 
may 2015 by jm
Lambda: Bees with Frickin' Laser Beams
a HTTP testing tool in AWS Lambda. nice enough, but still a toy...
lambda  aws  node  javascript  hacks  http  load-testing 
may 2015 by jm
Race conditions on Facebook, DigitalOcean and others
good trick -- exploit eventual consistency and a lack of distributed transactions by launching race-condition-based attacks
attacks  exploits  race-conditions  bugs  eventual-consistency  distributed-transactions  http  facebook  digitalocean  via:aphyr 
april 2015 by jm
(SEC307) Building a DDoS-Resilient Architecture with AWS
good slides on a "web application firewall" proxy service, deployable as an auto-scaling EC2 unit
ec2  aws  ddos  security  resilience  slides  reinvent  firewalls  http  elb 
april 2015 by jm
The official REST Proxy for Kafka
The REST Proxy is an open source HTTP-based proxy for your Kafka cluster. The API supports many interactions with your cluster, including producing and consuming messages and accessing cluster metadata such as the set of topics and mapping of partitions to brokers. Just as with Kafka, it can work with arbitrary binary data, but also includes first-class support for Avro and integrates well with Confluent’s Schema Registry. And it is scalable, designed to be deployed in clusters and work with a variety of load balancing solutions.

We built the REST Proxy first and foremost to meet the growing demands of many organizations that want to use Kafka, but also want more freedom to select languages beyond those for which stable native clients exist today. However, it also includes functionality beyond traditional clients, making it useful for building tools for managing your Kafka cluster. See the documentation for a more detailed description of the included features.
kafka  rest  proxies  http  confluent  queues  messaging  streams  architecture 
march 2015 by jm
demonstration of the importance of server-side request timeouts
from MongoDB, but similar issues often apply in many other TCP/HTTP-based systems
tcp  http  requests  timeout  mongodb  reliability  safety 
march 2015 by jm
grpc.io
Binary message marshalling, client/server stubs generated by an IDL compiler, bidirectional binary protocol. CORBA is back from the dead!
Intro blog post: http://googledevelopers.blogspot.ie/2015/02/introducing-grpc-new-open-source-http2.html

Relevant: Steve Vinoski's commentary on protobuf-rpc back in 2008: http://steve.vinoski.net/blog/2008/07/13/protocol-buffers-leaky-rpc/
http  rpc  http2  netty  grpc  google  corba  idl  messaging 
february 2015 by jm
Can HTTP/2 Replace MQTT?
MQTT definitely has a smaller size on the wire. It’s also simpler to parse (let’s face it, Huffman isn’t that easy to implement) and provides guaranteed delivery to cater to shaky wireless networks. On the other hand, it’s also not terribly extensible. There aren’t a whole lot of headers and options available, and there’s no way to make custom ones without touching the payload of the message.

It seems that HTTP/2 could definitely serve as a reasonable replacement for MQTT. It’s reasonably small, supports multiple paradigms (pub/sub & request/response) and is extensible. Its also supported by the IETF (whereas MQTT is hosted by OASIS). From conversations I’ve had with industry leaders in the embedded software and chip manufacturing, they only want to support standards from the IETF. Many of them are still planning to support MQTT, but they’re not happy about it.

I think MQTT is better at many of the things it was designed for, but I’m interested to see over time if those advantages are enough to outweigh the benefits of HTTP. Regardless, MQTT has been gaining a lot of traction in the past year or two, so you may be forced into using it while HTTP/2 catches up.
http2  mqtt  iot  pub-sub  protocols  ietf  embedded  push  http 
february 2015 by jm
Vaurien, the Chaos TCP Proxy — Vaurien 1.8 documentation
Vaurien is basically a Chaos Monkey for your TCP connections. Vaurien acts as a proxy between your application and any backend. You can use it in your functional tests or even on a real deployment through the command-line.

Vaurien is a TCP proxy that simply reads data sent to it and pass it to a backend, and vice-versa. It has built-in protocols: TCP, HTTP, Redis & Memcache. The TCP protocol is the default one and just sucks data on both sides and pass it along.

Having higher-level protocols is mandatory in some cases, when Vaurien needs to read a specific amount of data in the sockets, or when you need to be aware of the kind of response you’re waiting for, and so on.

Vaurien also has behaviors. A behavior is a class that’s going to be invoked everytime Vaurien proxies a request. That’s how you can impact the behavior of the proxy. For instance, adding a delay or degrading the response can be implemented in a behavior.

Both protocols and behaviors are plugins, allowing you to extend Vaurien by adding new ones.

Last (but not least), Vaurien provides a couple of APIs you can use to change the behavior of the proxy live. That’s handy when you are doing functional tests against your server: you can for instance start to add big delays and see how your web application reacts.
proxy  tcp  vaurien  chaos-monkey  testing  functional-testing  failures  sockets  redis  memcache  http 
february 2015 by jm
Kimono
'Turn websites into structured APIs from your browser in seconds' -- next-generation web scraping, recommended by conoro
via:conoro  scraping  web  http  kimono  rss  json  csv  data 
january 2015 by jm
wrk2
'A constant throughput, correct latency-recording variant of wrk. This is a must-have when measuring network service latency -- corrects for Coordinated Omission error:
wrk's model, which is similar to the model found in many current load generators, computes the latency for a given request as the time from the sending of the first byte of the request to the time the complete response was received. While this model correctly measures the actual completion time of individual requests, it exhibits a strong Coordinated Omission effect, through which most of the high latency artifacts exhibited by the measured server will be ignored. Since each connection will only begin to send a request after receiving a response, high latency responses result in the load generator coordinating with the server to avoid measurement during high latency periods.
wrk  latency  measurement  tools  cli  http  load-testing  testing  load-generation  coordinated-omission  gil-tene 
november 2014 by jm
Introducing Proxygen, Facebook's C++ HTTP framework
Facebook's take on libevent, I guess:
We are excited to announce the release of Proxygen, a collection of C++ HTTP libraries, including an easy-to-use HTTP server. In addition to HTTP/1.1, Proxygen (rhymes with "oxygen") supports SPDY/3 and SPDY/3.1. We are also iterating and developing support for HTTP/2.

Proxygen is not designed to replace Apache or nginx — those projects focus on building extremely flexible HTTP servers written in C that offer good performance but almost overwhelming amounts of configurability. Instead, we focused on building a high performance C++ HTTP framework with sensible defaults that includes both server and client code and that's easy to integrate into existing applications. We want to help more people build and deploy high performance C++ HTTP services, and we believe that Proxygen is a great framework to do so.
c++  facebook  http  servers  libevent  https  spdy  proxygen  libraries 
november 2014 by jm
Most page loads will experience the 99th percentile response latency
MOST of the page view attempts will experience the 99%'lie server response time in modern web applications. You didn't read that wrong.
latency  metrics  percentiles  p99  web  http  soa 
october 2014 by jm
Vodafone UK, Verizon add mandatory device-tracking token on all web requests
'Verizon Wireless is monitoring users' mobile internet traffic, using a token slapped onto web requests, to facilitate targeted advertising even if a user has opted out.

The unique identifier token header (UIDH) was launched two years ago, and has caused an uproar in tech circles after it was re-discovered Thursday by Electronic Frontier Foundation staffer Jacob Hoffman-Andrews.

The Relevant Mobile Advertising program, under which the UIDH was used, allowed a restaurant to advertised to locals only or for retail websites to promote to previous visitors, according to Verizon Wireless.'
uidh  verizon  vodafone  privacy  tracking  http  cookies  advertising 
october 2014 by jm
Jauter
This Java library can route paths to targets and create paths from targets and params (reverse routing). This library is tiny, without additional dependencies, and is intended for use together with an HTTP server side library. If you want to use with Netty, see netty-router.
java  jauter  scala  request-routing  http  netty  open-source 
october 2014 by jm
JCDecaux Developer API
web service API for Dublin Bikes data (and other similar bikesharing services run by JCD):
Two kinds of data are delivered by the platform:

Static data provides stable information like station position, number of bike stands, payment terminal availability, etc.
Dynamic data provides station state, number of available bikes, number of free bike stands, etc.
Static data can be downloaded manually in file format or accessed through the API. Dynamic data are refreshed every minute and can be accessed only through the API.


Ruby API: https://github.com/oisin/bikes
jcdecaux  bikesharing  dublin  dublin-bikes  api  web-services  http  json  open-data 
october 2014 by jm
Axel
A nice curl/wget replacement which supports multi-TCP-connection downloads of HTTP/FTP resources. packaged for most Linux variants and OSX via brew
axel  curl  wget  via:johnke  downloading  tcp  http  ftp  ubuntu  debian  unix  linux 
september 2014 by jm
Comcast Wi-Fi serving self-promotional ads via JavaScript injection | Ars Technica
Comcast is adding data into the broadband packet stream. In 2007, it was packets serving up disconnection commands. Today, Comcast is inserting JavaScript that is serving up advertisements, according to [Robb] Topolski, who reviewed Singel's data. "It's the duty of the service provider to pull packets without treating them or modifying them or injecting stuff or forging packets. None of that should be in the province of the service provider," he said. "Imagine every Web page with a Comcast bug in the lower righthand corner. It's the antithesis of what a service provider is supposed to do. We want Internet access, not another version of cable TV."


The company appears to be called Front Porch: http://arstechnica.com/tech-policy/2014/09/meet-the-tech-company-performing-ad-injections-for-big-cable/
comcast  ads  injection  security  javascript  http  network-neutrality  isps 
september 2014 by jm
UK piracy police arrest man suspected of running proxy server (Wired UK)
The site, Immunicity.org, offers a proxy server and a proxy autoconfiguration file (PAC) to tell browsers to access various blocked sites (PirateBay, KickassTorrents et al) via the proxy.
The Police Intellectual Property Crime Unit has arrested a 20-year-old man in Nottingham on suspicion of copyright infringement for running a proxy server providing access to other sites subject to legal blocking orders.


Is operating a proxy server illegal? Interesting. Seems unlikely that this will go to court though.

(Via TJ McIntyre)
immunicity  via:tjmcintyre  police  uk  piracy  proxies  http  pac  pipcu  copyright 
august 2014 by jm
How to take over the computer of any JVM developer
To prove how easy [MITM attacking Mavencentral JARs] is to do, I wrote dilettante, a man-in-the-middle proxy that intercepts JARs from maven central and injects malicious code into them. Proxying HTTP traffic through dilettante will backdoor any JARs downloaded from maven central. The backdoored version will retain their functionality, but display a nice message to the user when they use the library.
jars  dependencies  java  build  clojure  security  mitm  http  proxies  backdoors  scala  maven  gradle 
july 2014 by jm
REST Commander: Scalable Web Server Management and Monitoring
We dynamically monitor and manage a large and rapidly growing number of web servers deployed on our infrastructure and systems. However, existing tools present major challenges when making REST/SOAP calls with server-specific requests to a large number of web servers, and then performing aggregated analysis on the responses. We therefore developed REST Commander, a parallel asynchronous HTTP client as a service to monitor and manage web servers. REST Commander on a single server can send requests to thousands of servers with response aggregation in a matter of seconds. And yes, it is open-sourced at http://www.restcommander.com.

Feature highlights:

Click-to-run with zero installation;
Generic HTTP request template supporting variable-based replacement for sending server-specific requests;
Ability to send the same request to different servers, different requests to different servers, and different requests to the same server;
Maximum concurrency control (throttling) to accommodate server capacity;
Commander itself is also “as a service”: with its powerful REST API, you can define ad-hoc target servers, an HTTP request template, variable replacement, and a regular expression all in a single call. In addition, intuitive step-by-step wizards help you achieve the same functionality through a GUI.
rest  http  clients  load-testing  ebay  soap  async  testing  monitoring 
july 2014 by jm
Netflix/ribbon
a client side IPC library that is battle-tested in cloud. It provides the following features:

Load balancing;
Fault tolerance;
Multiple protocol (HTTP, TCP, UDP) support in an asynchronous and reactive model;
Caching and batching.

I like the integration of Eureka and Hystrix in particular, although I would really like to read more about Eureka's approach to availability during network partitions and CAP.

https://groups.google.com/d/msg/eureka_netflix/LXKWoD14RFY/-5nElGl1OQ0J has some interesting discussion on the topic. It actually sounds like the Eureka approach is more correct than using ZK: 'Eureka is available. ZooKeeper, while tolerant against single node failures, doesn't react well to long partitioning events. For us, it's vastly more important that we maintain an available registry than a necessary consistent registry. If us-east-1d sees 23 nodes, and us-east-1c sees 22 nodes for a little bit, that's OK with us.'

See also http://ispyker.blogspot.ie/2013/12/zookeeper-as-cloud-native-service.html which corroborates this:

I went into one of the instances and quickly did an iptables DROP on all packets coming from the other two instances. This would simulate an availability zone continuing to function, but that zone losing network connectivity to the other availability zones. What I saw was that the two other instances noticed that the first server “going away”, but they continued to function as they still saw a majority (66%). More interestingly the first instance noticed the other two servers “going away” dropping the ensemble availability to 33%. This caused the first server to stop serving requests to clients (not only writes, but also reads). [...]

To me this seems like a concern, as network partitions should be considered an event that should be survived. In this case (with this specific configuration of zookeeper) no new clients in that availability zone would be able to register themselves with consumers within the same availability zone. Adding more zookeeper instances to the ensemble wouldn’t help considering a balanced deployment as in this case the availability would always be majority (66%) and non-majority (33%).
netflix  ribbon  availability  libraries  java  hystrix  eureka  aws  ec2  load-balancing  networking  http  tcp  architecture  clients  ipc 
july 2014 by jm
"The Tail at Scale"
by Jeffrey Dean and Luiz Andre Barroso, Google. A selection of Google's architectural mechanisms used to defeat 99th-percentile latency spikes: hedged requests, tied requests, micro-partitioning, selective replication, latency-induced probation, canary requests.
google  architecture  distcomp  soa  http  partitioning  replication  latency  99th-percentile  canary-requests  hedged-requests 
july 2014 by jm
lookout/ngx_borderpatrol
BorderPatrol is an nginx module to perform authentication and session management at the border of your network. BorderPatrol makes the assumption that you have some set of services that require authentication and a service that hands out tokens to clients to access that service. You may not want those tokens to be sent across the internet, even over SSL, for a variety of reasons. To this end, BorderPatrol maintains a lookup table of session-id to auth token in memcached.
borderpatrol  nginx  modules  authentication  session-management  web-services  http  web  authorization 
june 2014 by jm
spoofing the samsung smart tv internet check
If this kind of bullshit -- a HTTP GET of an XML file from www.samsung.com -- is how the Samsung Smart TV firmware decides if the internet is working or not, I dread to think how crappy the rest of the code is. (At least in Netnote we performed a bunch of bigco-domain DNS lookups before giving up...)
smart-tv  samsung  fail  xml  http  internet  embedded-software  firmware  crap-code 
april 2014 by jm
Haywire
kellabyte's hack in progress -- 'an asynchronous HTTP server framework written in C. The goal of Haywire is to learn how to create a server with a minimal feature set that can handle a high rate of requests and connections with as low of latency and resource usage as possible. Haywire uses the event loop based libuv platform layer that node.js is built on top of (also written in C). libuv abstracts IOCP on Windows and epoll/kqueue/event ports/etc. on Unix systems to provide efficient asynchronous I/O on all supported platforms.'

Outperforms libevent handily, it seems. Apache-licensed.
server  http  asynchronous  libuv  haywire  kellabyte  c  events  open-source  asl2 
april 2014 by jm
Consul
Nice-looking new tool from Hashicorp; service discovery and configuration service, built on Raft for leader election, Serf for gossip-based messaging, and Go. Some features:

* Gossip is performed over both TCP and UDP;

* gossip messages are encrypted symmetrically and therefore secure from eavesdropping, tampering, spoofing and packet corruption (like the incident which brought down S3 for days: http://status.aws.amazon.com/s3-20080720.html );

* exposes both a HTTP interface and (even better) DNS;

* includes explicit support for long-distance WAN operation as well as on LANs.

It all looks very practical and usable. MPL-licensed.

The only potential risk I can see is that expecting to receive config updates from a blocking poll of the HTTP interface needs some good "best practice" docs, to ensure that people don't mishandle the scenario where there is a network partition between your calling code and the Consul server/agent. Without any heartbeating protocol behind the scenes, HTTP is vulnerable to "hung connections" which would result in a config change being silently missed by the client until the connection eventually is timed out, either by the calling code or the client-side kernel. This could potentially take minutes to occur, which in some usage scenarios could be a big, unforeseen problem.
configuration  service-discovery  distcomp  raft  consensus-algorithms  go  mpl  open-source  dns  http  gossip-protocol  hashicorp 
april 2014 by jm
Why no SSL ? — Varnish version 4.0.0 documentation
Poul-Henning Kemp details why Varnish doesn't do SSL -- basically due to the quality and complexity of open-source SSL implementations:
There is no other way we can guarantee that secret krypto-bits do not leak anywhere they should not, than by fencing in the code that deals with them in a child process, so the bulk of varnish never gets anywhere near the certificates, not even during a core-dump.


Now looking pretty smart, post-Heartbleed.
ssl  tls  varnish  open-source  poul-henning-kemp  https  http  proxies  security  coding 
april 2014 by jm
Daring Fireball: Rethinking What We Mean by 'Mobile Web'
We shouldn’t think of “the web” as only what renders in web browsers. We should think of the web as anything transmitted using HTTP and HTTPS. Apps and websites are peers, not competitors. They’re all just clients to the same services.

+1. Finally, a Daring Fireball post I agree with.
daring-fireball  apps  web  http  https  mobile  apple  android  browsers 
april 2014 by jm
Issue 122 - android-query - HTTP 204 Response results in Network Error (-101)
an empty 204 response to a HTTP PUT will trigger this. See also https://code.google.com/p/android/issues/detail?id=24672, '"java.io.IOException: unexpected end of stream" on HttpURLConnection HEAD call'.
http  urlconnection  httpurlconnection  java  android  dalvik  bugs  204  head  get  exceptions 
march 2014 by jm
The Microservice Declaration of Independence
"Microservices" seems to be yet another term for SOA; small, decoupled, independently-deployed services, with well-defined public HTTP APIs. Pretty much all the services I've worked on over the past few years have been built in this style. Still, let's keep an eye on this concept anyway.

Another definition seems to be a more FP-style one: http://www.slideshare.net/michaelneale/microservices-and-functional-programming -- where the "microservice" does one narrowly-defined thing, and that alone.
microservices  soa  architecture  handwaving  http  services  web  deployment 
march 2014 by jm
htcat/htcat
a utility to perform parallel, pipelined execution of a single HTTP GET. htcat is intended for the purpose of incantations like: htcat https://host.net/file.tar.gz | tar -zx

It is tuned (and only really useful) for faster interconnects: [....] 109MB/s on a gigabit network, between an AWS EC2 instance and S3. This represents 91% use of the theoretical maximum of gigabit (119.2 MiB/s).
go  cli  http  file-transfer  ops  tools 
march 2014 by jm
Good explanation of exponential backoff
I've often had to explain this key feature verbosely, and it's hard to do without handwaving. Great to have a solid, well-explained URL to point to
exponential-backoff  backoff  retries  reliability  web-services  http  networking  internet  coding  design 
march 2014 by jm
"Dapper, a Large-Scale Distributed Systems Tracing Infrastructure" [PDF]
Google paper describing the infrastructure they've built for cross-service request tracing (ie. "tracer requests"). Features: low code changes required (since they've built it into the internal protobuf libs), low performance impact, sampling, deployment across the ~entire production fleet, output visibility in minutes, and has been live in production for over 2 years. Excellent read
dapper  tracing  http  services  soa  google  papers  request-tracing  tracers  protobuf  devops 
march 2014 by jm
Traffic Graph – Google Transparency Report
this is cool. Google are exposing an aggregated 'all services' hit count time-series graph, broken down by country, as part of their Transparency Report pages
transparency  filtering  web  google  http  graphs  monitoring  syria 
february 2014 by jm
Video Processing at Dropbox
On-the-fly video transcoding during live streaming. They've done a great job of this!
At the beginning of the development of this feature, we entertained the idea to simply pre-transcode all the videos in Dropbox to all possible target devices. Soon enough we realized that this simple approach would be too expensive at our scale, so we decided to build a system that allows us to trigger a transcoding process only upon user request and cache the results for subsequent fetches. This on-demand approach: adapts to heterogeneous devices and network conditions, is relatively cheap (everything is relative at our scale), guarantees low latency startup time.
ffmpeg  dropbox  streaming  video  cdn  ec2  hls  http  mp4  nginx  haproxy  aws  h264 
february 2014 by jm
CJEU in #Svensson says that in general it is OK to hyperlink to protected works without permission
IPKat says 'this morning the Court of Justice of the European Union issued its keenly awaited decision in Case C-466/12 Svensson [...]: The owner of a website may, without the authorisation of the copyright holders, redirect internet users, via hyperlinks, to protected works available on a freely accessible basis on another site. This is so even if the internet users who click on the link have the impression that the work is appearing on the site that contains the link.'

This is potentially big news. Not so much for the torrent-site scenario, but for the NNI/NLI linking-to-newspaper-stories scenario.
ip  svensson  cjeu  eu  law  linking  hyperlinks  pirate-bay  internet  web  links  http  copyright 
february 2014 by jm
Home · linkedin/rest.li Wiki
Rest.li is a REST+JSON framework for building robust, scalable service architectures using dynamic discovery and simple asynchronous APIs. Rest.li fills a niche for building RESTful service architectures at scale, offering a developer workflow for defining data and REST APIs that promotes uniform interfaces, consistent data modeling, type-safety, and compatibility checked API evolution.


The new underlying comms layer for Voldemort, it seems.
voldemort  d2  rest.li  linkedin  json  rest  http  api  frameworks  java 
february 2014 by jm
Sky parental controls break many JQuery-using websites
An 11 hour outage caused by a false positive in Sky's anti-phishing filter; all sites using the code.jquery.com CDN for JQuery would have seen errors.
Sky still appears to be blocking code.jquery.com and all files served via the site, and more worryingly is that if you try to report the incorrect category, once signing in on the Sky website you an error page. We suspect the site was blocked due to being linked to by a properly malicious website, i.e. code.jquery.com and some javascript files were being used on a dodgy website and every domain mentioned was subsequently added to a block list.


(via Tony Finch)
via:fanf  sky  filtering  internet  uk  anti-phishing  phish  jquery  javascript  http  web  fps  false-positives 
january 2014 by jm
UK porn filter blocks game update that contained 'sex' in URL
Staggeringly inept. The UK national porn filter blocks based on a regexp match of the URL against /.*sex.*/i -- the good old "Scunthorpe problem". Better, it returns a 404 response. This is also a good demonstration of how web filtering has unintended side effects, breaking third-party software updates with its false positives.
The update to online strategy game League of Legends was disrupted by the internet filter because the software attempted to access files that accidentally include the word “sex” in the middle of their file names. The block resulted in the update failing with “file not found” errors, which are usually created by missing files or broken updates on the part of the developers.
uk  porn  filtering  guardian  regular-expressions  false-positives  scunthorpe  http  web  league-of-legends  sex 
january 2014 by jm
Chartbeat's Lessons learned tuning TCP and Nginx in EC2
a good writeup of basic sysctl tuning for an internet-facing HTTP proxy fleet running in EC2. Nothing groundbreaking here, but it's well-written
nginx  amazon  ec2  tcp  ip  tuning  sysctl  linux  c10k  ssl  http 
january 2014 by jm
websocketd
'like inetd, but for WebSockets' -- 'a small command line tool that will wrap an existing command line interface program, and allow it to be accessed via a WebSocket. It provides a quick mechanism for allowing web-applications to interact with existing command line tools.'

Awesome idea. BSD-licensed. (Via Mike Loukides)
websockets  cli  server  tools  unix  inetd  web  http  open-source 
december 2013 by jm
wrk
a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU. It combines a multithreaded design with scalable event notification systems such as epoll and kqueue.  An optional LuaJIT script can perform HTTP request generation, response processing, and custom reporting.


Written in C, ASL2 licensed.
wrk  benchmarking  http  performance  testing  lua  load-testing  load-generation 
december 2013 by jm
« earlier      
per page:    204080120160

related tags

3g  4g  99th-percentile  ack  ad-trackers  ads  advertising  agpl  alarming  alerting  amazon  android  anti-phishing  apache  api  api-gateway  api-services  apis  apns  apple  apps  archaius  architecture  archive  archives  asl2  assembly  async  asynchronous  atmosphere  atom  attacks  audio  australia  authentication  authorization  automation  availability  aws  axel  backbone  backdoors  backoff  baker-street  bandwidth  bbc  benchmarking  beta  bikesharing  bitcoin  blockchain.info  bloom-cookies  bloom-filters  book  borderpatrol  browser  browsers  bufferbloat  buffering  bugs  build  c  c++  c10k  c10m  callbacks  canary-requests  capture  captures  capturing  cask  cdn  chaos-monkey  chrome  chunked-encoding  circuit-breakers  cjeu  cli  clients  clojure  closures  cloud  cloudflare  clowns  cnbc  codecs  codel  coding  comcast  comet  compression  configuration  confluent  consensus-algorithms  consul  conversion  cookies  coordinated-omission  copyright  corba  crap-code  csv  curl  customer-care  d2  dalvik  dapper  daring-fireball  data  data-structures  datawire  dates  daylight-savings  ddos  debian  debugging  delay  demos  dependencies  deployment  design  devops  diff  diffy  digitalocean  distcomp  distributed  distributed-systems  distributed-transactions  dns  docker  dot-net  download  downloading  dropbox  dropwizard  dst  dublin  dublin-bikes  ebay  ebooks  ec2  edge-services  elasticsearch  elb  embedded  embedded-software  entropy  epoll  error-checking  error-handling  eu  eureka  event  events  eventual-consistency  exceptions  exploits  exponential-backoff  fabio  facebook  fail  failure  failures  false-positives  fastmail  fault-tolerance  feeds  ffmpeg  file-transfer  files  filtering  firewalls  firmware  fps  fq_codel  framework  frameworks  friendfeed  frontline  ftp  functional-testing  funny  games  gce  gchq  get  gil-tene  github  go  golang  google  gor  gossip-protocol  gradle  graphs  groovy  grpc  guardian  guava  h264  hacking  hacks  handwaving  haproxy  har  hardware  hashicorp  haywire  head  hedged-requests  hls  hosting  hsdpa  http  http-push  http2  httpbin  httpry  https  httpurlconnection  humor  humour  hyperlinks  hystrix  idl  ids  ietf  ilya-grigorik  immunicity  inept  inetd  infrastructure  injection  internet  iot  ip  ipc  iso8601  isps  jackson  jars  jauter  java  javascript  jax-rs  jcdecaux  jdk  jersey  jetty  jim-gettys  jitter  jmeter  jmx  jokes  jquery  json  jvm  kafka  kellabyte  kimono  lambda  latency  law  lbs  league-of-legends  leaks  left-pad  leonard-richardson  libevent  libraries  libuv  linkedin  linking  links  linux  live  load-balancers  load-balancing  load-generation  load-testing  local  logging  long-poll  long-polling  lua  mail.ru  martin-fowler  martin-thompson  maven  measurement  mechanical-sympathy  memcache  messaging  metrics  microservices  mirror  mitm  mobile  mocking  modules  mongodb  monitoring  monkey-patching  mp3  mp4  mpl  mqtt  music  nagle  nchan  netflix  netty  network  network-neutrality  networking  nginx  node  node.js  npm  nsa  oauth  okhttp  open-data  open-source  opensource  ops  opsgenie  oscon  p99  pac  packages  packet-capture  packet-injection  packets  papers  partitioning  passwords  percentiles  performance  personalization  phish  phones  ping  pipcu  piracy  pirate-bay  players  pokemon  pokemon-go  police  politics  porn  poul-henning-kemp  pound  presentations  privacy  prng  production  protobuf  protocols  proxies  proxy  proxygen  proxying  pub-sub  pubsub  pubsubhubbub  push  push-notifications  pushpin  pushy  python  queues  quora  race-conditions  raft  random  randomness  realtime  recording  redis  reference  regular-expressions  reinvent  reliability  replication  request-routing  request-tracing  requests  resilience  rest  rest.li  restful  retries  reverse-proxy  reversing  rfc  ribbon  round-trip  router  routing  rpc  rss  rtt  ruby  rwasa  saas  safety  samsung  scala  scalability  scale  scaling  scraping  scunthorpe  search  securerandom  security  server  servers  service-discovery  services  session-management  sessions  sex  shopping  signalfx  sky  slides  slow-start  slowdown  smart-tv  smartstack  snabb-switch  sniffing  snooping  snowflake  soa  soap  sockets  software  soundcloud  spdy  speed  spotify  sse  ssh  ssl  stack  staging  standards  streaming  streams  strings  supertweet  surveillance  svensson  sync  syria  sysctl  talks  tcp  tcpdump  tee  tesco  testing  tests  thrift  timeout  times  timezones  tls  tools  tornado  toxy  trace  tracer-requests  tracers  tracing  tracking  traefik  transparency  transport  transports  tuning  tunneling  tutorials  twitter  ubuntu  udp  ui  uidh  uk  unit-tests  unix  url  urlconnection  urllib  urls  usability  user-agent  user-tracking  varnish  vaurien  verizon  via:aphyr  via:conoro  via:eoin-brazil  via:eoinbrazil  via:fanf  via:jacob  via:johnke  via:mikeste  via:pixelbeat  via:tjmcintyre  video  vodafone  voldemort  web  web-services  webdesign  webdev  webhooks  webrtc  websites  websockets  wget  wrk  wtf  x86_64  xhr  xml  yammer  zipkin  zuul 

Copy this bookmark:



description:


tags: