jm + incast   2

TCP incast
a catastrophic TCP throughput collapse that occurs as the number of storage servers sending data to a client increases past the ability of an Ethernet switch to buffer packets. In a clustered file system, for example, a client application requests a data block striped across several storage servers, issuing the next data block request only when all servers have responded with their portion (Figure 1). This synchronized request workload can result in packets overfilling the buffers on the client's port on the switch, resulting in many losses. Under severe packet loss, TCP can experience a timeout that lasts a minimum of 200ms, determined by the TCP minimum retransmission timeout (RTOmin).
incast  networking  performance  tcp  bandwidth  buffering  switch  ethernet  capacity 
november 2014 by jm
TCP incast vs Riak
An extremely congested local network segment causes the "TCP incast" throughput collapse problem -- packet loss occurs, and TCP throughput collapses as a side effect. So far, this is pretty unsurprising, and anyone designing a service needs to keep bandwidth requirements in mind.

However it gets worse with Riak. Due to a bug, this becomes a serious issue for all clients: the Erlang network distribution port buffers fill up in turn, and the Riak KV vnode process (in its entirety) will be descheduled and 'cannot answer any more queries until the A-to-B network link becomes uncongested.'

This is where EC2's fully-uncontended-1:1-network compute cluster instances come in handy, btw. ;)
incast  tcp  networking  bandwidth  riak  architecture  erlang  buffering  queueing 
february 2014 by jm

Copy this bookmark: