jm + realtime   29

Determinism in League of Legends
Once again, deterministic replay/reruns of online games proves useful. John Carmack wrote a .plan about this many years ago:

(via Nelson)
clock  realtime  time  determinism  testing  replay  games  league-of-legends  via:nelson 
4 weeks ago by jm
TIL: clock skew exists
good roundup of real-world clock skew links
clocks  clock-skew  ntp  realtime  time  bugs  distcomp  reliability  skew 
february 2016 by jm
WebSockets, caution required!
This, so much.
There are very valid technical reasons many of the biggest sites on the Internet have not adopted them. Twitter use HTTP/2 + polling, Facebook and Gmail use Long Polling. Saying WebSockets are the only way and the way of the future, is wrongheaded. HTTP/2 may end up winning this battle due to the huge amount of WebSocket connections web browsers allow, and HTTP/3 may unify the protocols
http  realtime  websockets  long-polling  http2  protocols  transport  web  internet 
january 2016 by jm
Scaling Analytics at Amplitude
Good blog post on Amplitude's lambda architecture setup, based on S3 and a custom "real-time set database" they wrote themselves.

antirez' comment from a Redis angle on the set database:

HN thread:
lambda-architecture  analytics  via:hn  redis  set-storage  storage  databases  architecture  s3  realtime 
august 2015 by jm
Adrian Colyer reviews the Twitter Heron paper
ouch, really sounds like Storm didn't cut the muster. 'It’s hard to imagine something more damaging to Apache Storm than this. Having read it through, I’m left with the impression that the paper might as well have been titled “Why Storm Sucks”, which coming from Twitter themselves is quite a statement.'

If I was to summarise the lessons learned, it sounds like: backpressure is required; and multi-tenant architectures suck.

Update: response from Storm dev ptgoetz here:
storm  twitter  heron  big-data  streaming  realtime  backpressure 
june 2015 by jm
Twitter ditches Storm
in favour of a proprietary ground-up rewrite called Heron. Reading between the lines it sounds like Storm had problems with latency, reliability, data loss, and supporting back pressure.
analytics  architecture  twitter  storm  heron  backpressure  streaming  realtime  queueing 
june 2015 by jm
A Case Study of Toyota Unintended Acceleration and Software Safety
I drive a Toyota, and this is scary stuff. Critical software systems need to be coded with care, and this isn't it -- they don't even have a bug tracking system!
Investigations into potential causes of Unintended Acceleration (UA) for Toyota vehicles have made news several times in the past few years. Some blame has been placed on floor mats and sticky throttle pedals. But, a jury trial verdict was based on expert opinions that defects in Toyota's Electronic Throttle Control System (ETCS) software and safety architecture caused a fatal mishap.  This talk will outline key events in the still-ongoing Toyota UA litigation process, and pull together the technical issues that were discovered by NASA and other experts. The results paint a picture that should inform future designers of safety critical software in automobiles and other systems.
toyota  safety  realtime  coding  etcs  throttle-control  nasa  code-review  embedded 
january 2015 by jm
"Aeron: High-Performance Open Source Message Transport" [slides, PDF]
a new networked pub/sub library from Martin "Disruptor" Thompson, based around a replicated, persistent log of messages, with exceptionally low latency. Apache-licensed. Very similar to the realtime messaging stack we've built in Swrve. ;)
realtime  messaging  pub-sub  ipc  queues  transports  martin-thompson  slides  latencies  open-source  java  libraries 
november 2014 by jm
Announcing Confluent, A Company for Apache Kafka And Realtime Data
Jay Kreps, Neha Narkhede, and Jun Rao are leaving LinkedIn to form a Kafka-oriented realtime event processing company
realtime  event-processing  logs  kafka  streaming  open-source  jay-kreps  jun-rao  confluent 
november 2014 by jm
Integrating Kafka and Spark Streaming: Code Examples and State of the Game
Spark Streaming has been getting some attention lately as a real-time data processing tool, often mentioned alongside Apache Storm. [...] I added an example Spark Streaming application to kafka-storm-starter that demonstrates how to read from Kafka and write to Kafka, using Avro as the data format and Twitter Bijection for handling the data serialization. In this post I will explain this Spark Streaming example in further detail and also shed some light on the current state of Kafka integration in Spark Streaming. All this with the disclaimer that this happens to be my first experiment with Spark Streaming.
spark  kafka  realtime  architecture  queues  avro  bijection  batch-processing 
october 2014 by jm
"Left-Right: A Concurrency Control Technique with Wait-Free Population Oblivious Reads" [pdf]
'In this paper, we describe a generic concurrency control technique with Blocking write operations and Wait-Free Population Oblivious read operations, which we named the Left-Right technique. It is of particular interest for real-time applications with dedicated Reader threads, due to its wait-free property that gives strong latency guarantees and, in addition, there is no need for automatic Garbage Collection.
The Left-Right pattern can be applied to any data structure, allowing concurrent access to it similarly to a Reader-Writer lock, but in a non-blocking manner for reads. We present several variations of the Left-Right technique, with different versioning mechanisms and state machines. In addition, we constructed an optimistic approach that can reduce synchronization for reads.'

See also for java implementation code.
left-right  concurrency  multithreading  wait-free  blocking  realtime  gc  latency  reader-writer  locking  synchronization  java 
september 2014 by jm
Fighting spam with BotMaker
Some vague details of the antispam system in use at Twitter.
The main challenges in supporting this type of system are evaluating rules with low enough latency that they can run on the write path for Twitter’s main features (i.e., Tweets, Retweets, favorites, follows and messages), supporting computationally intense machine learning based rules, and providing Twitter engineers with the ability to modify and create new rules instantaneously.
spam  realtime  scaling  twitter  anti-spam  botmaker  rules 
august 2014 by jm
Real time analytics with Netty, Storm, Kafka
Arch of a fairly typical Kafka/Storm realtime ad-tracking setup, from eClick/mc2ads, via Trustin Lee
via:trustinlee  kafka  storm  netty  architecture  ad-tracking  ads  realtime 
august 2014 by jm
Why Disqus made the Python->Go switchover
for their realtime component, from the horse's mouth:
at higher contention, the CPU was choking everything. Switching over to Go removed that contention for us, which was the primary issue that we were seeing.
python  languages  concurrency  go  threading  gevent  scalability  disqus  realtime  hn 
may 2014 by jm
Scaling Realtime at DISQUS
Disqus' realtime architecture -- nginx PushStream module doing the heavy lifting, basically. See for the production nginx configs they use. I am very impressed that push-stream has grown to be so solid; it's a great way to deal with push from the sounds of it. now notes that some of the realtime backends are in Go. ("C1M and Nginx") is a more up to date presentation. It notes that PushStream supports "EventSource, WebSocket, Long Polling, and forever iframe". More sysctls and nginx tuning in that prez.
sysctl  nginx  tuning  go  disqus  realtime  push  eventsource  websockets  long-polling  iframe  python 
april 2014 by jm
SAMOA, an open source platform for mining big data streams
Yahoo!'s streaming machine learning platform, built on Storm, implementing:

As a library, SAMOA contains state-of-the-art implementations of algorithms for distributed machine learning on streams. The first alpha release allows classification and clustering. For classification, we implemented a Vertical Hoeffding Tree (VHT), a distributed streaming version of decision trees tailored for sparse data (e.g., text). For clustering, we included a distributed algorithm based on CluStream. The library also includes meta-algorithms such as bagging.
storm  streaming  big-data  realtime  samoa  yahoo  machine-learning  ml  decision-trees  clustering  bagging  classification 
november 2013 by jm
a realtime processing engine, built on a persistent queue and a set of workers. 'The main goal is data availability and persistency. We created grape for those who cannot afford losing data'. It does this by allowing infinite expansion of the pending queue in Elliptics, their Dynamo-like horizontally-scaled storage backend.
kafka  queue  queueing  storage  realtime  fault-tolerance  grape  cep  event-processing 
november 2013 by jm
Toyota's killer firmware: Bad design and its consequences
This is exactly what you do NOT want to read about embedded systems controlling acceleration in your car:

The Camry electronic throttle control system code was found to have 11,000 global variables. Barr described the code as “spaghetti.” Using the Cyclomatic Complexity metric, 67 functions were rated untestable (meaning they scored more than 50). The throttle angle function scored more than 100 (unmaintainable).
Toyota loosely followed the widely adopted MISRA-C coding rules but Barr’s group found 80,000 rule violations. Toyota's own internal standards make use of only 11 MISRA-C rules, and five of those were violated in the actual code. MISRA-C:1998, in effect when the code was originally written, has 93 required and 34 advisory rules. Toyota nailed six of them. Barr also discovered inadequate and untracked peer code reviews and the absence of any bug-tracking system at Toyota.

On top of this, there was no error-correcting RAM in use; stack-killing recursive code; a quoted 94% stack usage; risks of unintentional RTOS task shutdown; buffer overflows; unsafe casting; race conditions; unchecked error code return values; and a trivial watchdog timer check. Crappy, unsafe coding.
firmware  horror  embedded-systems  toyota  camry  safety  acceleration  misra-c  coding  code-verification  spaghetti-code  cyclomatic-complexity  realtime  rtos  c  code-reviews  bug-tracking  quality 
october 2013 by jm
Storm at - London Storm Meetup 2013-06-18
Not just a Storm success story. Interesting slides indicating where a startup *stopped* using Storm as realtime wasn't useful to their customers
storm  realtime  hadoop  cascading  python  cep  anti-spam  events  architecture  distcomp  low-latency  slides  rabbitmq 
october 2013 by jm
Facebook announce Wormhole
Over the last couple of years, we have built and deployed a reliable publish-subscribe system called Wormhole. Wormhole has become a critical part of Facebook's software infrastructure. At a high level, Wormhole propagates changes issued in one system to all systems that need to reflect those changes – within and across data centers.

Facebook's Kafka-alike, basically, although with some additional low-latency guarantees. FB appear to be using it for multi-region and multi-AZ replication. Proprietary.
pub-sub  scalability  facebook  realtime  low-latency  multi-region  replication  multi-az  wormhole 
june 2013 by jm
Google Drive SDK
realtime collaboration API. nifty! but can it collaborate on a per-app shared doc, or does it require that the app user auth to Google and access their own docs?
collaboration  api  realtime  google  javascript 
march 2013 by jm
Trident: a high-level abstraction for realtime computation
built on Storm:

Trident is a new high-level abstraction for doing realtime computing on top of Twitter Storm, available in Storm 0.8.0. It allows you to seamlessly mix high throughput (millions of messages per second), stateful stream processing with low latency distributed querying. If you're familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar - Trident has joins, aggregations, grouping, functions, and filters. In addition to these, Trident adds primitives for doing stateful, incremental processing on top of any database or persistence store. Trident has consistent, exactly-once semantics, so it is easy to reason about Trident topologies.
distributed  realtime  twitter  storm  trident  distcomp  stream-processing  low-latency  nathan-marz 
october 2012 by jm
How to beat the CAP theorem
Nathan "Storm" Marz on building a dual realtime/batch stack. This lines up with something I've been building in work, so I'm happy ;)
nathan-marz  realtime  batch  hadoop  storm  big-data  cap 
october 2011 by jm
Storm: distributed and fault-tolerant realtime computation
intro slideshow to this really nifty-looking distcomp platform
distcomp  distributed  realtime  storm  slides  twitter 
september 2011 by jm
'The past decade has seen a revolution in data processing. MapReduce, Hadoop, and related technologies have made it possible to store and process data at scales previously unthinkable. Unfortunately, these data processing technologies are not realtime systems, nor are they meant to be. There's no hack that will turn Hadoop into a realtime system; realtime data processing has a fundamentally different set of requirements than batch processing.

However, realtime data processing at massive scale is becoming more and more of a requirement for businesses. The lack of a "Hadoop of realtime" has become the biggest hole in the data processing ecosystem. Storm fills that hole.'
data  scaling  twitter  realtime  scalability  storm  queueing 
september 2011 by jm
The Secrets of Building Realtime Big Data Systems
great slides, via HN. recommends a canonical Hadoop long-term store and a quick, realtime, separate datastore for "not yet processed by Hadoop" data
hadoop  big-data  data  scalability  datamining  realtime  slides  presentations 
may 2011 by jm
Semi-Realtime Satellite Desktop Backgrounds
Russ Garrett with another set of near-realtime desktop weather imagery (cf. )
weather  desktop  image  satellite  realtime  backgrounds  from delicious
january 2010 by jm

related tags

acceleration  ad-tracking  ads  analytics  anti-spam  api  architecture  async  atom  avro  backgrounds  backpressure  bagging  batch  batch-processing  big-data  bijection  blocking  botmaker  bug-tracking  bugs  c  camry  cap  cascading  cep  classification  clock  clock-skew  clocks  clustering  code-review  code-reviews  code-verification  coding  collaboration  comet  concurrency  confluent  cyclomatic-complexity  darach-ennis  data  databases  datamining  decision-trees  desktop  determinism  disqus  distcomp  distributed  embedded  embedded-systems  erlang  etcs  event-processing  events  eventsource  facebook  fault-tolerance  feeds  firmware  games  gc  gevent  go  google  grape  hadoop  heron  hn  horror  http  http-push  http2  iframe  image  internet  interviews  ipc  java  javascript  jay-kreps  jun-rao  kafka  lambda-architecture  languages  latencies  latency  league-of-legends  left-right  libraries  locking  logs  long-polling  low-latency  machine-learning  martin-thompson  messaging  misra-c  ml  multi-az  multi-region  multithreading  nasa  nathan-marz  netty  nginx  ntp  open-source  ping  presentations  protocols  pub-sub  pubsub  pubsubhubbub  push  python  qcon  quality  queue  queueing  queues  rabbitmq  reader-writer  realtime  redis  reliability  replay  replication  rtos  rules  rx  s3  safety  samoa  satellite  scalability  scaling  set-storage  skew  slides  spaghetti-code  spam  spark  standards  storage  storm  stream-processing  streaming  synchronization  sysctl  testing  threading  throttle-control  time  toyota  transport  transports  trident  tuning  twitter  via:hn  via:nelson  via:trustinlee  wait-free  weather  web  websockets  wormhole  yahoo 

Copy this bookmark: