jm + messaging   28

Delivering Billions of Messages Exactly Once · Segment Blog
holy crap, this is exactly the wrong way to build a massive-scale deduplication system -- with a monster random-access "is this random UUID in the db" lookup
deduping  architecture  horror  segment  messaging  kafka 
25 days ago by jm
Bots won't replace apps. Better apps will replace apps
As I’ll explain, messenger apps’ apparent success in fulfilling such a surprising array of tasks does not owe to the triumph of “conversational UI.” What they’ve achieved can be much more instructively framed as an adept exploitation of Silicon Valley phone OS makers’ growing failure to fully serve users’ needs, particularly in other parts of the world. Chat apps have responded by evolving into “meta-platforms.” Many of the platform-like aspects they’ve taken on to plaster over gaps in the OS actually have little to do with the core chat functionality. Not only is “conversational UI” a red herring, but as we look more closely, we’ll even see places where conversational UI has breached its limits and broken down.
apps  bots  chatops  chat  ui  messaging  silicon-valley  agents  alexa  siri  phones 
april 2016 by jm
GCM XMPP delivery receipt not always received - Google Groups
Good to know:
'GCM delivery receipts don't have an SLA at this time. Having your connection open longer will increase the odds that delivery receipts will arrive. 10 seconds seems a bit short. I'm glad it works. I would recommend longer like 10 min or an hour. The real design of this system is for persistent connections, hence connections that setup and tear down frequently will have difficulty receiving delivery receipts.'
gcm  xmpp  receipts  messaging  push-notifications  google 
march 2016 by jm
View & diagnose Google Cloud Messaging (GCM) statistics
Looks like GCM now offers a way to determine if a message got delivered, via the GCM diagnostics console
gcm  google  push-notifications  android  messaging 
february 2016 by jm
5 subtle ways you're using MySQL as a queue, and why it'll bite you
Excellent post from Percona. I particularly like that they don't just say "don't use MySQL" -- they give good advice on how it can be made work: 1) avoid polling; 2) avoid locking; and 3) avoid storing your queue in the same table as other data.
database  mysql  queueing  queue  messaging  percona  rds  locking  sql  architecture 
january 2016 by jm
Topics in High-Performance Messaging
'We have worked together in the field of high-performance messaging for many years, and in that time, have seen some messaging systems that worked well and some that didn't. Successful deployment of a messaging system requires background information that is not easily available; most of what we know, we had to learn in the school of hard knocks. To save others a knock or two, we have collected here the essential background information and commentary on some of the issues involved in successful deployments. This information is organized as a series of topics around which there seems to be confusion or uncertainty. Please contact us if you have questions or comments.'
messaging  scalability  scaling  performance  udp  tcp  protocols  multicast  latency 
december 2015 by jm
What's confusing about Kafka: a list
At a recent call, Neha said “The most confusing behavior we have is how producing to a topic can return errors for few seconds after the topic was already created”. As she said that, I remembered that indeed, this was once very confusing, but then I got used to it.  Which got us thinking: What other things that Kafka does are very confusing to new users, but we got so used to them that we no longer even see the issue?
kafka  messaging  tips  gotchas 
march 2015 by jm
The official REST Proxy for Kafka
The REST Proxy is an open source HTTP-based proxy for your Kafka cluster. The API supports many interactions with your cluster, including producing and consuming messages and accessing cluster metadata such as the set of topics and mapping of partitions to brokers. Just as with Kafka, it can work with arbitrary binary data, but also includes first-class support for Avro and integrates well with Confluent’s Schema Registry. And it is scalable, designed to be deployed in clusters and work with a variety of load balancing solutions.

We built the REST Proxy first and foremost to meet the growing demands of many organizations that want to use Kafka, but also want more freedom to select languages beyond those for which stable native clients exist today. However, it also includes functionality beyond traditional clients, making it useful for building tools for managing your Kafka cluster. See the documentation for a more detailed description of the included features.
kafka  rest  proxies  http  confluent  queues  messaging  streams  architecture 
march 2015 by jm
grpc.io
Binary message marshalling, client/server stubs generated by an IDL compiler, bidirectional binary protocol. CORBA is back from the dead!
Intro blog post: http://googledevelopers.blogspot.ie/2015/02/introducing-grpc-new-open-source-http2.html

Relevant: Steve Vinoski's commentary on protobuf-rpc back in 2008: http://steve.vinoski.net/blog/2008/07/13/protocol-buffers-leaky-rpc/
http  rpc  http2  netty  grpc  google  corba  idl  messaging 
february 2015 by jm
Slack's coming to Dublin
Butterfield insists that Slack improves on the basic messaging functionality offered by its predecessors. The company plans to expand from 100 employees to 250 this year, open an office in Dublin, and launch a version that supports large companies with multiple teams.
slack  messaging  chat  dublin  ireland  jobs  tech 
february 2015 by jm
Comparing Message Queue Architectures on AWS
A good overview -- I like the summary table. tl;dr:
If you are light on DevOps and not latency sensitive use SQS for job management and Kinesis for event stream processing. If latency is an issue, use ELB or 2 RabbitMQs (or 2 beanstalkds) for job management and Redis for event stream processing.
amazon  architecture  aws  messaging  queueing  elb  rabbitmq  beanstalk  kinesis  sqs  redis  kafka 
february 2015 by jm
David Cameron in 'cloud cuckoo land' over encrypted messaging apps ban | Technology | The Guardian
One insider at a major US technology firm told the Guardian that “politicians are fond of asking why it is that tech companies don’t base themselves in the UK” ... “I think if you’re saying that encryption is the problem, at a time when consumers and businesses see encryption as a very necessary part of trust online, that’s a very indicative point of view.”
business  guardian  david-cameron  uk-politics  crypto  ripa  messaging  internet  privacy 
january 2015 by jm
sclasen/event-shuttle
UNIX system service [jmason: ie a sidecar] that collects events and reliably delivers them to kafka, relieving other services on the same system from having to do so. Journals events through bolt-db so that in the event of an kafka outage, events can still be accepted, and will be delivered when kafka becomes available.
kafka  messaging  ruby  go  events  fault-tolerance  queueing 
november 2014 by jm
Exactly-Once Delivery May Not Be What You Want
An extremely good explanation from Marc Brooker that exactly-once delivery in a distributed system is very hard.
And so on. There's always a place to slot in one more turtle. The bad news is that I'm not aware of a nice solution to the general problem for all side effects, and I suspect that no such solution exists. On the bright side, there are some very nice solutions that work really well in practice. The simplest is idempotence. This is a very simple idea: we make the tasks have the same effect no matter how many times they are executed.
architecture  messaging  queues  exactly-once-delivery  reliability  fault-tolerance  distcomp  marc-brooker 
november 2014 by jm
"Aeron: High-Performance Open Source Message Transport" [slides, PDF]
a new networked pub/sub library from Martin "Disruptor" Thompson, based around a replicated, persistent log of messages, with exceptionally low latency. Apache-licensed. Very similar to the realtime messaging stack we've built in Swrve. ;)

https://github.com/real-logic/Aeron
realtime  messaging  pub-sub  ipc  queues  transports  martin-thompson  slides  latencies  open-source  java  libraries 
november 2014 by jm
The Saga pattern
'a distribution of long-living [distributed] transactions where steps may interleave, each with associated compensating transactions providing a compensation path across databases in the occurrence of a fault that may or may not compensate the entire chain back to the originator.'
distributed  messaging  saga  patterns  architecture  transactions  distributed-transactions  distcomp 
october 2014 by jm
Moquette MQTT
a Java implementation of an MQTT 3.1 broker. Its code base is small. At its core, Moquette is an events processor; this lets the code base be simple, avoiding thread sharing issues. The Moquette broker is lightweight and easy to understand so it could be embedded in other projects.
mqtt  moquette  netty  messaging  queueing  push-notifications  iot  internet  push  eclipse 
may 2014 by jm
iMessage purgatory
Oh Apple, you asshats. This is some seriously shitty programming. iMessage on iOS devices caches the "iMessage-capable" flag for all numbers, indefinitely, so if you switch from iPhone to Android, messages from your friends' iPhones won't get delivered to you henceforth -- and to add insult to injury, it claims they do with a "Delivered." status appearing under the message. This is happening to me right now...
apple  sms  messaging  phones  mobile  imessage  android  fail  bad-programming  bugs 
may 2014 by jm
Simple Binary Encoding
an OSI layer 6 presentation for encoding/decoding messages in binary format to support low-latency applications. [...] SBE follows a number of design principles to achieve this goal. By adhering to these design principles sometimes means features available in other codecs will not being offered. For example, many codecs allow strings to be encoded at any field position in a message; SBE only allows variable length fields, such as strings, as fields grouped at the end of a message.

The SBE reference implementation consists of a compiler that takes a message schema as input and then generates language specific stubs. The stubs are used to directly encode and decode messages from buffers. The SBE tool can also generate a binary representation of the schema that can be used for the on-the-fly decoding of messages in a dynamic environment, such as for a log viewer or network sniffer.

The design principles drive the implementation of a codec that ensures messages are streamed through memory without backtracking, copying, or unnecessary allocation. Memory access patterns should not be underestimated in the design of a high-performance application. Low-latency systems in any language especially need to consider all allocation to avoid the resulting issues in reclamation. This applies for both managed runtime and native languages. SBE is totally allocation free in all three language implementations.

The end result of applying these design principles is a codec that has ~25X greater throughput than Google Protocol Buffers (GPB) with very low and predictable latency. This has been observed in micro-benchmarks and real-world application use. A typical market data message can be encoded, or decoded, in ~25ns compared to ~1000ns for the same message with GPB on the same hardware. XML and FIX tag value messages are orders of magnitude slower again.

The sweet spot for SBE is as a codec for structured data that is mostly fixed size fields which are numbers, bitsets, enums, and arrays. While it does work for strings and blobs, many my find some of the restrictions a usability issue. These users would be better off with another codec more suited to string encoding.
sbe  encoding  protobuf  protocol-buffers  json  messages  messaging  binary  formats  low-latency  martin-thompson  xml 
may 2014 by jm
Making Storm fly with Netty | Yahoo Engineering
Y! engineer doubles the speed of Storm's messaging layer by replacing the zeromq implementation with Netty
netty  async  zeromq  storm  messaging  tcp  benchmarks  yahoo  clusters 
october 2013 by jm
Chronicle
an ultra low latency, high throughput, persisted, messaging and event driven in memory database. The typical latency is as low as 80 nano-seconds and supports throughputs of 5-20 million messages/record updates per second.

This library also supports distributed, durable, observable collections (Map, List, Set) The performance depends on the data structures used, but simple data structures can achieve throughputs of 5 million elements or key/value pairs in batches (eg addAll or putAll) and 500K elements or key/values per second when added/updated/removed individually.

It uses almost no heap, trivial GC impact, can be much larger than your physical memory size (only limited by the size of your disk) and can be shared between processes with better than 1/10th latency of using Sockets over loopback. It can change the way you design your system because it allows you to have independent processes which can be running or not at the same time (as no messages are lost) This is useful for restarting services and testing your services from canned data. e.g. like sub-microsecond durable messaging. You can attach any number of readers, including tools to see the exact state of the data externally.
library  messaging  performance  java  chronicle  disk  mmap 
july 2013 by jm
Google Cloud Messaging for Android
GCM is a service that allows you to send data from your server to your users' Android-powered device, and also to receive messages from devices on the same connection. The GCM service handles all aspects of queueing of messages and delivery to the target Android application running on the target device. GCM is completely free no matter how big your messaging needs are, and there are no quotas.
gcm  messaging  android  google  push 
july 2013 by jm
Cap'n Proto
Cap’n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except faster. In fact, in benchmarks, Cap’n Proto is INFINITY TIMES faster than Protocol Buffers.


Basically, marshalling like writing an aligned C struct to the wire, QNX messaging protocol-style. Wasteful on space, but responds to this by suggesting compression (which is a fair point tbh). C++-only for now. I'm not seeing the same kind of support for optional data that protobufs has though. Overall I'm worried there's some useful features being omitted here...
serialization  formats  protobufs  capn-proto  protocols  coding  c++  rpc  qnx  messaging  compression  compatibility  interoperability  i14y 
april 2013 by jm
Pushover: Simple Mobile Notifications for Android and iOS
'Pushover makes it easy to send real-time notifications to your Android and iOS devices.' extremely simple HTTPS API; 'Pushover has no monthly subscription fees and users will always be able to receive unlimited messages for free. Most applications can send messages for free, subject to monthly limits.' Also supported by ifttt.com
ios  android  iphone  push  messaging 
january 2013 by jm
Apache Kafka
'Kafka provides a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to an offline analysis system like Hadoop, but is very limiting for building real-time processing. Kafka aims to unify offline and online processing by providing a mechanism for parallel load into Hadoop as well as the ability to partition real-time consumption over a cluster of machines.' neat
kafka  linkedin  apache  distributed  messaging  pubsub  queue  incubator  scaling 
february 2012 by jm
Storage Infrastructure Behind Facebook Messages
HBase and Haystack; all data LZO-compressed; very interesting approach to testing -- they 'shadow the real production workload into the test cluster to test before going into production'. This catches a 'high percentage' of issues before production. nice
testing  shadowing  haystack  hbase  facebook  scalability  lzo  messaging  sms  via:james-hamilton 
october 2011 by jm

related tags

agents  alexa  amazon  android  apache  apple  apps  architecture  async  atom  aws  bad-programming  beanstalk  benchmarks  binary  bots  bugs  business  c++  capn-proto  cep  chat  chatops  chronicle  clusters  coding  comet  compatibility  compression  confluent  corba  crypto  darach-ennis  database  david-cameron  deduping  disk  distcomp  distributed  distributed-transactions  dublin  eclipse  elb  encoding  erlang  events  exactly-once-delivery  facebook  fail  fault-tolerance  feeds  formats  gcm  go  google  gotchas  grpc  guardian  haystack  hbase  horror  http  http-push  http2  i14y  idl  imessage  incubator  internet  interoperability  interviews  ios  iot  ipc  iphone  ireland  java  jobs  json  kafka  kinesis  latencies  latency  libraries  library  linkedin  locking  low-latency  lzo  marc-brooker  martin-thompson  messages  messaging  mmap  mobile  moquette  mqtt  multicast  mysql  netty  open-source  patterns  percona  performance  phones  ping  privacy  protobuf  protobufs  protocol-buffers  protocols  proxies  pub-sub  pubsub  pubsubhubbub  push  push-notifications  qcon  qnx  queue  queueing  queues  rabbitmq  rds  realtime  receipts  redis  reliability  rest  ripa  rpc  ruby  rx  saga  sbe  scalability  scaling  segment  serialization  shadowing  silicon-valley  siri  slack  slides  sms  sql  sqs  standards  storm  stream-processing  streams  tcp  tech  testing  tips  transactions  transports  udp  ui  uk-politics  via:james-hamilton  web  xml  xmpp  yahoo  zeromq 

Copy this bookmark:



description:


tags: