jm + sbe   3

Cap'n Proto, FlatBuffers, and SBE
a feature comparison of these new serialization formats from Kenton, the capnp dude
serialization  protobuf  capnproto  sbe  flatbuffers  google  coding  storage 
june 2014 by jm
Simple Binary Encoding
an OSI layer 6 presentation for encoding/decoding messages in binary format to support low-latency applications. [...] SBE follows a number of design principles to achieve this goal. By adhering to these design principles sometimes means features available in other codecs will not being offered. For example, many codecs allow strings to be encoded at any field position in a message; SBE only allows variable length fields, such as strings, as fields grouped at the end of a message.

The SBE reference implementation consists of a compiler that takes a message schema as input and then generates language specific stubs. The stubs are used to directly encode and decode messages from buffers. The SBE tool can also generate a binary representation of the schema that can be used for the on-the-fly decoding of messages in a dynamic environment, such as for a log viewer or network sniffer.

The design principles drive the implementation of a codec that ensures messages are streamed through memory without backtracking, copying, or unnecessary allocation. Memory access patterns should not be underestimated in the design of a high-performance application. Low-latency systems in any language especially need to consider all allocation to avoid the resulting issues in reclamation. This applies for both managed runtime and native languages. SBE is totally allocation free in all three language implementations.

The end result of applying these design principles is a codec that has ~25X greater throughput than Google Protocol Buffers (GPB) with very low and predictable latency. This has been observed in micro-benchmarks and real-world application use. A typical market data message can be encoded, or decoded, in ~25ns compared to ~1000ns for the same message with GPB on the same hardware. XML and FIX tag value messages are orders of magnitude slower again.

The sweet spot for SBE is as a codec for structured data that is mostly fixed size fields which are numbers, bitsets, enums, and arrays. While it does work for strings and blobs, many my find some of the restrictions a usability issue. These users would be better off with another codec more suited to string encoding.
sbe  encoding  protobuf  protocol-buffers  json  messages  messaging  binary  formats  low-latency  martin-thompson  xml 
may 2014 by jm
Simple Binary Encoding
'SBE is an OSI layer 6 representation for encoding and decoding application messages in binary format for low-latency applications.'

Licensed under ASL2, C++ and Java supported.
sbe  encoding  codecs  persistence  binary  low-latency  open-source  java  c++  serialization 
december 2013 by jm

Copy this bookmark:



description:


tags: