jm + json   24

How eBay’s Shopping Cart used compression techniques to solve network I/O bottlenecks
compressing data written to MongoDB using LZ4_HIGH --dropped oplog write rates from 150GB/hour to 11GB/hour. Snappy and Gzip didn't fare too well by comparison
lz4  compression  gzip  json  snappy  scaling  ebay  mongodb 
10 days ago by jm - Parsing JSON is a Minefield 💣
Crockford chose not to version [the] JSON definition: 'Probably the boldest design decision I made was to not put a version number on JSON so there is no mechanism for revising it. We are stuck with JSON: whatever it is in its current form, that’s it.' Yet JSON is defined in at least six different documents.

"Boldest". ffs. :facepalm:
bold  courage  json  parsing  coding  data  formats  interchange  fail  standards  confusion 
october 2016 by jm
'a small library to manage encrypted secrets using asymmetric encryption.'
The main benefits provided by ejson are:

Secrets can be safely stored in a git repo.
Changes to secrets are auditable on a line-by-line basis with git blame.
Anyone with git commit access has access to write new secrets.
Decryption access can easily be locked down to production servers only.
Secrets change synchronously with application source (as opposed to secrets provisioned by Configuration Management).
Simple, well-tested, easily-auditable source.
crypto  security  credentials  encryption  ejson  json  configuration  config 
july 2016 by jm
Avro, mail # dev - bytes and fixed handling in Python implementation - 2014-09-04, 22:54
More Avro trouble with "bytes" fields! Avoid using "bytes" fields in Avro if you plan to interoperate with either of the Python implementations; they both fail to marshal them into JSON format correctly. This is the official "avro" library, which produces UTF-8 errors when a non-UTF-8 byte is encountered
bytes  avro  marshalling  fail  bugs  python  json  utf-8 
march 2015 by jm
'Turn websites into structured APIs from your browser in seconds' -- next-generation web scraping, recommended by conoro
via:conoro  scraping  web  http  kimono  rss  json  csv  data 
january 2015 by jm
an [XPath-style] query language for JSON. You can extract and transform elements from a JSON document.

Supported by the "aws" CLI tool, and in boto.
aws  boto  jmespath  json  xpath  querying  languages  documents 
november 2014 by jm
The problem of managing schemas
Good post on the pain of using CSV/JSON as a data interchange format:
eventually, the schema changes. Someone refactors the code generating the JSON and moves fields around, perhaps renaming few fields. The DBA added new columns to a MySQL table and this reflects in the CSVs dumped from the table. Now all those applications and scripts must be modified to handle both file formats. And since schema changes happen frequently, and often without warning, this results in both ugly and unmaintainable code, and in grumpy developers who are tired of having to modify their scripts again and again.
schema  json  avro  protobuf  csv  data-formats  interchange  data  hadoop  files  file-formats 
november 2014 by jm
JCDecaux Developer API
web service API for Dublin Bikes data (and other similar bikesharing services run by JCD):
Two kinds of data are delivered by the platform:

Static data provides stable information like station position, number of bike stands, payment terminal availability, etc.
Dynamic data provides station state, number of available bikes, number of free bike stands, etc.
Static data can be downloaded manually in file format or accessed through the API. Dynamic data are refreshed every minute and can be accessed only through the API.

Ruby API:
jcdecaux  bikesharing  dublin  dublin-bikes  api  web-services  http  json  open-data 
october 2014 by jm
on using JSON as a config file format
Ben Hughes on twitter:

"JSON is fine for config files, if you don't want to comment your config file. Which is a way of saying, it isn't fine for config files."
ben-hughes  funny  json  file-formats  config-files  configuration  software  coding 
september 2014 by jm
Postman - REST Client
a Chrome extension to aid working with REST APIs. Formats XML and JSON responses, supports file uploads, key/value editors, autocomplete, open source under ASL2
open-source  chrome  extensions  browser  postman  rest  hateoas  api  xml  json  web-services  via:eonnen 
july 2014 by jm
FlatBuffers: Main Page
A new serialization format from Google's Android gaming team, supporting C++ and Java, open source under the ASL v2. Reasons to use it:
Access to serialized data without parsing/unpacking - What sets FlatBuffers apart is that it represents hierarchical data in a flat binary buffer in such a way that it can still be accessed directly without parsing/unpacking, while also still supporting data structure evolution (forwards/backwards compatibility).
Memory efficiency and speed - The only memory needed to access your data is that of the buffer. It requires 0 additional allocations. FlatBuffers is also very suitable for use with mmap (or streaming), requiring only part of the buffer to be in memory. Access is close to the speed of raw struct access with only one extra indirection (a kind of vtable) to allow for format evolution and optional fields. It is aimed at projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the benchmarks for details.
Flexible - Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for long-lived games: don't have to update all data with each new version!). It also means you have a lot of choice in what data you write and what data you don't, and how you design data structures.
Tiny code footprint - Small amounts of generated code, and just a single small header as the minimum dependency, which is very easy to integrate. Again, see the benchmark section for details.
Strongly typed - Errors happen at compile time rather than manually having to write repetitive and error prone run-time checks. Useful code can be generated for you.
Convenient to use - Generated C++ code allows for terse access & construction code. Then there's optional functionality for parsing schemas and JSON-like text representations at runtime efficiently if needed (faster and more memory efficient than other JSON parsers).

Looks nice, but it misses the language coverage of protobuf. Definitely more practical than capnproto.
c++  google  java  serialization  json  formats  protobuf  capnproto  storage  flatbuffers 
june 2014 by jm
Simple Binary Encoding
an OSI layer 6 presentation for encoding/decoding messages in binary format to support low-latency applications. [...] SBE follows a number of design principles to achieve this goal. By adhering to these design principles sometimes means features available in other codecs will not being offered. For example, many codecs allow strings to be encoded at any field position in a message; SBE only allows variable length fields, such as strings, as fields grouped at the end of a message.

The SBE reference implementation consists of a compiler that takes a message schema as input and then generates language specific stubs. The stubs are used to directly encode and decode messages from buffers. The SBE tool can also generate a binary representation of the schema that can be used for the on-the-fly decoding of messages in a dynamic environment, such as for a log viewer or network sniffer.

The design principles drive the implementation of a codec that ensures messages are streamed through memory without backtracking, copying, or unnecessary allocation. Memory access patterns should not be underestimated in the design of a high-performance application. Low-latency systems in any language especially need to consider all allocation to avoid the resulting issues in reclamation. This applies for both managed runtime and native languages. SBE is totally allocation free in all three language implementations.

The end result of applying these design principles is a codec that has ~25X greater throughput than Google Protocol Buffers (GPB) with very low and predictable latency. This has been observed in micro-benchmarks and real-world application use. A typical market data message can be encoded, or decoded, in ~25ns compared to ~1000ns for the same message with GPB on the same hardware. XML and FIX tag value messages are orders of magnitude slower again.

The sweet spot for SBE is as a codec for structured data that is mostly fixed size fields which are numbers, bitsets, enums, and arrays. While it does work for strings and blobs, many my find some of the restrictions a usability issue. These users would be better off with another codec more suited to string encoding.
sbe  encoding  protobuf  protocol-buffers  json  messages  messaging  binary  formats  low-latency  martin-thompson  xml 
may 2014 by jm
Google's Open Bidder stack moving from Jetty to Netty
Open Bidder traditionally used Jetty as an embedded webserver, for the critical tasks of accepting connections, processing HTTP requests, managing service threads, etc. Jetty is a robust, but traditional stack that carries the weight and tradeoffs of Servlet’s 15 years old design. For a maximum performance RTB agent that must combine very large request concurrency with very low latencies, and often benefit also from low-level control over the transport, memory management and other issue, a different webserver stack was required. Open Bidder now supports Netty, an asynchronous, event-driven, high-performance webserver stack.

For existing code, the most important impact is that Netty is not compatible with the Servlet API. Its own internal APIs are often too low-level, not to mention proprietary to Netty; so Open Bidder v0.5 introduces some new, stack-neutral APIs for things like HTTP requests and responses, cookies, request handlers, and even simple HTML templating based on Mustache. These APIs will work with both Netty and Jetty. This means you don’t need to change any code to switch between Jetty and Netty; on the other hand, it also means that existing code written for Open Bidder 0.4 may need some changes even if you plan to keep using Jetty.

[....] Netty's superior efficiency is very significant; it supports 50% more traffic in the same hardware, and it maintains a perfect latency distribution even at the peak of its supported load.

This doc is noteworthy on a couple of grounds:

1. the use of Netty in a public API/library, and the additional layer in place to add a friendlier API on top of that. I hope they might consider releasing that part as OSS at some point.

2. I also find it interesting that their API uses protobufs to marshal the message, and they plan in a future release to serialize those to JSON documents -- that makes a lot of sense.
apis  google  protobufs  json  documents  interoperability  netty  jetty  servlets  performance  java 
april 2014 by jm
Home · linkedin/ Wiki is a REST+JSON framework for building robust, scalable service architectures using dynamic discovery and simple asynchronous APIs. fills a niche for building RESTful service architectures at scale, offering a developer workflow for defining data and REST APIs that promotes uniform interfaces, consistent data modeling, type-safety, and compatibility checked API evolution.

The new underlying comms layer for Voldemort, it seems.
voldemort  d2  linkedin  json  rest  http  api  frameworks  java 
february 2014 by jm
Gap's application of Knockout.js and the MVVM model
Interesting, first time I'd heard of it; the Model-View-View Model pattern.
mvvm  architecture  javascript  web  ui  knockout-js  martin-fowler  json 
april 2013 by jm
like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text. [it] is written in portable C, and it has zero runtime dependencies. You can download a single binary, scp it to a far away machine, and expect it to work.

Nice tool. Needs to get into the Debian/Ubuntu apt repos pronto ;)
jq  tools  cli  via:peakscale  json  coding  data  sed  unix 
april 2013 by jm
Written by Google, this library is a flexible, efficient, and powerful Java client library for accessing any resource on the web via HTTP. It features a pluggable HTTP transport abstraction that allows any low-level library to be used, such as, Apache HTTP Client, or URL Fetch on Google App Engine. It also features efficient JSON and XML data models for parsing and serialization of HTTP response and request content. The JSON and XML libraries are also fully pluggable, including support for Jackson and Android's GSON libraries for JSON.

Not quite as simple an API as Python's requests, sadly, but still an improvement on the verbose Apache HttpComponent API. Good support for unit testing via a built-in mock-response class. Still in beta
google  beta  software  http  libraries  json  xml  transports  protocols 
april 2013 by jm
'a Java framework for developing ops-friendly, high-performance, RESTful web services. Developed by Yammer to power their JVM-based backend services, Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, lightweight package that lets you focus on getting things done. Dropwizard has out-of-the-box support for sophisticated configuration, application metrics, logging, operational tools, and much more, allowing you and your team to ship a production-quality HTTP+JSON web service in the shortest time possible.' From Coda Hale/Yammer; includes Guava, Jetty, Jersey, Jackson, Metrics, slf4j. Pretty good baseline to start any new Java service with....
framework  http  java  rest  web  jersey  guava  jackson  jetty  json  web-services  yammer 
may 2012 by jm
SoundCloud Developers Manifesto
'We recognize that only through your apps and hacks, can SoundCloud fully realize its potential as the audio platform.'
apps  hacks  soundcloud  mp3  music  hosting  files  json  rest  oauth  apis  http  from delicious
may 2010 by jm
nifty; Apache-licensed distributed, RESTful, JSON-over-HTTP, schemaless search server with multi-tenancy
search  distributed  rest  json  apache  elasticsearch  http  from delicious
february 2010 by jm
'free, open, developer-generated APIs for a wide variety of websites. is a place to create and share them. [..] Check out [..] ways to use parselets from our web service, Ruby, Python, C/C++, or the *nix command-line.'
parselets  scraping  html  web  regexps  sitescooper  json  from delicious
december 2009 by jm
JSON Format
'your online JSON Formatter'. useful. via JKeyes
via:jkeyes  json  formatting  tools  useful  format  debugging  from delicious
november 2009 by jm
Nelson Minar plugs a new way to write web apps
Every HTML page is static -- the dynamic parts are entirely DOM-injected from server-delivered JSON by client-side Javascript. No dynamic data is delivered in HTML. I'm thinking about this, and it does seem to bring a lot of positives. hmm
dom  javascript  json  web-apps  web  nelson-minar  ajax  from delicious
september 2009 by jm

related tags

ajax  apache  api  apis  apps  architecture  avro  aws  ben-hughes  beta  bikesharing  binary  bold  boto  browser  bugs  bytes  c++  capnproto  chrome  cli  coding  compression  config  config-files  configuration  confusion  courage  credentials  crypto  csv  d2  data  data-formats  debugging  distributed  documents  dom  dublin  dublin-bikes  ebay  ejson  elasticsearch  encoding  encryption  extensions  fail  file-formats  files  flatbuffers  format  formats  formatting  framework  frameworks  funny  google  guava  gzip  hacks  hadoop  hateoas  hosting  html  http  interchange  interop  interoperability  jackson  java  javascript  jcdecaux  jersey  jetty  jmespath  jq  json  kimono  knockout-js  languages  libraries  linkedin  low-latency  lz4  marshalling  martin-fowler  martin-thompson  messages  messaging  mongodb  mp3  music  mvvm  nelson-minar  netty  oauth  open-data  open-source  parselets  parsing  performance  postman  protobuf  protobufs  protocol-buffers  protocols  python  querying  regexps  rest  rss  sbe  scaling  schema  scraping  search  security  sed  serialization  servlets  sitescooper  snappy  soa  software  soundcloud  standards  storage  tools  transports  ui  unix  useful  utf-8  via:conoro  via:eonnen  via:jkeyes  via:marc  via:peakscale  voldemort  web  web-apps  web-services  xml  xpath  yammer 

Copy this bookmark: