jm + lz4   3

Scylla compression benchmarks
ScyllaDB tested out LZ4, Snappy, DEFLATE, and ZStandard at several different levels on a decently real-world-ish workload. tl;dr:
Use compression. Unless you are using a really (but REALLY) fast hard drive, using the default compression settings will be even faster than disabling compression, and the space savings are huge.

When running a data warehouse where data is mostly being read and only rarely updated, consider using DEFLATE. It provides very good compression ratios while maintaining high decompression speeds; compression can be slower, but that might be unimportant for your workload.

If your workload is write-heavy but you really care about saving disk space, consider using ZStandard on level 1. It provides a good middle-ground between LZ4/Snappy and DEFLATE in terms of compression ratios and keeps compression speeds close to LZ4 and Snappy. Be careful however: if you often want to read cold data (from the SSTables on disk, not currently stored in memory, so for example data that was inserted a long time ago), the slower decompression might become a problem.
compression  scylladb  storage  deflate  zstd  zstandard  lz4  snappy  gzip  benchmarks  tests  performance 
9 weeks ago by jm
airlift/aircompressor: A port of Snappy, LZO and LZ4 to Java
This library contains implementations of LZ4, Snappy, and LZO written in pure Java. They are typically 10-40% faster than the JNI wrapper for the native libraries.
lz4  lzo  lzop  snappy  java  libraries  airlift  compression  performance 
january 2018 by jm
How eBay’s Shopping Cart used compression techniques to solve network I/O bottlenecks
compressing data written to MongoDB using LZ4_HIGH --dropped oplog write rates from 150GB/hour to 11GB/hour. Snappy and Gzip didn't fare too well by comparison
lz4  compression  gzip  json  snappy  scaling  ebay  mongodb 
february 2017 by jm

Copy this bookmark: