lmdb   111

« earlier    

Creating an LMDB database in Python · Deep learning at the University of Chicago
LMDB is the database of choice when using Caffe with large datasets. This is a tutorial of how to create an LMDB database from Python. First, let’s look at the pros and cons of using LMDB over HDF5.

Reasons to use HDF5:

Simple format to read/write.
Reasons to use LMDB:

LMDB uses memory-mapped files, giving much better I/O performance.
Works well with really large datasets. The HDF5 files are always read entirely into memory, so you can’t have any HDF5 file exceed your memory capacity. You can easily split your data into several HDF5 files though (just put several paths to h5 files in your text file). Then again, compared to LMDB’s page caching the I/O performance won’t be nearly as good.
python  lmdb  hdf5  database 
june 2017 by griddell
lmdb — lmdb 0.92 documentation
LMDB is a tiny database with some excellent properties:

Ordered map interface (keys are always lexicographically sorted).
Reader/writer transactions: readers don’t block writers, writers don’t block readers. Each environment supports one concurrent write transaction.
Read transactions are extremely cheap.
Environments may be opened by multiple processes on the same host, making it ideal for working around Python’s GIL.
Multiple named databases may be created with transactions covering all named databases.
Memory mapped, allowing for zero copy lookup and iteration. This is optionally exposed to Python using the buffer() interface.
Maintenance requires no external process or background threads.
No application-level caching is required: LMDB fully exploits the operating system’s buffer cache.
python  lmdb  database 
april 2017 by griddell
LDAP at Lightning Speed - LMDB
Howard Chu covers highlights of the LMDB design and discusses some of the internal improvements in slapd due to LMDB, as well as the impact of LMDB on other projects.
lmdb  rocksdb 
december 2016 by vdm
Using to write a HUGE dataset into . Its quite fast processing 16bit ima…
LMDB  deeplearning  from twitter_favs
november 2016 by kyosha
Gmane -- RE: newstore direction
For people who can do math - LSM write-amp is O(N^2). B+tree write amp is essentially O(1).
LMDB  LevelDB  RocksDB  from twitter_favs
october 2015 by avifreedman

« earlier    

related tags

access  acid-state  acid  alternative  api  b+tree  b-tree  bdb  benchmark  benchmarks  berkeley-db  berkeleydb  binding  bloat  boltdb  bookmarks_bar  btree  c++  c  cabinet  code  comparison  complexity  compression  consistency  data  database  databases  datastore  datastores  datastructure  db  dbms  deeplearning  delicious  design  dev  development  documentation  edu  efficiency  embedded  encapsulation  engine  external  fast  filesystem  gc  go  golang  good  graph  haskell  hdf5  heroes  http  hyperleveldb  implementation  index  influxdb  information  interface  java  javascript  jni  kafka  key-value-store  key-value  key  keyvalue  kv-store  kv  ldap  leveldb  levelup  library  lightning  log  logs  lsm  lua  lucene  luxio  mapped  mdb  memory  memorymapped  method  mmap  mongo  mvcc  nanomsg  node.js  nosql  nsa  objective-c  open-source  open  openldap  opensource  oss  performance  persistence  persistent  piperesearch  port  powerdns  presentation  programming  pypi  python  q&a  query  range  redis  retrieval  rocksdb  sample  samza  scheduler  secondary  serialization  server  simd  singleton  software  sort  source  sqlite  sstable  storage  store  streams  structure  succint  symas  task  tech_talk  tokukv  tokyo  tradeoffs  transaction  tree  value  variant  vcache  video  vs  ycsb  zambezi 

Copy this bookmark: