mpm + io   23

Package file handles write-ahead logs and space management of os.File-like entities
go  io 
6 weeks ago by mpm
Toward non-blocking asynchronous I/O
The Linux asynchronous I/O (AIO) layer tends to have many critics and few defenders, but most people at least expect it to actually be asynchronous. In truth, an AIO operation can block in the kernel for a number of reasons, making AIO difficult to use in situations where the calling thread truly cannot afford to block.
linux  io  concurrency 
october 2018 by mpm
Diskspd Utility
A feature-rich and versatile storage testing tool, Diskspd (version 2.0.15) combines robust and granular IO workload definition with flexible runtime and output options, creating an ideal tool for synthetic storage subsystem testing and validation.
performance  testing  io 
april 2016 by mpm
Welcome to the open source project for creating new interfaces for non-volatile memory (like flash).
io  storage 
april 2016 by mpm
UNIX pipes as IO monads
We will show uncanny similarities between monadic i/o in Haskell, and UNIX filter compositions based on pipes and redirections
io  haskell  unix 
january 2016 by mpm
From ARIES to MARS: Reengineering Transaction Management for Next-Generation, Solid-State Drives
Systems that provide powerful transaction mechanisms often rely on write-ahead logging (WAL) implementations that were designed with slow, disk-based systems in mind. The emerging class of fast, byte-addressable, non-volatile memory (NVM) technologies (e.g., phase change memories, spin-torque MRAMs, and the memristor), however, present performance characteristics very different from both disks and flash-based SSDs. This paper addresses the problem of designing a WAL scheme optimized for these fast NVM-based storage systems. We examine the features that a system like ARIES, a WAL algorithm popular for databases, must provide and separate them from the implementation decisions ARIES makes to optimize for disk-based systems. We design a new NVMoptimized WAL scheme (called MARS) in tandem with a novel SSD multi-part atomic write primitive that combine to provide the same features as ARIES does without any of the disk-centric limitations. The new atomic write primitive makes the log’s contents visible to the application, allowing for a simpler and faster implementation. MARS provides atomicity, durability, and high performance by leveraging the enormous internal bandwidth and high degree of parallelism that advanced SSDs will provide. We have implemented MARS and the novel visible atomic write primitive in a next-generation SSD. This paper demonstrates the overhead of the primitive is minimal compared to normal writes, and our hardware provides large speedups for transactional updates to hash tables, b-trees, and large graphs. MARS outperforms ARIES by up to 3.7 ⇥ while reducing software complexity
database  io  performance 
january 2015 by mpm
Muninn: a Versioning Key-Value Store using Object-based Storage Model
While non-volatile memory (NVRAM) devices have the potential to alleviate the trade-off between performance, scalability, and energy in storage and memory subsystems, a block interface and storage subsystems designed for slow I/O devices make it difficult to efficiently exploit NVRAMs in a portable and extensible way.

We propose an object-based storage model as a way of addressing the shortfalls of the current interfaces. Through the design of Muninn, an object-based versioning key-value store, we demonstrate that an in-device NVRAM management layer can be as efficient as that of NVRAM-aware key-value stores while not requiring host resources or host changes, and enabling tightly-coupled optimizations. As a key-value store, Muninn is designed to achieve better lifetime and low read/write amplification by eliminating internal data movements and per-object metadata updates using Bloom filters and hash functions. By doing so, it achieves as little as 1.5 flash page reads per look up and 0.26 flash page writes per insert on average with 50 million 1 KB key-value pairs without incurring data re-organization. This is close to the minimum number of flash accesses required to read and store the 1 KB key-value pairs, thus increasing performance and lifetime of flash media.
storage  io  performance 
july 2014 by mpm
Implementing a Key-Value Store – Part 6: Open-Addressing Hash Tables
In this article, I will compare several open-addressing hash tables: Linear Probing, Hopscotch hashing, and Robin Hood hashing. I have already done some work on this topic, and in this article I want to gather data for more metrics in order to decide which hash table I will use for my key-value store.
datastructure  performance  io 
july 2014 by mpm
Detecting (write) failures when using memory mapped files in Java
What happens in case of a disk failure? How can the OS notify your process that it failed to write to disk what you’ve written to memory?
java  io 
march 2014 by mpm
This tool lets you monitor I/O latency in real time. It shows disk latency in the same way as ping shows network latency
monitoring  io 
august 2013 by mpm
On the Efficiency of Durable State Machine Replication
State Machine Replication (SMR) is a fundamental technique for ensuring the dependability of critical services in modern internet-scale infrastructures. SMR alone does not protect from full crashes, and thus in practice it is employed together with secondary storage to ensure the durability of the data managed by these services. In this work we show that the classical durability enforcing mechanisms—logging, checkpointing, state transfer—can have a high impact on the performance of SMR-based services even if SSDs are used instead of disks. To alleviate this impact, we propose three techniques that can be used in a transparent manner, i.e., without modifying the SMR programming model or requiring extra resources: parallel logging, sequential checkpointing, and collaborative state transfer. We show the benefits of these techniques experimentally by implementing them in an open-source replication library, and evaluating them in the context of a consistent key-value store and a coordination service
performance  io  paxos 
june 2013 by mpm
Inside HyperLevelDB
While stock LevelDB is an excellent foundation, our experience with HyperDex identified several opportunities for further performance improvements. This article describes the changes we've made to LevelDB meet HyperDex clients' demanding needs
database  io  concurrency  storage  non-blocking 
june 2013 by mpm
Block I/O Layer Tracing using blktrace
blktrace is a really useful tool to see what I/O operations are going on inside the Linux block I/O layer
linux  io  monitoring 
october 2012 by mpm
Lua for Event-Based IO. lev allows developers to work with IO streams without the raw data ever touching the VM
lua  io 
september 2012 by mpm
SSTable and Log Structured Storage: LevelDB
SSTable is a simple abstraction to efficiently store large numbers of key-value pairs while optimizing for high throughput, sequential read/write workloads
database  datastructure  io  storage 
july 2012 by mpm
Journal.IO is a zero-dependency, fast and easy-to-use journal storage implementation based on append-only rotating logs and checksummed variable-length records, supporting concurrent reads and writes, dynamic batching, tunable durability and data compaction.

Journal.IO has been forked from the HawtJournal project, in order to provide faster development and release cycles, as well as foster open collaboration
java  io 
june 2012 by mpm
The purpose of this document is to explain how to use the Linux AIO interface, namely the function family io_setup, io_submit, io_getevents, io_destroy
linux  io  performance 
april 2012 by mpm
Java Sequential IO Performance
Java has a number of means by which a file can be sequentially written to, or read back again. This article explores some of these mechanisms to understand their performance characteristics
java  performance  io 
december 2011 by mpm
Understanding the code inside Tornado, the asynchronous web server
My goal here is to have a walk through the lower layers of the Tornado asynchronous web server. I take a bottom-up approach, starting with the polling loop and going up to the application layer, pointing out the interesting things I see on my way.
python  http  io  concurrency 
august 2011 by mpm
Lottery and stride scheduling
two very related approaches to doing efficient proportional-share scheduling. I believe this is the canonical way of doing things, since mClock (by Gulati et al., presented at OSDI 2010) used stride scheduling successfully to schedule disk I/O in a hypervisor.
io  scheduling 
july 2011 by mpm
Java Nio Complete Scala Server
A complete multi-client stateful application server in Scala using Java NIO non-blocking IO for both reading and writing, and delimited continuations as coroutines for both IO and application processing.
java  io 
april 2011 by mpm
HawtDispatch is a small ( less than 100k ) thread pooling and NIO event notification framework API modeled after the libdispatch API that Apple created to power the Grand Central Dispatch (GCD) technology in OS X. It allows you to easily develop multi-threaded applications without having to deal with the problems that traditionally plague multi-threaded application development
java  io  concurrency 
january 2011 by mpm
Parallel Disk IO
a description of the parallel disk IO subsystem that I used in my Wide-Finder 2 entry, however the solution is pretty general and highlights the main principles.

Do not create threads to perform specific functions, threads and functions are orthogonal - any thread can perform any function. Do not leave critical aspects of behavior to the chance, instead proactively force desired behavior. Let each thread decide what is currently the most important/optimal thing to do next
concurrency  io 
january 2011 by mpm

Copy this bookmark: