jm + filesystems   16

Instapaper Outage Cause & Recovery
Hard to see this as anything other than a pretty awful documentation fail by the AWS RDS service:
Without knowledge of the pre-April 2014 file size limit, it was difficult to foresee and prevent this issue. As far as we can tell, there’s no information in the RDS console in the form of monitoring, alerts or logging that would have let us know we were approaching the 2TB file size limit, or that we were subject to it in the first place. Even now, there’s nothing to indicate that our hosted database has a critical issue.
limits  aws  rds  databases  mysql  filesystems  ops  instapaper  risks 
11 days ago by jm
Collecting my thoughts about Torus
Worryingly-optimistic communications about CoreOS' recently-announced distributed storage system. I had similar thoughts, but Jeff Darcy is actually an expert on this stuff so he's way more worth listening to on the topic ;)
jeff-darcy  distcomp  filesystems  coreos  torus  storage 
june 2016 by jm
CharybdeFS: a new fault-injecting filesystem for software testing
a FUSE-based filesystem from ScyllaDB to test filesystem-related failure scenarios. great idea
fuse  software  testing  scylladb  filesystems  disk  charybdefs  fault-injection  tests 
february 2016 by jm
Files Are Hard
This is basically terrifying. A catalog of race conditions and reliability horrors around the POSIX filesystem abstraction in Linux -- it's a wonder anything works.

'Where’s this documented? Oh, in some mailing list post 6-8 years ago (which makes it 12-14 years from today). The fs devs whose posts I’ve read are quite polite compared to LKML’s reputation, and they generously spend a lot of time responding to basic questions, but it’s hard for outsiders to troll [sic] through a decade and a half of mailing list postings to figure out which ones are still valid and which ones have been obsoleted! I don’t mean to pick on filesystem devs. In their OSDI 2014 talk, the authors of the paper we’re discussing noted that when they reported bugs they’d found, developers would often respond “POSIX doesn’t let filesystems do that”, without being able to point to any specific POSIX documentation to support their statement. If you’ve followed Kyle Kingsbury’s Jepsen work, this may sound familiar, except devs respond with “filesystems don’t do that” instead of “networks don’t do that”.I think this is understandable, given how much misinformation is out there. Not being a filesystem dev myself, I’d be a bit surprised if I don’t have at least one bug in this post.'
filesystems  linux  unix  files  operating-systems  posix  fsync  osdi  papers  reliability 
december 2015 by jm
Low-latency journalling file write latency on Linux
great research from LMAX: xfs/ext4 are the best choices, and they explain why in detail, referring to the code
linux  xfs  ext3  ext4  filesystems  lmax  performance  latency  journalling  ops 
december 2015 by jm
S3QL
a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provides a hard disk of dynamic, infinite capacity that can be accessed from any computer with internet access running Linux, FreeBSD or OS-X.
S3QL is a standard conforming, full featured UNIX file system that is conceptually indistinguishable from any local file system. Furthermore, S3QL has additional features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival.
S3QL is designed to favor simplicity and elegance over performance and feature-creep. Care has been taken to make the source code as readable and serviceable as possible. Solid error detection and error handling have been included from the very first line, and S3QL comes with extensive automated test cases for all its components.
filesystems  aws  s3  storage  unix  google-storage  openstack 
september 2015 by jm
danilop/yas3fs · GitHub
YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster.
aws  s3  s3fs  yas3fs  filesystems  fuse  sns 
july 2015 by jm
/dev/full - Wikipedia, the free encyclopedia
This is handy!

'In Linux, /dev/full or the always full device[1][2] is a special file that always returns the error code ENOSPC (meaning "No space left on device") on writing, and provides an infinite number of null characters to any process that reads from it (similar to /dev/zero). This device is usually used when testing the behaviour of a program when it encounters a "disk full" error.'
dev  /dev/full  filesystems  devices  linux  testing  enospc  error-handling 
november 2014 by jm
The State of ZFS on Linux
Linux users familiar with other filesystems or ZFS users from other platforms will often ask whether ZFS on Linux (ZoL) is “stable”. The short answer is yes, depending on your definition of stable. The term stable itself is somewhat ambiguous.


Oh dear. that's not a good start. Good reference page, though
zfs  linux  filesystems  ops  solaris 
september 2014 by jm
Revisiting How We Put Together Linux Systems
Building a running OS out of layered btrfs filesystems. This sounds awesome.
Instantiating a new system or OS container (which is exactly the same in this scheme) just consists of creating a new appropriately named root sub-volume. Completely naturally you can share one vendor OS copy in one specific version with a multitude of container instances.

Everything is double-buffered (or actually, n-fold-buffered), because usr, runtime, framework, app sub-volumes can exist in multiple versions. Of course, by default the execution logic should always pick the newest release of each sub-volume, but it is up to the user keep multiple versions around, and possibly execute older versions, if he desires to do so. In fact, like on ChromeOS this could even be handled automatically: if a system fails to boot with a newer snapshot, the boot loader can automatically revert back to an older version of the OS.


(via Tony Finch)
via:fanf  linux  docker  btrfs  filesystems  unionfs  copy-on-write  os  hacking  unix 
september 2014 by jm
S3QL
a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack. S3QL effectively provides a hard disk of dynamic, infinite capacity that can be accessed from any computer with internet access running Linux, FreeBSD or OS-X.

S3QL is a standard conforming, full featured UNIX file system that is conceptually indistinguishable from any local file system. Furthermore, S3QL has additional features like compression, encryption, data de-duplication, immutable trees and snapshotting which make it especially suitable for online backup and archival.
s3  s3ql  backup  aws  filesystems  linux  freebsd  osx  ops 
march 2014 by jm
vmtouch - the Virtual Memory Toucher
'vmtouch is a tool for learning about and controlling the file system cache of unix and unix-like systems. It is BSD licensed.'
vmtouch  vm  linux  unix  fs  filesystems  instagram 
december 2011 by jm
A short history of btrfs [LWN.net]
wow, sounds good! looking forward to this hitting production-ready status
btrfs  history  zfs  linux  open-source  licensing  storage  sysadmin  b-trees  b+trees  algorithms  fs  filesystems 
august 2009 by jm

Copy this bookmark:



description:


tags: