Apache Iceberg (incubating)
5 weeks ago by jm
Coming to presto soon apparently....
excellent -- this will let me obsolete so much of our own code :)
presto
storage
s3
hive
iceberg
apache
asf
data
architecture
Iceberg tracks individual data files in a table instead of directories. This allows writers to create data files in-place and only adds files to the table in an explicit commit.
Table state is maintained in metadata files. All changes to table state create a new metadata file and replace the old metadata with an atomic operation. The table metadata file tracks the table schema, partitioning config, other properties, and snapshots of the table contents.
The atomic transitions from one table metadata file to the next provide snapshot isolation. Readers use the latest table state (snapshot) that was current when they load the table metadata and are not affected by changes until they refresh and pick up a new metadata location.
excellent -- this will let me obsolete so much of our own code :)
5 weeks ago by jm
Copy this bookmark: