jm + presto   3

Apache Iceberg (incubating)
Coming to presto soon apparently....
Iceberg tracks individual data files in a table instead of directories. This allows writers to create data files in-place and only adds files to the table in an explicit commit.

Table state is maintained in metadata files. All changes to table state create a new metadata file and replace the old metadata with an atomic operation. The table metadata file tracks the table schema, partitioning config, other properties, and snapshots of the table contents.

The atomic transitions from one table metadata file to the next provide snapshot isolation. Readers use the latest table state (snapshot) that was current when they load the table metadata and are not affected by changes until they refresh and pick up a new metadata location.


excellent -- this will let me obsolete so much of our own code :)
presto  storage  s3  hive  iceberg  apache  asf  data  architecture 
4 weeks ago by jm

Copy this bookmark:



description:


tags: