Iceberg Deep Dive, Part 1: What a Table Actually Is on Disk
Contents
Iceberg Deep Dive — a series where I stop repeating the docs and start opening the files. Part 1 (this post): the metadata tree, top to bottom. Part 2: manifest internals and the Parquet underneath. Part 3: time travel, rollback, and who actually keeps the history. Part 4: branches, tags, write-audit-publish — and who gets to have them.
Updated 2026-09-01: the stack moved from Spark 3.5 to Spark 4.1, so I reran the whole experiment. All the output below is from the new run.
For this post I created a tiny table, inserted six rows across two commits, and then opened every file it produced. Instead of trusting SELECT, I went at the files directly: raw HTTP calls against the catalog, Avro parsers against object storage. Then I ran the same experiment again against a Hive Metastore, so I could separate what Iceberg itself does from what the catalog adds. The stack is Spark 4.1.3 with Iceberg 1.11.0, Nessie as the REST catalog, MinIO for storage. Everything below is real output.
The mental model
An Iceberg table makes more sense if you think about it backwards from what you might expect. The table itself lives in the metadata, and the data files are just immutable blobs that the metadata points at.
catalog (Nessie / Hive Metastore / Glue / ...)
└─ pointer to the current metadata.json ← the only mutable thing
└─ metadata.json
├─ schema, partition specs, properties
└─ snapshots (one per commit)
└─ manifest list (snap-*.avro) ← one row per manifest
└─ manifest (*.avro) ← one row per data file
└─ Parquet data files ← immutable — never edited in place; changes write new filesEach append writes new files at every level of the tree and then atomically moves the pointer, leaving the old files in place. Because those old files stay readable, this is also where time travel, concurrent writers, and instant rollback come from.
The setup
The table is deliberately tiny so every file stays human-sized:
CREATE TABLE lakehouse.iceberg_lab.anatomy (
id INT,
name STRING,
amount DOUBLE,
ts TIMESTAMP
) USING icebergDESCRIBE EXTENDED already shows quite a bit:
DESCRIBE EXTENDED lakehouse.iceberg_lab.anatomy+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
|col_name |data_type |comment|
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+
|id |int |NULL |
|name |string |NULL |
|amount |double |NULL |
|ts |timestamp |NULL |
| | | |
|# Metadata Columns | | |
|_spec_id |int | |
|_partition |struct<> | |
|_file |string | |
|_pos |bigint | |
|_deleted |boolean | |
| | | |
|# Detailed Table Information| | |
|Name |lakehouse.iceberg_lab.anatomy | |
|Type |MANAGED | |
|Location |s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4 | |
|Provider |iceberg | |
|Owner |spark | |
|Table Properties |[created-at=2026-09-01T08:32:53.662484625Z,current-snapshot-id=none,format=iceberg/parquet,format-version=2,gc.enabled=false,nessie.catalog.content-id=3f64d851-592f-4d74-ae70-f0045502c343,nessie.commit.id=326949af0359533b67d030983eb44cf16e31ae8f11f3b6a01cf835a4a9f0c05f,nessie.commit.ref=main]| |
|Statistics |0 bytes, 0 rows |NULL |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+Some of these details come from Iceberg itself, and some are Nessie’s fingerprints.
- Spark exposes Iceberg’s hidden metadata columns even on a brand-new table: the base five,
_spec_id,_partition,_file,_pos, and_deleted. The row-lineage columns (_row_idand_last_updated_sequence_number) are missing, and that’s expected: they only exist on format-version-3 tables, and this one is V2. format-version=2is the default. V3 adds deletion vectors and row lineage, but you have to opt into it.- Nessie stamped its own properties into the table (
nessie.commit.id,nessie.commit.ref=main) and turned garbage collection off withgc.enabled=false. The catalog wants to manage cleanup itself here. - The
Locationreadsanatomy_062818ac-...rather than plainanatomy, because Nessie appends a UUID to the table directory. If you go looking for a Nessie table in the bucket, look for the suffix.
Commit #1: insert three rows
With the table in place, the first commit is a plain INSERT of three rows, plus a SELECT to watch them land:
INSERT INTO lakehouse.iceberg_lab.anatomy VALUES
(1, 'espresso', 3.50, TIMESTAMP '2026-08-16 08:15:00'),
(2, 'cortado', 4.25, TIMESTAMP '2026-08-16 09:02:00'),
(3, 'cold brew', 5.00, TIMESTAMP '2026-08-16 10:30:00');
SELECT * FROM lakehouse.iceberg_lab.anatomy ORDER BY id;+---+---------+------+-------------------+
|id |name |amount|ts |
+---+---------+------+-------------------+
|1 |espresso |3.5 |2026-08-16 08:15:00|
|2 |cortado |4.25 |2026-08-16 09:02:00|
|3 |cold brew|5.0 |2026-08-16 10:30:00|
+---+---------+------+-------------------+Iceberg exposes its own metadata as SQL tables. After that one commit:
SELECT committed_at, snapshot_id, parent_id, operation, manifest_list, summary
FROM lakehouse.iceberg_lab.anatomy.snapshots+-----------------------+-------------------+---------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|committed_at |snapshot_id |parent_id|operation|manifest_list |summary |
+-----------------------+-------------------+---------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|2026-09-01 08:32:58.613|3840990993138382909|NULL |append |s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/snap-3840990993138382909-1-3eb548f0-7a7c-4f5f-b390-b34da6569d12.avro|{spark.app.id -> app-20260901083246-0000, manifests-created -> 1, manifests-kept -> 0, manifests-replaced -> 0, added-data-files -> 2, added-records -> 3, added-files-size -> 2414, changed-partition-count -> 1, total-records -> 3, total-files-size -> 2414, total-data-files -> 2, total-delete-files -> 0, total-position-deletes -> 0, total-equality-deletes -> 0, engine-version -> 4.1.3, app-id -> app-20260901083246-0000, engine-name -> spark, iceberg-version -> Apache Iceberg 1.8.1 (commit 9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d), app-name -> iceberg-exp01-anatomy}|
+-----------------------+-------------------+---------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+That’s one commit, one snapshot, with parent_id NULL because it’s the root of the chain. The manifest_list value points at a real file (snap-3840990993138382909-1-3eb548f0-....avro), and summary is a map of everything the commit did: added-data-files -> 2, added-records -> 3, total-data-files -> 2, along with engine fingerprints like engine-version -> 4.1.3 and iceberg-version -> Apache Iceberg 1.8.1 (commit 9ce0fcf...).
The 1.8.1 in that fingerprint looks wrong at first, because the stack actually runs Iceberg 1.11.0. The explanation is that my Spark image also pins iceberg-hive-metastore at 1.8.1 for the Hive wiring, and on Spark 4’s classpath that jar happens to win the version lookup for both catalogs. So the summary stamps 1.8.1 while the runtime writing the table is genuinely 1.11.0.
The summary also records that my single INSERT statement produced two Parquet files. The three rows went in as two parallel tasks, and each task closed its own file, so one file got a single row and the other got two. This is the small-files problem in its earliest form: every write produces at least one file per task, and nothing merges them until you ask it to. The .files table confirms the split:
SELECT content, file_path, file_format, spec_id, record_count, file_size_in_bytes
FROM lakehouse.iceberg_lab.anatomy.files+-------+------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-------+------------+------------------+
|content|file_path |file_format|spec_id|record_count|file_size_in_bytes|
+-------+------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-------+------------+------------------+
|0 |s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/data/00000-0-d509f0b4-c80e-4cde-b4d2-7ce5a8dd1adf-0-00001.parquet|PARQUET |0 |1 |1206 |
|0 |s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/data/00001-1-d509f0b4-c80e-4cde-b4d2-7ce5a8dd1adf-0-00001.parquet|PARQUET |0 |2 |1208 |
+-------+------------------------------------------------------------------------------------------------------------------------------------------------+-----------+-------+------------+------------------+The content column distinguishes file kinds: 0 is a plain data file, while 1 and 2 are reserved for delete files, the two mechanisms (position deletes and equality deletes) Iceberg uses to remove rows without rewriting the data files that hold them. This tiny table has none. Each file weighs about 1.2 KB, which is almost all Parquet overhead for so few rows.
Commit #2: three more rows, and the chain appears
The second commit looks the same from the SQL side:
INSERT INTO lakehouse.iceberg_lab.anatomy VALUES
(4, 'flat white', 4.75, TIMESTAMP '2026-08-17 08:05:00'),
(5, 'cappuccino', 4.50, TIMESTAMP '2026-08-17 09:20:00'),
(6, 'affogato', 6.25, TIMESTAMP '2026-08-17 11:45:00');There are now four data files across the two commits, and .manifests shows two manifests, one per commit:
SELECT path, length, added_snapshot_id, added_data_files_count,
existing_data_files_count, deleted_data_files_count,
partition_summaries
FROM lakehouse.iceberg_lab.anatomy.manifests+------------------------------------------------------------------------------------------------------------------------------------+------+-------------------+----------------------+-------------------------+------------------------+-------------------+
|path |length|added_snapshot_id |added_data_files_count|existing_data_files_count|deleted_data_files_count|partition_summaries|
+------------------------------------------------------------------------------------------------------------------------------------+------+-------------------+----------------------+-------------------------+------------------------+-------------------+
|s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/19f7cfb3-95f0-49af-9f18-582f77bb13d9-m0.avro|7291 |6719067934236400536|2 |0 |0 |[] |
|s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/3eb548f0-7a7c-4f5f-b390-b34da6569d12-m0.avro|7289 |3840990993138382909|2 |0 |0 |[] |
+------------------------------------------------------------------------------------------------------------------------------------+------+-------------------+----------------------+-------------------------+------------------------+-------------------+Commit #2 never touched commit #1’s manifest: it wrote a new manifest for its own files, and the new snapshot’s manifest list simply references both. On an append, nothing that already exists gets rewritten; that’s what append-only means in practice.
The .snapshots table, however, does not fit that story: it still shows only one row after commit #2, even though two commits happened, and the snapshot from commit #1 is missing from the SQL-visible history. The gap is Nessie’s doing: Iceberg itself would keep both snapshots, and the Hive rerun at the end of this post shows it.
Leaving SQL: ask the catalog where the table is
Everything so far went through Iceberg’s SQL facade, so none of it has touched the files themselves. To get to those, the first question is where the table lives, and the catalog is the component that knows: it holds the pointer to the current metadata.json. Nessie speaks the standard Iceberg REST protocol; the curl below asks its loadTable endpoint, and the jq filter narrows the answer to the three fields that matter here:
curl -s -H 'Accept: application/json' \
'http://127.0.0.1:19120/iceberg/v1/main%7Cwarehouse/namespaces/iceberg_lab/tables/anatomy' \
| jq -r '"metadata-location: \(."metadata-location")",
"current-snapshot-id: \(.metadata."current-snapshot-id")",
"location: \(.metadata.location)"'The raw response is one big JSON document describing the whole table; most of the rest is what we’ll meet again when we open the metadata file below.
metadata-location: s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/00000-ef999169-075a-45b4-9246-2fea88d74896.metadata.json
current-snapshot-id: 6719067934236400536
location: s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4(The main%7Cwarehouse segment is Nessie’s URL-encoded {ref}|{warehouse} prefix; git-style refs are baked into the REST path.)
Almost everything that follows hangs off that one metadata-location field: a single pointer that Iceberg swaps on every commit, and the place where every engine that reads the table (Spark, DuckDB, Trino) starts.
The physical table: 11 objects
With the pointer in hand, the next step is to look at what the table actually consists of. The files sit in MinIO, which speaks plain S3, so the AWS CLI can list everything under the table’s prefix. The credentials are just the MinIO root user and password from the lakehouse stack:
AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD \
aws s3api list-objects-v2 \
--endpoint-url http://127.0.0.1:9000 \
--region us-east-1 \
--bucket warehouse \
--prefix nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/The whole six-row table comes to 11 objects:
{
"Contents": [
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/data/00000-0-d509f0b4-c80e-4cde-b4d2-7ce5a8dd1adf-0-00001.parquet",
"LastModified": "2026-09-01T08:32:58.149Z",
"ETag": "\"00bad43fce96178e5b7a1d482333de16\"",
"Size": 1206,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/data/00000-7-92354f1c-18ea-44cd-a599-5b5d20221d35-0-00001.parquet",
"LastModified": "2026-09-01T08:33:01.888Z",
"ETag": "\"cf320eba88287ca189a814812f57e826\"",
"Size": 1220,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/data/00001-1-d509f0b4-c80e-4cde-b4d2-7ce5a8dd1adf-0-00001.parquet",
"LastModified": "2026-09-01T08:32:58.149Z",
"ETag": "\"a623b2756c73fa6a81c9f851f43fce32\"",
"Size": 1208,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/data/00001-8-92354f1c-18ea-44cd-a599-5b5d20221d35-0-00001.parquet",
"LastModified": "2026-09-01T08:33:01.887Z",
"ETag": "\"899d3ad46813cfba704f1d167b1ce5f4\"",
"Size": 1218,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/00000-62b609b7-8b27-4990-bab0-e01003accf9d.metadata.json",
"LastModified": "2026-09-01T08:32:54.161Z",
"ETag": "\"50bf79840cd40147331b7d5f720b6224\"",
"Size": 916,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/00000-ef999169-075a-45b4-9246-2fea88d74896.metadata.json",
"LastModified": "2026-09-01T08:33:02.023Z",
"ETag": "\"9a5c83a4d330960ebc2351e70030ebd3\"",
"Size": 1941,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/00000-f7996371-526b-4729-b70b-4d2708458085.metadata.json",
"LastModified": "2026-09-01T08:32:58.665Z",
"ETag": "\"4a1e9d020580142b99bd255d8b395628\"",
"Size": 1941,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/19f7cfb3-95f0-49af-9f18-582f77bb13d9-m0.avro",
"LastModified": "2026-09-01T08:33:01.947Z",
"ETag": "\"14d7a6840b9b011c42810108eff345d9\"",
"Size": 7291,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/3eb548f0-7a7c-4f5f-b390-b34da6569d12-m0.avro",
"LastModified": "2026-09-01T08:32:58.540Z",
"ETag": "\"5f8b35ebe5640cb361e67e34a889f0a3\"",
"Size": 7289,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/snap-3840990993138382909-1-3eb548f0-7a7c-4f5f-b390-b34da6569d12.avro",
"LastModified": "2026-09-01T08:32:58.611Z",
"ETag": "\"5dd981d3bb0ae3bbc4ad5b19dfe6b33b\"",
"Size": 4481,
"StorageClass": "STANDARD"
},
{
"Key": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/snap-6719067934236400536-1-19f7cfb3-95f0-49af-9f18-582f77bb13d9.avro",
"LastModified": "2026-09-01T08:33:01.999Z",
"ETag": "\"851f502342f3eb0578079650baf9731a\"",
"Size": 4553,
"StorageClass": "STANDARD"
}
],
"RequestCharged": null,
"Prefix": "nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/"
}You can read the whole history off the filenames:
data/*.parquet— the four data files, two per commit.snap-<snapshot-id>-*.avro— the two manifest lists, one per snapshot, with the snapshot ID right there in the filename.*-m0.avro— the two manifests (the-m0suffix means “manifest 0 of that snapshot’s write”).- Three
metadata.jsonfiles, one per state change: CREATE TABLE (916 B, no snapshots yet), commit #1, and commit #2.
This listing also shows how much of the table is metadata: 7 of the 11 objects, roughly 28 KB out of 33. For a six-row table the metadata tree dwarfs the actual data. On a real table the ratio flips, of course, but the tree never goes away, which is why manifest compaction exists.
All three metadata.json files are named 00000-*. Under a Hive Metastore table you’d see 00000-, 00001-, 00002-: a version counter you can read straight off the filename, and the comparison below shows exactly that. Nessie instead names every version 00000-<uuid> and keeps the ordering in its own commit log. If you stare at a Nessie table’s bucket, the filenames tell you nothing about which version is current; only the catalog’s pointer does.
Inside metadata.json
The catalog’s answer named one specific metadata file, so let’s fetch it. It’s another object in the same bucket; MinIO expects signed requests like any S3, so curl signs with its --aws-sigv4 flag, and jq formats the result:
curl -s --aws-sigv4 'aws:amz:us-east-1:s3' \
-u "$MINIO_ROOT_USER:$MINIO_ROOT_PASSWORD" \
'http://127.0.0.1:9000/warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/00000-ef999169-075a-45b4-9246-2fea88d74896.metadata.json' \
| jq .The file is long, so it sits collapsed below, with ← marks on the fields we’ll walk through underneath:
Full metadata.json
{
"format-version": 2, ← the spec version; sequence numbers and delete files follow from it
"table-uuid": "3a559150-e8bf-4827-b6ce-c898bc7f0e39",
"location": "s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4",
"last-sequence-number": 2,
"last-updated-ms": 1788251582011,
"last-column-id": 4, ← the counter that hands out the next field id
"schemas": [
{
"type": "struct",
"schema-id": 0,
"fields": [
{
"id": 1,
"name": "id",
"required": false,
"type": "int"
},
{
"id": 2,
"name": "name",
"required": false,
"type": "string"
},
{
"id": 3,
"name": "amount",
"required": false,
"type": "double"
},
{
"id": 4,
"name": "ts",
"required": false,
"type": "timestamptz"
}
] ← columns are identified by id, so renaming one is a metadata-only edit
}
],
"current-schema-id": 0,
"partition-specs": [
{
"spec-id": 0,
"fields": []
}
],
"default-spec-id": 0,
"last-partition-id": 999, ← reserved even for unpartitioned tables
"default-sort-order-id": 0,
"sort-orders": [
{
"order-id": 0,
"fields": []
}
],
"properties": {
"owner": "spark",
"gc.enabled": "false",
"created-at": "2026-09-01T08:32:53.662484625Z"
},
"current-snapshot-id": 6719067934236400536,
"refs": {
"main": {
"snapshot-id": 6719067934236400536,
"type": "branch"
}
}, ← main is a branch; tags and WAP branches are just more entries here
"snapshots": [
{
"sequence-number": 2,
"snapshot-id": 6719067934236400536,
"timestamp-ms": 1788251582002,
"summary": {
"operation": "append",
"spark.app.id": "app-20260901083246-0000",
"manifests-created": "1",
"manifests-kept": "1",
"manifests-replaced": "0",
"added-data-files": "2",
"added-records": "3",
"added-files-size": "2438",
"changed-partition-count": "1",
"total-records": "6",
"total-files-size": "4852",
"total-data-files": "4",
"total-delete-files": "0",
"total-position-deletes": "0",
"total-equality-deletes": "0",
"engine-version": "4.1.3",
"app-id": "app-20260901083246-0000",
"engine-name": "spark",
"iceberg-version": "Apache Iceberg 1.8.1 (commit 9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d)",
"app-name": "iceberg-exp01-anatomy"
},
"manifest-list": "s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/snap-6719067934236400536-1-19f7cfb3-95f0-49af-9f18-582f77bb13d9.avro",
"schema-id": 0 ← the next level down: every snapshot owns its manifest-list pointer
}
], ← three state changes, yet only the CURRENT snapshot — Nessie keeps the present
"statistics": [],
"partition-statistics": [],
"snapshot-log": [
{
"timestamp-ms": 1788251582002,
"snapshot-id": 6719067934236400536
}
],
"metadata-log": []
}(Verbatim jq output, plus the ← annotations.)
format-version: 2is the spec version. Everything downstream (sequence numbers, delete files) follows from it.schemasis a list, and every field has anid. Each column’s numeric ID never changes (tsis field 4 forever), and since the data files reference columns by those IDs, renaming a column is a metadata-only edit.last-column-id: 4is the counter that hands out the next one. These IDs are what make column renames safe.partition-specsis also a list, with an empty spec here. Partitioning is metadata too, and a table can carry multiple specs over its life as partitions evolve.last-partition-id: 999is the spec’s internal counter, reserved even for unpartitioned tables.refsholds one entry:main, a branch pointing at the current snapshot. Tags and extra branches, including write-audit-publish, are just more entries in this map.snapshotsis the commit history. Each snapshot carries its commitsummaryand itsmanifest-listpointer, the next level down.
You can also see the Nessie behavior from earlier right here in the file. The table has gone through three state changes, yet snapshots holds only the current snapshot and snapshot-log only the last entry. Nessie deliberately keeps only the present in the metadata and stores the history in its own commit log; its docs describe the single-snapshot view as a design choice that preserves its cross-branch consistency guarantees. A Hive Metastore table keeps every snapshot in this file until you run expire_snapshots, as the comparison below shows. This is why the .snapshots table showed one row after two commits, and it’s also why anything that needs snapshot history, like time travel or rollback, has to go through the catalog.
The manifest list: one row per manifest
The snapshot in that file carries a manifest-list field, and following it takes us one level further down the tree. The file it names is Avro rather than JSON, so opening it needs a decoder: the first command below downloads it, and the second prints it with fastavro, which uvx runs without installing anything up front:
curl -s --aws-sigv4 'aws:amz:us-east-1:s3' \
-u "$MINIO_ROOT_USER:$MINIO_ROOT_PASSWORD" \
-o snap-list.avro \
'http://127.0.0.1:9000/warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/snap-6719067934236400536-1-19f7cfb3-95f0-49af-9f18-582f77bb13d9.avro'
uvx --quiet --with fastavro python -c '
import fastavro, json, sys
print(json.dumps(list(fastavro.reader(open(sys.argv[1], "rb"))), indent=2, default=str))
' snap-list.avro[
{
"manifest_path": "s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/19f7cfb3-95f0-49af-9f18-582f77bb13d9-m0.avro",
"manifest_length": 7291,
"partition_spec_id": 0,
"content": 0,
"sequence_number": 2,
"min_sequence_number": 2,
"added_snapshot_id": 6719067934236400536,
"added_files_count": 2,
"existing_files_count": 0,
"deleted_files_count": 0,
"added_rows_count": 3,
"existing_rows_count": 0,
"deleted_rows_count": 0,
"partitions": [],
"key_metadata": null
},
{
"manifest_path": "s3://warehouse/nessie/iceberg_lab/anatomy_062818ac-b7af-4319-8482-0ddd5f4e2cc4/metadata/3eb548f0-7a7c-4f5f-b390-b34da6569d12-m0.avro",
"manifest_length": 7289,
"partition_spec_id": 0,
"content": 0,
"sequence_number": 1,
"min_sequence_number": 1,
"added_snapshot_id": 3840990993138382909,
"added_files_count": 2,
"existing_files_count": 0,
"deleted_files_count": 0,
"added_rows_count": 3,
"existing_rows_count": 0,
"deleted_rows_count": 0,
"partitions": [],
"key_metadata": null
}
](Two entries, one per manifest.)
This file is the table’s planning layer: a query planner reads it first (a few KB, one row per manifest) and already knows:
- which snapshot added each manifest (
added_snapshot_id), - how many files each manifest tracks (
added_files_count+existing_files_count+deleted_files_count), - what partition values live inside (
partitions, empty here because the table is unpartitioned), - and that
content: 0means this manifest tracks data files, not delete files.
On a real table this is the difference between planning that reads thousands of manifests and planning that skips most of them. The per-column min/max bounds that make file skipping possible live one level down, inside the manifests themselves, and Part 2 opens those. First, though, a control experiment that separates what Iceberg does from what the catalog does.
Same table, Hive catalog: what Iceberg does by default
Nearly everything surprising above was Nessie’s doing, and to show that I ran the identical experiment against a Hive Metastore: same Spark 4.1.3, same six rows, same two commits, with only the catalog changing. The catalog this time is Iceberg’s HiveCatalog over thrift, with warehouse = s3://warehouse/hive. Hive Metastore is the classic pointer store: it remembers where the current metadata lives, and leaves history to Iceberg.
Under Hive, the pointer lives in a database row. Nessie answered “where’s the table?” with an Iceberg REST call, but Hive Metastore has no REST endpoint: the pointer is a plain table property in the metastore’s backing database, queried with psql:
SELECT tp."PARAM_VALUE"
FROM "DBS" d
JOIN "TBLS" t ON t."DB_ID" = d."DB_ID"
JOIN "TABLE_PARAMS" tp ON tp."TBL_ID" = t."TBL_ID"
WHERE d."NAME" = 'iceberg_lab' AND t."TBL_NAME" = 'anatomy'
AND tp."PARAM_KEY" = 'metadata_location';s3://warehouse/hive/iceberg_lab.db/anatomy/metadata/00002-42c90e3c-5b73-4637-81de-5674261e2795.metadata.jsonIt’s the same arrangement as Nessie (one pointer, swapped atomically on each commit), except this one lives in a relational row instead of a REST response.
The bucket holds the same objects under different filenames. The physical table is the same shape, 11 objects in all: four Parquet, two manifests, two manifest lists, three metadata.json. Listing them is the same command as before, with the Hive prefix in place of the Nessie one:
AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD \
aws s3api list-objects-v2 \
--endpoint-url http://127.0.0.1:9000 \
--region us-east-1 \
--bucket warehouse \
--prefix hive/iceberg_lab.db/anatomy/{
"Contents": [
{
"Key": "hive/iceberg_lab.db/anatomy/data/00000-208-ea708a97-e3e7-4995-a26c-0bd46dc381a1-0-00001.parquet",
"LastModified": "2026-09-01T08:33:44.632Z",
"ETag": "\"90439210f7e92faaa656d74ba424788b\"",
"Size": 1178,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/data/00000-215-ea8aeb3f-7c6c-417c-90d0-68ed12a144d0-0-00001.parquet",
"LastModified": "2026-09-01T08:33:47.197Z",
"ETag": "\"1a13d69f322c02726019d3ba3d4620de\"",
"Size": 1192,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/data/00001-209-ea708a97-e3e7-4995-a26c-0bd46dc381a1-0-00001.parquet",
"LastModified": "2026-09-01T08:33:44.632Z",
"ETag": "\"c1605bf611200db5d0779ea3a33270e9\"",
"Size": 1190,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/data/00001-216-ea8aeb3f-7c6c-417c-90d0-68ed12a144d0-0-00001.parquet",
"LastModified": "2026-09-01T08:33:47.204Z",
"ETag": "\"a4d4b396c531fa5ab7fa0dbddfd5e7ac\"",
"Size": 1196,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/metadata/00000-94654b34-ecee-4793-95a3-e6c2728cc4e6.metadata.json",
"LastModified": "2026-09-01T08:33:43.046Z",
"ETag": "\"622ad59f4cbff8d62efb9db06b7f8f06\"",
"Size": 864,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/metadata/00001-72eee596-f259-4268-aa43-36494750acf7.metadata.json",
"LastModified": "2026-09-01T08:33:44.943Z",
"ETag": "\"e32b28717a7b710cdba04580739e0fa6\"",
"Size": 2006,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/metadata/00002-42c90e3c-5b73-4637-81de-5674261e2795.metadata.json",
"LastModified": "2026-09-01T08:33:47.318Z",
"ETag": "\"d192a43337749291976e474f569bb103\"",
"Size": 3117,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/metadata/83e7f5ed-828d-4a7f-86d8-942df53fa7ae-m0.avro",
"LastModified": "2026-09-01T08:33:44.875Z",
"ETag": "\"53ac31e81d9337d7b11ec3d8984d2d97\"",
"Size": 7264,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/metadata/c5ee174f-1abb-496d-92bc-1d93322fcca2-m0.avro",
"LastModified": "2026-09-01T08:33:47.270Z",
"ETag": "\"41a154d19bb30df7eb06adc00ae6cd55\"",
"Size": 7264,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/metadata/snap-1392298835344445060-1-c5ee174f-1abb-496d-92bc-1d93322fcca2.avro",
"LastModified": "2026-09-01T08:33:47.307Z",
"ETag": "\"1eeff09b5f71ec44c818aec42e80c5c1\"",
"Size": 4522,
"StorageClass": "STANDARD"
},
{
"Key": "hive/iceberg_lab.db/anatomy/metadata/snap-2731291236511896720-1-83e7f5ed-828d-4a7f-86d8-942df53fa7ae.avro",
"LastModified": "2026-09-01T08:33:44.934Z",
"ETag": "\"e45f5d1558fe4d866bf7fe2bda7ff478\"",
"Size": 4456,
"StorageClass": "STANDARD"
}
],
"RequestCharged": null,
"Prefix": "hive/iceberg_lab.db/anatomy/"
}The metadata filenames are where the two catalogs part ways. There’s the version counter Nessie never wrote: 00000, 00001, 00002. And notice that the files grow (864 B, then 2 KB, then 3 KB) because each version carries the accumulating history with it. Nessie’s two post-commit files were both exactly 1941 B, because each one carried only the present. The table directory is different too: iceberg_lab.db/anatomy, because Hive namespaces map to <namespace>.db and there’s no UUID suffix.
Inside the current file, the whole chain is intact. To see that we open 00002 itself, fetched the same way as the Nessie metadata above. Where Nessie’s file held a single snapshot, this one keeps both:
curl -s --aws-sigv4 'aws:amz:us-east-1:s3' \
-u "$MINIO_ROOT_USER:$MINIO_ROOT_PASSWORD" \
'http://127.0.0.1:9000/warehouse/hive/iceberg_lab.db/anatomy/metadata/00002-42c90e3c-5b73-4637-81de-5674261e2795.metadata.json' \
| jq .The file is longer than Nessie’s, so it also sits collapsed, with ← marks on the fields that differ:
Full 00002 metadata.json
{
"format-version": 2,
"table-uuid": "081a7091-765c-4260-bd44-4f8f77b0f263",
"location": "s3://warehouse/hive/iceberg_lab.db/anatomy", ← no UUID suffix: Hive maps the namespace to iceberg_lab.db
"last-sequence-number": 2,
"last-updated-ms": 1788251627311,
"last-column-id": 4,
"current-schema-id": 0,
"schemas": [
{
"type": "struct",
"schema-id": 0,
"fields": [
{
"id": 1,
"name": "id",
"required": false,
"type": "int"
},
{
"id": 2,
"name": "name",
"required": false,
"type": "string"
},
{
"id": 3,
"name": "amount",
"required": false,
"type": "double"
},
{
"id": 4,
"name": "ts",
"required": false,
"type": "timestamptz"
}
]
}
],
"default-spec-id": 0,
"partition-specs": [
{
"spec-id": 0,
"fields": []
}
],
"last-partition-id": 999,
"default-sort-order-id": 0,
"sort-orders": [
{
"order-id": 0,
"fields": []
}
],
"properties": {
"owner": "spark",
"write.parquet.compression-codec": "zstd"
}, ← none of Nessie's fingerprints: no gc.enabled=false, no nessie.commit.*
"current-snapshot-id": 1392298835344445060,
"refs": {
"main": {
"snapshot-id": 1392298835344445060,
"type": "branch"
}
},
"snapshots": [
{
"sequence-number": 1,
"snapshot-id": 2731291236511896720,
"timestamp-ms": 1788251624936,
"summary": {
"operation": "append",
"spark.app.id": "app-20260901083323-0001",
"manifests-created": "1",
"manifests-kept": "0",
"manifests-replaced": "0",
"added-data-files": "2",
"added-records": "3",
"added-files-size": "2368",
"changed-partition-count": "1",
"total-records": "3",
"total-files-size": "2368",
"total-data-files": "2",
"total-delete-files": "0",
"total-position-deletes": "0",
"total-equality-deletes": "0",
"engine-version": "4.1.3",
"app-id": "app-20260901083323-0001",
"engine-name": "spark",
"iceberg-version": "Apache Iceberg 1.8.1 (commit 9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d)",
"app-name": "iceberg-exp01c-hive-anatomy"
},
"manifest-list": "s3://warehouse/hive/iceberg_lab.db/anatomy/metadata/snap-2731291236511896720-1-83e7f5ed-828d-4a7f-86d8-942df53fa7ae.avro",
"schema-id": 0
}, ← commit #1's snapshot is still here (Nessie had dropped it)
{
"sequence-number": 2,
"snapshot-id": 1392298835344445060,
"parent-snapshot-id": 2731291236511896720, ← the parent link Nessie severed
"timestamp-ms": 1788251627310,
"summary": {
"operation": "append",
"spark.app.id": "app-20260901083323-0001",
"manifests-created": "1",
"manifests-kept": "1",
"manifests-replaced": "0",
"added-data-files": "2",
"added-records": "3",
"added-files-size": "2388",
"changed-partition-count": "1",
"total-records": "6",
"total-files-size": "4756",
"total-data-files": "4",
"total-delete-files": "0",
"total-position-deletes": "0",
"total-equality-deletes": "0",
"engine-version": "4.1.3",
"app-id": "app-20260901083323-0001",
"engine-name": "spark",
"iceberg-version": "Apache Iceberg 1.8.1 (commit 9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d)",
"app-name": "iceberg-exp01c-hive-anatomy"
},
"manifest-list": "s3://warehouse/hive/iceberg_lab.db/anatomy/metadata/snap-1392298835344445060-1-c5ee174f-1abb-496d-92bc-1d93322fcca2.avro",
"schema-id": 0
}
], ← both snapshots, not just the current one
"statistics": [],
"partition-statistics": [],
"snapshot-log": [
{
"timestamp-ms": 1788251624936,
"snapshot-id": 2731291236511896720
},
{
"timestamp-ms": 1788251627310,
"snapshot-id": 1392298835344445060
}
], ← two entries (Nessie kept only the last)
"metadata-log": [
{
"timestamp-ms": 1788251623028,
"metadata-file": "s3://warehouse/hive/iceberg_lab.db/anatomy/metadata/00000-94654b34-ecee-4793-95a3-e6c2728cc4e6.metadata.json"
},
{
"timestamp-ms": 1788251624938,
"metadata-file": "s3://warehouse/hive/iceberg_lab.db/anatomy/metadata/00001-72eee596-f259-4268-aa43-36494750acf7.metadata.json"
}
] ← remembers the two earlier files (this field was empty under Nessie)
}(Verbatim jq output, plus the ← annotations.)
Both commits are there, chained by parent-snapshot-id, and the snapshot-log and metadata-log at the bottom are populated too. This is the Iceberg default: every commit appends a snapshot, and the metadata accumulates until something explicitly expires it.
The SQL side agrees. After commit #2, .snapshots shows two rows, and the parent link that was severed under Nessie is intact:
SELECT committed_at, snapshot_id, parent_id, operation, manifest_list, summary FROM lakehouse_hive.iceberg_lab.anatomy.snapshots+-----------------------+-------------------+-------------------+---------+------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|committed_at |snapshot_id |parent_id |operation|manifest_list |summary |
+-----------------------+-------------------+-------------------+---------+------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|2026-09-01 08:33:44.936|2731291236511896720|NULL |append |s3://warehouse/hive/iceberg_lab.db/anatomy/metadata/snap-2731291236511896720-1-83e7f5ed-828d-4a7f-86d8-942df53fa7ae.avro|{spark.app.id -> app-20260901083323-0001, manifests-created -> 1, manifests-kept -> 0, manifests-replaced -> 0, added-data-files -> 2, added-records -> 3, added-files-size -> 2368, changed-partition-count -> 1, total-records -> 3, total-files-size -> 2368, total-data-files -> 2, total-delete-files -> 0, total-position-deletes -> 0, total-equality-deletes -> 0, engine-version -> 4.1.3, app-id -> app-20260901083323-0001, engine-name -> spark, iceberg-version -> Apache Iceberg 1.8.1 (commit 9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d), app-name -> iceberg-exp01c-hive-anatomy}|
|2026-09-01 08:33:47.31 |1392298835344445060|2731291236511896720|append |s3://warehouse/hive/iceberg_lab.db/anatomy/metadata/snap-1392298835344445060-1-c5ee174f-1abb-496d-92bc-1d93322fcca2.avro|{spark.app.id -> app-20260901083323-0001, manifests-created -> 1, manifests-kept -> 1, manifests-replaced -> 0, added-data-files -> 2, added-records -> 3, added-files-size -> 2388, changed-partition-count -> 1, total-records -> 6, total-files-size -> 4756, total-data-files -> 4, total-delete-files -> 0, total-position-deletes -> 0, total-equality-deletes -> 0, engine-version -> 4.1.3, app-id -> app-20260901083323-0001, engine-name -> spark, iceberg-version -> Apache Iceberg 1.8.1 (commit 9ce0fcf0af7becf25ad9fc996c3bad2afdcfd33d), app-name -> iceberg-exp01c-hive-anatomy}|
+-----------------------+-------------------+-------------------+---------+------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+and .history has two entries this time:
SELECT * FROM lakehouse_hive.iceberg_lab.anatomy.history+-----------------------+-------------------+-------------------+-------------------+
|made_current_at |snapshot_id |parent_id |is_current_ancestor|
+-----------------------+-------------------+-------------------+-------------------+
|2026-09-01 08:33:44.936|2731291236511896720|NULL |true |
|2026-09-01 08:33:47.31 |1392298835344445060|2731291236511896720|true |
+-----------------------+-------------------+-------------------+-------------------+None of Nessie’s fingerprints show up here either: the properties block in the metadata file above has no gc.enabled=false and no nessie.commit.*. Under Hive, garbage collection is back in Iceberg’s hands (expire_snapshots), and the snapshot chain survives until you actually run it.
So the picture that comes out of the two runs is this: the metadata tree (pointer, metadata.json, manifest lists, manifests, Parquet) is Iceberg, and it has the same shape under both catalogs. What the catalog decides is how much history the tree gets to keep. Nessie keeps the present and moves the history into its commit log, while Hive keeps the whole chain inside the metadata itself, so the same six rows end up with very different histories.
What I’ll take away
- A table is a tree of immutable files plus one mutable pointer. Six rows and two commits produced 11 objects (4 Parquet, 2 manifests, 2 manifest lists, 3 metadata.json), more metadata objects than data objects.
- One INSERT doesn’t mean one file. My three-row insert became two Parquet files because it ran as two tasks, and one of those files holds a single row. Small files are the default behavior; compaction is what you run to fix it.
- The catalog is essentially a pointer store. Schema, snapshots, refs, and commit summaries all live in
metadata.json, and any engine that can read that file can read the table. - Schema and partitioning are lists keyed by IDs. That’s what makes renaming a column and evolving partitions metadata-only operations.
- Catalogs are not interchangeable, and the Hive rerun showed it. The tree itself is Iceberg and identical everywhere; what differs is how much history the catalog lets it keep. Nessie names every
metadata.json00000-*, keeps only the current snapshot, and stores history in its commit log. Hive keeps the whole chain (versioned metadata files, both snapshots, an intact parent link) untilexpire_snapshotsruns. It’s worth picking a catalog knowing which of the two behaviors you need.
Next: Part 2 opens the manifests, decodes the per-column stats Iceberg uses to skip files, and reads the Parquet itself with no Iceberg involved.