Introduction
A DuckLake keeps detailed records about itself: every change to a
table is a snapshot, every changed row is in the change feed, and every
Parquet file is tracked in the catalog. All of that comes back to R as
data frames, but past a handful of rows a picture is easier to scan.
This vignette walks through the package’s three plotting functions:
plot_snapshots() draws a table’s history as a
commit-log style timelineplot_table_changes() shows how many rows each snapshot
changedplot_table_files() shows how each table’s data is laid
out on disk
These require the ggplot2 package, which is suggested (not required)
by ducklake.
Building Some History
We’ll create a small lake and put a table through a few changes,
recording authors and commit messages along the way.
# The ducklake extension only needs installing once per machine:
# install_ducklake()
attach_ducklake(
ducklake_name = "lake_viz_demo",
lake_path = vignette_temp_dir
)
fleet <- tibble(
car_id = 1:5,
model = c("Corolla", "Civic", "Model 3", "F-150", "Outback"),
mileage = c(42000, 38000, 12000, 67000, 55000)
)
with_transaction(
create_table(fleet, "fleet"),
author = "Data Engineer",
commit_message = "Initial fleet load"
)
#> Transaction started.
#> Transaction committed.
Now a couple of routine updates:
new_cars <- tibble(
car_id = 6:7,
model = c("CX-5", "Ioniq 5"),
mileage = c(21000, 8000)
)
with_transaction(
rows_insert(get_ducklake_table("fleet"), new_cars, by = "car_id"),
author = "Fleet Manager",
commit_message = "Add March arrivals"
)
#> Transaction started.
#> Transaction committed.
serviced <- tibble(
car_id = c(1, 3, 5),
mileage = c(42750, 13400, 55900)
)
with_transaction(
rows_update(get_ducklake_table("fleet"), serviced, by = "car_id"),
author = "Fleet Manager",
commit_message = "Record spring odometer readings"
)
#> Transaction started.
#> Transaction committed.
sold <- tibble(car_id = 4L)
with_transaction(
rows_delete(get_ducklake_table("fleet"), sold, by = "car_id"),
author = "Fleet Manager",
commit_message = "Remove sold F-150"
)
#> Transaction started.
#> Transaction committed.
The audit trail so far:
list_table_snapshots("fleet") |>
select(snapshot_id, snapshot_time, author, commit_message)
#> snapshot_id snapshot_time author commit_message
#> 1 1 2026-08-27 14:26:03 Data Engineer Initial fleet load
#> 2 2 2026-08-27 14:26:03 Fleet Manager Add March arrivals
#> 3 3 2026-08-27 14:26:03 Fleet Manager Record spring odometer readings
#> 4 4 2026-08-27 14:26:03 Fleet Manager Remove sold F-150
Plotting the Timeline
plot_snapshots() turns that data frame into a
picture:
Reading it like a git log: the newest snapshot is at the top, each
row shows the snapshot’s timestamp, author, and commit message, and the
color shows whether the snapshot created the table, changed data,
changed the schema, or was maintenance activity (like flushing inlined
data or compacting files). Snapshots are spaced evenly by order rather
than by clock time, so a table that sat untouched for months stays
readable: a long idle stretch shows up as an inline marker (like “103
days later”) instead of stretching the whole plot.
How the Data Is Stored
Behind the scenes, table data lives in Parquet files (very small
writes may be inlined into the catalog until they are flushed out). To
make the storage view interesting, let’s add a larger table alongside
fleet and grow it in two batches, so it ends up with two
data files:
set.seed(42)
telemetry <- tibble(
reading_id = 1:5000,
car_id = sample(1:7, 5000, replace = TRUE),
speed = round(runif(5000, 0, 80)),
fuel = round(runif(5000, 0, 100))
)
create_table(telemetry, "telemetry")
more_readings <- tibble(
reading_id = 5001:10000,
car_id = sample(1:7, 5000, replace = TRUE),
speed = round(runif(5000, 0, 80)),
fuel = round(runif(5000, 0, 100))
)
rows_insert(get_ducklake_table("telemetry"), more_readings, by = "reading_id")
The fleet table’s handful of rows were small enough to be inlined, so
we flush them into a Parquet file to make them visible to the storage
statistics:
flush_inlined_data()
#> Flushed 7 rows from 1 table to Parquet.
#> schema_name table_name rows_flushed
#> 1 main fleet 7
get_table_info() reports each table’s file count and
size:
get_table_info()
#> table_name schema_id table_id table_uuid file_count
#> 1 fleet 0 1 01a0439d-15ce-7869-b573-bd49f0fcadad 2
#> 2 telemetry 0 2 01a0439d-1a0d-739d-8935-8abbb0f55fc4 2
#> file_size_bytes delete_file_count delete_file_size_bytes
#> 1 1662 1 1349
#> 2 65534 0 0
And plot_table_files() draws it, with delete files (rows
removed but not yet compacted away) stacked in a second color:
This is the picture to watch as a lake ages: many small files or a
growing delete-file share mean it is time for the compaction tools
covered in vignette("storage-and-backups"), like
merge_adjacent_files() and
rewrite_data_files().
Plotting the Whole Lake
With two tables in the lake, we can step back and look at everything
at once. Calling plot_snapshots() without a table name
switches to a swimlane view: one row per table, one point per snapshot,
with recently active tables at the top. It answers a different question
than the single-table timeline – not “what happened to this table” but
“which tables are active, and what kind of churn does each one see”:
Snapshots that don’t belong to any table, like the initial schema
creation, appear in a (lake) lane. As in the timeline,
points are spaced by order rather than clock time; the x axis labels
show each snapshot’s date.
Customizing the Plots
Each function returns a regular ggplot object, so you can restyle it
like any other plot. (The single-table timeline draws on a blank canvas,
so it is best customized with labs() and
theme() tweaks rather than a full theme swap like
theme_classic(), which would bring back axes that have no
meaning there.)
plot_snapshots("fleet") +
ggplot2::labs(title = "Fleet table audit trail")
