Storage Options
DuckLake works with any filesystem backend that DuckDB supports,
including:
- Local files and folders: Fast access, ideal for
single-machine workflows
- Cloud object storage:
- AWS S3 (and S3-compatible services like Cloudflare R2, MinIO)
- Google Cloud Storage
- Azure Blob Storage
- Network-attached storage: NFS, SMB, FUSE-based
filesystems
Storage Patterns
Here are how some common storage patterns may look:
# Local storage - fastest, but not shared
attach_ducklake(
ducklake_name = "local_lake",
lake_path = "~/data/my_ducklake"
)
# PostgreSQL catalog with S3 data - multi-client, scalable
attach_ducklake(
ducklake_name = "shared_lake",
backend = "postgres",
catalog_connection_string = "dbname=ducklake_catalog host=localhost",
lake_path = "s3://my-bucket/ducklake/data"
)
# SQLite catalog - lightweight multi-client option
attach_ducklake(
ducklake_name = "team_lake",
backend = "sqlite",
catalog_connection_string = "~/data/metadata.sqlite",
lake_path = "~/data/parquet_files"
)
Key considerations:
- Latency vs. accessibility: Local storage is fast
but not shareable; cloud storage is accessible but has higher
latency
- Scalability vs. cost: Object stores scale easily
but may charge for data transfer
- Security: Consider using DuckLake’s encryption
features for cloud storage
Cloud Storage Credentials
Object storage needs credentials before a remote
lake_path will work. create_storage_secret()
registers them with DuckDB’s secrets manager:
# Explicit keys, scoped to one bucket
create_storage_secret(
"s3",
key_id = Sys.getenv("AWS_ACCESS_KEY_ID"),
secret = Sys.getenv("AWS_SECRET_ACCESS_KEY"),
region = "us-east-1",
scope = "s3://my-bucket"
)
# Or let the AWS credential chain find them (env vars, profiles,
# instance metadata) -- no keys in code
create_storage_secret("s3", provider = "credential_chain")
# Then attach as usual
attach_ducklake("shared_lake", lake_path = "s3://my-bucket/ducklake/data")
Secrets are in-memory by default and disappear with the session; pass
persistent = TRUE only if you are comfortable with DuckDB
writing them, unencrypted, under ~/.duckdb/. GCS,
Cloudflare R2, and Azure use the same function with
type = "gcs", "r2", or
"azure".
Note that backup_ducklake() works on local data paths
only. For a lake on object storage, use your provider’s replication or
sync tooling (bucket versioning, aws s3 sync, and similar)
for the data files, and back up the catalog database with the tools for
its backend.
Inspecting DuckLake Files
Let’s create a sample DuckLake and explore what files it
generates:
# Create a temporary directory for our demo
lake_dir <- file.path(vignette_temp_dir, "storage_demo")
dir.create(lake_dir, showWarnings = FALSE, recursive = TRUE)
# Install ducklake extension
# The ducklake extension only needs installing once per machine:
# install_ducklake()
# Create and populate a DuckLake
attach_ducklake(
ducklake_name = "demo_lake",
lake_path = lake_dir
)
# Add some data with transactions
with_transaction(
create_table(mtcars[1:15, ], "cars"),
author = "Demo User",
commit_message = "Initial load"
)
#> Transaction started.
#> Transaction committed.
with_transaction(
get_ducklake_table("cars") |>
mutate(hp_per_cyl = hp / cyl) |>
replace_table("cars"),
author = "Demo User",
commit_message = "Add hp_per_cyl metric"
)
#> Transaction started.
#> Transaction committed.
with_transaction(
get_ducklake_table("cars") |>
mutate(mpg_adjusted = if_else(cyl == 4, mpg * 1.1, mpg)) |>
replace_table("cars"),
author = "Demo User",
commit_message = "Add adjusted MPG for 4-cylinder cars"
)
#> Transaction started.
#> Transaction committed.
Catalog Files
The catalog is a single database file containing all metadata:
dir_tree(lake_dir)
#> /var/folders/b7/664jmq55319dcb7y4jdb39zr0000gq/T//RtmpBfCpem/storage_backups_vignette/storage_demo
#> ├── demo_lake.ducklake
#> ├── demo_lake.ducklake.wal
#> └── main
#> └── cars
#> ├── ducklake-01a0439c-faa2-703b-ac19-a0576afb538c.parquet
#> ├── ducklake-01a0439c-faca-764b-afb5-bc3895007dbb.parquet
#> └── ducklake-01a0439c-faf6-7691-9eb7-a1b052ea5196.parquet
The catalog files (demo_lake.ducklake and
.wal) contain all metadata about tables, snapshots, and
transactions.
Storage (Data) Files
Data files are stored in Parquet format in a structured
directory:
# Data files are organized by schema and table
main_dir <- file.path(lake_dir, "main")
dir_tree(main_dir, recurse = 2)
#> /var/folders/b7/664jmq55319dcb7y4jdb39zr0000gq/T//RtmpBfCpem/storage_backups_vignette/storage_demo/main
#> └── cars
#> ├── ducklake-01a0439c-faa2-703b-ac19-a0576afb538c.parquet
#> ├── ducklake-01a0439c-faca-764b-afb5-bc3895007dbb.parquet
#> └── ducklake-01a0439c-faf6-7691-9eb7-a1b052ea5196.parquet
# Get details about parquet files
parquet_files <- dir_ls(main_dir, recurse = TRUE, regexp = "\\.parquet$")
for (f in parquet_files) {
cat(sprintf(" %s (%s bytes)\n",
path_file(f),
file.size(f)))
}
#> ducklake-01a0439c-faa2-703b-ac19-a0576afb538c.parquet (2306 bytes)
#> ducklake-01a0439c-faca-764b-afb5-bc3895007dbb.parquet (2503 bytes)
#> ducklake-01a0439c-faf6-7691-9eb7-a1b052ea5196.parquet (2726 bytes)
Understanding File Organization
Each table’s data is organized by schema and table, with each
transaction creating new Parquet files:
# List all snapshots to see the version history
snapshots <- list_table_snapshots("cars")
snapshots |>
select(snapshot_id, author, commit_message)
#> snapshot_id author commit_message
#> 1 1 Demo User Initial load
#> 2 2 Demo User Add hp_per_cyl metric
#> 3 3 Demo User Add adjusted MPG for 4-cylinder cars
The key insight is that DuckLake never modifies or deletes
existing Parquet files. Each change creates new files,
preserving the complete history for time travel queries.
Routine Maintenance
A lake that sees regular writes accumulates small Parquet files (one
per insert) and old snapshots whose files can no longer be reclaimed
until the snapshots are expired. The one-stop command is
checkpoint_ducklake(), which flushes inlined data, merges
small files, expires old snapshots, and cleans up unreferenced files in
a single call. For finer control, each step has its own function:
# Compact small adjacent Parquet files into larger ones
merge_adjacent_files()
# Preview a retention policy, then apply it
expire_snapshots(older_than = Sys.time() - 30 * 24 * 60 * 60, dry_run = TRUE)
expire_snapshots(older_than = Sys.time() - 30 * 24 * 60 * 60)
# Expired snapshots only *schedule* file deletion; this reclaims the storage
cleanup_old_files(cleanup_all = TRUE)
# Rewrite data files whose rows have mostly been deleted
rewrite_data_files(delete_threshold = 0.5)
# Remove untracked files from the data path -- always dry-run this one first
delete_orphaned_files(dry_run = TRUE, cleanup_all = TRUE)
The typical cycle is merge, then expire, then clean up: merging and
expiring both mark files as unreferenced, and
cleanup_old_files() deletes them. Expiring a snapshot gives
up time travel to it, so choose older_than to match how far
back you need to audit or restore.
One task lives outside DuckLake itself: the catalog database. If you
use a PostgreSQL or SQLite catalog, occasionally run VACUUM
there with that database’s own tooling so metadata queries stay fast.
The default DuckDB-file catalog does not need this.
Maintenance Considerations
When planning backups, coordinate with maintenance operations:
- Compaction (merging adjacent files): Run before
backups to ensure consistent file layout
- Cleanup (removing obsolete files): Run before
backups to avoid backing up unnecessary files
# Recommended backup sequence
# 1. Run maintenance operations (if needed)
merge_adjacent_files()
expire_snapshots(older_than = Sys.time() - 30 * 24 * 60 * 60)
cleanup_old_files(cleanup_all = TRUE)
# 2. Ensure all transactions are committed
# (no pending work)
# 3. Release file locks before copying the catalog
detach_ducklake("demo_lake", shutdown = TRUE)
# 4. Back up catalog
dir.create(file.path(lake_dir, "backups"), showWarnings = FALSE)
file.copy(
from = file.path(lake_dir, "demo_lake.ducklake"),
to = file.path(lake_dir, "backups",
paste0("backup_", format(Sys.time(), "%Y%m%d_%H%M%S"), ".ducklake"))
)
# 5. Back up data files
dir_copy(
path = file.path(lake_dir, "main"),
new_path = file.path(lake_dir, "backups", "main_latest")
)
# 6. Re-attach and continue working
attach_ducklake("demo_lake", lake_path = lake_dir)
Complete Backup Example
DuckLake provides a convenient backup_ducklake()
function for creating timestamped backups:
# Create a complete backup with timestamp
backup_dir <- backup_ducklake(
ducklake_name = "demo_lake",
lake_path = lake_dir,
backup_path = file.path(lake_dir, "backups")
)
#> Catalog backed up successfully.
#> Data files backed up successfully (1 directory).
#> Backup completed:
#> '/var/folders/b7/664jmq55319dcb7y4jdb39zr0000gq/T//RtmpBfCpem/storage_backups_vignette/storage_demo/backups/backup_20260827_072557'
# The function returns the backup directory path
print(backup_dir)
#> [1] "/var/folders/b7/664jmq55319dcb7y4jdb39zr0000gq/T//RtmpBfCpem/storage_backups_vignette/storage_demo/backups/backup_20260827_072557"
The backup_ducklake() function: - Creates a timestamped
backup directory - Copies the catalog database file (releasing file
locks first) - Copies the data files from every schema directory in the
lake - Returns the backup directory path for reference