First CRAN release.
New ducklake_extension_available() reports whether
the ducklake DuckDB extension is installed and loadable. It
probes with automatic extension installation switched off, so it never
downloads anything, and it is what the package’s own examples, tests,
and vignettes gate on.
Every example that can run against a temporary lake now does,
guarded by @examplesIf ducklake_extension_available(). Only
the cases that need outside infrastructure stay unevaluated: Quack
servers, PostgreSQL and MySQL catalogs, and remote URLs.
backup_ducklake() no longer needs the fs package. It
was the one place in the package that called a suggested dependency
unconditionally, so backups failed for anyone without fs
installed.
The package now tells you before it installs a DuckDB extension
for you. attach_ducklake() and the functions that load
httpfs, quack, or a backend extension used to
download into the extension cache in your home directory without a
word.
Fixed the with_transaction() rollback example, which
reused a table name that already existed and so failed before reaching
the error it meant to demonstrate.
New rows_upsert() completes the dplyr
rows_* family: rows that match on the key columns are
updated and the rest are inserted, as one atomic MERGE INTO
statement (DuckLake tables have no primary keys, so the
ON CONFLICT upsert other databases use does not apply). One
call is one snapshot, with the updates and inserts recorded individually
in the change feed.
New merge_into() exposes the full SQL MERGE surface
for the cases rows_upsert() cannot express: conditional
matched and not-matched clauses, deleting matched rows, and
delete_missing = TRUE to drop target rows absent from the
source (a staging-table sync). DuckLake currently allows one
update/delete action per MERGE statement, so syncs that need both run as
MERGE plus DELETE inside a single transaction and snapshot.
New table documentation family, built for labelled-data
workflows: set_table_comment() and
set_column_comments() store descriptions in the lake’s
catalog (COMMENT ON), and get_table_comments()
reads them back as a tidy data frame. create_table() gains
a labels argument (default TRUE) that stores
haven/labelled variable labels as column comments at load time, and
collect() on a lake table reattaches stored comments as
label attributes – so gtsummary, gt, and other label-aware
tools work as if the data never left R, and every other client of the
lake can read the same documentation.
New create_view() stores a dplyr pipeline as a SQL
view in the lake: shared business logic that reads current data and that
every client – R, Python, or plain SQL – sees identically.
drop_view() removes one. SQL macros stay unwrapped on
purpose (a macro body is raw SQL and unreachable from dplyr pipelines);
the cookbook shows the DBI::dbExecute() escape
hatch.
New list_ducklake_tables() answers “what is in this
lake?” with a tidy frame of tables and views.
New schema evolution family: add_table_column()
(with optional default, which DuckLake applies to existing
rows too), drop_table_column(),
rename_table_column(), set_column_type()
(widening promotions only, with the add-copy-drop-rename recipe in the
error when a change would narrow), and
rename_ducklake_table(). All are metadata-only
ALTER TABLE operations: no data files are rewritten, and
earlier snapshots keep the earlier schema. Until now schema changes went
through replace_table(), which collects the whole table
into R; its documentation now points here, and it remains the tool for
bulk data transformations. Derived columns combine the two
styles: add_table_column() then a mutate()
pipeline through ducklake_exec() fills the column with an
in-database UPDATE.
New plotting functions, all requiring the suggested ggplot2
package and shown in action in a new “Visualizing Your Lake” vignette:
plot_snapshots() draws a table’s snapshot history as a
commit-log timeline (snapshots in order, with authors, commit messages,
and inline markers for long idle gaps) or, without a table name, the
whole lake as a swimlane with one row per table;
plot_table_changes() draws the rows each snapshot inserted,
updated, and deleted as diverging bars; and
plot_table_files() draws each table’s Parquet file count
and size on disk.
New get_table_info() returns per-table file
statistics (data and delete file counts and sizes) from the DuckLake
catalog, wrapping DuckLake’s ducklake_table_info()
function.
New add_data_files() registers existing Parquet
files with a table without copying or rewriting them – the migration
path for data that is already in Parquet. A vector of files is
registered atomically in one snapshot, and create = TRUE
can bootstrap a new target table directly from the Parquet schema.
list_ducklake_files() shows the files backing a table,
optionally as of a past snapshot.
New sorted-table support: set_table_sorting() and
reset_table_sorting() manage a table’s declared sort order,
the complement to partitioning for pruning on high-cardinality
columns.
New set_ducklake_option() and
get_ducklake_options() expose DuckLake’s full option system
(parquet_compression, target_file_size,
sort_on_insert, require_commit_message, …) at
lake, schema, or table scope. set_inlining_row_limit() now
builds on the same internals.
New create_storage_secret() stores object-storage
credentials (S3, GCS, R2, Azure) via DuckDB’s secrets manager, so a
lake’s lake_path can live on cloud storage.
backup_ducklake() now errors clearly for remote data paths
instead of failing partway through.
attach_ducklake() gains
snapshot_version and snapshot_time arguments
to attach a lake pinned to a historical snapshot – a frozen, read-only
view for reproducing past analyses.
replace_table() now runs its drop and create as one
transaction, so a failed create no longer leaves the table dropped. When
the caller has already opened a transaction,
replace_table() defers to it as before.
set_snapshot_metadata() now validates
ducklake_name like the rest of the package, and resolves
the catalog backend from that name instead of the current
database.
create_table() no longer leaves a temporary view
registered on the shared connection when the CREATE statement
fails.
replace_table(.quiet = FALSE) reports progress via
messages (cli) rather than printing to the console, so it can be
suppressed and captured like the rest of the package’s output.
The package now declares R (>= 4.1) explicitly (the tests and examples use the base pipe), and is prepared for CRAN: extension-dependent tests skip on CRAN, and vignettes evaluate only where the ducklake DuckDB extension can be loaded.
New targeted maintenance wrappers complement
checkpoint_ducklake(): expire_snapshots()
(with older_than, versions, and
dry_run), merge_adjacent_files(),
cleanup_old_files(), delete_orphaned_files(),
and rewrite_data_files() (#16, suggested by @stefanlinner).
New partitioning support: set_table_partitioning()
and reset_table_partitioning() manage a table’s partition
keys (identity,
year/month/day/hour,
and bucket transforms), and
get_table_partitions() lists the keys from the metadata
catalog (#16, suggested by @stefanlinner).
New get_table_changes() exposes DuckLake’s data
change feed: the exact inserts, deletes, and update pre/post images
between two snapshots, as a lazy table that composes with dplyr verbs
(#16, suggested by @stefanlinner).
Timestamps passed to the new functions as POSIXct are converted to UTC before interpolation, matching how DuckLake records snapshot times.
get_ducklake_table_asof() and
restore_table_version() now also convert POSIXct timestamps
to UTC. Previously they rendered local time, which DuckLake reads as
UTC, silently shifting the queried instant by the UTC offset – a bare
Sys.time() looked hours in the past (or future) unless the
session’s timezone was UTC. Timestamps taken from
list_table_snapshots()$snapshot_time are
unaffected.
attach_ducklake() now collapses duplicate slashes in
lake_path (remote URIs are untouched). DuckLake compares
file paths as exact strings, so a doubled slash – which R’s
tempdir() produces on macOS – made
delete_orphaned_files() treat every live data file as
orphaned.
The dplyr-to-DuckLake translation behind
ducklake_exec() and show_ducklake_query() is
now built from dbplyr’s structured query objects
(dbplyr::sql_build()) instead of pattern-matching rendered
SQL text. Classification no longer depends on what the SQL happens to
look like, which fixes several latent bugs:
get_ducklake_table("staging") |> filter(...) |> ducklake_exec("target"))
was translated into a DELETE on the target table; it now
appends the matching rows, as intended.filter(note != "WHERE is it")) were refused as “too
complex”; they now translate fine.mutate() that adds a new column is refused upfront
with a pointer to replace_table(), instead of failing with
a database binder error.INSERT translations now list columns explicitly, so
appends from another table match columns by name rather than by
position, and joined or unioned sources can be appended in one
step.
Pipelines that compile to a subquery over the target table
(grouped filters, filtering on a just-mutated column), and clauses with
no in-place equivalent (arrange(), head(),
distinct()), are detected structurally and refused with a
clear message rather than mistranslated.
dbplyr (>= 2.5.0) is now required; both dbplyr 2.5.x and the select-list format introduced in dbplyr 2.6.0 are supported.
This release focuses on production hardiness: self-contained connection management, working detach/restore, SQL identifier safety, Quack remote access, and a documentation overhaul.
Added support for Quack, DuckDB’s client-server protocol, which became a core extension in DuckDB 1.5.3 (#20, @JavOrraca). A DuckLake served by one DuckDB instance can now be queried and modified by other R sessions over the network. For concurrent access this is a lighter-weight option than a PostgreSQL or SQLite catalog, since the whole setup stays inside DuckDB and DuckLake.
attach_quack() connects to a remote Quack server and
attaches it as a catalog in the current session.detach_quack() disconnects from a remote Quack
server.install_quack() installs the Quack DuckDB
extension.quack_query() runs a one-off query against a remote
Quack server and returns a data.frame.quack_serve() serves the current session, including an
attached DuckLake, to other clients over Quack.quack_stop() stops a running Quack server.attach_ducklake() gains an encrypted
argument: pass encrypted = TRUE to have DuckLake encrypt
the Parquet files it writes (#18). Note that the encryption keys are
stored in the catalog database, so protect the catalog. The httpfs
extension is loaded automatically for encrypted lakes, since on some
platforms (notably Windows) DuckDB’s built-in crypto module is
read-only.restore_table_version() now works. It previously
generated a RESTORE TABLE statement that does not exist in
DuckLake and failed on every call. It now recreates the table from a
time-travel read inside a transaction, recording the restore as a new
snapshot so history is preserved. It also gains author and
commit_message arguments so the restore snapshot carries
full audit-trail metadata.get_ducklake_backend() gains a
ducklake_name argument and tracks each attached lake
separately, so sessions with several lakes on different catalog backends
resolve backend-specific behaviour correctly.detach_ducklake() now actually detaches. Previously the
DETACH ran while the lake was still the session’s current
database, which DuckDB refuses, and the error was silently swallowed –
the lake stayed attached. The session now switches back to the
connection’s own catalog first. Relatedly, restoring a backup to a new
location requires override_data_path = TRUE (as
documented); the storage vignette example has been corrected.DBI::dbQuoteIdentifier() and friends), so names with
spaces or quotes no longer produce malformed statements.rows_insert(), rows_update(), and
rows_delete() now also dispatch as S3 methods on tables
returned by get_ducklake_table(). Previously, if dplyr was
loaded after ducklake, dplyr’s generics masked ducklake’s
wrappers and calls failed with conflict = "error"
complaints; load order no longer matters.rows_insert(), rows_update(), and
rows_delete() now work inside
with_transaction(), so several row operations can be
grouped into a single snapshot with an author and commit message.
Previously, passing a local data frame made dbplyr copy it to a
temporary table inside its own transaction, which DuckDB rejects when
one is already open. Local data frames are now sent as inline queries
(dbplyr::copy_inline()), which is also faster for the small
changesets these functions are designed for.backup_ducklake() backs up every schema directory, not
just main.create_table() now converts factor columns to character
(with a message) instead of failing with “unsupported type ENUM” –
DuckLake does not support DuckDB’s ENUM type, which is what factors
become.ducklake_exec() no longer uses sink() (which
could leak diverted output on error), and now refuses queries with
subqueries or multiple WHERE clauses instead of generating
incorrect SQL.ducklake_exec() no longer executes its statement twice.
The internal translation step also executed the SQL before
ducklake_exec() ran it again, so every call created two
snapshots and non-idempotent updates (e.g. v = v + 1) were
applied twice.show_ducklake_query() is now a true preview: it
previously executed the translated statement against the lake
while displaying it.ducklake_exec() now translates any
mutate() into an UPDATE, not just those that compile to
CASE WHEN. Previously a simple transformation like
mutate(v = round(v, 1)) fell through to an INSERT of the
table’s own rows, silently duplicating the table. Plain self-reads with
nothing to translate are now refused for the same reason, and UPDATE
assignments containing commas inside function calls are parsed
correctly.list_table_snapshots(table_name) no longer misses
snapshots created by rows_insert(),
rows_update(), and rows_delete(). DuckLake
records row-level changes against the table’s numeric id rather than its
name; the filter now resolves and matches those ids, so the per-table
audit trail is complete. Filtered listings also number their rows from 1
instead of leaking the row positions of the unfiltered result.ducklake now creates and manages its own DuckDB connection instead of
reaching into duckplyr’s unexported internals. This removes the
package’s last ::: calls and the duckplyr dependency
entirely.
set_ducklake_connection().set_ducklake_connection() (returning by popular demand,
now safer): point ducklake at any DuckDB connection you manage — for
example one shared with other DBI tools. Connections you supply are
never closed by ducklake; only its own automatically created connection
is shut down by detach_ducklake(shutdown = TRUE) and at
session exit.This release aligns the package with the DuckLake v1.0 stable specification, which requires DuckDB v1.5.2+ (compatible with duckdb R package >= 1.5.1).
DuckDB version requirement bumped from 1.3.0 to
1.5.1 (duckdb R package) / 1.5.2
(DuckDB engine/CLI) to match DuckLake v1.0.
install_ducklake() now enforces this at the engine
level.
commit_transaction() and
with_transaction() now use the official
CALL ducklake.set_commit_message() API to set commit
metadata within the transaction before
COMMIT, consistent with the v1.0 specification.
set_snapshot_metadata() retroactively updates the
ducklake_snapshot_changes metadata table directly.
DuckLake now supports PostgreSQL, SQLite, and MySQL as catalog backends in addition to DuckDB (#15, @stefanlinner). This aligns with the DuckLake 1.0 specification and enables concurrent multi-client access when using PostgreSQL or SQLite.
attach_ducklake() gains backend,
catalog_connection_string, read_only, and
override_data_path parameters for multi-backend
support.install_ducklake() gains a backend
parameter to pre-install backend extensions (e.g.,
install_ducklake(backend = "postgres")).get_ducklake_backend() returns the active catalog
backend type.detach_ducklake() gains a shutdown
parameter. By default it now performs a soft detach (SQL
DETACH + USE memory;) instead of shutting down
the connection, allowing backend switching within a session.backup_ducklake() is now backend-aware: file-based
backends (DuckDB, SQLite) get catalog + data copied; PostgreSQL/MySQL
get data only with guidance to use
pg_dump/mysqldump. Also fixes a pre-existing
bug where catalog backups were silently 0 bytes due to DuckDB holding
file locks during file.copy().attach_ducklake() now requires
lake_path (previously optional).set_ducklake_connection() has been removed. The package
now exclusively uses duckplyr’s singleton DuckDB connection.detach_ducklake() no longer shuts down the DuckDB
connection by default. Pass shutdown = TRUE for the
previous behaviour.get_metadata_table(), time_travel.R,
transactions.R) to handle PostgreSQL/MySQL backends that
don’t use the .main. schema prefix.build_attach_sql(),
ensure_extensions(),
shutdown_and_reset_singleton().Initial release of ducklake, an R package for versioned data lake infrastructure built on DuckDB and DuckLake.
create_table() - Create new tables in the data
lakeget_ducklake_table() - Retrieve tables as tibblesreplace_table() - Replace entire table contents with
versioningrows_insert() - Insert new rows with automatic
versioningrows_update() - Update existing rows with audit
trailrows_delete() - Delete rows while maintaining
historywith_transaction() - Execute code blocks within
transactionsbegin_transaction(), commit_transaction(),
rollback_transaction() - Manual transaction controlget_ducklake_table_asof() - Query table state at
specific timestampsget_ducklake_table_version() - Retrieve specific table
versionslist_table_snapshots() - View complete version
historyrestore_table_version() - Roll back to previous
versionsget_metadata_table() - Access comprehensive
metadataset_snapshot_metadata() - Add author, commit messages,
and tagsinstall_ducklake() - Install/update DuckLake
extensionattach_ducklake() - Initialize data lake
connectionsdetach_ducklake() - Clean up connectionsget_ducklake_connection() - Retrieve the active DuckDB
connectionducklake_exec() - Execute SQL with automatic assignment
handlingshow_ducklake_query() - Preview translated SQL
queriesextract_assignments_from_sql() - Parse SQL table
assignmentsbackup_ducklake() - Create incremental backupsThis package is currently in experimental status. The API may change as we gather feedback from early users, but core functionality is stable and ready for pilot projects.