---
title: "Provenance and archiving: an agent run as evidence"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Provenance and archiving: an agent run as evidence}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include=FALSE}
knitr::opts_chunk$set(
  collapse = TRUE, comment = "#>",
  eval = identical(tolower(Sys.getenv("LLMRAGENT_RUN_VIGNETTES", "false")), "true")
)
```

When a language model produces the data you will analyze, the model is an
instrument. Instruments need a record: what was asked, which model answered,
how many tokens it cost, and whether anything failed. Without that record a
result cannot be checked, and a study that cannot be checked is hard to defend.
This vignette shows how each LLMRagent run records itself, and how to write an
archive a reader can open months later.

```{r setup}
library(LLMRagent)
cfg <- LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0.7)
```

## One object, five views

Anything you run, whether a single chat, a deliberation, or an experiment,
converts to one object through `as_agent_run()`. That object answers five
questions, and you ask each by naming a level:

```{r run}
panel <- list(
  agent("Optimist", cfg, persona = "You see opportunity first."),
  agent("Skeptic",  cfg, persona = "You probe for hidden costs.")
)
debate <- deliberate(panel, "Should the department adopt a four-day week?",
                     rounds = 1)

run <- as_agent_run(debate)
```

The levels are deliberately plain tibbles, so the rest of your analysis is
ordinary data work:

```{r levels}
tibble::as_tibble(run, level = "utterance")  # who said what, in order
tibble::as_tibble(run, level = "call")       # one row per model call
tibble::as_tibble(run, level = "event")      # the full event graph
tibble::as_tibble(run, level = "tool")       # every tool invocation
tibble::as_tibble(run, level = "state")      # each agent's memory at the end
```

The `utterance` level is the transcript: the thing you usually read. The `call`
level is the instrument log: every model call as a row, with the served model
version, the token counts, a success flag, and a `request_hash` that uniquely
identifies the exact request that produced it. The agent recorded all of this
while it answered.

## Diagnostics for a run

`diagnostics()` reads the call level and returns a one-row summary. The numbers
come from the same place the wider 'LLMR' ecosystem reads them, so they agree
across tools rather than reporting two different figures:

```{r diag}
diagnostics(run)
```

A high failure count, an unexpected model version, a token bill larger than the
work warrants -- these surface here before they surface in your conclusions.

## Reporting within scope

`report()` writes a short methods paragraph and, importantly, refuses to
overstate. An agent run is model-conditioned output, not a sample from a human
population, and `report()` says so unless you have attached a calibration (see
the validity vignette). The report carries that scope condition unless a
calibration is attached.

```{r report}
report(run)
```

## The manifest: identity without the transcript

Two runs are "the same study" when their design is the same: same personas,
same tools, same turn policy, same model and settings. They are emphatically
not the same when any of those change, even if the transcripts happen to read
alike. `agent_manifest()` captures design identity as a set of content hashes,
keeping outcome (the transcript) separate from apparatus (the design):

```{r manifest}
m <- agent_manifest(run)
m$manifest_hash    # changes iff the design changes
```

A reworded persona, a changed tool body, or a different temperature moves the
hash. Re-run the identical design and it stays put. This is the same split
'LLMR' makes between a request's identity and its content, lifted to the level
of a whole study.

## Archiving: seal the evidence

`archive_agent_study()` writes a self-describing directory. The transcript, the
event graph, the manifest, the artifacts (votes, verdicts, designs), a drafted
methods note, and a checksum file that seals the whole thing. The centerpiece is
`calls.jsonl`, which is a genuine 'LLMR' audit log: you can read it back with
'LLMR' itself.

```{r archive}
dir <- file.path(tempdir(), "four_day_week_study")
archive_agent_study(run, dir)
list.files(dir)
```

The archive keeps the original hashes. The hashes are computed over the
original text, so they match the live run even if you redact the stored copy for
privacy. And the request hash in `calls.jsonl` joins to the `request_hash` you
saw at the call level. That join is the integrity check: it lets a skeptic
confirm that the archived log describes the run it claims to describe.

```{r join}
calls   <- tibble::as_tibble(run, level = "call")
archived <- LLMR::llm_log_read(file.path(dir, "calls.jsonl"))

# every archived call is one of the calls the run actually made
all(archived$manifest$request_hash %in% calls$request_hash)
```

If you keep a live audit log during the run (call
`LLMR::llm_log_enable("calls.jsonl")` before you start), the archive copies that
real file, filtered to this run's calls, rather than reconstructing it, and so
preserves the log's own record hashes.

## Redaction without lying

Survey responses and interview text often cannot be stored verbatim. `redact`
scrubs the free-text columns of the stored copy, while the hashes stay computed
over the original. The archive remains internally consistent: it does not claim
a request body it did not send, and the join still holds.

```{r redact}
archive_agent_study(run, file.path(tempdir(), "redacted_study"),
                    redact = c("four-day", "salary"))
```

For privacy that must omit the request entirely, pass `include_messages = FALSE`:
the archive then carries metadata and hashes but no message text at all.

## Why archive

A reviewer asks which model you used and whether you cherry-picked. A
co-author wants to re-run last month's study and get a comparable result. A
replication team, a year on, has only your files. In each case the answer is the
archive: the run, the manifest, and the log that joins them. The archive makes a
model-generated result something you can inspect, rerun, and defend.