Package {ragR}

ragR: Retrieval-Augmented Generation and Evaluation Tools

Description

The ragR package provides tools for building and evaluating retrieval-augmented generation systems in R. It includes:

• A data-ingestion pipeline for PDF, text, and Word documents

• An R-native vector store for document chunks and embeddings

• A configurable retrieval-augmented generation pipeline

• Persistent question-answer logging for reproducible evaluation

• Large-language-model-scored evaluation metrics, including context precision, context recall, answer relevance, and faithfulness

• A Plumber-based HTTP API suitable for chatbot and evaluation frontends

Architecture

The package is organized into several modules:

• Ingestion (ingestion_files.R)

  Extracts text from PDF, TXT, and DOCX files, divides the text into chunks, and prepares those chunks for embedding and storage. The vector-store path must be supplied explicitly by the caller.

• Embeddings and chat (embeddings_openai.R)

  Wraps the OpenAI REST API for embedding models, such as "text-embedding-3-small", and chat models, such as "gpt-4o-mini". The API key is read from the OPENAI_API_KEY environment variable.

• Vector store (vectorstore_interface.R)

  Implements an R-native vector store using tibbles and numeric matrices. Functions are provided to load the store, add or update records, retrieve similar chunks, delete collections, and clear the complete store.

  All vector-store functions require an explicit storage path.

• Retrieval-augmented generation pipeline (rag_pipeline.R)

  Given a user question, the pipeline:

  1. Computes an embedding for the question

  2. Retrieves similar chunks from the selected vector-store collection

  3. Builds a grounded, context-aware prompt

  4. Calls a chat model to generate an answer

• Question-answer logging and persistence (qa_logging.R, qa_persistence.R)

  Stores interactions including the question, model answer, retrieved chunks, final grounded prompt, model names, collection name, and timestamp. Helpers are provided to load, save, and clear logs.

  Log locations must be supplied explicitly.

• Evaluation metrics and summaries (ragas_metrics.R, ragas_summary.R)

  Computes large-language-model-scored evaluation metrics, including context precision, context recall, answer relevance, faithfulness, and an overall score. Summary helpers aggregate metric results across question-answer pairs.

• Reports (ragas_report.R)

  Generates an evaluation report by:

  • Computing metrics from a saved question-answer log

  • Saving the resulting metrics to an explicitly supplied RDS path

  • Writing a summary CSV to an explicitly supplied output directory

  • Writing a bar chart of mean metric values to that output directory

• API handlers (api_handlers.R)

  Connect the core package functions to HTTP endpoints. Storage paths are supplied by the server configuration rather than by client request bodies or hidden package defaults.

HTTP API

The development server in scripts/dev/run_api.R exposes:

• POST /chat

  Runs the retrieval-augmented generation pipeline for a user question and appends the interaction to the configured question-answer log.

• GET /ragas

  Computes and returns evaluation metrics and a summary for the logged question-answer interactions.

• POST /ragas/report

  Computes metrics from the configured question-answer log and writes the metrics, summary CSV, and plot to server-configured locations.

• POST /ragas/clear

  Clears the configured question-answer log, metrics file, and report directory.

• POST /clear

  Clears one collection from the configured vector store.

• POST /clear_all

  Clears all collections from the configured vector store using api_clear_all_handler() and vectorstore_clear_all().

The development server uses a temporary storage directory unless a persistent location is supplied through the RAGR_API_DATA_DIR environment variable.

Usage

Programmatic usage inside R:

library(ragR)

vectorstore_path <- tempfile(fileext = ".rds")

# Documents must first be ingested into this vector store.
res <- query_rag(
  question = "What is this document about?",
  collection = "default",
  vectorstore_path = vectorstore_path,
  top_k = 5L,
  embedding_model = "text-embedding-3-small",
  chat_model = "gpt-4o-mini",
  system_prompt = "You are a helpful assistant."
)

cat(res$answer)

To run the development HTTP server from the project root:

source("scripts/dev/run_api.R")

Interactive API documentation is then available at ⁠http://127.0.0.1:8000/__docs__/⁠.

Author(s)

Maintainer: Muhammad Aimal Rehman rehman.aimal@gmail.com

Authors:

• Muhammad Aimal Rehman rehman.aimal@gmail.com

• Zhili Lu

• Chi-Kuang Yeh

See Also

Useful links:

• https://github.com/aimalrehman92/ragR

• Report bugs at https://github.com/aimalrehman92/ragR/issues

API handler: chat with the RAG pipeline

Description

This handler receives a parsed JSON request body, runs the RAG pipeline, appends the interaction to the QA log, and returns the model answer.

Usage

api_chat_handler(body, qa_log_path, vectorstore_path)

Arguments

Details

File locations must be supplied explicitly. The handler does not choose default locations in the user's working directory.

Value

A list suitable for JSON serialization.

API handler: clear the entire vector store

Description

Removes all collections and stored embeddings from the vector store. The vector-store path must be supplied explicitly.

Usage

api_clear_all_handler(vectorstore_path)

Arguments

Value

A list containing status and message.

Examples

path <- tempfile(fileext = ".rds")

api_clear_all_handler(
  vectorstore_path = path
)

API handler: clear a vector-store collection

Description

Deletes a collection from the R-native vector store. The request body may specify a collection name; otherwise, "default" is used. The vector-store path must be supplied explicitly.

Usage

api_clear_handler(body, vectorstore_path)

Arguments

Value

A list suitable for JSON serialization, containing status, collection, and message.

Examples

path <- tempfile(fileext = ".rds")

api_clear_handler(
  body = list(collection = "example"),
  vectorstore_path = path
)

API handler: clear RAG evaluation files

Description

Clears the saved question-answering log, saved evaluation metrics, and generated report directory. All locations must be supplied explicitly.

Usage

api_ragas_clear_handler(qa_log_path, qa_metrics_path, output_dir)

Arguments

Value

A list containing status and message.

Examples

qa_log_path <- tempfile(fileext = ".rds")
qa_metrics_path <- tempfile(fileext = ".rds")
output_dir <- tempfile()

save_qa_log(qa_log_empty(), qa_log_path)
save_qa_metrics(qa_metrics_empty(), qa_metrics_path)
dir.create(output_dir)

api_ragas_clear_handler(
  qa_log_path = qa_log_path,
  qa_metrics_path = qa_metrics_path,
  output_dir = output_dir
)

API handler: compute RAG evaluation metrics

Description

Loads a saved question-answering log, computes evaluation metrics, and returns both the per-interaction metrics and their summary. The QA-log path must be supplied explicitly.

Usage

api_ragas_handler(path, judge_model = "gpt-4o-mini")

Arguments

Value

A list containing status, n_qa, metrics, and summary.

Examples

path <- tempfile(fileext = ".rds")
save_qa_log(qa_log_empty(), path)

api_ragas_handler(
  path = path,
  judge_model = "gpt-4o-mini"
)

API handler: generate a RAG evaluation performance report

Description

Loads a saved question-answering log, computes evaluation metrics, and writes the metrics, summary CSV, and plot to explicitly supplied locations.

Usage

api_ragas_report_handler(
  qa_log_path,
  qa_metrics_path,
  output_dir,
  judge_model = "gpt-4o-mini"
)

Arguments

Value

A list containing status, n_qa, qa_metrics_path, summary_csv_path, and plot_path. If the QA log is empty, the returned list instead includes an explanatory message.

Build a RAG prompt from question and retrieved chunks

Description

Build a RAG prompt from question and retrieved chunks

Usage

build_rag_prompt(question, retrieved)

Arguments

Value

Character scalar: the constructed grounded RAG prompt.

Chunk text into overlapping character segments

Description

Chunk text into overlapping character segments

Usage

chunk_text_character(text, chunk_size = 500L, chunk_overlap = 50L)

Arguments

Value

Character vector of chunks.

Chunk text into sentences (strict)

Description

Splits text into individual sentences. Each returned element is intended to be exactly one sentence (best-effort based on punctuation).

Usage

chunk_text_sentence(text)

Arguments

Value

Character vector; one sentence per element.

Examples

text <- paste(
  "Retrieval-augmented generation retrieves relevant information.",
  "The retrieved information is then supplied to a language model.",
  "This can help produce more grounded responses."
)

chunk_text_sentence(text)

Clean raw extracted text (minimal)

Description

• Forces UTF-8

• Replaces carriage returns and newlines with spaces

• Removes control characters

• Collapses multiple whitespace

• Trims

Usage

clean_raw_text(text)

Arguments

Value

Cleaned character scalar

Clean text for embeddings (minimal, deterministic)

Description

Clean text for embeddings (minimal, deterministic)

Usage

clean_text_for_embeddings(text)

Arguments

Value

Cleaned character scalar

Clear the QA log on disk

Description

This helper overwrites the QA log file with an empty QA log, as created by qa_log_empty(). It is intended for starting a fresh evaluation session.

Usage

clear_qa_log(path)

Arguments

Value

Invisibly, the path to the saved file.

Examples

path <- tempfile(fileext = ".rds")

clear_qa_log(path)
load_qa_log(path)

Clear QA metrics on disk

Description

This helper overwrites the QA metrics file with an empty metrics tibble, as created by qa_metrics_empty().

Usage

clear_qa_metrics(path)

Arguments

Value

Invisibly, the path to the saved file.

Examples

path <- tempfile(fileext = ".rds")

clear_qa_metrics(path)
load_qa_metrics(path)

Compute RAGAS metrics

Description

Convenience wrapper for computing LLM-scored RAGAS-style metrics.

Usage

compute_ragas_metrics(
  qa_log,
  judge_model = "gpt-4o-mini",
  seed = NULL,
  embedding_model = "text-embedding-3-small",
  answer_relevance_strictness = 3L
)

Arguments

Value

A metrics tibble.

Compute RAGAS-style metrics (LLM scored)

Description

This implementation mirrors the structured Python RAGAS workflow rather than asking the judge model for one direct scalar score per metric.

Usage

compute_ragas_metrics_llm(
  qa_log,
  judge_model = "gpt-4o-mini",
  seed = NULL,
  embedding_model = "text-embedding-3-small",
  answer_relevance_strictness = 3L
)

Arguments

Details

Notes:

• context_precision uses answer_reference if available; otherwise it uses answer_model, matching the Python with-reference / without-reference split.

• answer_relevance requires an embedding helper to be available in the package environment.

Value

A tibble with one row per qa_id and metric columns.

Embed texts robustly using OpenAI embeddings helper

Description

Calls get_openai_embeddings() on the provided texts. On transient failures, retries with exponential backoff. On HTTP 400 (often request too large), the batch is split recursively.

Usage

embed_texts_batched(
  texts,
  model,
  retry = TRUE,
  max_retries = 5L,
  verbose = TRUE
)

Arguments

Value

A numeric matrix of embeddings aligned with texts

Extract text from a document

Description

Detects file type and extracts raw text from PDF, DOCX, or TXT.

Usage

extract_text(path)

Arguments

Value

Character scalar containing extracted text.

Generate a chat completion from OpenAI

Description

Used by the RAG pipeline and RAGAS-style metric prompts to generate text from a constructed prompt.

Usage

generate_openai_chat(
  prompt,
  model = "gpt-4o-mini",
  system_message = NULL,
  temperature = 0,
  max_output_tokens = 512L,
  seed = NULL,
  api_key = NULL
)

Arguments

Value

Character scalar containing the model's answer.

Generate a RAG evaluation performance report

Description

Loads a question-answering log from disk, computes large language model evaluation metrics, and writes:

Usage

generate_ragas_report(
  qa_log_path,
  qa_metrics_path,
  output_dir,
  judge_model = "gpt-4o-mini"
)

Arguments

Details

• metrics to an RDS file;

• a summary to ragas_summary.csv;

• a bar plot of mean metrics to ragas_means.png.

All input and output locations must be supplied explicitly.

Value

A list containing status, n_qa, qa_metrics_path, summary_csv_path, and plot_path.

Generate a RAG evaluation report using large language model scoring

Description

This is an explicit alias for generate_ragas_report().

Usage

generate_ragas_report_llm(
  qa_log_path,
  qa_metrics_path,
  output_dir,
  judge_model = "gpt-4o-mini"
)

Arguments

Details

All input and output locations must be supplied explicitly.

Value

A list containing status, n_qa, qa_metrics_path, summary_csv_path, and plot_path.

Get an environment variable or fail with a clear error

Description

Get an environment variable or fail with a clear error

Usage

get_env_or_stop(name)

Arguments

Value

The value of the environment variable as a character scalar.

Get embeddings from OpenAI

Description

Calls the OpenAI embeddings endpoint to obtain vector representations for one or more input texts. Used by both ingestion and query-time retrieval.

Usage

get_openai_embeddings(texts, model = "text-embedding-3-small", api_key = NULL)

Arguments

Value

A numeric matrix with one row per input text.

Ingest documents into the vector store

Description

Reads PDF, DOCX, or TXT files, extracts text, cleans it lightly, chunks it, embeds chunks, and stores them in a vector-store collection.

Usage

ingest_documents(
  paths,
  vectorstore_path,
  collection = "default",
  chunk_size = 500L,
  chunk_overlap = 50L,
  chunking_strategy = c("character", "sentence"),
  embedding_model = "text-embedding-3-small",
  embedding_batch_size = 128L,
  embedding_max_chars = 8000L,
  retry = TRUE,
  max_retries = 5L,
  resume = TRUE,
  checkpoint_path = NULL,
  use_openai = TRUE,
  verbose = TRUE
)

Arguments

Details

Chunking strategies:

• "character": overlapping fixed-size character windows.

• "sentence": strict sentence splitting via chunk_text_sentence().

Cleaning strategy:

• Replace carriage returns and newlines with spaces

• Remove control characters

• Collapse multiple whitespace characters into a single space

• Trim leading and trailing whitespace

Embedding strategy:

• Embeddings are computed in batches.

• Batches are retried with exponential backoff after transient failures.

• A batch is automatically divided after certain request-size failures.

• Each chunk is truncated to a specified maximum length before embedding.

• Ingestion can optionally be resumed using a local checkpoint file.

Value

A tibble containing a per-file ingestion summary with columns collection, path, and n_chunks.

Examples

document_path <- tempfile(fileext = ".txt")
writeLines(
  "Retrieval-augmented generation combines retrieval and generation.",
  document_path
)

store_path <- tempfile(fileext = ".rds")

result <- ingest_documents(
  paths = document_path,
  vectorstore_path = store_path,
  collection = "example",
  use_openai = FALSE,
  resume = FALSE,
  verbose = FALSE
)

result
vectorstore_load(store_path)

Load a QA log from disk

Description

This helper loads a QA log tibble from an RDS file. If the file does not exist, it returns an empty QA log created by qa_log_empty().

Usage

load_qa_log(path)

Arguments

Value

A tibble containing the QA log.

Examples

path <- tempfile(fileext = ".rds")

load_qa_log(path)

Load QA metrics from disk

Description

This helper loads QA metrics from an RDS file. If the file does not exist, it returns an empty metrics tibble created by qa_metrics_empty().

Usage

load_qa_metrics(path)

Arguments

Value

A tibble containing the QA metrics.

Examples

path <- tempfile(fileext = ".rds")

load_qa_metrics(path)

Load RAG configuration

Description

Placeholder for a configuration loader. In the future, this can read a YAML file and merge it with environment variables and defaults.

Usage

load_rag_config(path = NULL)

Arguments

Value

A list of configuration values. Currently returns an empty list.

Append one RAG interaction to the QA log

Description

Adds one row to an in-memory QA log tibble.

Usage

log_rag_interaction(
  qa_log,
  question,
  rag_result,
  collection,
  chat_model = "gpt-4o-mini",
  embedding_model = "text-embedding-3-small",
  qa_id = NULL,
  timestamp = Sys.time()
)

Arguments

Value

Updated QA log tibble.

Plot mean RAG evaluation metrics to a PNG

Description

Creates a bar chart of mean metric values and writes it to a PNG file. The output path must be supplied explicitly.

Usage

plot_ragas_means(summary_df, output_path, width = 1200, height = 700)

Arguments

Value

Invisibly, output_path.

Examples

summary_df <- tibble::tibble(
  metric = c("Context precision", "Faithfulness"),
  mean = c(0.8, 0.9)
)

output_path <- tempfile(fileext = ".png")

plot_ragas_means(
  summary_df = summary_df,
  output_path = output_path
)

file.exists(output_path)

Create an empty QA log tibble

Description

Standard schema for storing Q/A interactions produced by the RAG pipeline.

Usage

qa_log_empty()

Value

A tibble with zero rows and the standard QA log columns.

Create an empty QA metrics tibble

Description

This helper creates an empty tibble with the columns used to store RAGAS-style evaluation metrics for each QA interaction.

Usage

qa_metrics_empty()

Details

All metric values are expected to be in [0,1].

Value

A tibble with zero rows and the standard QA metrics columns.

Query the RAG pipeline

Description

Takes a user question, embeds it, retrieves relevant chunks from a vector store, constructs a grounded RAG prompt, and calls a large language model to generate an answer.

Usage

query_rag(
  question,
  collection,
  vectorstore_path,
  top_k = 4L,
  embedding_model = "text-embedding-3-small",
  chat_model = "gpt-4o-mini",
  temperature = 0,
  max_output_tokens = NULL,
  score_threshold = 0,
  system_prompt = ""
)

Arguments

Value

A list with:

Save a QA log to disk

Description

This helper saves a QA log tibble, as produced by log_rag_interaction(), to an RDS file. The output path must be supplied explicitly.

Usage

save_qa_log(qa_log, path)

Arguments

Value

Invisibly, the path to the saved file.

Examples

qa_log <- qa_log_empty()
path <- tempfile(fileext = ".rds")

save_qa_log(qa_log, path)
file.exists(path)

Save QA metrics to disk

Description

This helper saves a QA metrics tibble, as produced by compute_ragas_metrics(), to an RDS file. The output path must be supplied explicitly.

Usage

save_qa_metrics(qa_metrics, path)

Arguments

Value

Invisibly, the path to the saved file.

Examples

qa_metrics <- qa_metrics_empty()
path <- tempfile(fileext = ".rds")

save_qa_metrics(qa_metrics, path)
file.exists(path)

Summarize RAGAS metrics across QA pairs

Description

Computes mean, standard deviation, minimum, and maximum for each metric column in a QA-metrics tibble.

Usage

summarize_ragas(qa_metrics)

Arguments

Value

A tibble with columns: metric, mean, sd, min, max.

Delete all collections from the R-native vector store

Description

Replaces the vector store with an empty tibble. The vector-store path must be supplied explicitly.

Usage

vectorstore_clear_all(path)

Arguments

Value

Invisibly, TRUE.

Examples

path <- tempfile(fileext = ".rds")

embeddings <- matrix(c(1, 0), nrow = 1)

vectorstore_upsert(
  collection = "example",
  ids = "doc_1",
  embeddings = embeddings,
  documents = "Example document",
  path = path
)

vectorstore_clear_all(path)
vectorstore_load(path)

Delete a collection from the R-native vector store

Description

Removes all stored records belonging to a specified collection. The vector-store path must be supplied explicitly.

Usage

vectorstore_delete_collection(collection, path)

Arguments

Value

Invisibly, TRUE on success.

Examples

path <- tempfile(fileext = ".rds")

embeddings <- matrix(c(1, 0), nrow = 1)

vectorstore_upsert(
  collection = "example",
  ids = "doc_1",
  embeddings = embeddings,
  documents = "Example document",
  path = path
)

vectorstore_delete_collection(
  collection = "example",
  path = path
)

vectorstore_load(path)

Load the R-native vector store

Description

Loads the current contents of an R-native vector store from an RDS file. If the file does not exist, the function returns an empty tibble with the expected columns: collection, id, text, embedding, and metadata.

Usage

vectorstore_load(path)

Arguments

Details

The vector-store path must be supplied explicitly.

Value

A tibble with one row per stored chunk, or zero rows if the store does not yet exist.

Examples

path <- tempfile(fileext = ".rds")

vectorstore_load(path)

Query the R-native vector store for nearest neighbors

Description

Retrieves the stored chunks whose embeddings have the highest cosine similarity to a supplied query embedding. The vector-store path must be supplied explicitly.

Usage

vectorstore_query(collection, query_embedding, path, top_k = 4L)

Arguments

Value

A tibble with columns collection, id, text, score, and metadata.

Examples

path <- tempfile(fileext = ".rds")

embeddings <- matrix(
  c(
    1, 0,
    0, 1
  ),
  nrow = 2,
  byrow = TRUE
)

vectorstore_upsert(
  collection = "example",
  ids = c("doc_1", "doc_2"),
  embeddings = embeddings,
  documents = c("First document", "Second document"),
  path = path
)

vectorstore_query(
  collection = "example",
  query_embedding = c(1, 0),
  path = path,
  top_k = 1L
)

Upsert embeddings into the R-native vector store

Description

Adds new records to a vector store or replaces existing records having the same collection and identifier. The vector-store path must be supplied explicitly.

Usage

vectorstore_upsert(
  collection,
  ids,
  embeddings,
  documents,
  metadatas = NULL,
  path
)

Arguments

Value

Invisibly, TRUE on success.

Examples

path <- tempfile(fileext = ".rds")

embeddings <- matrix(
  c(
    1, 0,
    0, 1
  ),
  nrow = 2,
  byrow = TRUE
)

vectorstore_upsert(
  collection = "example",
  ids = c("doc_1", "doc_2"),
  embeddings = embeddings,
  documents = c("First document", "Second document"),
  path = path
)

vectorstore_load(path)