First CRAN release since 0.2.0: this version also carries everything in 0.3.0 and 0.3.1 below (SI-PNI on the healthbr-data mirror, SI-PNI dictionary corrections), which were GitHub-only.
vignette("healthbr-vs-microdatasus") compares healthbR
with microdatasus
on the DATASUS systems both read (SIM, SINASC, SIH, SIA, SINAN, CNES):
coverage table, what each package does with the raw codes (labels
vs. codes plus dictionary), where the bytes come from (DATASUS FTP
.dbc vs. the healthbr-data Parquet mirror with provenance),
lazy evaluation, cache, and a measured run of the same three downloads
(SIH RR 2023-01, SIM and SINASC AC 2022) in both packages. README and
DESCRIPTION now name DATASUS and the neighbouring packages
(microdatasus, PySUS, basedosdados) so the comparison is findable.sih_status() tibble cost ~4.6 s and ran on every
sih_data(source = "r2") call, so a one-file read from a
warm cache took ~5.5 s against 0.9 s from the FTP. The per-partition
output-file lookup is now done once and the summary is memoised per
manifest content (rlang::hash()), the way the parsed
manifest already was: the same warm read now takes ~0.8 s and
sih_status() ~0.75 s. Found while measuring the comparison
vignette.The healthbr-data
mirror (hive-partitioned Parquet on Cloudflare R2, one partition per
DATASUS file RD{UF}{yy}{mm}.dbc, values byte-identical,
1992-present, provenance record per file) is now the default source of
the SIH module, the same design the SI-PNI module adopted in 0.3.0:
source argument in
sih_data(): c("r2", "datasus") (default) reads
from the mirror and falls back to the DATASUS FTP when the mirror yields
nothing; a single value pins a source. r2_credentials
points at another bucket. Without the FTP is used directly. Both
transports share the local partitioned cache.sih_status() (new): one row per
published partition (competence x UF) from the mirror’s manifest, with
the DATASUS URL, MD5 and size of the source file, record count and
processing timestamp – the way to see which competences exist and
whether a file was re-issued. It also carries the Parquet behind each
partition (parquet_path, parquet_sha256,
parquet_size_bytes) and the healthbr-data pipeline that
wrote it (pipeline_version, git_commit), with
the manifest’s last_updated and
manifest_version as attributes – enough for a derived
product to record exactly which files it was built from (the sih-br-mcp
cubes do).attr(x, "healthbr_source") says which source served the
tibble; from the mirror, attr(x, "healthbr_provenance")
lists the source files behind it.sih_data(lazy = TRUE) now returns a remote arrow (or
duckdb) dataset over R2 – nothing is transferred until
collect(), and filters and column selections are pushed
down. The mirror’s 14 historical schemas are handled by imposing the
schema of the newest requested year, so only the requested years are
ever read.year/month
are the AIH’s billing competence, not the admission
date (DT_INTER), and that the four competences after a year
close 99.7-99.9% of its admissions (measured on the whole mirror).*_clear_cache() now clears the partitioned
cache too. The shared .clear_cache() removed only
flat <module>_*.parquet|rds files and left the
hive-partitioned dataset directories (sih_data/,
sim_data/, sipni_r2_data/, …) behind, so
sih_clear_cache() answered “No cached SIH files to clear”
while sih_cache_status() still listed the partitions – and
a “cold” download timed right after clearing was served from the cache.
It now removes the same directories .cache_status() reports
and says how many partition files went. Affects every module with a
partitioned cache. Found while measuring the comparison vignette.PRELIM/ to FINAIS/; the package still
built PRELIM/ URLs for 2023-2024, which no longer exist on
the FTP, so sinan_data(2023)/sinan_data(2024)
failed. Year lists updated: final = 2007-2025, preliminary = 2026.Detected by the portfolio source monitor (edition probing against the
DATASUS FTP); the year lists in R/*_data_internal.R remain
the source of truth until discovery is made dynamic.
09 is Haemophilus influenzae tipo b (Hib) per
the official .cnv, not BCG (BCG is data code 02) —
and used DOSE/FX_ETARIA codes that do not occur in the data at all. Any
decoding done with the built-in dictionary of previous versions should
be redone. Regeneration script:
data-raw/sipni-dictionaries.R.lookup = TRUE in
sipni_dictionary(): returns a data-code lookup
(one row per value as it appears in the data, expanding the .cnv
source_codes — including comma lists like
88,45 and ranges like 11-31), ready to join
against sipni_data() results. Joining data against the .cnv
code column decodes to wrong labels; the vignette examples
were corrected accordingly. Overlapping categories are resolved by
specificity — explicitly listed codes beat range catch-alls — so
residuals like FX_ETARIA “Idade ignorada” (source codes
00-99) only label codes no specific category claimed.unify_schemas = TRUE (all modules): when cached
years carry different column sets — DPNI 1994 has 7 columns, 2019 has 12
— reads used the first file’s schema and silently dropped the extra
columns of other years.year, month, uf_source) first,
matching fresh downloads.sipni_cache_status() (and every module’s
*_cache_status()) now also reports hive-partitioned cache
datasets, which were previously invisible.\u escapes into Rd
files (R CMD check WARNING).pnadc_apply_survey_design() error tests skip
without srvyr.The old OpenDataSUS host
(arquivosdadosabertos.saude.gov.br) was decommissioned and,
as of 2026, the Ministry removed the 2020-2025 microdata files from the
new location as well — sipni_data() for years >= 2020
had been failing with 404 (#1). This release makes the healthbr-data
mirror (hive-partitioned Parquet on Cloudflare R2, values byte-identical
to the Ministry’s files, complete 2020+ series) the default source for
the whole SI-PNI module:
source argument in
sipni_data() and sipni_dictionary():
c("r2", "datasus") (default) reads from the R2 mirror and
falls back to the official DATASUS/OpenDataSUS sources automatically;
pass a single value to pin a source. r2_credentials allows
pointing at another S3-compatible bucket.sipni_status() (new): month/UF-level
availability and provenance of the mirror, read from its
manifest.json files (processing timestamp, Ministry source
URL, record counts). Manifests are cached locally and revalidated by
ETag.healthbr_source (which source served each era) and, for R2
reads, healthbr_provenance (per-partition processing
timestamp + source URL).dt_vacina, ds_vacina, …), while the
OpenDataSUS CSVs use different names (data_vacina,
descricao_vacina, …). Each source returns its columns
exactly as published — no renaming. See
sipni_variables(type = "API", source =).lazy = TRUE with R2 returns the remote
arrow dataset itself: dplyr verbs are pushed down and only the touched
partitions are transferred (partition columns keep the bucket layout:
ano, mes, uf).source = "datasus"); the R2
dictionaries are the full versions converted from the Ministry’s
original .cnv/.dbf files, with a source_codes traceability
column.arrow package (in
Suggests); without it, sipni_data() falls back to DATASUS
with a message.CNES (National Health Facility Registry): Health facility data from DATASUS FTP as .dbc files (2005-2024). 13 file types (ST, LT, PF, DC, EQ, SR, HB, EP, RC, IN, EE, EF, GM), monthly data per UF.
SINAN (Notifiable Diseases Surveillance): Disease notification microdata from DATASUS FTP as .dbc files (2007-2024). 31 diseases including DENG, CHIK, ZIKA, TUBE, HANS, HEPA, MALA, and more.
SI-PNI (National Immunization Program): Aggregated vaccination data from DATASUS FTP as .DBF files (1994-2019), plus individual-level microdata from OpenDataSUS CSV (2020-2025). Transparent year-based routing between sources.
SISAB (Primary Care Health Information): Coverage indicators from the relatorioaps REST API. 4 report types (aps, sb, acs, pns) at 4 geographic levels (brazil, region, uf, municipality).
ANS (Supplementary Health): Beneficiary,
complaint, and financial data from the ANS open data portal. Plus
ans_operators() for operator registry.
ANVISA (Health Surveillance): Product registrations, pharmacovigilance, hemovigilance, technovigilance, and controlled substance sales (SNGPC) from the ANVISA open data portal. 14 data types total.
POF (Household Budget Survey): Food security, consumption, and health expenditure microdata from IBGE FTP (2002-2003, 2008-2009, 2017-2018).
PNADC (PNAD Continua): Health-related supplementary modules from PNAD Continua (deficiencia, habitacao, moradores, APS).
PNS (National Health Survey): Microdata and SIDRA tabulated data (2013, 2019).
Censo Demografico: Population denominators from the IBGE SIDRA API, covering Census years 1970-2022 and intercensitary estimates 2001-2021.
Hive-style partitioned parquet caching for all DATASUS modules.
Lazy evaluation with lazy = TRUE and
backend = "arrow" or "duckdb" for
out-of-memory queries.
Parallel downloads via .map_parallel() (furrr when
configured, purrr fallback).
Download progress bars for all modules using
cli::cli_progress_bar(). Shows spinner, current/total
count, progress bar, and estimated time remaining (ETA) during
multi-file downloads.
Concurrent HTTP downloads for SI-PNI multi-month CSV requests via
curl::multi_download().
Resumable HTTP downloads via Range headers for large files.
Smart type parsing (parse = TRUE) for all 7 DATASUS
modules, with automatic numeric and date column conversion. Use
col_types for fine-grained control or
parse = FALSE for all-character.
Extracted shared helpers for validation, search, cache management, and return logic, eliminating ~1100 lines of duplication across modules.
Reduced cyclomatic complexity across 11 high-complexity functions by extracting internal subfunctions. All exported and internal functions now have cyclomatic complexity below 15.
Removed deprecated flat cache migration infrastructure.
Moved readxl and haven from Imports to
Suggests.
Removed unused vroom from Suggests.
Added “Parallel downloads” section to all 12
*_data() help pages (?sim_data,
?sih_data, etc.) explaining how to enable parallel
processing via future::plan().
Added “Parallel downloads” section to README with a working example.
Added CONTRIBUTING.md with contributor
guidelines.
Added CODE_OF_CONDUCT.md (Contributor Covenant
v2.1).
Complete refactoring of VIGITEL functions due to Ministry of Health website restructuring. Data is now distributed as a single consolidated file containing all years (2006-2024) instead of separate files per year.
vigitel_data() API changed: new format
parameter, year defaults to
NULL (all years), removed
lazy/parallel/force_download
parameters.
Removed vigitel_download() and
vigitel_convert_to_parquet() functions.
pns_sidra_data() unknown table warning.file.exists() guard before
file.size() check).Moved arrow package from Imports to
Suggests for better cross-platform compatibility. The
package now checks for arrow availability and provides
informative error messages with installation instructions when
needed.
Added cache_dir parameter to all data fetching
functions (vigitel_data(), vigitel_download(),
vigitel_variables(), vigitel_dictionary(),
vigitel_cache_status(),
vigitel_clear_cache()). This allows using
tempdir() for temporary storage that doesn’t persist after
the R session.
Updated examples to use cache_dir = tempdir() to
avoid leaving files on the system during CRAN checks.
vigitel_years() - list available VIGITEL survey
yearsvigitel_variables() - list variables available in a
specific yearvigitel_dictionary() - get the data dictionary with
variable descriptionsvigitel_data() - download and load VIGITEL data with
multiple options:
furrrvigitel_convert_to_parquet() - convert cached Excel
files to Parquet formatfurrr package