Input files

The project’s routing, auxiliary-variable and logical-edit metadata use ; as the separator; items_mapping.csv uses commas. Configuration specifies these paths, the metadata separator, missing_codes and record_id (DB030 in the included example). Do not infer a file’s schema from its name: check its columns and key against the actual rules.

source(file.path("scripts", "00_config.R"))
stopifnot(all(file.exists(c(config$routing_file, config$vaux_file,
                            config$logical_edit_rules_file,
                            config$items_mapping_file))))
utils::head(utils::read.csv(config$items_mapping_file, check.names = FALSE))

Exogenous auxiliary variables and the observation unit

An auxiliary variable requiring information on individual household members cannot be inferred from a file containing only the household respondent. In this project, extfile3.csv contains DB030 and EXIST_MINOR; the latter indicates whether the household contains at least one individual with PRESEN_B = 1 and ETAINT < 15. Auxiliary rules must refer to the correct external information, linked through DB030.

Active exogenous auxiliary variables retain their source value during correction; if routing makes them inactive, they may be blanked to NA. Check missing source values before interpreting an error as an incorrect questionnaire response.

Defining auxiliary-variable categories

vaux_rules.csv has nine columns without a header, separated by ;. Each row contributes to a category definition; multiple rows with the same variable name and category form its condition.

For logical conditions, the parser automatically opens a parenthesis at the start of each condition and closes it when it encounters NULL.

For example, the purely logical rule A = 1 AND (D = 1 OR E = 3) is represented as follows:

In the actual file, V4 and V5 are empty for variables already present in the data. If V4 specifies extfile1, extfile2 or extfile3, V5 specifies the key (DB030 in the project example).

VAUX7;1;EXIST_MINOR;extfile3;DB030;1;.;.;NULL
VAUX7;2;EXIST_MINOR;extfile3;DB030;0;.;.;NULL

VAUX_PROP2;1;QPROP1;;;;;;+
VAUX_PROP2;1;QPROP2;;;;;;<100
VAUX_PROP2;2;QPROP1;;;;;;+
VAUX_PROP2;2;QPROP2;;;;;;=100

# Run from the root of your project copy.
d <- readRDS("output/prepared-data/prepared_questionnaire_data.rds")
table(d$VAUX7, useNA = "ifany")
table(d$VAUX_PROP2, useNA = "ifany")

Dedicated routing guide

The complete routing_file.csv schema, variable types, categories, conditional and unconditional jumps, and OR/XOR groups are described in Building the routing file. Do not infer routing conventions from the auxiliary-variable file: the two formats have different purposes and fields.

Consistency of external sources

Before preparation, check key uniqueness, key types, coverage of questionnaire DB030 values and availability of the values required by the rules. A complete questionnaire file combined with reduced external sources can produce many missing categories even when the original responses have not changed. When extracting a sample, subset each source using the same keys and document links that were already absent from the source. After replacing sources, regenerate prepared data, checks and extended data, and verify cache validity before correction.

Defining semantic edits: logical_edit_rules.csv

This is a headerless nine-column file separated by semicolons. The columns have the same roles V1–V9 as the VAUX table, but the LEDIT evaluator has its own semantics. Each edit must define both accepted categories 1 and 2. Both are valid outcomes: 2 does not mean error. If no condition assigns an accepted category, the result remains NA and the edit is violated. Choose and document what the two categories mean for each edit.

For example, require A + B = 100 when amounts are reported, and allow both amounts to be blank when the edit is inapplicable. Define category 2 first for the all-blank case, then category 1 for the total:

LEDIT_TOTAL;2;A;;;BLANK;;;AND
LEDIT_TOTAL;2;B;;;BLANK;;;NULL
LEDIT_TOTAL;1;A;;;;;;+
LEDIT_TOTAL;1;B;;;;;;=100

V4 and V5 are empty because A and B are input columns. V6 specifies an exact value or BLANK; V7 and V8 specify inclusive bounds instead. Arithmetic rows leave V6–V8 empty. V9 joins rows and closes a predicate with NULL or an arithmetic expression with a comparison. Keep each condition contiguous and use the same V1 and V2 throughout that condition.

Missing arithmetic operands are replaced by zero in the LEDIT evaluator. Consequently A = 100 and B = NA satisfies the total in this example. If both amounts must be observed, define additional presence requirements. This differs from the VAUX arithmetic behavior described above. Logical conditions evaluating to NA do not assign a category. Conditions are processed in file order; the first matching condition wins. Later conditions cannot overwrite an assigned category. Avoid unintended overlaps.

local({
  edits <- read.table(text = paste(
    "LEDIT_TOTAL;2;A;;;BLANK;;;AND",
    "LEDIT_TOTAL;2;B;;;BLANK;;;NULL",
    "LEDIT_TOTAL;1;A;;;;;;+",
    "LEDIT_TOTAL;1;B;;;;;;=100", sep = "\n"),
    sep = ";", header = FALSE, colClasses = "character",
    na.strings = character(0))
  x <- data.frame(ID = 1:5, A = c(40, NA, 40, 100, NA),
                  B = c(60, NA, 50, NA, 20))
  create_edits <- getFromNamespace("create_logical_edit_variables", "SSPICE")
  out <- create_edits(x, edits)
  print(out)
  stopifnot(identical(out$LEDIT_TOTAL, c(1, 2, NA_real_, 1, NA_real_)))
  print(attr(out, "logical_edit_definitions"))
})
#>   ID   A  B LEDIT_TOTAL
#> 1  1  40 60           1
#> 2  2  NA NA           2
#> 3  3  40 50          NA
#> 4  4 100 NA           1
#> 5  5  NA 20          NA
#>   logical_edit assigned_value statement_number
#> 1  LEDIT_TOTAL              2                1
#> 2  LEDIT_TOTAL              1                2
#>                                                                                                                                                   expression
#> 1                                                                                                                                  (is.na(`A`) & is.na(`B`))
#> 2 ((ifelse(is.na(`A`), 0, suppressWarnings(as.numeric(as.character(`A`)))) + ifelse(is.na(`B`), 0, suppressWarnings(as.numeric(as.character(`B`))))) == 100)
#>   condition_true_records newly_assigned_records
#> 1                      1                      1
#> 2                      2                      2

The output categories are 1, 2, NA, 1, NA. The logical_edit_definitions attribute records the expressions and assignment counts. The internal evaluator is called explicitly here to illustrate the metadata; the operational entry point is Step 09, which also appends LEDIT nodes with accepted domain {1, 2} to the routing representation.

For membership expressions, IN(A,B) compares the current variable with other columns in the same record, not with a list of numeric constants. IN(BLANK) tests missingness. Logical tokens include AND, OR, XOR and NOT; NOT in a connector position means AND NOT of the following predicate. For mixed arithmetic and logical groups the comparison connector has the form >=5) AND (. Do not copy the more permissive VAUX token syntax without checking it against the LEDIT parser.

V4 specifies an external source basename and V5 its linkage key. External keys must be unique. Missing links remain missing, subject to the arithmetic zero convention above. If a referenced column already exists in the input, the evaluator retains that column; avoid conflicting source-column names.

Item mapping and semantic predicates

Item mapping explains the correspondence between observed categories and equidistant structural codes. Keep observed codes in the data and in predicates on those variables in VAUX/LEDIT metadata. Routing bounds and jump conditions use structural codes. Mapping does not rewrite semantic statements or change their arithmetic meaning.