Read these before upgrading: some numbers, names, labels and missing values are now different (and correct).
V7 37-43 (1)
in SPSS, V7 37-43 .1 in SAS) used to be read as whole
numbers, so a value of 0000785 was 785 rather than 78.5.
They now get their decimal point, with a message listing each column and
its number of decimal places. With coerce_numeric = FALSE
the point is added to the text (“000078.5”). Values that already have a
decimal point are kept as they are, as in SPSS and SAS. Value labels and
missing values of these columns, which setup files write as numbers
(1.0, -9.00), are now matched by value.
parse_setup()$setup has a new decimals
column.CLOTHING_V93, and PSID’s repeated “RELEASE NUMBER” becomes
RELEASE_NUMBER_TA050001 for the 2005 wave. Names are
compared ignoring case, as SPSS and SAS do, so Age and
AGE count as the same name. Before, the data had duplicate
names, so data$CLOTHING only gave the first of them and
dplyr and data.table stopped with an error. A message lists the renamed
columns.= inside a label no longer cuts the name
short. “SEP FROM FATHER < OR = 5” was named X5;
it is now SEP_FROM_FATHER_under_OR_5.% in a label becomes
_percent_ in the name, as <
already became _under_ (e.g. “1919 RT % VOTES CAST” is
X1919_RT_percent_VOTES_CAST, not
X1919_RT_VOTES_CAST, which “1919 RT VOTES CAST” could also
be).9, 09 or
0000009 matches any of them in the data. Before, they had
to be written exactly as in the data, so
IF V156=0000009 THEN V156=.; never matched the
9s in a 1-column field. Columns the setup file says are
text (SPSS (A), SAS $) still need exactly the
same text.0 now labels data written
0000000 (e.g. 7,997 “Not updated” values in the 1960 UCR
file that used to stay as 0000000). A column that now has
some labelled values is text rather than numbers: in the 1960 UCR file,
columns such as JAN_LAST_UPDATE have “Not updated” where
they had 0. use_value_labels = FALSE keeps the codes.NA in text columns get
their value labels: e.g. 166,650 blanks in the 1960 UCR file
are now “No return received” and 15,555 in the 2000 NIBRS batch header
“Inactive”, where they used to be NA. (SPSS treats a blank
text value as a value; SAS treats it as missing but still gives it a
format’s label for a blank.) A blank or NA text value
without a label is still NA. In number columns a blank,
NA or a lone . is missing (a lone
. used to be kept).(A) or an A format, SAS $),
as in SPSS and SAS, even when every value is a number, so codes such as
ZIP code 01001 keep their leading zeros. Before, such a
column became numeric with coerce_numeric = TRUE. In the
SHR example data, the rarely filled-in ages of victims 5 to 11 are now
text like those of victims 1 to 4, and empty text columns are character
NA rather than numeric NA. Use
as.numeric() on a column you want as numbers.LFM and LFF
were ..._LOCAL_AUTHORITY and
..._LOCAL_AUTHORITY_LFF (one name, made unique) and are now
..._LOCAL_AUTHORITY_MALE and
..._LOCAL_AUTHORITY_FEMALE. 24 names in 6 of the 157 setup
files used to check this release changed this way.ADD VALUE LABELS adds to them
instead). Before, the first set was kept. A message lists the columns
whose earlier, different labels were replaced.F7.1 don’t imply decimal places.%, =, full SAS labels).
parse_setup(setup_file)$setup lists every column’s short
name (column_number) and descriptive name
(column_name); short names haven’t changed, and
use_clean_names = FALSE gives them.10^decimals, using its decimals in
parse_setup(setup_file)$setup.use_value_labels = FALSE keeps the codes, and
as.numeric() turns a text column of numbers into
numbers.missing_values = "all", including ones that are commented
out in the setup file).read_ascii_setup(), spss_ascii_reader()
and sas_ascii_reader() have a missing_values
argument. Many setup files have their missing values section commented
out (* MISSING VALUES in SPSS, inside /* */ in
SAS) so the codes stay in the data unless you choose otherwise:
"all" (the default, as before) uses every missing value in
the file, "active" only those that aren’t commented out,
and "none" none of them. When commented-out missing values
are used, a message says where they are and how many values they made
NA. parse_setup()$missing has a new commented
column.-9 in a
(2) column is -0.09, not the code -9 ‘Blank’). As in SPSS
and SAS the code isn’t used, but it suggests the data doesn’t match the
setup file’s decimal places."asciiSetupReader_message" (and one for each kind), so they
can be silenced with suppressMessages(). Missing values
that also have a value label keep the label (as before); the
documentation now says so.V1 V2 1 'Yes',
V1 TO V5 (9)) and starting on the command’s own line,
ADD VALUE LABELS, several MISSING VALUES sections in one
file, commented-out VALUE LABELS between other sections, variables named
VALUE or LABELS, and SAS formatted input
(@1 V1 $CHAR2. @3 V2 4.1).V1 () clears
them, ALL means every column, / can separate
columns on one line, THRU can be in any case, and long or
decimal ranges (1000000 THRU 9999999,
99.5 THRU 99.9) are compared as numbers instead of being
listed one value at a time.>=/GE alone
(98 thru highest), <=/LE,
GE a AND LE b, EQ, lower-case names and
statements over several lines. Statements that aren’t “IF v = … THEN v =
.;” (e.g. a recode to another value, or a condition on another column)
used to be misread as missing values; they’re now skipped, with a
message.IF v = 1 THEN v = 5;, SPSS RECODE v (1=5))
aren’t applied; a message lists them. Only a setup file’s layout, names,
labels and missing values are used, as before, but such statements used
to be misread.SKIP (SPSS DATA LIST), /FIRSTCASE (GET
DATA) and FIRSTOBS (SAS INFILE) skip header lines in the
data.@ pointers
(INPUT V1 2. V2 $3.;), INPUT on the same line as the first
column, blank lines inside INPUT and commented-out INPUT statements are
handled."asciiSetupReader_error": a file for delimited data
(GET DATA /ARRANGEMENT = DELIMITED,
DATA LIST FREE or LIST, or SAS
DLM=/DSD, GitHub #13), a file of value labels
without column positions (such as PSID’s separate formats files), a file
without a DATA LIST or INPUT section, or a layout the package can’t
read: several records per case (RECORDS=, /2,
SAS #2), columns given by format only
(ID (F3) NAME (A5)) or SAS variable lists in parentheses.
These used to fail with “invalid ‘row.names’ length” or “argument of
length 0”, or read the wrong columns.FORMAT V1 V2 YNF.;), numbered and
positional lists (V1-V5, Q08-Q10,
A1_1-A1_3, V1--V5), formats without value
labels (8.2, DATE9.) and
ATTRIB ... FORMAT= (also after LABEL='...').
Format names match in any case, and a format defined twice uses the
later definition, as in SAS.VAR LABELS), several columns on one line, /
between columns, the first column on the command’s own line and labels
joined with +; in SAS LABEL, several columns on one line,
blank lines, labels over several lines, a ; inside a label,
LABEL after another statement on its line, and
ATTRIB V1 V2 LABEL = '...' (for every column listed)
whatever else ATTRIB sets. A LABEL statement inside a /* */
comment isn’t used.VALUE F 1, 2, 3 = 'Low';) give each
code the label. Value ranges (1 - 97 = 'Actual hours') and
OTHER aren’t applied, so the real numbers are kept; a
message lists them.AA 1-4(A) BB 5-8(A)),
formats and decimals anywhere on a line
(V1 1-5 (F,2) V2 6-7), columns on the DATA LIST line
itself, a name and its position on different lines, /* */
comments, names with @, #, $ or
., and a missing closing period. As in SPSS, names listed
before one range share it equally (A B C 1-6,
Q1 TO Q3 3-5). GET DATA formats such as AUTO
and ADATE10 and /VARIABLES= without spaces are
read, and a second record given by a bare / stops with the
several-records error. All 157 setup files used to check this release
are read exactly as before.inst/testdata to tests/testthat/testdata, so
they’re no longer installed with the package (about 4 MB less).select_columns: a column number past the last column
stops with an error instead of crashing R (negative numbers still drop
columns); a column asked for twice is read once, with its implied
decimals; a name that isn’t found is listed, with a note on which names
changed in this version; and selecting a name that another column also
had says what that column is called now.97 THRU HI, LO THRU -1, SAS
IF V1 >= 97 THEN V1 = .;) no longer stops
read_ascii_setup() with “‘names’ attribute [1] must be the
same length as the vector [0]”.INPUT V1 1-2 V2 $ 3-4 V3 5-8 .2;) is
read token by token, as formatted input is, so a statement or comment
after the ; on INPUT’s line
(INPUT V1 1-2 V2 3-4; RUN;), a /* */ comment
inside INPUT and single-spaced columns
(SEX $ 8 INCOME 9-15) no longer drop or misplace
columns.97 THRU 99. parse_setup()$missing still lists
a range of whole numbers value by value."anti-jehovah's witness" became “anti-jehovah s witness”
and 'Don''t know' became “Dont know”). This affected both
SPSS and SAS setup files.'Wife/"Wife"') is kept, and so are = and
;, which used to become a space or be dropped.2 'Wife' 3 'Common-law husband'
gave a code “2 Wife 3”), a code written '''', quoted words
inside a label ('farmer, ''profits''',
'Twelfth grade, "High School", GED'), and SAS labels split
over several quoted pieces, whose end became a separate code.+ (at the end of a
line or the start of the next) are joined into one label, and a
double-quoted code followed by more pairs on the same line
("02" "two" 3 'three') no longer runs into the next
pair.coerce_numeric = TRUE now checks every row before
converting a column to numeric. Previously files over 50,000 rows only
had a random sample of rows checked (and a file of exactly 50,000 rows
only 5,000), so a column with rare text values could be converted to
numeric, silently turning those values into NA, differently
on every run. The sampling also changed the user’s random number
stream.make_sps_setup() works again on R >= 4.5.0 (it
failed with “NA in coercion to boolean”), works when
col_labels isn’t given, and no longer leaves a
sink() open if it fails. A file_name ending in
“.SPS” no longer gets a second extension, and one merely ending in “sps”
(e.g. “datasps”) is no longer cut short.labels must be unique” when the value X itself appears in
the data.make_sps_setup()’s missing_values accepts
column names as well as labels. Previously a column name was treated as
a missing value of the column before it.DATA LIST can now
be read.spss_ascii_reader() and sas_ascii_reader()
now call read_ascii_setup(); results and arguments are
unchanged.vroom not handling embedded
nulls.readr
to vroom to significantly increase read time.read_ascii_setup() function
interactively. This allows users to select every option in the popup
window.parse_setup
to be in same order as in setup file.read_ascii_setup) that will
read in data for both .dat+.sps and .dat+.sas pairs. This is just a
helper function that calls spss_ascii_reader or
sas_ascii_reader.