Nothing is written outside tempdir() unless the user
asks. The Chicago full-dataset cache lives under tempdir()
by default and moves to a persistent directory only through
options(rmoriedata.cache_dir = );
clear_chicago_cache() empties it.
load_chicago_data(as = "parquet_path") also writes under
tempdir(): it wrote the bundled sample into the user’s
R_user_dir() cache, and a bounded full = TRUE
fetch could land on the full-dataset cache path. The schema names the
language column of siu_directors_reports and
siu_drid_manifest _language, as the files do;
it carried the X_language that check.names had
made of it, so the typed store and load_siu_reports()
disagreed on the name. load_siu_reports() documents that it
returns the corpus as text ("" for empty cells) while
morie_data_load("siu_directors_reports") returns the typed
frame.
Every bundled table ships as Parquet again, next to its CSV
(inst/extdata/parquet/<slug>.parquet, plus
_catalog.parquet and
siu_directors_reports_corpus.parquet).
morie_data_load(format = "parquet") and
load_siu_reports(format = "parquet") read those copies; the
new morie_data_path(slug, format) returns the verified path
of either copy, which is the bridge to
pandas.read_parquet(). The catalog gains a
parquet_path column and the signed manifest covers the new
files.
The Parquet writer no longer mangles non-ASCII text in a C locale
(enc2utf8() on an unmarked string turned each byte into its
<c3><a9> display form, and
load_chicago_data(full = TRUE) wrote that into the
cross-session cache). load_siu_reports() reads its CSV as
UTF-8 like the store does, so both copies agree in every locale; a data
page v2 file is reported as such before any decompression; and CI runs
the check in a C locale. Every CSV reader in the package now declares
UTF-8, so a live Chicago fetch and its cached copy are
identical() in any locale. A string that is not valid UTF-8
and carries no encoding mark is refused by column name (a Parquet string
is UTF-8; escaping lost the data, passing the bytes through made a file
no other reader opens), and duplicate column names are refused on write
and kept apart on read. The cross-session Chicago cache carries a
SHA-256 sidecar checked on every read.
load_cihi_data_tables() rejects an
archived_only that is not TRUE or FALSE instead of ignoring
it.
morie_data_checksums() gains a path
column relative to the extdata root; the bare file
basenames it returned were not unique and 20 of them did not resolve
from the root (morie_data_verify() already returned
path).
ask() quotes every argument it hands to the
rmorie CLI. The preamble contains a parenthesis, so with
the CLI installed every call had died in the shell before the binary
ran; a ; in the question would have run as a command.
model and backend are validated, and a blank
or NA question is rejected like "" and
NULL. A stub binary on PATH now exercises the
branch the tests never reached.
morie_data_load() on a dictionary slug says to use
morie_data_dictionary() instead of pointing back at the
catalog.
load_chicago_data(full = TRUE) no longer caches a
response that is not the dataset. A 200 carrying an outage page, a
header-only body or an export shorter than the row count the service
reports is refused with the reason; before, an outage was written to the
cross-session cache and every later session read the empty frame back.
The cache is written to a temporary file and renamed, so a concurrent
reader never sees a half-written file; a cache that cannot be read is
discarded and refetched; and a new refresh = TRUE argument
refetches on demand. When the service’s count endpoint is down the data
is returned but not cached across sessions.
fetch_cihi_table() checks the downloaded file’s size
and signature against the catalogue’s declared format and validates
which (a whole number within the catalogue, or a non-empty
title substring) and timeout before any request. A live URL
that answers 200 with an outage page now falls through to the Wayback
copy.
The verifier pins the signing key’s XMSS root and public seed in package code. A store re-signed with another key, even with a consistent manifest and shipped public key, no longer verifies; the signature now means more than a checksum against local tampering.
CSV tables are read with encoding = "UTF-8" instead
of fileEncoding = "UTF-8-BOM": in a C locale the latter
truncated 22 of the 99 tables (5 to zero rows); the bytes are now left
alone and marked as UTF-8 in every locale.
The data store is signed. _checksums.csv lists every
shipped file with its SHA-256, and the manifest’s own SHA-256 carries an
XMSS (RFC 8391, SHA-256) signature whose public key ships as
_signing_key.json. The signature is checked once per
session and each file when first read, so a modified or corrupted
install errors instead of returning altered data;
morie_data_verify() checks the whole store at once. Needs
rmoriebricklayer 0.4.0 or newer for the verification.
load_siu_reports(format = "parquet") pointed at the
parquet copy this release removed; the option now errors with a note
that the gzip CSV is the same corpus. rmorie reads the corpus through
the default "csv".
morie_dp_gaussian_mean() now rejects infinite
lower or upper with the error its message
already promised, instead of returning NaN with a warning
from rnorm().
The data store ships every table once.
morie_data_load() reads the table’s CSV (the file rmorie
reads by name) and applies a bundled column schema, so each table comes
back with exactly the names and classes the retired parquet copies had
(verified identical for all 99 tables), and keeps it in a session cache:
repeated loads are instant. The parquet directory, a duplicate
samples/ directory and a second copy of the SIU reports are
gone, which takes the source tarball from 6.8 MB to under the 5 MB CRAN
cap. The ten Victoria tables that existed only as parquet now ship as
CSV under vic/. morie_data_catalog() has
n_rows and n_cols for every table;
morie_data_dictionary() returns the shipped JSON.
Documentation is generated with markdown roxygen; backticks in
the help pages are now \code{} (they rendered as stray
opening quotes in the PDF manual).
The narrative corpus behind morie_describe() was
excluded from the 0.3.1 tarball to get this package under 5 MB, which
left rmorie carrying the only shipped copy. rmorie has no room for it
either, so the corpus is back where it belongs, and both packages sit in
the 5-10 MB range CRAN accepts with a written justification.
The package shipped 9.2 MB, and most of the excess was duplication:
rmoriedata.sqlite (15 MB uncompressed) was retired by
the SQLite-to-Parquet migration and no shipped code reads it. Kept in
the repository, excluded from the tarball..csv.gz. load_siu_reports(format = "parquet")
now reads the store copy that morie_data_load() already
serves, so the top-level duplicate is gone. The CSV remains: it is the
default format and a genuinely different encoding, not a third copy of
the same one.describe_corpus.Rds (1.6 MB, and it does not compress)
is referenced by no R code, Rd page or test. Excluded.Verified after the change: the Parquet path, the CSV path and the
siu_directors_reports store slug all return the same 5,157
x 65 corpus.
nanoparquet is gone from Imports. The bundled store is
now read and written by this package’s own codec
(R/aaa_parquet.R), so a plain install no longer pulls a
compiled Parquet dependency, and
load_siu_reports(format = "parquet") works on a machine
that never had one – previously it turned a missing optional package
into a hard stop, despite the corpus being bundled in exactly that
format.morie_data_load("vic_<key>"); they appear in
morie_data_catalog() like any other slug.data-raw/build_vic_tables.R.morie_datasets_vic_table() defaults to
offline = TRUE, and with nothing bundled it returned a
0-row frame on any machine that had not already downloaded the
workbook.load_chicago_data(full = TRUE) gains
limit (exact row cap) and fraction (share of
the live dataset, total looked up first); bounded fetches never touch
the full-dataset cache. Network examples are now
\donttest{} with try() so they degrade
gracefully offline.
SIU corpus also bundled as Parquet;
load_siu_reports(format = "parquet") reads it via
nanoparquet (columnar / SQL-friendly). CSV remains the default (no extra
dependency).
panel_reviewed flag column (65 cols).inst/extdata/rmoriedata.sqlite (retired by the
sqlite-to-parquet migration; no shipped code read it — the loader API is
Parquet-only via nanoparquet, and the file remains backed
up outside the package). Tarball drops from 5.5 MB to 4.25 MB.AGPL (>= 3) (the LICENSE
file was a verbatim copy of the standard license, which CRAN
flags).CITATION.cff, NOTICE, and
LICENSE are excluded from the built package
(.Rbuildignore), clearing the top-level-files NOTE.LinkingTo: rmoriebricklayer (>= 0.2.0) and links the
ecosystem’s shared compiled core instead of duplicating any C code.morie_core_sha256() and
morie_core_mean() call the shared kernels directly (fast
data-integrity hashing + summaries for the integrated fixtures, with no
dependency on rmorie). Tests assert they are byte-identical to
rmoriebricklayer’s own core_sha256() /
core_mean().Six small, base-R-only helpers for analysts releasing aggregate statistics from the integrated fixtures (or any other dataset) without re-identification risk:
morie_dp_laplace_count() — (epsilon, 0)-DP count via
the Laplace mechanism (sensitivity 1).morie_dp_gaussian_mean() — approximate (epsilon,
delta)-DP mean of a bounded numeric vector via the analytic Gaussian
mechanism.morie_dp_laplace_histogram() — per-bin Laplace noise on
a histogram.morie_k_anonymity_verify() — checks k-anonymity over a
set of quasi-identifiers and returns the offending equivalence
classes.morie_l_diversity_verify() — per-class distinct-count
check on a sensitive attribute.morie_cell_suppress() — small-cell suppression with
optional complementary suppression so primary-suppressed cells can’t be
reconstructed from row/column marginals (StatCan-style).All helpers use only base R + the existing stats import;
no new runtime dependencies. Inputs are validated and edge cases (empty
vectors, NAs, out-of-bounds inputs, invalid privacy parameters) throw
informative errors.
tests/testthat/test-dp.R — Monte-Carlo convergence +
variance-scaling sanity tests for the DP mechanisms; input-validation
coverage.tests/testthat/test-k-anonymity.R — hand-built fixtures
with known equivalence-class structure; complementary-suppression
correctness.