basetable

basetable is a fast in-memory data-manipulation package for R with a base-R interface and no dependencies. You write subset(), transform(), aggregate(), merge(), split(), and the work runs on the package’s own C++ engine. There is no data.table, no dplyr, no Arrow underneath, and nothing in Imports beyond the base and recommended packages (parallel, stats, utils).

Every verb returns a basetable: an ordinary data.frame with one extra class so it prints compactly and [ keeps the class. as.data.frame() strips it back to a plain frame.

This is a deliberately focused tool. It is aimed at

• people teaching or learning base R who want speed without the cognitive load of tidy evaluation or [i, j, by], and
• codebases already built on subset() / merge() / aggregate() that want a faster engine without a rewrite.

Design

• Base-style naming and semantics. Functions read like subset(), transform(), aggregate(), merge(), split().
• A native C++ engine. Projection, filtering, ordering, distinct, grouping, all join kinds, row-bind and subset() predicate evaluation run in compiled .Call kernels. There is no third-party compute backend.
• Explicit, standard-evaluation interfaces. Column names are strings, not captured symbols (with the marked exceptions subset() and transform() inherit from base R).
• Zero hard dependencies. data.table and dplyr appear only in Suggests, and only as competitors in the benchmark vignette.

Installation

Install the released version from CRAN:

install.packages("basetable")

Install the development version from GitHub:

# install.packages("pak")
pak::pak("ielbadisy/basetable")

Minimal examples

library(basetable)

# nested
describe(
  transform(
    subset(mtcars, cyl == 6, select = c("mpg", "hp", "wt", "cyl")),
    power = hp / wt
  )
)

# pipe
mtcars |>
  pick(c("mpg", "hp", "wt", "cyl")) |>
  transform(power = hp / wt) |>
  aggregate(by = "cyl", value = c("mpg", "power"), fun = mean)

# table-1 style summary
summarytab(
  transform(mtcars, am = factor(am, labels = c("Automatic", "Manual"))),
  vars = c("mpg", "hp"), by = "am", p_value = TRUE
)

Performance

Timing and memory below come from the bench package at 1,000,000 rows on one Linux machine (inst/benchmarks/make-readme-figures.R regenerates the figures; the Benchmarks vignette has the full reproducible report). basetable is compared with data.table, dplyr and collapse. The table uses all cores, with basetable, data.table and collapse given the same thread budget (dplyr has no parallel path); the figures below are the single-thread pass. Timings vary a few milliseconds from run to run, so read differences of that size as ties.

Speed

Memory

(equi join pins data.table to sort = FALSE, matching basetable::merge(), which returns rows in input order.)

basetable is the fastest engine on every operation here when all cores are used. It is faster than collapse on all seven (see basetable against collapse below), faster than data.table on all seven, and faster than dplyr on all seven, by more than 100x on high-cardinality count.

With everything pinned to one thread (the figure above), basetable still leads on six of the seven. The exception is equi join, where data.table is level or slightly ahead (2 ms against 3 ms).

Memory, ranked by advantage

basetable allocates the least (or tied least) on every operation measured. The size of the edge splits in two: overwhelming on grouped reductions, where the result is tiny and nothing intermediate is materialised in R; modest on operations that return a full table, where the output frame itself sets a floor.

These are R-level allocations as reported by bench. The C++ engine also uses malloc’d scratch buffers (radix keys, per-thread row-position vectors) that bench does not count, so peak process memory during a sort or filter is higher than the figure above; data.table does the same, and collapse allocates in C as well.

basetable against collapse

collapse was the fastest engine on most of these operations in earlier releases, so it is the reference basetable is now tuned against. inst/benchmarks/profile-core.R runs the exact README expressions for the two engines only, checks that both return the same values, and times them in three alternating batches so neither engine always runs first. Times are the median across batches; the speedup is collapse time over basetable time.

basetable is faster on all seven operations at both thread counts, and in every case its slowest batch still beats the fastest collapse batch. The narrowest margin is single-thread string sort, at about 10%. To reproduce:

BT_FIG_REPS=25 Rscript inst/benchmarks/profile-core.R

Setting BT_PROFILE=1 also prints the time spent in each native phase (lookup, scatter, gather, and so on) for every operation.

Nested tables

nest() collapses each group into a data frame held in a list-column, and unnest() expands it back. They are compared here with tidyr::nest() and dplyr::group_nest() for nesting, and tidyr::unnest() for unnesting, at 1,000,000 rows and four columns (integer key, double, integer, character), varying the number of groups. Medians and R-level allocations come from bench; inst/benchmarks/benchmark-nest.R reproduces them.

basetable is faster than both packages on every row. The edge is modest for nest() (about 1.3x to 1.5x over tidyr) and grows with the number of groups for unnest(), where it is roughly 9x faster at 1e5 groups because the nested frames are stacked in one native pass. As elsewhere, the C++ engine also uses malloc’d scratch buffers that bench does not count.

Positioning

basetable now leads the engines measured above on speed, but data.table and collapse cover far more operations and data.table has a far larger ecosystem; dplyr is the tidyverse standard. basetable is a good fit when you want:

• base-R syntax and semantics, not [i, j, by], tidy evaluation, or a method-chained frame object;
• no dependencies to install, pin, or reason about;
• a package small enough to read end to end, teach from, and hand to a language model as a stable target;
• the fastest times and the lowest memory use of the engines measured on the everyday operations (filter, group, join, distinct) without changing how you write code.

Grouping is a by argument on the verb that needs it (aggregate(), count(), summaries(), transform(), subset(), samplerows(), firstby(), …), not a stateful group_by(). The group is named at the call and never persists, so there is no ungroup() to forget.

Using basetable alongside dplyr and data.table

basetable reuses base-R verb names (subset(), merge(), transform(), split(), aggregate()) on purpose. It does not ship the dplyr-coined verbs (filter(), select(), mutate(), arrange(), summarise(), distinct(), glimpse(), …), so it can be attached next to dplyr without shadowing its grammar. The two names it shares with dplyr are count() and pick(), kept because they read as base-style verbs; with both packages attached, whichever was attached last wins for those (and for the base-R names data.table also defines). Two fixes:

• call it explicitly: basetable::transform(...);
• or conflicted::conflict_prefer("transform", "basetable") once per session.

Operation dictionary

btread() memory-maps the file and, with lazy = TRUE, returns columns as ALTREP vectors parsed on first access. aggregate(), count(), uniquerows() and freq() accept a single file path as their first argument and fuse the parse with the grouping, so unused columns are never materialised.

Status

Every exported function has direct test coverage. Vignettes cover getting started, data manipulation, exploration, a complete function reference, and benchmarks. CI checks release R on Linux, macOS and Windows plus oldrel and devel.

Base R compatibility and join diagnostics

The interface is inspired by base R; it is not a drop-in replacement for every base function. In particular, aggregate() accepts column names in by and value, rather than the by list or formula interface of stats::aggregate(). merge() defaults to sort = FALSE (base R defaults to TRUE) and requires at least one join column. Results carry the basetable class; use as.data.frame() when a plain data frame is required. Consult each function’s help for its arguments and supported behavior.

Before joining, joinrelationship(x, y, by) reports whether keys are unique on each side. It checks the entire input, including keys that do not match; repeated missing keys count as duplicates. matchedkeys(x, y, by) and unmatchedkeys(x, y, by) return rows of x with keys present or absent in y, respectively, preserving duplicate rows from x. These tools help expose unexpected multiplicity and missing matches before a join.