collapse: Speed Without Complexity
Faster grouped calculations in R with the collapse package
Introduction
R has a mature ecosystem for statistical computing, and packages like dplyr and data.table drive modern data manipulation with expressive, user-friendly syntax.
As datasets grow into the millions of rows, however, performance and memory efficiency become critical. What works well on small data can add noticeable overhead at scale.
The collapse package addresses this: it offers fast, memory-efficient data manipulation designed for large-scale workflows.
What is the collapse package?
collapse is a high-performance R package for fast, memory-efficient operations on vectors and data frames. Its core functions are written in C, so they run much faster than many higher-level abstractions.
By making grouped computations direct and explicit, collapse reduces overhead, especially on large datasets.
Installing and loading the packages
install.packages(c("dplyr", "collapse", "bench"))
library(dplyr) # Used for comparison (standard tidy data manipulation)
library(collapse) # Used for fast, C-optimised grouped computations
library(bench) # Used for fair, repeated benchmarkingExample data frame
To show the difference in approach and performance, we build a large panel-style dataset:
set.seed(123)
n_subjects <- 100000
n_periods <- 10
df <- data.frame(
id = rep(1:n_subjects, each = n_periods),
period = rep(1:n_periods, times = n_subjects),
value = rnorm(n_subjects * n_periods, 100, 15)
)
head(df) id period value
1 1 1 91.59287
2 1 2 96.54734
3 1 3 123.38062
4 1 4 101.05763
5 1 5 101.93932
6 1 6 125.72597This dataset has:
- 100,000 unique subjects
- 10 repeated measurements per subject
- 1,000,000 rows in total
Structures like this are common in longitudinal research, finance (a time series per entity), econometrics, machine learning pipelines and healthcare analytics. At this scale, performance differences become measurable and meaningful.
Grouped transformations
Suppose that, for each subject, we want the group mean, the group standard deviation, a centred value (the deviation from the group mean) and a standardised z-score. This is a common preprocessing step in many modelling workflows.
Using dplyr
dplyr_way <- function(d) {
d %>%
group_by(id) %>%
mutate(
mean_id = mean(value),
sd_id = sd(value),
centered = value - mean_id,
z_score = centered / sd_id
) %>%
ungroup()
}
df_dplyr <- dplyr_way(df)The dplyr syntax is readable and declarative: it clearly expresses grouping the data, computing statistics and transforming values within each group.
Internally, though, dplyr builds grouped objects, performs tidy evaluation and manages extra layers of abstraction. This makes it easier to use, but it adds computational overhead that becomes noticeable on very large datasets.
Using collapse
collapse_way <- function(d) {
g <- GRP(d$id)
d$mean_id <- fmean(d$value, g, TRA = "replace")
d$sd_id <- fsd(d$value, g, TRA = "replace")
d$centered <- d$value - d$mean_id
d$z_score <- d$centered / d$sd_id
d
}
df_collapse <- collapse_way(df)The collapse version is more explicit and lower level. GRP() creates a lightweight grouping index, and functions such as fmean() and fsd() run in optimised C code. With TRA = "replace", each group's result is copied back to every row of that group, without building grouped data frames or extra layers of abstraction.
collapse also offers dplyr-style verbs
If you prefer pipe-friendly syntax, collapse provides its own fast equivalents of the dplyr verbs (fgroup_by(), fmutate(), fungroup()):
collapse_verbs <- function(d) {
d |>
fgroup_by(id) |>
fmutate(mean_id = fmean(value), sd_id = fsd(value)) |>
fungroup() |>
fmutate(centered = value - mean_id, z_score = centered / sd_id)
}
df_verbs <- collapse_verbs(df)This keeps most of dplyr's readability while still running at collapse speed.
Verifying numerical equivalence
To keep the comparison fair, we first confirm that all three implementations give the same results:
cols <- c("mean_id", "sd_id", "centered", "z_score")
all.equal(as.data.frame(df_dplyr[, cols]), as.data.frame(df_collapse[, cols]), tolerance = 1e-8)
all.equal(as.data.frame(df_dplyr[, cols]), as.data.frame(df_verbs[, cols]), tolerance = 1e-8)[1] TRUE
[1] TRUEThe difference lies not in correctness but in how the work is carried out.
Timing comparison
A single system.time() call is easily affected by whatever else the computer is doing. We therefore use bench::mark(), which runs each version 10 times and reports the minimum, the median and the memory allocated:
bench::mark(
dplyr = dplyr_way(df),
collapse_GRP = collapse_way(df),
collapse_verbs = collapse_verbs(df),
iterations = 10, check = FALSE
)We set check = FALSE because dplyr returns a tibble and collapse returns a data frame; we already confirmed above that the values are identical. bench prints a wide table, summarised below, and may warn about garbage collection, which is harmless here.
| Approach | Minimum | Median | Memory allocated |
|---|---|---|---|
| dplyr | 3.18 s | 3.31 s | 56.8 MB |
collapse (GRP + TRA) | 16.5 ms | 25.8 ms | 42.3 MB |
collapse (fgroup_by / fmutate) | 16.2 ms | 23.6 ms | 42.4 MB |
On this one-million-row dataset, collapse was more than 100 times faster than dplyr (median 3.31 s against about 25 ms) and allocated about a quarter less memory. Both collapse styles performed almost identically.
Your exact numbers will differ with your hardware and package versions; these results come from R 4.3.3, dplyr 1.1.4 and collapse 2.0.10 on a 2-core cloud machine. The size of the gap also depends on the task: it is largest for grouped operations over many small groups, as here, and smaller for simple whole-column operations.
Why collapse performs better
- Minimal abstraction. Operations run close to the underlying data structures, avoiding the overhead of complex evaluation frameworks.
- C-level optimisation. Core functions such as
fmean()andfsd()are written in optimised C code. - Explicit control of group behaviour. The
TRAargument states exactly how group results are returned, with no hidden behaviour. - Memory efficiency. Less copying and fewer intermediate objects reduce memory use, as the benchmark shows.
Comparison with other packages
| Feature | dplyr | data.table | collapse |
|---|---|---|---|
| Syntax style | Declarative | Compact | Explicit, or dplyr-like verbs |
| Grouped-operation speed | Moderate | High | High |
| Learning curve | Low | Steeper | Moderate (low with the verbs) |
| Best suited to | Readable everyday analysis | Large data, in-place updates | Large data, repeated grouped statistics |
These ratings are general guidance rather than results from this article's benchmark, which compared only dplyr and collapse. dplyr remains a good default for most everyday work, and backends such as dtplyr and duckplyr can speed up dplyr code on large data. Each package serves a different purpose, and the right choice depends on data size, performance needs and coding preferences.
When should you use collapse?
Use collapse when you are:
- Working with very large datasets
- Running the same grouped calculations in a scheduled or production pipeline
- Running repeated grouped operations
- Optimising performance and memory use
Conclusion
collapse does not replace the tidyverse; it is a focused tool for when speed and memory matter. On large datasets with repeated grouped calculations it can cut run times sharply (more than 100 times in this example), and its dplyr-style verbs keep the code readable.
All code in this article was tested with R 4.3.3, dplyr 1.1.4, collapse 2.0.10 and bench 1.1.3.
Written in conjunction with Sankhya Analytics.