Speed Up Grouped R Operations in R by Reducing Data Frame Overhead
Why grouped R code gets slow
R’s familiar advice is to vectorize wherever possible. The reason is practical: each iteration of an R loop carries interpreter overhead, while vectorized operations such as `*`, `cumsum()` and `match()` do their looping in compiled code. A simple benchmark makes the gap clear. On 100,000 random values, an R loop that doubles each value takes a median 2.22 ms and allocates 819 KB. Multiplying the vector by two directly takes 59.02 μs and allocates 781 KB. The answers and memory use are similar; the loop is nearly 40 times slower. `lapply()` and `vapply()` still call R code once per item, so their cost depends on how many calls they make and how much work each call performs. Grouped operations complicate the usual advice. An expression such as `x - mean(x)` must be evaluated for each group, and because that expression can be arbitrary R code, the group loop can’t simply disappear. The useful target is the overhead around it: keep the per-group work small and do the rest with vectorized operations. That’s the idea behind the changes to {poorman}, a base R implementation of dplyr-like operations. Its earlier grouped verbs used split-apply-combine: create a data frame for each group, run the operation, then bind the pieces back together. That approach is easy to follow, but slows sharply once the number of groups climbs past a few hundred. The author first described that implementation in [an earlier post about grouped operations in {poorman}](https://nathaneastwood.github.io/2020/02/27/building-a-base-dplyr-with-primitives-grouped-operations-pipes-and-more/). The revised approach keeps the unavoidable group loop but cuts down the work surrounding it. The techniques use base R, so they’re useful beyond {poorman}.Less data-frame machinery, better timings
Consider centering a numeric column within each group. A conventional implementation splits the whole data frame, adds a result column to each piece, binds those data frames together, and restores row order. The faster version assigns integer IDs to groups, splits only the column being calculated, applies the calculation, and combines the results with one `unlist()` call. The benchmark uses 100,000 rows and compares 10, 1,000 and 10,000 groups. At 10 groups, the conventional method takes 29.62 ms and allocates 26.1 MB; the streamlined version takes 3.76 ms and allocates 7.57 MB. At 1,000 groups, the times are 86.71 ms and 6.23 ms, with 61.97 MB and 7.67 MB allocated. At 10,000 groups, they’re 522.85 ms and 26.27 ms, with 55.61 MB and 6.18 MB allocated. The trend matters more than any one timing: the streamlined method’s advantage grows as groups multiply. That’s a useful reminder that grouped performance often depends less on the calculation itself than on the data structures built around it. For the examples that follow, the article uses 100,000 rows with 1,000 groups, plus a small sales data set with `shop`, `day` and `amount` columns to make the mechanics visible.1. Assign group IDs with match()
One way to find group membership is to compare every row against every distinct group. For `n` rows and `G` groups, that means `n × G` comparisons, with a new logical vector allocated for each comparison. `match()` does the job in one pass: each value receives the position of its first appearance in `unique()`, so group IDs follow the order in which groups appear.
In the article’s benchmark, comparing rows against each unique value takes 162 ms and allocates 764.87 MB. `match(g, unique(g))` takes 653 μs and allocates 2.16 MB. That gap is substantial, and it comes from using base R’s C-level hash-table work instead of repeatedly scanning the data in R.
Multiple grouping columns require an ID for each combination. The article points out a pitfall in `interaction()`: it pastes values together with a period, so distinct pairs such as `("a.b", "c")` and `("a", "b.c")` can both become `a.b.c`. It can also create levels for combinations absent from the data, wasting memory. The alternative is to number each column with `match()`, combine those numbers arithmetically, then call `match()` again to produce compact IDs from 1 through the number of observed groups. Using double arithmetic also avoids integer overflow in the intermediate calculation.
2. Make the factor from IDs you already have
`split()` expects a factor. But `factor(ids)` would sort and match the IDs again, even though `match()` has already produced consecutive integers from 1 to the number of groups. The article builds the factor directly with `structure()`, supplying its levels and class attributes. For 100,000 rows and 1,000 groups, splitting after calling `factor()` takes 5.63 ms and allocates 4.2 MB. Constructing the factor with `structure()` takes 1.28 ms and allocates 1.2 MB. This shortcut relies on the IDs being gapless integers starting at 1, which `match()` guarantees.3. Split columns instead of data frames
Splitting a data frame calls `[.data.frame` once per group. That method handles many indexing cases and checks row names, work that adds up when repeated. Splitting columns individually with `split.default()` avoids much of that machinery: in the article’s benchmark, it takes 2.04 ms and allocates 2.02 MB, versus 13.01 ms and 3.9 MB for splitting the data frame. There’s another choice: once you know each group’s row indices, you can extract a column by indexing those rows, or split the column directly. The faster option depends on group size. With 100 groups, indexing takes 232.35 μs and splitting takes 964.07 μs. At 3,000 groups they’re about even, at 1.1 ms each. By 10,000 groups, splitting is quicker: 1.18 ms compared with 2.9 ms for indexing. The reason is straightforward. Splitting scans the full column and costs about 1 ms in these benchmarks; indexing makes one R call per group. Indexing wins when groups are larger, while splitting wins when there are many small groups. The implementation uses indexing when groups average at least 32 rows, and `split.default()` below that threshold.4. Combine plain results with one unlist()
Binding one-row data frames with `do.call(rbind, ...)` triggers repeated data-frame checks for column order, factor levels and row names. If each group produces a plain vector, `unlist()` can combine the pieces directly. In the example, `rbind` takes 7.11 ms and allocates 267.74 KB; `unlist()` takes 7.38 μs and allocates 7.86 KB. With 1,000 pieces, that’s roughly a thousandfold difference in time.
There is a trade-off: `unlist()` drops attributes. Numeric and character results are simple cases; factors and dates need additional care to preserve their classes.
5. Restore row order with a permutation
Combining group results puts them in group order, but `mutate()` needs the original row order. The article solves this by recording the row indices in group order, then building a reverse index: for each original row, store where its result sits in the combined vector. A single subset with that index restores the results to their original positions. The index can be reused across columns of different types, so it only needs to be computed once. That makes row-order restoration a small, reusable step rather than another round of per-group data-frame work.6. Skip the group loop when base R can do the job
Some grouped calculations have a direct base R equivalent. For group means, `rowsum()` computes sums in C and `tabulate()` supplies group counts; subtracting the corresponding mean from each row avoids a loop over groups. Across 10, 1,000 and 10,000 groups, that method takes 1.64, 1.64 and 6.59 ms, respectively. The more general method takes 3.81, 5.55 and 22.65 ms. A general verb such as `mutate()` can’t make that substitution because it must evaluate whatever expression the caller provides. Verbs with a fixed operation have more room to use specialized base R functions. Filling missing values is one example. For a vector, multiply each position by a flag for non-missing values, then apply `cummax()` to track the position of the most recent observed value. Within groups, mark the first position of each group as valid too. That prevents a value from carrying across a group boundary, while leaving a leading `NA` in place when its group has no earlier value.Where the speedup comes from
The biggest change in {poorman} 0.3.0 is how it handles groups. The benchmark’s sharpest result is group_by(): with 10,000 groups, it drops from 15.2 seconds and 18.64 GB of memory in version 0.2.8 to 5.35 milliseconds and 5.83 MB in 0.3.0. That’s more than a 2,800-fold reduction in time. These figures come from 100,000-row data, tested with three benchmark iterations for each operation.
The same pattern holds across all seven verbs and all three group counts. At 10 groups, 0.3.0 runs between four and 22 times faster. At 10,000 groups, the gains range from 21-fold to roughly 3,800-fold. mutate() at that scale goes from 11.8 seconds and 18.72 GB to 115.23 milliseconds and 11.63 MB. And fill() falls from 12.38 seconds to 3.25 milliseconds.
Here’s the thing: these gains come from avoiding repeated work, using base R. The older version repeatedly compared rows against groups—during group_by() and again inside grouped operations. The new version stores each group’s row numbers once and reuses them. A loop remains where an operation needs one, but it no longer surrounds work that can be done across the full data set. That shift also explains why memory use in the old version ballooned as group counts rose.
How the grouped verbs fit together
{poorman} records group membership in a .rows list column, one entry per group. For example, grouping sales by shop produces rows 2 and 5 for shop a, rows 1, 3 and 6 for b, and row 4 for c. The groups are sorted by their values, and grouped verbs read those saved row indices instead of rediscovering the groups.
For verbs such as mutate(), the package evaluates an expression inside a small environment containing that group’s column values. R can then resolve column names there, while names it doesn’t find—such as a total defined by the caller—remain available through the environment’s parent. Results from the groups are combined and restored to the original row order. The same internal machinery serves summarise(), mutate(), transmute(), filter() and slice().
The implementation also avoids building a group data frame unless an expression actually reads .data. An active binding creates that object only when needed, so an expression such as sum(amount) skips the cost entirely. In filter(), group-level conditions are assembled into one logical vector and applied to the data in a single subset operation; the saved group rows are then updated. Other verbs take similarly direct routes: fill() orders rows by group and uses cummax() across the column, while distinct() identifies unique group-and-value combinations without looping over groups. slice_head() still loops, but only over short vectors of row indices before subsetting once.
The earlier comparison of two fill() implementations shows the tradeoff behind that approach. With 10,000 groups, processing the full reordered column takes 2.09 milliseconds, versus 11.53 milliseconds for splitting into groups. The whole-column method allocates more memory—9.16 MB versus 3.45 MB in that run—but avoids the repeated group loop.
Trying version 0.3.0
The benchmark compared {poorman} 0.3.0 with CRAN version 0.2.8, installed in separate R sessions. Version 0.3.0 was available from GitHub at the time of writing, with a CRAN release expected soon. If you’re working with grouped data in R, the results make a persuasive case to try it: the improvement is especially striking when the number of groups is high. The measurements are for one 100,000-row benchmark setup, though, so your own workload may behave differently.
install.packages("poorman")Until the CRAN release, the package can be installed from GitHub:
remotes::install_github("nathaneastwood/poorman")Report bugs or differences from {dplyr} on GitHub.
To comment for the author, visit Random R Ramblings.
R-bloggers.com offers daily email updates about R news, tutorials about learning R, and other topics. You can post or find an R/data-science job, or share a blog post on R-bloggers. If you don’t have a blog, visit r-posts.com.
Continue reading: {poorman}: Tips and Tricks for Faster Grouped Operations in base R