Yet Another Blog in Statistical Computing

I can calculate the motion of heavenly bodies but not the madness of people. -Isaac Newton

Surprising Performance of data.table in Data Aggregation

data.table (http://datatable.r-forge.r-project.org/) inherits from data.frame and provides functionality in fast subset, fast grouping, and fast joins. In previous posts, it is shown that the shortest CPU time to aggregate a data.frame with 13,444 rows and 14 columns for 10 times is 0.236 seconds with summarize() in Hmisc package. However, after the conversion from data.frame to data.table, the CPU time of aggregation improves significantly, as shown in the example below.

> library(data.table)
data.table 1.8.6  For help type: help("data.table")
> class(df)
[1] "data.frame"
> dt <- data.table(df)
> class(dt)
[1] "data.table" "data.frame"
> system.time({
+   for (i in 1:10){
+     summ <- dt[, list(INCOME = mean(INCOME), BAD = mean(BAD)),by = list(SELFEMPL, OWNRENT)]
+   }
+ })
   user  system elapsed 
  0.060   0.000   0.062 
> print(summ)
   SELFEMPL OWNRENT   INCOME        BAD
1:        0       0 2133.314 0.08470957
2:        0       1 2881.201 0.06293210
3:        1       1 3487.910 0.05316973
4:        1       0 2742.247 0.06896552
Advertisements

Written by statcompute

December 29, 2012 at 12:11 am

Posted in S+/R

Tagged with ,

%d bloggers like this: