R – data.table: Using with=False and transforming function/summary function

data.tablegroup-summarieslapplyrsummarization

I want to summarise several variables in data.table, output in wide format, output possibly as a list per variable. Since several other approaches did not work, I tried to do an outer lapply, giving the names of the variables as character vectors. I wanted to pass these in, using with=FALSE.

carsx=as.data.table(cars)
lapply( list(speed="speed",dist= "dist"), #error object 'ansvals' not found
    function(x)  carsx[,list(mean(x), min(x), max(x) ), with=FALSE ] )

Since this does not work, I tried the more simple approach without lapply.

carsx[,list(mean("speed"), min("speed"), max("speed") ), with=FALSE ] #error object 'ansvals' not found

This does not work either. Is there any way to do something like this? Is this behaviour of 'with' wanted? (I am aware that ?data.table mentions with only to select columns, but in my case it would be useful to be able to transform them as well)

When with=FALSE, j is a vector of names or positions to select, similar to a data.frame. with=FALSE is often useful in data.table to select columns dynamically.

EDIT
My aim is to get a summary per group in wide format, for different variables.
I tried to extend the following, which works only for one variable, for a list of variables.

carsx[,list(mean(speed), min(speed), max(speed) ) ,by=(dist>50)

Lamentably SO doesnt let me post my other question. There I described that I want an output similiar to:

lapply( list(speed="speed",dist= "dist"),
        function(x) do.call("as.data.frame", aggregate(cars[,x], list(class=cars$dist>50), FUN=summary) ) )

Expected Output would be something like:

$speed 
         V1       V2 V3
1: FALSE 12.96970  4 20
2:  TRUE 20.11765 14 25

$dist
         V1       V2 V3
1: FALSE 12.96970  4 20
2:  TRUE 20.11765 14 25

Best Answer

You can specify the columns with the .SDcols parameter:

carsx[ , lapply(.SD, function(x) c(mean(x), min(x), max(x))), 
      .SDcols = c("speed", "dist")]
#    speed   dist
# 1:  15.4  42.98
# 2:   4.0   2.00
# 3:  25.0 120.00

carsx[ , lapply(.SD, function(x) c(mean(x), min(x), max(x))), 
      .SDcols = "speed"]
#    speed
# 1:  15.4
# 2:   4.0
# 3:  25.0

Related Solutions

R – How to one work fully generically in data.table in R with column names in variables

Problem you are describing is not strictly related to data.table.
Complex queries cannot be easily translated to code that machine can parse, thus we are not able to escape complexity in writing a query for complex operations.
You can try to imagine how to programmatically construct a query for the following data.table query using dplyr or SQL:

DT[, c(f1(v1, v2, opt=TRUE),
       f2(v3, v4, v5, opt1=FALSE, opt2=TRUE),
       lapply(.SD, f3, opt1=TRUE, opt2=FALSE))
   , by=.(id1, id2)]

Assuming that all columns (id1, id2, v1...v5) or even options (opt, opt1, opt2) should be passed as variables.

Because of complexity in expression of queries I don't think you could easily accomplish requirement stated in your question:

is simpler, more elegant, shorter, or easier to design or implement or understand than the one above or others that require frequent quote-ing and eval-ing.

Although, comparing to other programming languages, base R provides very useful tools to deal with such problems.

You already found suggestions to use get, mget, DT[[col_name]], parse, quote, eval.

As you mentioned DT[[col_name]] might not play well with data.table optimizations, thus is not that useful here.
parse is probably the easiest way to construct complex queries as you can just operate on strings, but it doesn't provide basic language syntax validation. So you can ended up trying to parse a string that R parser does not accept. Additionally there is a security concern as presented in 2655#issuecomment-376781159.
get/mget are the ones most commonly suggested to deal with such problems. get and mget are internally catch by [.data.table and translated to expected columns. So you are assuming your arbitrary complex query will be able to be decomposed by [.data.table and expected columns properly inputted.
Since you asked this question few years back, the new feature - dot-dot prefix - is being rolled out in recently. You prefix variable name using dot-dot to refer to a variable outside of the scope of current data.table. Similarly as you refer parent directory in file system. Internals behind dot-dot will be quite similar to get, variables having prefix will be de-referenced inside of [.data.table. . In future releases dot-dot prefix may allow calls like:

col1="a"; col2="b"; col3="g"; col4="x"; col5="y"
DT[..col4==..col5, .(s1=sum(..col1), s2=sum(..col2)), by=..col3]

Personally I prefer quote and eval instead. quote and eval is interpreted almost as written by hand from scratch. This method does not rely on data.table abilities to manage references to columns. We can expect all optimizations to work the same way as if you would write those queries by hand. I found it also easier to debug as at any point you can just print quoted expression to see what is actually passed to data.table query. Additionally there is a less space for bugs to occur. Constructing complex queries using R language object is sometimes tricky, it is easy to wrap the procedure into function so it can be applied in different use cases and easily re-used. Important to note that this method is independent from data.table. It uses R language constructs. You can find more information about that in official R Language Definition in Computing on the language chapter.
What else?
- I submitted proposal of a new concept called macro in #1579. In short it is a wrapper on DT[eval(qi), eval(qj), eval(qby)] so you still have to operate on R language objects. You are welcome to put your comment there.
- Recently I proposed another approach for metaprogramming interface in PR#4304. In short it plugs base R substitute functionality into [.data.table using new argument env.

Going to the example. Below I will show two ways to solve it. First one will use base R metaprogramming, second one will use metaprogramming for data.table proposed in PR#4304 (see above).

Base R computing on the language

I will wrap all logic into do_vars function. Calling do_vars(donot=TRUE) will print expressions to be computed on data.table instead of eval them. Below code should be run just after the OP code.

expected = copy(new.table)
new.table = the.table[, list(asofdate=seq(from=ymd((year)*10^4+101), length.out=12, by="1 month")), by=year]

do_vars = function(x, y, vars, donot=FALSE) {
  name.suffix = function(x, suffix) as.name(paste(x, suffix, sep="."))
  do_var = function(var, x, y) {
    substitute({
      x[, .anntot := y[, rep(.var, each=12)]]
      x[, .monthly := y[, rep(.var/12, each=12)]]
      x[, .rolling := rollapply(.monthly, mean, width=12, fill=c(head(.monthly,1), tail(.monthly,1)))]
      x[, .scaled := .anntot/sum(.rolling)*.rolling, by=year]
    }, list(
      .var=as.name(var),
      .anntot=name.suffix(var, "annual.total"),
      .monthly=name.suffix(var, "monthly"),
      .rolling=name.suffix(var, "rolling"),
      .scaled=name.suffix(var, "scaled")
    ))
  }
  ql = lapply(setNames(nm=vars), do_var, x, y)
  if (donot) return(ql)
  lapply(ql, eval.parent)
  invisible(x)
}
do_vars(new.table, the.table, c("var1","var2","var3"))
all.equal(expected, new.table)
#[1] TRUE

we can preview queries

do_vars(new.table, the.table, c("var1","var2","var3"), donot=TRUE)
#$var1
#{
#    x[, `:=`(var1.annual.total, y[, rep(var1, each = 12)])]
#    x[, `:=`(var1.monthly, y[, rep(var1/12, each = 12)])]
#    x[, `:=`(var1.rolling, rollapply(var1.monthly, mean, width = 12, 
#        fill = c(head(var1.monthly, 1), tail(var1.monthly, 1))))]
#    x[, `:=`(var1.scaled, var1.annual.total/sum(var1.rolling) * 
#        var1.rolling), by = year]
#}
#
#$var2
#{
#    x[, `:=`(var2.annual.total, y[, rep(var2, each = 12)])]
#    x[, `:=`(var2.monthly, y[, rep(var2/12, each = 12)])]
#    x[, `:=`(var2.rolling, rollapply(var2.monthly, mean, width = 12, 
#        fill = c(head(var2.monthly, 1), tail(var2.monthly, 1))))]
#    x[, `:=`(var2.scaled, var2.annual.total/sum(var2.rolling) * 
#        var2.rolling), by = year]
#}
#
#$var3
#{
#    x[, `:=`(var3.annual.total, y[, rep(var3, each = 12)])]
#    x[, `:=`(var3.monthly, y[, rep(var3/12, each = 12)])]
#    x[, `:=`(var3.rolling, rollapply(var3.monthly, mean, width = 12, 
#        fill = c(head(var3.monthly, 1), tail(var3.monthly, 1))))]
#    x[, `:=`(var3.scaled, var3.annual.total/sum(var3.rolling) * 
#        var3.rolling), by = year]
#}
#

Proposed data.table metaprogramming

expected = copy(new.table)
new.table = the.table[, list(asofdate=seq(from=ymd((year)*10^4+101), length.out=12, by="1 month")), by=year]

name.suffix = function(x, suffix) as.name(paste(x, suffix, sep="."))
do_var2 = function(var, x, y) {
  x[, .anntot := y[, rep(.var, each=12)],
    env = list(
      .anntot = name.suffix(var, "annual.total"),
      .var = var
    )]
  x[, .monthly := y[, rep(.var/12, each=12)],
    env = list(
      .monthly = name.suffix(var, "monthly"),
      .var = var
    )]
  x[, .rolling := rollapply(.monthly, mean, width=12, fill=c(head(.monthly,1), tail(.monthly,1))),
    env = list(
      .rolling = name.suffix(var, "rolling"),
      .monthly = name.suffix(var, "monthly")
    )]
  x[, .scaled := .anntot/sum(.rolling)*.rolling, by=year,
    env = list(
      .scaled = name.suffix(var, "scaled"),
      .anntot = name.suffix(var, "annual.total"),
      .rolling = name.suffix(var, "rolling")
    )]
  TRUE
}

sapply(setNames(nm=var.names), do_var2, new.table, the.table)
#var1 var2 var3 
#TRUE TRUE TRUE 
all.equal(expected, new.table)
#[1] TRUE

Data and updated OP code

library(data.table)
library(lubridate)
library(zoo)

the.table <- data.table(year=1991:1996,var1=floor(runif(6,400,1400)))
the.table[,`:=`(var2=var1/floor(runif(6,2,5)),
                var3=var1/floor(runif(6,2,5)))]

# Replicate data across months
new.table <- the.table[, list(asofdate=seq(from=ymd((year)*10^4+101),
                                           length.out=12,
                                           by="1 month")),by=year]

# Do a complicated procedure to each variable in some group.
var.names <- c("var1","var2","var3")

for(varname in var.names) {
  #As suggested in an answer to Link 3 above
  #Convert the column name to a 'quote' object
  quote.convert <- function(x) eval(parse(text=paste0('quote(',x,')')))
  
  #Do this for every column name I'll need
  varname <- quote.convert(varname)
  anntot <- quote.convert(paste0(varname,".annual.total"))
  monthly <- quote.convert(paste0(varname,".monthly"))
  rolling <- quote.convert(paste0(varname,".rolling"))
  scaled <- quote.convert(paste0(varname,".scaled"))
  
  #Perform the relevant tasks, using eval()
  #around every variable columnname I may want
  new.table[,paste0(varname,".annual.total"):=
              the.table[,rep(eval(varname),each=12)]]
  new.table[,paste0(varname,".monthly"):=
              the.table[,rep(eval(varname)/12,each=12)]]
  new.table[,paste0(varname,".rolling"):=
              rollapply(eval(monthly),mean,width=12,
                        fill=c(head(eval(monthly),1),
                               tail(eval(monthly),1)))]
  new.table[,paste0(varname,".scaled"):=
              eval(anntot)/sum(eval(rolling))*eval(rolling),
            by=year]
}

R – using mean with .SD and .SDcols in data.table

I think the appropriate way of approaching what you want is to just use the standard syntax:

dt[ , lapply(.SD, mean), .SDcols = "a"]

Alternatively, you can pass a variable by name as follows:

col_to_pass = "a"
dt[ , mean(get(col_to_pass)) ]

Eventually, you can generalized this approach to multiple columns as follows:

col_to_pass = c("a", "d")
dt[ , lapply( mget(col_to_pass), mean) ]

Best Answer

Related Solutions

R – How to one work fully generically in data.table in R with column names in variables

R – using mean with .SD and .SDcols in data.table

Related Topic