Remove multiple columns from data.table

data.tabler

What's the correct way to remove multiple columns from a data.table? I'm currently using the code below, but was getting unexpected behavior when I accidentally repeated one of the column names. I wasn't sure if this was a bug, or if I shouldn't be removing columns this way.

library(data.table)
DT <- data.table(x = letters, y = letters, z = letters)
DT[ ,c("x","y") := NULL]
names(DT)
[1] "z"

The above works fine, but

DT <- data.table(x = letters, y = letters, z = letters)
DT[ ,c("x","x") := NULL]
names(DT)
[1] "z"

Best Answer

This looks like a solid, reproducible bug. It's been filed as Bug #2791.

It appears that repeating the column attempts to delete the subsequent columns.
If no columns remain, then R crashes.

UPDATE : Now fixed in v1.8.11. From NEWS :

Assigning to the same column twice in the same query is now an error rather than a crash in some circumstances; e.g., DT[,c("B","B"):=NULL] (delete by reference the same column twice). Thanks to Ricardo (#2751) and matt_k (#2791) for reporting. Tests added.

Related Solutions

R – How to sort a dataframe by multiple column(s)

You can use the order() function directly without resorting to add-on tools -- see this simpler answer which uses a trick right from the top of the example(order) code:

R> dd[with(dd, order(-z, b)), ]
    b x y z
4 Low C 9 2
2 Med D 3 1
1  Hi A 8 1
3  Hi A 9 1

Edit some 2+ years later: It was just asked how to do this by column index. The answer is to simply pass the desired sorting column(s) to the order() function:

R> dd[order(-dd[,4], dd[,1]), ]
    b x y z
4 Low C 9 2
2 Med D 3 1
1  Hi A 8 1
3  Hi A 9 1
R>

rather than using the name of the column (and with() for easier/more direct access).

R – Summarizing multiple columns with data.table

You can use a simple lapply statement with .SD

dt[, lapply(.SD, sum, na.rm=TRUE), by=category ]

   category index        a        b        z         c        d
1:        c    19 51.13289 48.49994 42.50884  9.535588 11.53253
2:        b     9 17.34860 20.35022 10.32514 11.764105 10.53127
3:        a    27 25.91616 31.12624  0.00000 29.197343 31.71285

If you only want to summarize over certain columns, you can add the .SDcols argument

#  note that .SDcols also allows reordering of the columns
dt[, lapply(.SD, sum, na.rm=TRUE), by=category, .SDcols=c("a", "c", "z") ] 

   category        a         c        z
1:        c 51.13289  9.535588 42.50884
2:        b 17.34860 11.764105 10.32514
3:        a 25.91616 29.197343  0.00000

This of course, is not limited to sum and you can use any function with lapply, including anonymous functions. (ie, it's a regular lapply statement).

Lastly, there is no need to use i=T and j= <..>. Personally, I think that makes the code less readable, but it is just a style preference.

Documentation

See ?.SD, ?data.table and its .SDcols argument, and the vignette Using .SD for Data Analysis.

Also have a look at data.table FAQ 2.1.

Best Answer

Related Solutions

R – How to sort a dataframe by multiple column(s)

R – Summarizing multiple columns with data.table

Documentation

Related Topic