I'm having a little trouble understanding the pass-by-reference properties of data.table
. Some operations seem to 'break' the reference, and I'd like to understand exactly what's happening.
On creating a data.table
from another data.table
(via <-
, then updating the new table by :=
, the original table is also altered. This is expected, as per:
?data.table::copy
and stackoverflow: pass-by-reference-the-operator-in-the-data-table-package
Here's an example:
library(data.table)
DT <- data.table(a=c(1,2), b=c(11,12))
print(DT)
# a b
# [1,] 1 11
# [2,] 2 12
newDT <- DT # reference, not copy
newDT[1, a := 100] # modify new DT
print(DT) # DT is modified too.
# a b
# [1,] 100 11
# [2,] 2 12
However, if I insert a non-:=
based modification between the <-
assignment and the :=
lines above, DT
is now no longer modified:
DT = data.table(a=c(1,2), b=c(11,12))
newDT <- DT
newDT$b[2] <- 200 # new operation
newDT[1, a := 100]
print(DT)
# a b
# [1,] 1 11
# [2,] 2 12
So it seems that the newDT$b[2] <- 200
line somehow 'breaks' the reference. I'd guess that this invokes a copy somehow, but I would like to understand fully how R is treating these operations, to ensure I don't introduce potential bugs in my code.
I'd very much appreciate if someone could explain this to me.
Best Answer
Yes, it's subassignment in R using
<-
(or=
or->
) that makes a copy of the whole object. You can trace that usingtracemem(DT)
and.Internal(inspect(DT))
, as below. Thedata.table
features:=
andset()
assign by reference to whatever object they are passed. So if that object was previously copied (by a subassigning<-
or an explicitcopy(DT)
) then it's the copy that gets modified by reference.Notice how even the
a
vector was copied (different hex value indicates new copy of vector), even thougha
wasn't changed. Even the whole ofb
was copied, rather than just changing the elements that need to be changed. That's important to avoid for large data, and why:=
andset()
were introduced todata.table
.Now, with our copied
newDT
we can modify it by reference :Notice that all 3 hex values (the vector of column points, and each of the 2 columns) remain unchanged. So it was truly modified by reference with no copies at all.
Or, we can modify the original
DT
by reference :Those hex values are the same as the original values we saw for
DT
above. Typeexample(copy)
for more examples usingtracemem
and comparison todata.frame
.Btw, if you
tracemem(DT)
thenDT[2,b:=600]
you'll see one copy reported. That is a copy of the first 10 rows that theprint
method does. When wrapped withinvisible()
or when called within a function or script, theprint
method isn't called.All this applies inside functions too; i.e.,
:=
andset()
do not copy on write, even within functions. If you need to modify a local copy, then callx=copy(x)
at the start of the function. But, rememberdata.table
is for large data (as well as faster programming advantages for small data). We deliberately don't want to copy large objects (ever). As a result we don't need to allow for the usual 3* working memory factor rule of thumb. We try to only need working memory as large as one column (i.e. a working memory factor of 1/ncol rather than 3).