memory - merging really not that large data.tables immediately results in R being killed -

-

i have 32gb of ram on machine, can r killed faster ;)

example

the goal here achieve rbind() of 2 data.tables using functions make use of data.table's efficiency.

input:

rm(list=ls()) gc()

output:

          used (mb) gc trigger   (mb)  max used   (mb) ncells 1604987 85.8    2403845  128.4   2251281  120.3 vcells 3019405 23.1  537019062 4097.2 468553954 3574.8

input:

tmp.table <- data.table(x1=sample(1:7,4096000,replace=true),                            x2=as.factor(sample(1:2,4096000,replace=true)),                            x3=sample(1:1000,4096000,replace=true),                            x4=sample(1:256,4096000,replace=true),                            x5=sample(1:16,4096000,replace=true),                            x6=rnorm(4096000))  setkey(tmp.table,x1,x2,x3,x4,x5,x6)  join.table <- data.table(x1 = integer(), x2 = factor(),                           x3 = integer(), x4=integer(),                          x5 = integer(), x6 = numeric())  setkey(join.table,x1,x2,x3,x4,x5,x6)  tables()

output:

     name            nrow  mb cols              key               [1,] join.table         0   1 x1,x2,x3,x4,x5,x6 x1,x2,x3,x4,x5,x6 [2,] tmp.table  4,096,000 110 x1,x2,x3,x4,x5,x6 x1,x2,x3,x4,x5,x6 total: 111mb

input:

join.table <- merge(join.table,tmp.table,all.y=true)

output:

ha! nope. rstudio restarts session.

question

what's going on here? explicitly setting factor levels in join.table had no effect. rbind() instead of merge() didn't help--exact same behavior. have done way more complicated , bulky things related data without problems.

version info

$platform [1] "x86_64-pc-linux-gnu"  $arch [1] "x86_64"  $os [1] "linux-gnu"  $system [1] "x86_64, linux-gnu"  $version.string [1] "r version 3.0.2 (2013-09-25)"  $nickname [1] "frisbee sailing"  > rstudio::versioninfo() $version [1] ‘99.9.9’  $mode [1] "server"

data.table version 1.8.11.

update: has been fixed in commit 1123 of v1.8.11. news:

o rbindlist @ least 1 factor column along presence of @ least 1 empty data.table resulted in segfault (or in linux/mac reported error related hash tables). fixed, #5355. trevor alexander reporting on (and mnel filing bug report): merging not large data.tables results in r being killed

this can reproduced single row data.table factor column , zero-row data.table factor column.

library(data.table) <- data.table(x=factor(1), key='x') b <- data.table(x=factor(), key='x') merge(b, a, all.y=true)  # rstudio -> r encountered fatal error #  r gui -> r windoze gui has stopped working

using debugonce(data.table:::merge.data.table) can traced line rbind(dt,yy) equivalent of

rbind(b,a)

which, if run it, give same error.

this has been reported package authors issue #5355


Comments

Post a Comment

Popular posts from this blog

python - Subclassed QStyledItemDelegate ignores Stylesheet -

-
i'm trying subclass qstyleditemdelegate remove focus rectangle in qcombobox . even though i'm calling base implementation of paint function , nothing else, result different. looks if parts of style-sheet influence bounding box of item taken consideration. class pstyleditemdelegate(qstyleditemdelegate): def __init__(self, *args, **kwds): super(pstyleditemdelegate, self).__init__(*args, **kwds) def paint(self, *args, **kwargs): qstyleditemdelegate.paint(*args, **kwargs) what have paint unmodified qstyleditemdelegate ? as suggested tried replacing pyside pyqt4 , works, appears bug. update pyside 1.1.2 1.2.1 result same. unfortunately switch breaks other parts of code, if there no other suggestions i'm going accept answer. edit bug tracked here
Read more

java - HttpClient 3.1 Connection pooling vs HttpClient 4.3.2 -

-
i have used httpclient3.1(java) connection pooling below : import org.apache.commons.httpclient.httpclient; import org.apache.commons.httpclient.multithreadedhttpconnectionmanager; import org.apache.commons.httpclient.params.httpclientparams; import org.apache.commons.httpclient.params.httpmethodparams; import org.apache.log4j.logger; public class httpclientmanager { private static logger logger = logger.getlogger(httpclientmanager.class); private static multithreadedhttpconnectionmanager connectionmanager = new multithreadedhttpconnectionmanager(); private static httpclientparams clientparams = new httpclientparams(); static { try { clientparams.makelenient(); clientparams.setauthenticationpreemptive(false); clientparams.setparameter(httpmethodparams.so_timeout, new integer(30000)); // set so_timeout clientparams.setparameter(httpmethodparams.head_body_check_timeout, new integer(3000)); // head responses ...
Read more

node.js - StackOverflow API not returning JSON -

-
i using node.js script gather data stackoverflow. know responses gzipped, put code in should take care of that. script follows: var request = require('request'); var zlib = require('zlib'); function getstackoverflowresponse(url, callback){ request(url, {encoding: null}, function(err, response, body){ if(response.headers['content-encoding'] == 'gzip'){ zlib.gunzip(body, function(err, dezipped) { callback(dezipped); }); } else { callback(body); } }); } var url = "https://api.stackexchange.com/docs/questions#pagesize=2&order=desc&min=2014-01-04&max=2014-02-02&sort=activity&tagged=apigee&filter=default&site=stackoverflow&run=true"; getstackoverflowresponse(url, function(questions) { console.log(questions); }); instead of getting json output, i'm getting following response: buffer 0d 0a 0d 0a 0d 0a 0d 0a 3c 21 44 4f 43 54 59 50 45 20 48...
Read more