RevoScaleR: get XDF info from all nodes RRS feed

  • Question

  • I use Microsoft R Server 2016 on a Hadoop cluster. I am trying to get XDF information (rxGetInfo) from all nodes, but it seems it doesn't work even though I set the allNodes = TRUE. 

    > higgsInfoAllNodes <- rxGetInfo(higgsXdfDS, getVarInfo = TRUE, getBlockSizes = TRUE, allNodes = TRUE)

    > higgsInfoAllNodes
    File name: /user/share/higgs/higgsXDF100000
    Number of composite data files: 60
    Number of observations: 1.1e+07
    Number of variables: 29
    Number of blocks: 120
    Rows per block in first data file: 1e+05 83733
    Compression type: zlib
    Variable information: ...

     > higgsInfoAllNodes$rowsPerBlock
    [1] 100000  83733

    Do you have any idea how to solve this problem. For example, how can I get block sizes of other composite data files?  

    Thank You!

    Tuesday, November 8, 2016 5:22 AM