Monitoring Parallel Processes RRS feed

  • Question

  • I (am pretty sure I) have been running my code in parallel for about 10 months.  However, recently during (what I thought was) a 32-core parallel computation, my sysadmin looked at my usage and informed me that I was only using three cores.  He claimed that my use of makeCluster() was not doing what I thought it should be doing.  But in order to confirm whether or not he is correct, I need a way to look deeper into what exactly makeCluster() is doing behind the scenes.  Note that this will require a more involved solution than merely calling detectCores().  Unfortunately, the documentation for parallel computing in R is quite scant.  It gives a very nice introduction to parallel computing in R, but doesn't specify how to dig deeper and find out how exactly the different cores are coming into play.  For someone like myself, who feels much safer using software if they know how it works, this high level introduction is not enough.

    Here is the code I am using to set up the parallel computation at the beginning of my script:

    no_cores <- 32
    cl <- makeCluster(no_cores)

    After this, I use a "foreach" loop with the %dopar% operator to fork the computation to the 32 different cores.  My questions are:

    (a) Is the code chunk doing what I think it is doing?  (i.e. will the subsequent "foreach" loop fork as I intend?)

    (b) Can someone point me in a direction that will either explain what makeCluster() is doing behind the scenes or give me the tools to figure out how it works on my own?  I particular, how can I monitor the different cores in real time?  I have tried htop from the command line, but not really sure how to interpret the htop pane.  Specifically, the htop pane shows that I am using a ton of different processes; many more than 32.  

    Thank you,


    Wednesday, November 15, 2017 5:39 PM

All replies

  • Here are some pointers, that may be helpful.  doParallel() only makes use of physical cores in parallel computations - virtual cores do not count.  One way to check how many cores doParallel will use after setting up your parallel backend is to call the function 'getDoParWorkers()', to verify the number of cores in use.

    Also, you need to make sure that R has been installed on each worker node.

    I would take a look at this vignette for some additional information:

    Friday, November 17, 2017 6:36 PM