Parallelization in R

Overview

parallelization
future
plans
- sequential
- multisession/multicluster
- cluster

parallelization

Save time by doing atomic tasks in parallel

Divide tasks or datasets into smaller pieces

Can bottleneck if tasks are directly dependent

R-package: future

Other packages decide the parallelization method during development. With future the code is the same and the USER decides parallelization method.

Very simple
Uniform code, no matter the strategy
User defined parallelization
Unblocked R process during resolving of futures process
Works well on multiple architectures

R package developers rarely know who the end-users are and what compute resources they have. With future, instead of programming for one architecture, the code should work on most architectures. We will get back to the Unblocked R process during resolving but basically what it means is that even if multiple things are being computed in parallel the R code can continue unblocked until the resolved values are needed.

Text from article: The state of a future can either be unresolved or resolved. As soon as it is resolved, the value is available instantaneously. If the value is queried while the future is still unresolved, the current process is blocked until the future is resolved.

R package developers rarely know who the end-users are and what compute resources they have. Regardless, developers who wish to support parallel processing still face the problem of deciding which parallel framework to target, a decision which often has to be done early in the development cycle. This means deciding on what type of parallelism to support, This decision is critical because it limits the end-user’s options and any change, later on, might be expensive because of, for instance, having to rewrite and retest part of the codebase. A developer who wishes to support multiple parallel backends has to implement support for each of them individually and provide the end-user with a mechanism to choose between them.

Building block: variable %<-% {expression(s)}

Plans

plan(sequential)

Building block: variable %<-% {expression(s)}

future::plan(sequential)

a %<-% {
  Sys.sleep(3)
  a <- 1
}
b %<-% {
  Sys.sleep(3)
  b <- 2
}

a + bfuture::plan(sequential)

a %<-% {
  Sys.sleep(3)
  a <- 1
}
b %<-% {
  Sys.sleep(3)
  b <- 2
}

a + bfuture::plan(sequential)

a %<-% {
  Sys.sleep(3)
  a <- 1
}
b %<-% {
  Sys.sleep(3)
  b <- 2
}

a + bfuture::plan(sequential)

a %<-% {
  Sys.sleep(3)
  a <- 1
}
b %<-% {
  Sys.sleep(3)
  b <- 2
}

a + b

[1] 3
   user  system elapsed 
  0.061   0.004   6.076

plan(multisession) & plan(multicore)

plan(multicore)

a %<-% {
  Sys.sleep(3)
  a <- 1
}
b %<-% {
  Sys.sleep(3)
  b <- 2
}

a + b

[1] 3
   user  system elapsed 
  0.129   0.072   3.154

availableCores()

system 
     6

plan(multisession) & plan(multicore)

plan(multicore)

a %<-% {
  Sys.sleep(3)
  a <- 1
}
b %<-% {
  Sys.sleep(3)
  b <- 2
}
c %<-% {
  Sys.sleep(3)
  c <- 3
}
...
k %<-% {
  Sys.sleep(3)
  e <- 5
}

a + b + c + d + e + f + g + h + i + j + k

[1] 60
   user  system elapsed 
  0.701   0.298   6.392

plan(cluster)

To some degree a wrapper around parallel::makeCluster()
For example:
- 3 connected nodes (computers) named n1:n3
- Each with 16 CPUs

plan(cluster, workers = c("n1", "n2", "n3"))

Specialized R package for interfacing with common HPC job schedulers exists: future.batchtools

Thank you! Questions?

         _                  
platform x86_64-pc-linux-gnu
os       linux-gnu          
major    4                  
minor    2.3

2023 • SciLifeLab • NBIS • RaukR