monitauR 🐮

Introducing MonitauR, a package to easily and remotely monitor the progress of your R scripts.

I like the idea of being able to see the progress of your analysis pipelines though some user friendly Interface i.e. not a qsub log file or command line output. I decide to combine this idea with my mission to begrudgingly learn more R programming.

I like to make Restful webAPIs (insert link to future Fasteve post here) and thought that would be a nice place to start with R as I already know a lot of the theory. However, I needed something to build my API around and thus monitauR was borne.

Plumber

Plumber is an API Generator for R. Plumber allows you to create a web API by merely decorating your existing R source code with special comments. The initialisation function from plumber builds a router that maps incoming requests to endpoints. Plumber files uses a special comment syntax to tell plumber how to build the API. Below is an example of a plumber file with the special comments (#*). For example you could go to http://localhost:8000/echo?msg=hello and the API would return hello. Not bad! I like the ability to quickly go from R code (like some statistical model) to an API with no real changes to the code.

# plumber.R

#* Echo back the input
#* @param msg The message to echo
#* @get /echo
function(msg="") {
  list(msg = paste0("The message is: '", msg, "'"))
}

Special comment hacks

This is not the first time I’ve ran into this kind of special comment syntax for modifying an R script. Roxygen2 also uses special comments (#') to help build packages and build documentation from R source code. These comments almost feel like hacks as they are a syntax built on top of the langue that has nothing to do with it, however, they don’t interfere with running scripts as they are after all just comments.

I started to think of other uses of these special comment hacks and landed on what I think is an elegant solution to script progress monitoring.

MonitauR

MonitauR uses a similar tactic to plumber and roxygen e.g. special comments define when some code should be executed.

The best way to explain might be with an example. Below is a script documented with the monitauR special comments.

# example_scripts/square.R

#< Setting up the square function
square <- function(x) {
  x*x
}
#< Computing the square
square(5)

Running the script with monitauR produces the following output.

Rscript -e "monitauR::monitor('example_scripts/square.R')"

Job 1: --- Initialised ---
Job 1: Setting up the square function (1/2)
Job 1: Computing the square (2/2)
Job 1: --- Finished ---

While evaluating the script when the special comment (#<) is reached a request is sent to a server telling it to log the comment.

This allows you to follow allowing with your script in real-time from anywhere with an internet connection.

curl 'https://monitaur-api.herokuapp.com/jobs' | json_pp

[
   {
      "created" : "2020-09-07 14:49:35",
      "id" : 1,
      "name" : "example_scripts/square.R",
      "status" : "finished",
      "updated" : "2020-09-07 14:49:35"
   }
]

curl 'https://monitaur-api.herokuapp.com/steps' | json_pp

[
   {
      "created" : "2020-09-07 14:49:35",
      "id" : 1,
      "job_id" : 1,
      "msg" : "Setting up the square function",
      "updated" : "2020-09-07 14:49:35"
   },
   {
      "created" : "2020-09-07 14:49:35",
      "id" : 2,
      "job_id" : 1,
      "msg" : "Computing the square",
      "updated" : "2020-09-07 14:49:35"
   }
]

The lifecycle of a monitauR script is as follows:

1. The script is parsed and the expressions are extracted
2. Job ID (and token) is generated (status set to initialised)
3. A future is created (using the Job ID) for each special comment step
4. Job status is set to running
5. Script is evaluated and futures are run in sequential order
6. Any errors are caught and sent to the server (status set to error)
7. When the script completes the Job status is set to finished

The real trick to monitauR is creating futures (step 3) for each of the comments so that the calls to the server only happen when the evaluation reaches that part of the script.

monitauR-api

There is a plumber api running on https://monitaur-api.herokuapp.com/ that receives and logs requests from monitauR::monitor i.e. when a special comment syntax (#<) is reached.

monitauR-webapp

To complete this project I needed a nice way to interact with the jobs being monitored. I made a webapp using 11ty, alpine.js and tailwind css (Details here. ).