Create a RESTful API for OpenPBS/Torque with Node.js and Express v4

In this post, we are going to create a API to interact with OpenPBS/Torque job scheduler sitting on the same machine as your web server or not using the nodejs-pbsjs module we distribute on GitHub in order to submit or control jobs in the queue and retrieve information under JSON form. This module can be used in two different ways: the node server interacts with the workload manager locally or remotely and then

- renders the info on the server-side

- or send the info to be rendered on the client-side in order to create a RESTful API.

This post is concerns the first method for now and another post will follow in order to create a RESTful API and render the info with Angularjs.

First, if you still haven't found a good IDE, you really should consider Cloud9 which is a really good editor coming in a web page (available also in the cloud } with really efficient auto-completion capabilities. You can pull our Docker container to automatically start a running Cloud9 environment on your machine, which is a fork of the kdelfour/cloud9-docker container but with node v5.

 

 1. Set-up

- To start our API, we will set-up a basic Express website with express generator. First clone the git repo of pbsjs and initialize an express app:

git clone https://github.com/quantumhpc/nodejs-pbsjs.git
express myapp

Go into the directory and install all the depencies

cd myapp
npm install 

For the purpose of this tutorial, we will use the ejs engine to render our pages which is good to loop on JSON data and output it in tables. Change the line:

app.set('view engine', 'jade' };

to

app.set('view engine', 'ejs' };

Install the ejs module

npm install ejs --save


and delete the files inside the views directory.

- Now create a simple index.ejs inside the views directory with the following content:


 

<%=title%>


Results

And to debug you app, an error.ejs file containing:


 

Error

<%= message %>

<%= error.status %>

<%= error.stack %>

- Replace the last line of the app.js file

module.exports = app;

by

app.listen(8080 };
console.log('The magic happens on port 8080' };

Now fire up the app.js file with node app.js or if you are in C9, by clicking on the Run icon, go to port 8080 on the same server and you should see:

Those are the basic steps to start a small express app. There are tons of tutorial on the web so it will mot be developed further so we can focus on the most interesting part!

 

2. Integrate the pbsjs module

The pbsjs module requires a config file under JSON format so that the information can be retrieved from a file, a request or a DB. It's up to you to configure it according to how you save those information. An config file example is available under the config dir of nodejs-pbsjs, modify it accordingly and open the routes/index.js file and insert the following line at the top:

var pbs_config = require("../nodejs-pbsjs /config/pbsserver.json" };

assuming you cloned the git repository in the same location you init the express app and reference the module with:

var pbsjs = require("../nodejs-pbsjs/pbsjs.js" };

 

3. Configuring routes

We will now create several routes according to the info we want to retrieve or send to OpenPBS/Torque.

a. The node list

The first one will display the result of the qnodes command. For that, create a new route similar to the index route with:

router.get('/nodelist', function(req, res, next } {
 res.render('index', { title: 'Server node list' } };
} };

Restart the server and go to yourserver:8080/nodelist to see the result.

Now let's get our information from the server. Each function of pbsjs requires the config parameters as the first arguments as well as one or more arguments related to the command to execute on the server and a nodejs-style callback. We will gather the info on the nodes with the command:

pbsjs.qnodes_js(pbs_config, callback(err,data } };

Inside the express router, it gives:

router.get('/nodelist', function(req, res, next } {
 pbsjs.qnodes_js(pbs_config, function(err,data }{
 if(err }{
 next(err };
 return;
 }
 res.render('index', { 
 title: 'Server node list', 
 nodelist: data
 } };
 } };
} };

The error if any is handled by the next controller with is defines inside the app.js file. We save here the result from the module inside the nodelist variable. To test the result, let's just display the plain JSON in the web page:

- Make a copy of the index.ejs file to nodes.ejs and change the router route for nodelist accordingly to obtain:

res.render('nodes', { 
 title: 'Server node list', 
 nodelist: data
 } };

- Now open nodes.ejs and insert below 'Results': 

<%-JSON.stringify(nodelist }%>

You should see the following results:

 

Now let's organize the results with a basic table. Replace the last line by the following code:

<% for (node in nodelist } {%>
 
<% for (key in nodelist[node] } {%> <% } %>
<%= key %> <%= nodelist[node][key] %>
<% } %>

Restart and you should be able to see:

Basic styling but the information are there! You should now be able to display those information in your local web server with a better rendering.

 

b. The job list

Following the same pattern, create a new route inside routes/index.js with the following code:

router.get('/joblist', function(req, res, next } {
 pbsjs.qstat_js(pbs_config, function(err,data }{
 if(err }{
 next(err };
 return;
 }
 res.render('jobs', { 
 title: 'Server job list', 
 joblist: data
 } };
 } };
} };

Duplicate the views/nodes.ejs file to create views/jobs.ejs and replace the part below 'Results' by the following:


 <% for (key in joblist[0] } {%>
 
 <% } %>
 
 <% for (job in joblist } {%>
 
 <% for (key in joblist[job] } {%>
 
 <% } %>
 
 <% } %>
 
<%= key %>
<%= joblist[job][key] %>

Since each object representing a job has the same structure, we use the joblist[0] to create the header of the table. You should now obtain:

 

c. Submitting a job

Now let's deal with the most interesting functionnality of the workload manager: submitting a job. To create a route handling the submission you will need a form gathering the options to submit your job as well as the main command to be executed. We will see how to send files along in another tutorial.

- First create a file inside views called submit.ejs and insert a form inside calling the route submitjob:

- Create inside routes/index.js the following two routes to display our form and submit it with a post method which gives:

router.get('/submit', function(req, res, next } {
 res.render('submit', { title: 'Submit a job' } };
} };

router.post('/submitjob', function(req, res, next } {
 
} };

- Now we need to use two actions inside pbsjs: 1 to create the script ( qscript_js(jobArgs, localJobDir, callback(err,data } } } and 1 to submit it to the server ( qsub_js(pbs_config, [submissionScript, jobFiles, ..], callback(err,data } } }

Here the correspondance between the arguments of qscript and an example form to submit those arguments:

JobArgs field Type Form element
shell String
jobName String
ressources String
walltime String
queue String
exclusive String
mail Boolean
mailAbort Boolean
mailBegins Boolean
mailTerminates Boolean
commands Array

Insert the following form inside submit.ejs and you should have :

 

To parse the content of the form, we need to look inside req.body inside the submitjob route and send the content to qscript_js along with a path to save the script. Add the following lines to the submitjob route to call qscript:

router.post('/submitjob', function(req, res, next } {
 pbsjs.qscript_js(req.body, '/tmp', function(err,data }{
 if (err }{
 next(err };
 return;
 }

If the job is well submitted, qscript should answer with data.message giving the status of the creation and data.path with the path of script with the following content based on the previous form:

root@dev:/tmp# cat testjob 
# Autogenerated script
#PBS -S /bin/bash
#PBS -N testjob
#PBS -l select=2
#PBS -l walltime=00:30:00
#PBS -q batch
#PBS -n
#PBS -M test@mail.com
#PBS -m be
parafoam -np 2

If you put several lines in the commands textarea, you should look at the result of the form to replace line breaks according to what your workload manager environment supports.

Now, still in the callback of qscript, we need to call qsub_js with the path of script. Add the following lines inside the qscript_js callback to submit the job:

router.post('/submitjob', function(req, res, next } {
 pbsjs.qscript_js(req.body, '/tmp', function(err,data }{
 if (err }{
 next(err };
 return;
 }
 pbsjs.qsub_js(pbs_config,[data.path],function(err,data }{
 if (err }{
 next(err };
 return;
 }
 res.send(data.message };
 } }
 } }
} };

If the submission is successful, data should contain the jobId and the success message that we send as a response to display on the web page. Restart the server and test the form, you should see:


That's it for now! You have a small web app to interact with your OpenPBS/Torque server and submit jobs through a simple web page. Pbsjs also includes a method to look inside the working folder, retrieve the files or get info on a specific job or cancel it. Do not hesitate to contribute on GitHub or ask questions about it.

Good coding!

Share this post

Comments (0)

Leave a comment