CCQ and HPC Jobs

HPC Job Script Introduction

A script is a set of instructions that specify how a job should be executed. These instructions are general in nature, since CloudyCluster can be used to run a wide variety of job types on GCP.

Torque and Slurm are job schedulers used to start, hold, monitor and cancel jobs submitted via the CloudyCluster interface. CCQ is a meta-scheduler provided with CloudyCluster that handles the instance selection and scaling the passes the jobs off to the configured scheduler, Torque or Slurm.

CloudyCluster Queue (CCQ)

CCQ is a meta-scheduler, or a wrapper that provides a single interface, for common schedulers such as Torque and Slurm. It is an overarching category of job utilities and the utility commands will be familiar to those who have used the qsub, qstat, qdel, sbatch, and other commands used by other HPC schedulers. Users interact with ccqsub in the command line interface of an instance in your environment.

CCQ gives the schedulers autoscaling capability within the Google Cloud. When a user submits a job using this script, the job script will be parsed, the number of instances needed is determined, and the instances will be launched for the job at run-time. These commands can be denoted inside of the jobscript itself by using the #CC directive.

The commands that are available through the #CC directive are shown in a following section.