↓ Skip to main content
Shota Ikari

Notes on a Small Slurm Cluster with Ansible

·444 words·3 mins
Author
Shota Ikari
Master’s student at the University of Tokyo.
Table of Contents

I cleaned up an Ansible playbook for provisioning a small Ubuntu-based Slurm cluster and put the sanitized version on GitHub.

Repository:

The playbook started as infrastructure code for a local lab environment. Before making it public, I removed site-specific inventories, host variables, generated hardware facts, SSH keys, Munge keys, local environment files, and old reference material. The repository now keeps those files out of Git and includes a Docker Compose environment for validating the roles safely.

Scope
#

The repository provisions:

  • a Slurm controller,
  • CPU and GPU compute nodes,
  • Munge authentication,
  • SlurmDBD accounting,
  • Prometheus,
  • Grafana,
  • Node Exporter,
  • optional DCGM Exporter for GPU telemetry.

The useful part is that the same Ansible structure can be tested locally before touching real machines.

Docker validation
#

The local test environment can be started with:

task dev:up
task dev:deploy
task status ENV=docker

The Docker inventory creates one controller and three compute containers:

NodeRole
controllerSlurm controller, SlurmDBD, Prometheus, Grafana
gpu1example GPU compute node
gpu2example GPU compute node
cpu1example CPU compute node

The Docker-only SSH and Munge keys are generated under secrets/, which is ignored by Git.

Slurm resource view
#

I also restored a small reference command named sres, which prints a compact view of Slurm node resources. In the Docker cluster, it produced:

┌───────────┬────┬─────────┬──────────────┬──────────────────────┐
│      node │ ST │   CPU   │    memory    │       CPU-JOBS       │
├───────────┼────┼─────────┼──────────────┼──────────────────────┤
│      cpu1 │ 🐟 │  6 / 6  │  23 / 23  GB │       CPU free       │
├───────────┼────┼─────────┼──────────────┼──────────────────────┤
│      gpu1 │ 🐟 │  6 / 6  │  23 / 23  GB │       CPU free       │
├───────────┼────┼─────────┼──────────────┼──────────────────────┤
│      gpu2 │ 🐟 │  6 / 6  │  23 / 23  GB │       CPU free       │
└───────────┴────┴─────────┴──────────────┴──────────────────────┘

Monitoring
#

Prometheus is configured to scrape the controller and compute nodes.

Prometheus targets showing node exporter endpoints are up
Prometheus targets in the Docker validation environment.

Grafana is provisioned with dashboards for the Slurm cluster and node metrics.

Grafana Slurm dashboard showing node status, CPU usage, and memory usage
Grafana Slurm dashboard rendered from the Docker validation environment.

Keeping the Grafana provisioning files and dashboard JSON in Git makes the monitoring layer easier to reproduce after rebuilding the environment.

Lab deployment example
#

In our lab, the same setup is used to keep an eye on the actual Slurm nodes and GPU resources. Ansible is valuable not only for its idempotent infrastructure management, but also as a practical tool for everyday system administration.

sres terminal output showing CPU, memory, GPU availability, and Slurm node states in the lab cluster
Actual lab cluster resource summary shown by sres.
Grafana dashboard showing node status, CPU, memory, GPU utilization, GPU memory, temperature, and power usage in the lab cluster
Grafana dashboard from the lab Slurm cluster.

References
#