Troubleshooting missing job records in Slurm accounting

Jobs that ran to completion are missing from Slurm accounting reports, such as the output of the sacct or sreport commands. Records are absent only for a limited time window, typically one that follows a period of very high job submission rate. Accounting reports are complete before and after that window. This topic applies to clusters that have Slurm accounting enabled. For more information about accounting, see Slurm accounting in AWS PCS.

Common cause

The Slurm controller daemon (slurmctld) doesn't write accounting data to the accounting database directly. It sends the data to the Slurm database daemon (slurmdbd) through an internal, in-memory queue. The queue can reach its maximum size if jobs are submitted faster than slurmdbd commits records to the database. When the queue is full, slurmctld discards new accounting messages instead of adding them to the queue.

Discarded messages aren't retried, so the job history that they carried is permanently absent from the accounting database. Only accounting data is affected. Job scheduling and job execution continue normally. Records that slurmdbd already committed remain in the database. Accounting resumes automatically when the queue drains.

Resolution

To confirm the cause and reduce the chance of recurrence

1. Search the slurmctld log of the cluster for entries similar to the following.

   error: agent queue is full (33794), discarding DBD_JOB_START:1425 request

   Each entry corresponds to one discarded accounting message. The message type identifies the record that was lost, such as DBD_JOB_START, DBD_JOB_COMPLETE, DBD_STEP_START, or DBD_STEP_COMPLETE. If the cluster doesn't deliver scheduler logs, set up log delivery. This lets you confirm the cause if the condition occurs again. For more information, see Scheduler logs in AWS PCS.

2. Set the CommitDelay parameter to 1 in the slurmdbd configuration of the cluster. With this setting, slurmdbd groups database commits instead of committing each record individually, which increases the rate at which it drains the queue. Use the SlurmdbdCustomSettings property of the cluster to apply the setting. For more information, see Configuring custom SlurmDBD settings in AWS PCS and CommitDelay in the Slurm documentation.

   Important

   SlurmdbdCustomSettings replaces the slurmdbd settings of the cluster instead of adding to them. Any setting that you omit from the update request is removed. If the cluster might already have slurmdbd settings, run aws pcs get-cluster first, add {parameterName=CommitDelay,parameterValue="1"} to the retrieved list, then submit the complete list.

   Example– Setting CommitDelay on a cluster

   aws pcs update-cluster --cluster-identifier my-cluster \
   --slurm-configuration \
   'SlurmdbdCustomSettings=[{parameterName=CommitDelay,parameterValue="1"}]'

3. Confirm that the setting is in place. In the AWS Management Console, view Additional scheduler settings for the cluster. In the AWS CLI, run the following command and check the slurmConfiguration field of the response.

   aws pcs get-cluster --cluster-identifier my-cluster

Note

Accounting records that slurmctld discarded can't be recovered. Jobs that ran during the affected window remain absent from accounting reports.

Prevention

To detect queue saturation while it happens instead of finding missing records later, deliver slurmctld logs to Amazon CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), or Amazon Data Firehose, and monitor them for agent queue is full entries. For more information, see Scheduler logs in AWS PCS.

Keep CommitDelay set to 1 on clusters that run workloads with a high job submission rate.