FreeBSD Manual Pages

home | help

sacctmgr(1) Slurm Commands sacctmgr(1)

NAME
sacctmgr - Used to view and modify Slurm account information.

SYNOPSIS
sacctmgr [OPTIONS...] [COMMAND...]

DESCRIPTION
sacctmgr is used to view or modify Slurm account information. The ac-
count information is maintained within a database with the interface
being provided by slurmdbd (Slurm Database daemon). This database can
serve as a central storehouse of user and computer information for mul-
tiple computers at a single site. Slurm account information is
recorded based upon four parameters that form what is referred to as an
association. These parameters are user, cluster, partition, and ac-
count. user is the login name. cluster is the name of a Slurm managed
cluster as specified by the ClusterName parameter in the slurm.conf
configuration file. partition is the name of a Slurm partition on that
cluster. account is the account for a job. The intended mode of oper-
ation is to initiate the sacctmgr command, add, delete, modify, and/or
list association records then commit the changes and exit.

NOTE: The contents of Slurm's database are maintained in lower case.
This may result in some sacctmgr output differing from that of other
Slurm commands.

OPTIONS
-s, --associations
Use with show or list to display associations with the entity.
This is equivalent to the associations command.

-h, --help
Print a help message describing the usage of sacctmgr. This is
equivalent to the help command.

-i, --immediate
Commit changes immediately without asking for confirmation.

--json, --json=list, --json=<data_parser>
Dump information as JSON using the default data_parser plugin or
explicit data_parser with parameters. Sorting and formatting ar-
guments will be ignored. This option is not available for every
command.

-n, --noheader
No header will be added to the beginning of the output.

-p, --parsable
Output will be '|' delimited with a '|' at the end.

-P, --parsable2
Output will be '|' delimited without a '|' at the end.

-Q, --quiet
Print no messages other than error messages. This is equivalent
to the quiet command.

-r, --readonly
Makes it so the running sacctmgr cannot modify accounting infor-
mation. The readonly option is for use within interactive mode.

--yaml, --yaml=list, --yaml=<data_parser>
Dump information as YAML using the default data_parser plugin or
explicit data_parser with parameters. Sorting and formatting ar-
guments will be ignored. This option is not available for every
command.

-v, --verbose
Enable detailed logging. This is equivalent to the verbose com-
mand.

-V , --version
Display version number. This is equivalent to the version com-
mand.

COMMANDS
add <ENTITY> <SPECS>
Add an entity. Identical to the create command.

archive {dump|load} <SPECS>
Write database information to a flat file or load information
that has previously been written to a file.

clear stats
Clear the server statistics.

create <ENTITY> <SPECS>
Add an entity. Identical to the add command.

delete <ENTITY> where <SPECS>
Delete the specified entities. Identical to the remove command.

dump <cluster>
Dump cluster data to the specified file. If the filename is not
specified it uses clustername.cfg filename by default.

help Display a description of sacctmgr options and commands.

list <ENTITY> [<SPECS>]
Display information about the specified entity. By default, all
entries are displayed, you can narrow results by specifying
SPECS in your query. Identical to the show command.

load <FILENAME>
Load cluster data from the specified file. This is a configura-
tion file generated by running the sacctmgr dump command. This
command does not load archive data, see the sacctmgr archive
load option instead.

modify <ENTITY> where <SPECS> set <SPECS>
Modify an entity.

ping Ping slurmdbd.

reconfigure
Reconfigures the SlurmDBD if running with one.

remove <ENTITY> where <SPECS>
Delete the specified entities. Identical to the delete command.

show <ENTITY> [<SPECS>]
Display information about the specified entity. By default, all
entries are displayed, you can narrow results by specifying
SPECS in your query. Identical to the list command.

shutdown
Shutdown the server.

version
Display the version number of sacctmgr.

INTERACTIVE COMMANDS
NOTE: All commands listed below can be used in the interactive mode,
but NOT on the initial command line.

exit Terminate sacctmgr interactive mode. Identical to the quit com-
mand.

quiet Print no messages other than error messages.

quit Terminate the execution of sacctmgr interactive mode. Identical
to the exit command.

verbose
Enable detailed logging. This includes time-stamps on data
structures, record counts, etc. This is an independent command
with no options meant for use in interactive mode.

!! Repeat the last command.

ENTITIES
account
An account, typically specified at job submit time using the
--account= option. These may be arranged in a hierarchical
fashion, for example accounts 'chemistry' and 'physics' may be
children of the account 'science'. The hierarchy may have an
arbitrary depth.

association
The entity used to group information consisting of four parame-
ters: account, cluster, partition (optional), and user. Used
only with the list or show command. Add, modify, and delete
should be done to a user, account or cluster entity, which will
in turn update the underlying associations. Modification of at-
tributes like limits is allowed for an association but not a
modification of the four core attributes of an association. You
cannot change the partition setting (or set one if it has not
been set) for an existing association. Instead, you will need to
create a new association with the partition included. You can
either keep the previous association with no partition defined,
or delete it. Note that these newly added associations are
unique entities and any existing usage information will not be
carried over to the new association.

cluster
The ClusterName parameter in the slurm.conf configuration file,
used to differentiate accounts on different machines.

configuration
Used only with the list or show command to report current system
configuration.

coordinator
A special privileged user, usually an account manager, that can
add users or sub-accounts to the account they are coordinator
over. This should be a trusted person since they can change
limits on account and user associations, as well as cancel, re-
queue or reassign accounts of jobs inside their realm.

event Events like downed or drained nodes on clusters. Note that this
does not include transitory states like DRAINING.

federation
A group of clusters that work together to schedule jobs.

job Used to modify specific fields of a job: Derived Exit Code, Com-
ment, AdminComment, Extra, SystemComment, TRES, or WCKey.

problem
Use with show or list to display entity problems.

qos Quality of Service.

reservation
A collection of resources set apart for use by a particular ac-
count, user or group of users for a given period of time.

resource
Software resources for the system. Those are software licenses
shared among clusters.

RunawayJobs
Used only with the list or show command to report current jobs
that have been orphaned on the local cluster and are now run-
away. If there are jobs in this state it will also give you an
option to "fix" them. This sets the end time for each job to
the latest of the job's start, eligible, and submit times, and
sets the state to completed by default. Once corrected, this
triggers the SlurmDBD to recalculate the usage from before the
earliest submit time of all the runaway jobs. NOTE: This could
take a long time and sreport may not return data until the re-
calculation is completed. NOTE: You must have an AdminLevel of
at least Operator to perform this.

stats Used with list or show command to view server statistics. Ac-
cepts optional argument of ave_time or total_time to sort on
those fields. By default, sorts on increasing RPC count field.

transaction
List of transactions that have occurred during a given time pe-
riod.

tres Used with list or show command to view a list of Trackable RE-
Sources configured on the system.

user The login name. Usernames are case-insensitive (forced to lower-
case) unless the PreserveCaseUser option has been set in the
SlurmDBD configuration file.

wckeys Workload Characterization Key. An arbitrary string for
grouping orthogonal accounts.

GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES
NOTE: The group limits (GrpJobs, GrpTRES, etc.) are tested when a job
is being considered for being allocated resources. If starting a job
would cause any of its group limit to be exceeded, that job will not be
considered for scheduling even if that job might preempt other jobs
which would release sufficient group resources for the pending job to
be initiated.

NOTE: TRES limit modifications of any kind using "+=" or "-=" syntax
will always produce a value between 0 and 18446744073709551600. Trying
to set a negative value using "-=" will set that TRES to 0, even if
there was not a TRES limit present beforehand.

DefaultQOS=<default_qos>
The QOS this association and its children will use by default if
allowed in the QosLevel list mentioned below. This is overrid-
den if set directly on a user. To clear an existing value, set
a new value of -1.

Fairshare={<fairshare_number>|parent}
Share={<fairshare_number>|parent}
Allocated shares used for fairshare calculation. Can also be the
string parent, which is interpreted differently if set on a user
or on an account. If set on a user, the parent association is
used for fairshare. If set on an account, that account's chil-
dren will be effectively re-parented for fairshare calculations
to the first parent of their parent that is not Fairshare=par-
ent. Limits remain the same, only its fairshare value is af-
fected. To clear an existing value, set a new value of -1.

GrpJobs=<max_jobs>
Maximum number of running jobs in aggregate for this association
and its children. To clear an existing value, set a new value
of -1.

GrpJobsAccrue=<max_jobs>
Maximum number of pending jobs in aggregate able to accrue age
priority for this association and its children. To clear an ex-
isting value, set a new value of -1.

GrpSubmit=<max_jobs>
GrpSubmitJobs=<max_jobs>
Maximum number of jobs in a pending or running state at any time
in aggregate for this association and its children. To clear an
existing value, set a new value of -1.

GrpTRES=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...]
Maximum number of TRES able to be allocated by running jobs in
aggregate for this association and its children.

Refer to the TRES information section below for further details.

GrpTRESMins=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...]
Maximum number of TRES minutes that can possibly be used by
past, present, and future jobs in this association and its chil-
dren.

Refer to the TRES information section below for further details.

NOTE: This limit is not enforced if set on the root association
of a cluster. So even though it may appear in sacctmgr output,
it will not be enforced.

NOTE: This limit only applies when using the Priority Multifac-
tor plugin. The time is decayed using the value of PriorityDe-
cayHalfLife or PriorityUsageResetPeriod as set in the
slurm.conf. When this limit is reached all associated jobs run-
ning will be killed and all future jobs submitted with associa-
tions in the group will be delayed until they are able to run
inside the limit.

GrpTRESRunMins=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...]
Maximum number of TRES minutes able to be allocated by running
jobs in this association and its children. This takes into con-
sideration time limit of running jobs and consumes it. If the
limit is reached no new jobs are started until other jobs finish
to allow time to free up.

Refer to the TRES information section below for further details.

GrpWall=<max_wall>
Maximum wall clock time able to be allocated by running jobs in
aggregate in this association and its children. GrpWall format
is <min> or <min>:<sec> or <hr>:<min>:<sec> or
<days>-<hr>:<min>:<sec> or <days>-<hr>. The value is recorded
in minutes with rounding as needed. To clear an existing value,
set a new value of -1.

NOTE: Although it may appear in sacctmgr output, this limit will
not be enforced if set on the root association of a cluster.

MaxJobs=<max_jobs>
Maximum number of running jobs per user in this association.
This is overridden if set directly on a user. Default is the
cluster's limit. To clear an existing value, set a new value of
-1.

MaxJobsAccrue=<max_jobs>
Maximum number of pending jobs able to accrue age priority at
any given time in this association. This is overridden if set
directly on a user. Default is the cluster's limit. To clear
an existing value, set a new value of -1.

MaxSubmit=<max_jobs>
MaxSubmitJobs=<max_jobs>
Maximum number of jobs in a pending or running state at any time
in this association. Default is the cluster's limit. To clear
an existing value, set a new value of -1.

MaxTRES=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...]
MaxTRESPJ=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...]
MaxTRESPerJob=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...]
Maximum number of TRES each job can use in this association.
This is overridden if set directly on a user. Default is the
cluster's limit.

Refer to the TRES information section below for further details.

MaxTRESMins=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...]
MaxTRESMinsPJ=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...]
MaxTRESMinsPerJob=TRES[+|-]=<minutes>[,TRES[+|-]=<minutes>,...]
Maximum number of TRES minutes each job can use in this associa-
tion. This is overridden if set directly on a user. Default is
the cluster's limit.

Refer to the TRES information section below for further details.

MaxTRESPN=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...]
MaxTRESPerNode=TRES[+|-]=<max_TRES>[,TRES[+|-]=<max_TRES>,...]
Maximum number of TRES each node in a job allocation can use in
this association. This is overridden if set directly on a user.
Default is the cluster's limit.

Refer to the TRES information section below for further details.

MaxWall=<max_wall>
MaxWallDurationPerJob=<max_wall>
Maximum wall clock time each job can use in this association.
This is overridden if set directly on a user. Default is the
cluster's limit. MaxWall format is <min> or <min>:<sec> or
<hr>:<min>:<sec> or <days>-<hr>:<min>:<sec> or <days>-<hr>. The
value is recorded in minutes with rounding as needed. To clear
an existing value, set a new value of -1.

NOTE: Changing this value will have no effect on any running or
pending job.

Priority
Association priority factor to be used by the priority/multifac-
tor plugin. This is overridden if set directly on a user. Un-
set by default, indicating that no extra priority is granted.
To clear an existing value, set a new value of -1.

QosLevel<operator><comma_separated_list_of_qos_names>
List of QOS names available to jobs running in this association.
To get a list of valid QOSs use 'sacctmgr list qos'. This value
will override its parents value and push down to its children as
the new default. Setting a QosLevel to '' (two single quotes
with nothing between them) restores its default setting. You can
also use the operator += and -= to add or remove certain QOSs
from the QOS list.

Valid <operator> values include:

=
Set QosLevel to the specified value. NOTE: the QOS that can
be used at a given account in the hierarchy are inherited
by the children of that account. By assigning QOS with the
= sign only the assigned QOS can be used by the account and
its children.
+=
Add the specified <qos> value to the current QosLevel.
The account will have access to this QOS and any others
previously assigned to it.
-=
Remove the specified <qos> value from the current
QosLevel.

See the EXAMPLES section below.

SPECIFICATIONS FOR ACCOUNTS
Accounts can be created, modified, and deleted with sacctmgr. These op-
tions allow you to set the corresponding attributes or filter on them
when querying for Accounts.

Cluster=<cluster>
Specific cluster to add account to. Default is all in system.

Description=<description>
An arbitrary string describing an account.

Flags=<flag>[,<flag>,...]
Valid options are:

NoUsersAreCoords
Remove the privilege UsersAreCoords sets.

UsersAreCoords
If set, all users in this account will have the coordina-
tor status here and of any sub-account in it's hierarchy.

Name=<name>
The name of an account. Note the name must be unique and can
not be represent different accounts at different points in the
account hierarchy.

Organization=<org>
Organization to which the account belongs.

Parent=<parent>
Parent account of this account. Default is the root account, a
top level account.

RawUsage=<value>
This allows an administrator to reset the raw usage accrued to
an account. The only value currently supported is 0 (zero). This
is a settable specification only - it cannot be used as a filter
to list accounts.

WithAssoc
Display all associations for this account.

WithCoord
Display all coordinators for this account.

WithDeleted
Display information with previously deleted data. Accounts that
are deleted within 24 hours of being created and did not have a
job run in the account during that time will be removed from the
database. Otherwise, the account will be marked as deleted and
will be viewable with the WithDeleted flag.

NOTE: If using the WithAssoc option you can also query against associa-
tion specific information to view only certain associations this ac-
count may have. These extra options can be found in the SPECIFICATIONS
FOR ASSOCIATIONS section. You can also use the general specifications
list above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES
section.

LIST/SHOW ACCOUNT FORMAT OPTIONS
Fields you can display when viewing Account records by using the for-
mat= option. The default format is:
Account,Description,Organization

Account
The name of an account.

Description
An arbitrary string describing an account.

Flags Flags set on the account.

Organization
Organization to which the account belongs.

Coordinators
List of users that are a coordinator of the account. (Only
filled in when using the WithCoordinator option.)

NOTE: If using the WithAssoc option you can also view the information
about the various associations the account may have on all the clusters
in the system. The association information can be filtered. Note that
all the accounts in the database will always be shown as filter only
takes effect over the association data. The Association format fields
are described in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

SPECIFICATIONS FOR ASSOCIATIONS
Associations can be created, modified, and deleted with sacctmgr. These
options allow you to set the corresponding attributes or filter on them
when querying for Associations.

Clusters=<cluster_name>[,<cluster_name>,...]
List the associations of the cluster(s).

Accounts=<account_name>[,<account_name>,...]
List the associations of the account(s).

Users=<user_name>[,<user_name>,...]
List the associations of the user(s).

Partitions=<partition_name>[,<partition_name>,...]
List the associations of the partition(s).

NOTE: Use Partitions="" or Partitions='' with no other names listed
when specifying the case where there is no partition. This can be use-
ful when using a command with an entity that has associations with and
without partitions. If given in a shell where the quotes will be con-
sumed then they must be quoted themselves. For example: Parti-
tions=\"\".

NOTE: You can also use the general specifications list above in the
GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.

Other options unique for listing associations:

OnlyDefaults
Display only associations that are default associations

Tree Display account names in a hierarchical fashion.

WithDeleted
Display information with previously deleted data. Associations
that are deleted within 24 hours of being created and did not
have a job run in the association during that time will be re-
moved from the database. Otherwise, the association will be
marked as deleted and will be viewable with the WithDeleted
flag.

WithSubAccounts
Display information with subaccounts. Only really valuable when
used with the account= option. This will display all the subac-
count associations along with the accounts listed in the option.

WOLimits
Display information without limit information. This is for a
smaller default format of "Cluster,Account,User,Partition".

WOPInfo
Display information without parent information (i.e. parent id,
and parent account name). This option also implicitly sets the
WOPLimits option.

WOPLimits
Display information without hierarchical parent limits (i.e.
will only display limits where they are set instead of propagat-
ing them from the parent).

LIST/SHOW ASSOCIATION FORMAT OPTIONS
Fields you can display when viewing Association records by using the
format= option.

Account
The name of an account in the association.

Cluster
The name of a cluster in the association.

DefaultQOS
The QOS this association and its children will use by default if
allowed in the QosLevel list mentioned below.

Fairshare
Share Allocated shares used for fairshare calculation. Can also be the
string parent, which is interpreted differently if set on a user
or on an account. If set on a user, the parent association is
used for fairshare. If set on an account, that account's chil-
dren will be effectively re-parented for fairshare calculations
to the first parent of their parent that is not Fairshare=par-
ent. Limits remain the same, only its fairshare value is af-
fected.

Flags Flags set on the association.

GrpJobs
Maximum number of running jobs in aggregate for this association
and its children.

GrpJobsAccrue
Maximum number of pending jobs in aggregate able to accrue age
priority for this association and its children.

GrpSubmit
GrpSubmitJobs
Maximum number of jobs in a pending or running state at any time
in aggregate for this association and its children.

GrpTRES
Maximum number of TRES able to be allocated by running jobs in
aggregate for this association and its children.

GrpTRESMins
Maximum number of TRES minutes that can possibly be used by
past, present, and future jobs in this association and its chil-
dren.

GrpTRESRunMins
Maximum number of TRES minutes able to be allocated by running
jobs in this association and its children. This takes into con-
sideration time limit of running jobs and consumes it. If the
limit is reached no new jobs are started until other jobs finish
to allow time to free up.

GrpWall
Maximum wall clock time able to be allocated by running jobs in
aggregate in this association and its children.

ID The id of the association.

Lineage
Complete path up the hierarchy to the root association.

MaxJobs
Maximum number of running jobs per user.

MaxJobsAccrue
Maximum number of pending jobs able to accrue age priority at
any given time.

MaxSubmit
MaxSubmitJobs
Maximum number of jobs in a pending or running state at any
time.

MaxTRES
MaxTRESPJ
MaxTRESPerJob
Maximum number of TRES each job can use.

MaxTRESMins
MaxTRESMinsPJ
MaxTRESMinsPerJob
Maximum number of TRES minutes each job can use.

MaxTRESPN
MaxTRESPerNode
Maximum number of TRES each node in a job allocation can use.

MaxWall
MaxWallDurationPerJob
Maximum wall clock time each job can use.

ParentID
The association id of the parent of this association.

ParentName
The account name of the parent of this association.

Partition
The name of a partition in the association.

Priority
Association priority factor to be used by the priority/multifac-
tor plugin.

Qos Valid QOSs for this association.

QosRaw Numeric IDs of valid QOSs for this association.

User The name of a user in the association.

WithRawQOSLevel
Display QosLevel in an unevaluated raw format, consisting of a
comma-separated list of QOS names prepended with '' (nothing),
'+' or '-' for the association. QOS names without +/- prepended
were assigned (ie, sacctmgr modify ... set QosLevel=qos_name)
for the entity listed or on one of its parents in the hierarchy.
QOS names with +/- prepended indicate the QOS was added/filtered
(ie, sacctmgr modify ... set QosLevel=[+-]qos_name) for the en-
tity listed or on one of its parents in the hierarchy. Including
WOPLimits will show exactly where each QOS was assigned, added
or filtered in the hierarchy.

SPECIFICATIONS FOR CLUSTERS
Clusters can be created, modified, and deleted with sacctmgr. These op-
tions allow you to set the corresponding attributes or filter on them
when querying for Clusters.

Classification=<classification>
Type of machine, current classifications are capability, capac-
ity and capapacity.

Features[+|-]=<comma_separated_list_of_feature_names>
Features that are specific to the cluster. Federated jobs can be
directed to clusters that contain the job requested features.

To add or remove individual features, use the += or -= opera-
tors. To clear all existing features, set a new value of ''
(two single quotes with nothing between them).

Federation=<federation>
The federation that this cluster should be a member of. A clus-
ter can only be a member of one federation at a time.

FedState=<state>
The state of the cluster in the federation.
Valid states are:

ACTIVE Cluster will actively accept and schedule federated jobs.

INACTIVE
Cluster will not schedule or accept any jobs.

DRAIN Cluster will not accept any new jobs and will let exist-
ing federated jobs complete.

DRAIN+REMOVE
Cluster will not accept any new jobs and will remove it-
self from the federation once all federated jobs have
completed. When removed from the federation, the cluster
will accept jobs as a non-federated cluster.

Name=<name>
The name of a cluster. This should be equal to the ClusterName
parameter in the slurm.conf configuration file for some
Slurm-managed cluster.

RPC=<rpc_list>
Comma-separated list of numeric RPC values.

WithDeleted
Display information with previously deleted data. Clusters that
are deleted within 24 hours of being created and did not have a
job run in the cluster during that time will be removed from the
database. Otherwise, the cluster will be marked as deleted and
will be viewable with the WithDeleted flag.

WithFed
Appends federation related columns to default format options
(e.g. Federation,ID,Features,FedState).

WOLimits
Display information without limit information. This is for a
smaller default format of Cluster,ControlHost,ControlPort,RPC

NOTE: You can also use the general specifications list above in the
GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES section.

LIST/SHOW CLUSTER FORMAT OPTIONS
Fields you can display when viewing Cluster records by using the for-
mat= option.

Classification
Type of machine, i.e. capability, capacity or capapacity.

Cluster
The name of the cluster.

ControlHost
When a slurmctld registers with the database the ip address of
the controller is placed here.

ControlPort
When a slurmctld registers with the database the port the con-
troller is listening on is placed here.

Features
The list of features on the cluster (if any).

Federation
The name of the federation this cluster is a member of (if any).

FedState
The state of the cluster in the federation (if a member of one).

FedStateRaw
Numeric value of the name of the FedState.

Flags Attributes possessed by the cluster. Current flags include Cray,
External and MultipleSlurmd.

External clusters are registration only clusters. A slurmctld
can designate an external slurmdbd with the AccountingStorageEx-
ternalHost slurm.conf option. This allows a slurmctld to regis-
ter to an external slurmdbd so that clusters attached to the ex-
ternal slurmdbd can communicate with the external cluster with
Slurm commands.

ID The ID assigned to the cluster when a member of a federation.
This ID uniquely identifies the cluster and its jobs in the fed-
eration.

NodeCount
The current count of nodes associated with the cluster.

NodeNames
The current Nodes associated with the cluster.

RPC When a slurmctld registers with the database the rpc version the
controller is running is placed here.

TRES Trackable RESources (Billing, BB (Burst buffer), CPU, Energy,
GRES, License, Memory, and Node) this cluster is accounting for.

NOTE: You can also view the information about the root association for
the cluster. The Association format fields are described in the
LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

SPECIFICATIONS FOR COORDINATOR
Coordinators can be created, modified, and deleted with sacctmgr. These
options allow you to set the corresponding attributes or filter on them
when querying for Coordinators.

Account=<account_name>[,<account_name>,...]
Account name to add this user as a coordinator to.

Names=<user_name>[,<user_name>,...]
Names of coordinators.

NOTE: To list coordinators use the WithCoordinator options with list
account or list user.

SPECIFICATIONS FOR EVENTS
Events are automatically generated and sent to slurmdbd to be stored.
These are options you can specify to filter for specific types of
events.

All_Clusters
Shortcut to get information on all clusters.

All_Time
Shortcut to get time period for all time.

Clusters=<cluster_name>[,<cluster_name>,...]
List the events of the cluster(s). Default is the cluster where
the command was run.

CondFlags=<flag>[,<flag>,...]
Optional list of flags to filter events by.
Valid options are:

Open If set, only open node events (currently down) will be
returned.

End=<OPT>
Period ending of events. Default is now.
Valid time formats are:

HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

Event=<OPT>
Specific types of events to look for. Valid options are Cluster
or Node. The default is both.

MaxCPUs=<OPT>
Max number of CPUs affected by an event.

MinCPUs=<OPT>
Min number of CPUs affected by an event.

Nodes=<node_name>[,<node_name>,...]
Node names affected by an event.

Reason=<reason>[,<reason>,...]
Reason associated with a node going down. A reason that contains
a space should be surrounded by quotes.

Start=<OPT>
Period start of events. Default is 00:00:00 of previous day, un-
less states are given with the States=<spec> events. If this is
the case the default behavior is to return events currently in
the states specified.
Valid time formats are:
HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

States=<state>[,<state>,...]
State of a node in a node event. If this is set, the event type
is set automatically to Node.

User=<user_name>[,<user_name>,...]
Query against users who set the event. If this is set, the event
type is set automatically to Node since only the slurm user can
perform a cluster event.

LIST/SHOW EVENT FORMAT OPTIONS
Fields you can display when viewing Event records by using the format=
option. The default format is:
Cluster,NodeName,TimeStart,TimeEnd,State,Reason,User

Cluster
The name of the cluster event happened on.

ClusterNodes
The hostlist of nodes on a cluster in a cluster event.

Duration
Time period the event was around for.

End Period when event ended.

Event Name of the event.

EventRaw
Numeric value of the name of the event.

NodeName
The node affected by the event. In a cluster event, this is
blank.

Reason The reason an event happened.

Start Period when event started.

State On a node event this is the formatted state of the node during
the event.

StateRaw
On a node event this is the numeric value of the state of the
node during the event.

TRES Number of TRES involved with the event.

User On a node event this is the user who caused the event to happen.

SPECIFICATIONS FOR FEDERATION
Federations can be created, modified, and deleted with sacctmgr. These
options allow you to set the corresponding attributes or filter on them
when querying for Federations.

Clusters[+|-]=<cluster_name>[,<cluster_name>,...]
List of clusters to add/remove to a federation. A blank value
(e.g. clusters=) will remove all federations for the federation.
NOTE: A cluster can only be a member of one federation.

Name=<name>
The name of the federation.

Tree Display federations in a hierarchical fashion.

WithDeleted
Display information with previously deleted data. Federations
that are deleted within 24 hours of being created will be re-
moved from the database. Federations that were created more than
24 hours prior to the deletion request are just marked as
deleted and will be viewable with the WithDeleted flag.

LIST/SHOW FEDERATION FORMAT OPTIONS
Fields you can display when viewing Federation records by using the
format= option. The default format is:
Federation,Cluster,Features,FedState

Cluster
Name of the cluster that is a member of the federation.

Features
The list of features on the cluster.

Federation
The name of the federation.

FedState
The state of the cluster in the federation.

FedStateRaw
Numeric value of the name of the FedState.

Index The index of the cluster in the federation.

SPECIFICATIONS FOR INSTANCES
Information about cloud node instances is sent to slurmdbd to be
stored. These are options you can specify to filter for specific in-
stances.

Clusters=<cluster_name>[,<cluster_name>,...]
Name of the cluster that the instance ran on. Default is the
cluster where the command was run.

End=<OPT>
Period ending of instances. Default is now.

Valid time formats are:
HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

Extra=<OPT>
Arbitrary string associated with node during life of the in-
stance.

InstanceId=<OPT>
Cloud instance ID.

InstanceType=<OPT>
Cloud instance type.

Nodes=<node_name>[,<node_name>,...]
The node on which the instance ran.

Start=<OPT>
Period start of instances. Default is 00:00:00 of previous day.

Valid time formats are:
HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

LIST/SHOW INSTANCE FORMAT OPTIONS
Fields you can display when viewing Instance records by using the for-
mat= option. The default format is:
Cluster,NodeName,Start,End,InstanceID,InstanceType,Extra

Cluster
Name of the cluster that the instance ran on.

End Time when instance ended.

Extra Arbitrary string associated with node during life of the in-
stance.

InstanceId
Cloud instance ID.

InstanceType
Cloud instance type.

NodeName
The node on which the instance ran.

Start Time when instance started.

SPECIFICATIONS FOR JOB
Job information is automatically sent to slurmdbd to be stored. These
are options you can specify to filter for specific jobs. There are also
some attributes you can modify for a job record.

AdminComment=<admin_comment>
Arbitrary descriptive string. Can only be modified by a Slurm
administrator. To clear an existing value, set a new value of
'' (two single quotes with nothing between them).

Comment=<comment>
The job's comment string when the AccountingStoreFlags parameter
in the slurm.conf file contains 'job_comment'. The user can only
modify the comment string of their own job. To clear an exist-
ing value, set a new value of '' (two single quotes with nothing
between them).

Cluster=<cluster_list>
List of clusters to alter jobs on, defaults to local cluster.

DerivedExitCode=<derived_exit_code>
The derived exit code can be modified after a job completes
based on the user's judgment of whether the job succeeded or
failed. The user can only modify the derived exit code of their
own job.

EndTime
Jobs must end before this time to be modified. Format output is,
YYYY-MM-DDTHH:MM:SS, unless changed through the SLURM_TIME_FOR-
MAT environment variable.

Extra=<extra>
The job's extra string when the AccountingStoreFlags parameter
in the slurm.conf file contains 'job_extra'. The user can only
modify the extra string of their own job. To clear an existing
value, set a new value of '' (two single quotes with nothing be-
tween them).

JobID=<jobid_list>
The id of the job to change. Not needed if altering multiple
jobs using wckey specification.

NewWCKey=<new_wckey>
Use to rename a wckey on job(s) in the accounting database

StartTime
Jobs must start at or after this time to be modified in the same
format as EndTime.

SystemComment=<system_comment>
Arbitrary descriptive string, usually managed by the Burst-
BufferPlugin. Can only be modified by a Slurm administrator.
To clear an existing value, set a new value of '' (two single
quotes with nothing between them).

TRES=<tres_name=value>
Use to set or modify a TRES on job(s) in the accounting database
that have already completed.

WARNING: This is permanent, the original value will be lost af-
terwards.

User=<user_list>
Used to specify the jobs of users jobs to alter.

WCKey=<wckey_list>
Used to specify the wckeys to alter.

The AdminComment, Comment, DerivedExitCode, Extra, SystemComment, and
WCKey fields are the only fields of a job record in the database that
can be modified after job completion.

LIST/SHOW JOB FORMAT OPTIONS
The sacct command is the exclusive command to display job records from
the Slurm database.

SPECIFICATIONS FOR QOS
A QOS can be created, modified, and deleted with sacctmgr. These op-
tions allow you to set the corresponding attributes or filter on them
when querying for a QOS.

NOTE: The group limits (GrpJobs, GrpTRES, etc.) are tested when a job
is being considered for being allocated resources. If starting a job
would cause any of its group limit to be exceeded, that job will not be
considered for scheduling even if that job might preempt other jobs
which would release sufficient group resources for the pending job to
be initiated.

NOTE: All TRES options (GrpTRES, MaxTRES, etc.) can also use the same
"[+|-]=" syntax available for association-based entities.

Description
An arbitrary string describing a QOS. Can only be modified by a
Slurm administrator.

Flags Used by the slurmctld to override or enforce certain character-
istics. To add or remove individual flags, use the += or -= op-
erators. To clear all existing flags, set a new value of -1.
Valid options are

DenyOnLimit
If set, jobs using this QOS will be rejected at submis-
sion time if they do not conform to the QOS 'Max' or
'Min' limits as stand-alone jobs. Jobs that exceed these
limits when other jobs are considered, but conform to the
limits when considered individually will not be rejected.
Instead they will pend until resources are available.
Group limits (e.g. GrpTRES) will also be treated like
'Max' limits (e.g. MaxTRESPerNode) and jobs will be de-
nied if they would violate the limit as stand-alone jobs.
This currently only applies to QOS and Association lim-
its.

EnforceUsageThreshold
If set, and the QOS also has a UsageThreshold, any jobs
submitted with this QOS that fall below the UsageThresh-
old will be held until their Fairshare Usage goes above
the Threshold.

NoDecay
If set, this QOS will not have its GrpTRESMins, GrpWall
and UsageRaw decayed by the slurm.conf PriorityDecay-
HalfLife or PriorityUsageResetPeriod settings. This al-
lows a QOS to provide aggregate limits that, once con-
sumed, will not be replenished automatically. Such a QOS
will act as a time-limited quota of resources for an as-
sociation that has access to it. Account/user usage will
still be decayed for associations using the QOS. The QOS
GrpTRESMins and GrpWall limits can be increased or the
QOS RawUsage value reset to 0 (zero) to again allow jobs
submitted with this QOS to be queued (if DenyOnLimit is
set) or run (pending with QOSGrp{TRES}MinutesLimit or
QOSGrpWallLimit reasons, where {TRES} is some type of
trackable resource).

NoReserve
If set and backfill scheduling is used, jobs using this
QOS will not reserve resources in the backfill schedule's
map of resources allocated through time. This flag is in-
tended for use with a QOS that may be preempted by jobs
associated with all other QOS (e.g use with a "standby"
QOS). If this flag is used with a QOS which can not be
preempted by all other QOS, it could result in starvation
of larger jobs.

OverPartQOS
If set, jobs using this QOS will be able to override any
limits used by the requested partition's QOS limits.

PartitionMaxNodes
If set, jobs using this QOS will be able to override the
requested partition's MaxNodes limit.

PartitionMinNodes
If set, jobs using this QOS will be able to override the
requested partition's MinNodes limit.

PartitionTimeLimit
If set, jobs using this QOS will be able to override the
requested partition's TimeLimit.

Relative
If set, the QOS limits will be treated as percentages of
the cluster or partition instead of absolute limits (num-
bers should be less than 100). The controller should be
restarted or reconfigured after adding the Relative flag
to the QOS.
If this is used as a partition QOS:

1. Limits will be calculated relative to the partition's
resources.
2. Only one partition may have this QOS as its partition
QOS.
3. Jobs will not be allowed to use it as a normal QOS.

Additional details are in the QOS documentation at
<https://slurm.schedmd.com/qos.html>.

RequiresReservation
If set, jobs using this QOS must designate a reservation
when submitting a job. This option can be useful in re-
stricting usage of a QOS that may have greater preemptive
capability or additional resources to be allowed only
within a reservation.

UsageFactorSafe
If set and AccountingStorageEnforce includes Safe, jobs
will only be able to run if the job can run to completion
with the UsageFactor applied.

GraceTime
Preemption grace time in seconds to be extended to a job which
has been selected for preemption. The default value is zero,
meaning no preemption grace time is allowed on this QOS. This
value is only applicable for QOS PreemptMode=CANCEL and Preempt-
Mode=REQUEUE.

GrpJobs
Maximum number of running jobs in aggregate for this QOS. To
clear an existing value, set a new value of -1.

GrpJobsAccrue
Maximum number of pending jobs in aggregate able to accrue age
priority for this QOS. This limit only applies to the job's QOS
and not the partition's QOS. To clear an existing value, set a
new value of -1.

GrpSubmit
GrpSubmitJobs
Maximum number of jobs in a pending or running state at any time
in aggregate for this QOS. To clear an existing value, set a
new value of -1.

GrpTRES
Maximum number of TRES able to be allocated by running jobs in
aggregate for this QOS.

Refer to the TRES information section below for further details.

GrpTRESMins
Maximum number of TRES minutes that can possibly be used by
past, present, and future jobs with this QOS.

Refer to the TRES information section below for further details.

NOTE: This limit only applies when using the Priority Multifac-
tor plugin. The time is decayed using the value of PriorityDe-
cayHalfLife or PriorityUsageResetPeriod as set in the
slurm.conf. When this limit is reached all associated jobs run-
ning will be killed and all future jobs submitted with this QOS
will be delayed until they are able to run inside the limit.

GrpTRESRunMins
Maximum number of TRES minutes able to be allocated by running
jobs with this QOS. This takes into consideration time limit of
running jobs and consumes it. If the limit is reached no new
jobs are started until other jobs finish to allow time to free
up.

Refer to the TRES information section below for further details.

GrpWall
Maximum wall clock time able to be allocated by running jobs in
aggregate for this QOS. If this limit is reached, job submis-
sions will be denied and the running jobs will be killed. Grp-
Wall format is <min> or <min>:<sec> or <hr>:<min>:<sec> or
<days>-<hr>:<min>:<sec> or <days>-<hr>. The value is recorded
in minutes with rounding as needed. To clear an existing value,
set a new value of -1.

NOTE: This limit only applies when using the Priority Multifac-
tor plugin. The time is decayed using the value of PriorityDe-
cayHalfLife or PriorityUsageResetPeriod as set in the
slurm.conf. When this limit is reached all associated jobs run-
ning will be killed and all future jobs submitted with this QOS
will be delayed until they are able to run inside the limit.

LimitFactor
A float that is factored into an association's [Grp|Max]TRES
limits. For example, if the LimitFactor is 2, then an associa-
tion with a GrpTRES of 30 CPUs, would be allowed to allocate 60
CPUs when running under this QOS. To clear an existing value,
set a new value of -1. NOTE: This factor is only applied to as-
sociations running in this QOS and is not applied to any limits
in the QOS itself.

MaxJobsAccruePA
MaxJobsAccruePerAccount
Maximum number of pending jobs an account (or subacct) can have
accruing age priority at any given time. This limit only ap-
plies to the job's QOS and not the partition's QOS. To clear an
existing value, set a new value of -1.

MaxJobsAccruePU
MaxJobsAccruePerUser
Maximum number of pending jobs a user can have accruing age pri-
ority at any given time. This limit only applies to the job's
QOS and not the partition's QOS. To clear an existing value,
set a new value of -1.

MaxJobsPA
MaxJobsPerAccount
Maximum number of running jobs per account. To clear an exist-
ing value, set a new value of -1.

MaxJobsPU
MaxJobsPerUser
Maximum number of running jobs per user. To clear an existing
value, set a new value of -1.

MaxSubmitJobsPA
MaxSubmitJobsPerAccount
Maximum number of jobs in a pending or running state per ac-
count. To clear an existing value, set a new value of -1.

MaxSubmitJobsPU
MaxSubmitJobsPerUser
Maximum number of jobs in a pending or running state per user.
To clear an existing value, set a new value of -1.

MaxTRES
MaxTRESPJ
MaxTRESPerJob
Maximum number of TRES each job can use.

Refer to the TRES information section below for further details.

MaxTRESMins
MaxTRESMinsPJ
MaxTRESMinsPerJob
Maximum number of TRES minutes each job can use.

Refer to the TRES information section below for further details.

MaxTRESPA
MaxTRESPerAccount
Maximum number of TRES each account can use.

Refer to the TRES information section below for further details.

MaxTRESPN
MaxTRESPerNode
Maximum number of TRES each node in a job allocation can use.

Refer to the TRES information section below for further details.

MaxTRESPU
MaxTRESPerUser
Maximum number of TRES each user can use.

Refer to the TRES information section below for further details.

MaxTRESRunMinsPA
MaxTRESRunMinsPerAccount
Maximum number of TRES minutes each account can use. This takes
into consideration the time limit of running jobs. If the limit
is reached, no new jobs are started until other jobs finish to
allow time to free up.

Refer to the TRES information section below for further details.

MaxTRESRunMinsPU
MaxTRESRunMinsPerUser
Maximum number of TRES minutes each user can use. This takes
into consideration the time limit of running jobs. If the limit
is reached, no new jobs are started until other jobs finish to
allow time to free up.

Refer to the TRES information section below for further details.

MaxWall
MaxWallDurationPerJob
Maximum wall clock time each job can use. MaxWall format is
<min> or <min>:<sec> or <hr>:<min>:<sec> or
<days>-<hr>:<min>:<sec> or <days>-<hr>. The value is recorded
in minutes with rounding as needed. To clear an existing value,
set a new value of -1.

MinPrioThreshold
Minimum priority required to reserve resources when scheduling.
To clear an existing value, set a new value of -1.

MinTRES
MinTRESPerJob
Minimum number of TRES each job running under this QOS must re-
quest. Otherwise the job will pend until modified.

Refer to the TRES information section below for further details.

Name Name of the QOS. Needed for creation.

Preempt
Other QOSs this QOS can preempt. To clear an existing value,
set a new value of '' (two single quotes with nothing between
them).

NOTE: The Priority of a QOS is NOT related to QOS preemption,
only Preempt is used to define which QOS can preempt others.

PreemptExemptTime
Specifies a minimum run time for jobs in this QOS before they
are considered for preemption. This QOS option takes precedence
over the global PreemptExemptTime. This is only honored for Pre-
emptMode=REQUEUE and PreemptMode=CANCEL.
Setting to -1 disables the option, allowing another QOS or the
global option to take effect. Setting to 0 indicates no minimum
run time and supersedes the lower priority QOS (see OverPartQOS)
and/or the global option in slurm.conf.

PreemptMode
Mechanism used to preempt jobs or enable gang scheduling for
this QOS when the cluster's PreemptType is set to preempt/qos.
This QOS-specific PreemptMode will override the cluster-wide
PreemptMode for this QOS. Unsetting the QOS specific Preempt-
Mode, by specifying "OFF", "" or "Cluster", makes it use the de-
fault cluster-wide PreemptMode.
The GANG option is used to enable gang scheduling independent of
whether preemption is enabled (i.e. independent of the Preempt-
Type setting). It can be specified in addition to a PreemptMode
setting with the two options comma-separated (e.g. Preempt-
Mode=SUSPEND,GANG).
See <https://slurm.schedmd.com/preempt.html> and
<https://slurm.schedmd.com/gang_scheduling.html> for more de-
tails.

NOTE: For performance reasons, the backfill scheduler reserves
whole nodes for jobs, not partial nodes. If during backfill
scheduling a job preempts one or more other jobs, the whole
nodes for those preempted jobs are reserved for the preemptor
job, even if the preemptor job requested fewer resources than
that. These reserved nodes aren't available to other jobs dur-
ing that backfill cycle, even if the other jobs could fit on the
nodes. Therefore, jobs may preempt more resources during a sin-
gle backfill iteration than they requested.
NOTE: For heterogeneous job to be considered for preemption all
components must be eligible for preemption. When a heterogeneous
job is to be preempted the first identified component of the job
with the highest order PreemptMode (SUSPEND (highest), REQUEUE,
CANCEL (lowest)) will be used to set the PreemptMode for all
components. The GraceTime and user warning signal for each com-
ponent of the heterogeneous job remain unique. Heterogeneous
jobs are excluded from GANG scheduling operations.

OFF Is the default value and disables job preemption and
gang scheduling. It is only compatible with Pre-
emptType=preempt/none at a global level.

CANCEL The preempted job will be cancelled.

GANG Enables gang scheduling (time slicing) of jobs in
the same partition, and allows the resuming of sus-
pended jobs. Configure the OverSubscribe setting to
FORCE for all partitions in which time-slicing is to
take place. Gang scheduling is performed indepen-
dently for each partition, so if you only want
time-slicing by OverSubscribe, without any preemp-
tion, then configuring partitions with overlapping
nodes is not recommended. Time-slicing won't happen
between jobs on different partitions.

NOTE: Heterogeneous jobs are excluded from GANG
scheduling operations.

REQUEUE Preempts jobs by requeuing them (if possible) or
canceling them. For jobs to be requeued they must
have the --requeue sbatch option set or the cluster
wide JobRequeue parameter in slurm.conf must be set
to 1.

SUSPEND The preempted jobs will be suspended, and later the
Gang scheduler will resume them. Therefore the SUS-
PEND preemption mode always needs the GANG option to
be specified at the cluster level. Also, because the
suspended jobs will still use memory on the allo-
cated nodes, Slurm needs to be able to track memory
resources to be able to suspend jobs.
If PreemptType=preempt/qos is configured and if the
preempted job(s) and the preemptor job are on the
same partition, then they will share resources with
the Gang scheduler (time-slicing). If not (i.e. if
the preemptees and preemptor are on different parti-
tions) then the preempted jobs will remain suspended
until the preemptor ends.

NOTE: Suspended jobs will not release GRES. Higher
priority jobs will not be able to preempt to gain
access to GRES.

WITHIN Allows for preemption between jobs sharing the same
qos. By default, PreemptType=preempt/qos will only
consider jobs to be eligible for preemption if they
do not share the same qos value.

Priority
QOS priority factor to be used by the priority/multifactor plu-
gin. Unset by default, indicating that no extra priority is
granted.

NOTE: The Priority of a QOS is NOT related to QOS preemption,
see Preempt instead.

RawUsage=<value>
This allows an administrator to set the raw usage accrued to a
QOS. Specifying a value of 0 (zero) will reset the raw usage.
This is a settable specification only - it cannot be used as a
filter to list accounts.

UsageFactor
A float that is factored into a job's TRES usage (e.g. RawUsage,
TRESMins, TRESRunMins). For example, if the usagefactor was 2,
for every TRESBillingUnit second a job ran it would count for 2.
If the usagefactor was .5, every second would only count for
half of the time. A setting of 0 would add no timed usage from
the job.

The usage factor only applies to the job's QOS and not the par-
tition QOS.

If the UsageFactorSafe flag is set and AccountingStorageEnforce
includes Safe, jobs will only be started if they can run to com-
pletion with the UsageFactor applied, and won't be killed due to
limits.

If the UsageFactorSafe flag is not set and AccountingStorageEn-
force includes Safe, jobs will be started if they can run to
completion without the UsageFactor applied, and won't be killed
due to limits.

If the UsageFactorSafe flag is not set and AccountingStorageEn-
force does not include Safe, jobs will be scheduled as long as
the limits are not reached, but could be killed due to limits.

See AccountingStorageEnforce in slurm.conf man page.

Default is 1. To clear an existing value, set a new value of -1.

UsageThreshold
A float representing the lowest fairshare of an association al-
lowed to run a job. If an association falls below this threshold
and has pending jobs or submits new jobs those jobs will be held
until the usage goes back above the threshold. Use sshare to see
current shares on the system. To clear an existing value, set a
new value of -1.

LIST/SHOW QOS FORMAT OPTIONS
Fields you can display when viewing QOS records by using the format=
option.

Description
An arbitrary string describing a QOS.

Flags Used by the slurmctld to override or enforce certain character-
istics.

GraceTime
Preemption grace time to be extended to a job which has been se-
lected for preemption in the format of hh:mm:ss.

GrpJobs
Maximum number of running jobs in aggregate for this QOS.

GrpJobsAccrue
Maximum number of pending jobs in aggregate able to accrue age
priority for this QOS. This limit only applies to the job's QOS
and not the partition's QOS.

GrpSubmit
GrpSubmitJobs
Maximum number of jobs in a pending or running state at any time
in aggregate for this QOS.

GrpTRES
Maximum number of TRES able to be allocated by running jobs in
aggregate for this QOS.

GrpTRESMins
Maximum number of TRES minutes that can possibly be used by
past, present, and future jobs with this QOS.

GrpTRESRunMins
Maximum number of TRES minutes able to be allocated by running
jobs with this QOS.

GrpWall
Maximum wall clock time able to be allocated by running jobs in
aggregate for this QOS.

ID The id of the QOS.

LimitFactor
A float that is factored into an association's [Grp|Max]TRES
limits.

MaxJobsAccruePA
MaxJobsAccruePerAccount
Maximum number of pending jobs an account (or subacct) can have
accruing age priority at any given time. This limit only applies
to the job's QOS and not the partition's QOS.

MaxJobsAccruePU
MaxJobsAccruePerUser
Maximum number of pending jobs a user can have accruing age pri-
ority at any given time. This limit only applies to the job's
QOS and not the partition's QOS.

MaxJobsPA
MaxJobsPerAccount
Maximum number of running jobs per account.

MaxJobsPU
MaxJobsPerUser
Maximum number of running jobs per user.

MaxSubmitJobsPA
MaxSubmitJobsPerAccount
Maximum number of jobs in a pending or running state per ac-
count.

MaxSubmitJobsPU
MaxSubmitJobsPerUser
Maximum number of jobs in a pending or running state per user.

MaxTRES
MaxTRESPJ
MaxTRESPerJob
Maximum number of TRES each job can use.

MaxTRESMins
MaxTRESMinsPJ
MaxTRESMinsPerJob
Maximum number of TRES minutes each job can use.

MaxTRESPA
MaxTRESPerAccount
Maximum number of TRES each account can use.

MaxTRESPN
MaxTRESPerNode
Maximum number of TRES each node in a job allocation can use.

MaxTRESPU
MaxTRESPerUser
Maximum number of TRES each user can use.

MaxTRESRunMinsPA
MaxTRESRunMinsPerAccount
Maximum number of TRES minutes each account can use.

MaxTRESRunMinsPU
MaxTRESRunMinsPerUser
Maximum number of TRES minutes each user can use.

MaxWall
MaxWallDurationPerJob
Maximum wall clock time each job can use.

MinPrioThreshold
Minimum priority required to reserve resources when scheduling.

MinTRES
Minimum number of TRES each job running under this QOS must re-
quest. Otherwise the job will pend until modified.

Name Name of the QOS.

Preempt
Other QOSs this QOS can preempt.

PreemptExemptTime
Specifies a minimum run time for jobs in this QOS before they
are considered for preemption.

PreemptMode
Mechanism used to preempt jobs or enable gang scheduling for
this QOS when the cluster's PreemptType is set to preempt/qos.
The default preemption mechanism is specified by the clus-
ter-wide PreemptMode configuration parameter.

Priority
QOS priority factor to be used by the priority/multifactor plu-
gin.

UsageFactor
A float that is factored into a job's TRES usage (e.g. RawUsage,
TRESMins, TRESRunMins).

UsageThreshold
A float representing the lowest fairshare of an association al-
lowed to run a job.

WithDeleted
Display information with previously deleted data. A QOS that is
deleted within 24 hours of being created and did not have a job
run in the QOS during that time will be removed from the data-
base. Otherwise, the QOS will be marked as deleted and will be
viewable with the WithDeleted flag.

SPECIFICATIONS FOR RESERVATIONS
Reservations are created with the scontrol command and information
about the reservations is sent to slurmdbd to be stored. These are op-
tions you can specify to filter for specific reservations.

Clusters=<cluster_name>[,<cluster_name>,...]
List the reservations of the cluster(s). Default is the cluster
where the command was run.

End=<OPT>
Period ending of reservations. Default is now.

Valid time formats are:
HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

ID=<OPT>
Comma-separated list of reservation ids.

Names=<OPT>
Comma-separated list of reservation names.

Nodes=<node_name>[,<node_name>,...]
Node names where reservation ran.

Start=<OPT>
Period start of reservations. Default is 00:00:00 of previous
day.

Valid time formats are:
HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

LIST/SHOW RESERVATION FORMAT OPTIONS
Fields you can display when viewing Reservation records by using the
format= option. The default format is:
Cluster,Name,TRES,Start,End,UnusedWall

Associations
The id's of the associations able to run in the reservation.

Cluster
Name of cluster reservation was on.

End End time of reservation.

Flags Flags set on the reservation.

ID Reservation ID.

Name Name of this reservation.

NodeNames
List of nodes in the reservation.

Start Start time of reservation.

TRES List of TRES in the reservation.

UnusedWall
Wall clock time in seconds unused by any job. A job's allocated
usage is its run time multiplied by the ratio of its CPUs to the
total number of CPUs in the reservation. For example, a job us-
ing all the CPUs in the reservation running for 1 minute would
reduce unused_wall by 1 minute.

SPECIFICATIONS FOR RESOURCE
Resources can be created, modified, and deleted with sacctmgr. These
options allow you to set the corresponding attributes or filter on them
when querying for Resources.

LastConsumed=<OPT>
Number of software resources of a specific name consumed out of
Count on the system being controlled by a resource manager.

Clusters=<name_list>
Comma-separated list of cluster names on which specified re-
sources are to be available. If no names are designated then the
clusters already allowed to use this resource will be altered.

Count=<OPT>
Number of software resources of a specific name configured on
the system being controlled by a resource manager.

Descriptions=
A brief description of the resource.

Flags[-|+]=<OPT>
Flags that identify specific attributes of the system resource.
Valid options are

Absolute
If set the resource will treat the counts for Allowed and
Allocated as absolute counts instead of percentages.

NOTE: If removing this with flags-=absolute there is no
effort to convert the numbers in the database back to
percentages. This is required by the user.

Names=<OPT>
Comma-separated list of the name of a resource configured on the
system being controlled by a resource manager. If this resource
is seen on the slurmctld its name will be name@server to distin-
guish it from local resources defined in a slurm.conf.

Allowed=<allowed>
Percentage/Count of a specific resource that can be used on
specified cluster.

Server=<OPT>
Arbitrary string indicating the name of the server serving up
the resource. Default is 'slurmdb' indicating the licenses are
being served by the database. This parameter is only for tag-
ging purposes.

ServerType=<OPT>
Arbitrary string used to tag the type of the software resource
manager providing the licenses. For example FlexNext Publisher
Flexlm license server or Reprise License Manager RLM. This does
not imply any kind of integration with license managers.

Type=<OPT>
The type of the resource represented by this record. Currently
the only valid type is License.

WithClusters
Display the clusters percentage/count of resources. If a re-
source hasn't been given to a cluster the resource will not be
displayed with this flag.

WithDeleted
Display information with previously deleted data. Resources
that are deleted within 24 hours of being created will be re-
moved from the database. Resources that were created more than
24 hours prior to the deletion request are just marked as
deleted and will be viewable with the WithDeleted flag.

NOTE: Resource is used to define each resource configured on a system
available for usage by Slurm clusters.

LIST/SHOW RESOURCE FORMAT OPTIONS
Fields you can display when viewing Resource records by using the for-
mat= option. The default format is:
Name,Server,Type,Count,LastConsumed,Allocated,ServerType,Flags

Allocated
The percent/count of licenses allocated to a cluster.

LastConsumed
The count of a specific resource consumed out of Count on the
system globally.

Cluster
Name of cluster resource is given to.

Count The count of a specific resource configured on the system glob-
ally.

Description
Description of the resource.

Name Name of this resource.

Server Server serving up the resource.

ServerType
The type of the server controlling the licenses.

Type Type of resource this record represents.

SPECIFICATIONS FOR RUNAWAYJOB
Under certain circumstances, jobs can complete without having that com-
pletion recorded by slurmdbd. This results in a "runaway job", where
slurmdbd is not going to record a completion time for that job without
intervention. This command will identify jobs that are in this state
and offer to have slurmdbd clean up the job record(s).

This particular variant of the "show" command also permits the use of
the "set" keyword to define the following specifications:

EndState=<state>
Desired state to use as the end state for fixed jobs. Supported
states are: Completed, Failed.

LIST/SHOW RUNAWAYJOB FORMAT OPTIONS
Fields you can display when viewing runaway job records by using the
format= option. The default format is:
ID,Name,Partition,Cluster,State,TimeSubmit,TimeStart,TimeEnd

Cluster
Name of cluster job ran on.

ID Id of the job.

Name Name of the job.

Partition
Partition job ran on.

State Current State of the job in the database.

TimeEnd
Current recorded time of the end of the job.

TimeStart
Time job started running.

TimeSubmit
Time job was submitted.

SPECIFICATIONS FOR TRANSACTIONS
Information about changes to clusters, resources, accounts, associa-
tions, etc., are recorded as transactions by slurmdbd. These are op-
tions you can specify to filter for specific transactions.

Accounts=<account_name>[,<account_name>,...]
Only print out the transactions affecting specified accounts.

Action=<Specific_action_the_list_will_display>
Only display transactions of the specified action type.

Actor=<Specific_name_the_list_will_display>
Only display transactions done by a certain person.

Clusters=<cluster_name>[,<cluster_name>,...]
Only print out the transactions affecting specified clusters.

End=<Date_and_time_of_last_transaction_to_return>
Return all transactions before this Date and time. Default is
now.

Start=<Date_and_time_of_first_transaction_to_return>
Return all transactions after this Date and time. Default is
epoch.

Valid time formats for End and Start are:
HH:MM[:SS] [AM|PM]
MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
MM/DD[/YY]-HH:MM[:SS]
YYYY-MM-DD[THH:MM[:SS]]
now[{+|-}count[seconds(default)|minutes|hours|days|weeks]]

Users=<user_name>[,<user_name>,...]
Only print out the transactions affecting specified users.

WithAssoc
Get information about which associations were affected by the
transactions.

LIST/SHOW TRANSACTIONS FORMAT OPTIONS
Fields you can display when viewing Transaction records by using the
format= option. The default format is:
Time,Action,Actor,Where,Info

Action Displays the type of Action that took place.

Actor Displays the Actor to generate a transaction.

Info Displays details of the transaction.

TimeStamp
Displays when the transaction occurred.

Where Displays details of the constraints for the transaction.

NOTE: If using the WithAssoc option you can also view the information
about the various associations the transaction affected. The Associa-
tion format fields are described in the LIST/SHOW ASSOCIATION FORMAT
OPTIONS section.

SPECIFICATIONS FOR USERS
Users can be created, modified, and deleted with sacctmgr. These op-
tions allow you to set the corresponding attributes or filter on them
when querying for Users.

It is important to recognize the difference between a User and an Asso-
ciation. There is a User entity that exists for each unique username.
However, there can be multiple User Associations for the same User. The
combination of a Cluster, Account, User, and optionally a Partition
constitute a User Association. When adding an existing User to another
Account, you are creating an additional User Association rather than
modifying an existing User.

Account=<account>
Account name to add this user to.

AdminLevel=<level>
Admin level of user. Valid levels are None, Operator, and Admin.

Cluster=<cluster>
Specific cluster to add user to the account on. Default is all
in system.

DefaultAccount=<account>
Identify the default account name to be used for a job if none
is specified at submission time.

DefaultWCKey=<defaultwckey>
Identify the default Workload Characterization Key.

Name=<name>
Name of user.

NewName=<newname>
Use to rename a user in the accounting database

Partition=<name>
Partition name.

NOTE: See also Partitions listed in the SPECIFICATIONS FOR ASSOCIATIONS
section.

RawUsage=<value>
This allows an administrator to reset the raw usage accrued to a
user. The only value currently supported is 0 (zero). This is a
settable specification only - it cannot be used as a filter to
list users.

WCKeys=<wckeys>
Workload Characterization Key values.

WithAssoc
Display all associations for this user.

WithCoord
Display all accounts a user is coordinator for.

WithDeleted
Display information with previously deleted data. Users that
are deleted within 24 hours of being created and did not have a
job run by the user during that time will be removed from the
database. Otherwise, the user will be marked as deleted and
will be viewable with the WithDeleted flag.

NOTE: If using the WithAssoc option you can also query against associa-
tion specific information to view only certain associations this user
may have. These extra options can be found in the SPECIFICATIONS FOR
ASSOCIATIONS section. You can also use the general specifications list
above in the GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES sec-
tion.

LIST/SHOW USER FORMAT OPTIONS
Fields you can display when viewing User records by using the format=
option. The default format is:
User,DefaultAccount,DefaultWCKey,AdminLevel

AdminLevel
Admin level of user.

Coordinators
List of users that are a coordinator of the account. (Only
filled in when using the WithCoordinator option.)

DefaultAccount
The user's default account.

DefaultWCKey
The user's default wckey.

User The name of a user.

NOTE: If using the WithAssoc option you can also view the information
about the various associations the user may have on all the clusters in
the system. The association information can be filtered. Note that all
the users in the database will always be shown as filter only takes ef-
fect over the association data. The Association format fields are de-
scribed in the LIST/SHOW ASSOCIATION FORMAT OPTIONS section.

LIST/SHOW WCKey
Fields you can display when viewing WCKey records by using the format=
option. The default format is:
WCKey,Cluster,User

Cluster
Specific cluster for the WCKey.

ID The ID of the WCKey.

User The name of a user for the WCKey.

WCKey Workload Characterization Key.

WithDeleted
Display information with previously deleted data. WCKeys that
are deleted within 24 hours of being created and did not have a
job run with the WCKey during that time will be removed from the
database. Otherwise, the WCKey will be marked as deleted and
will be viewable with the WithDeleted flag.

LIST/SHOW TRES
Fields you can display when viewing TRES records by using the format=
option. The default format is:
Type,Name,ID

ID The identification number of the trackable resource as it ap-
pears in the database.

Name The name of the trackable resource. This option is required for
TRES types BB (Burst buffer), GRES, and License. Types CPU, En-
ergy, Memory, and Node do not have Names. For example if GRES is
the type then name is the denomination of the GRES itself e.g.
GPU.

Type The type of the trackable resource. Current types are BB (Burst
buffer), CPU, Energy, GRES, License, Memory, and Node.

TRES information
Trackable RESources (TRES) are used in many QOS or Association limits
(GrpTRES* / MaxTRES*). When setting limits, one or more comma-separated
"TRES=value" assignments can be specified.

Each TRES can be one of the Slurm defaults (e.g., cpu, mem, node), or
any defined generic resource. You can see the list of available re-
sources by running sacctmgr show tres.

Each TRES has a different limit, e.g., "GrpTRES=cpu=16,mem=32G" would
establish 2 different limits: one for 16 CPUs and another for 32 GB of
memory.

Setting a new value for a TRES type does not affect any other TRES
types that were previously specified. For example, on the same entity
as above, a subsequent assignment of "GrpTRES=gres/gpu=2" will result
in "GrpTRES=cpu=16,mem=32G,gres/gpu=2" applying to that entity.

To remove a limit, assign -1 to the TRES type whose limit should be re-
moved, e.g., "GrpTRES=cpu=-1" would remove only the cpu TRES limit, re-
sulting in "GrpTRES=mem=32G,gres/gpu=2" on that entity.

Starting in Slurm 25.11, a TRES assignment can use a += or -= operator
to add to or remove from the previously set value for that TRES, e.g.,
"GrpTRES=mem-=8G,gres/gpu+=2" would result in "Grp-
TRES=mem=24G,gres/gpu=4" on that entity.

NOTE: When dealing with Memory as a TRES the default units are MB. A
suffix of G, T, or P can be added to specify a limit in larger units.

NOTE: The Billing TRES is calculated from a partition's TRESBilling-
Weights. It is temporarily calculated during scheduling for each parti-
tion to enforce billing TRES limits. The final Billing TRES value is
calculated after the job has been allocated resources and can be seen
in scontrol show jobs and sacct output.

GLOBAL FORMAT OPTION
When using the format option for listing various fields you can put a
%NUMBER afterwards to specify how many characters should be printed.

e.g. format=name%30 will print 30 characters of field name right justi-
fied. A -30 will print 30 characters left justified.

FLAT FILE DUMP AND LOAD
sacctmgr has the capability to load and dump Slurm association data to
and from a file. This method can easily add a new cluster or copy an
existing cluster's associations into a new cluster with similar ac-
counts. Each file contains Slurm association data for a single cluster.
Beginning with version 25.05, QOS information is included in the dump
file. Comments can be put into the file with the # character. Each
line of information must begin with one of the five titles; QOS, Clus-
ter, Parent, Account or User. Following the title is a space, dash,
space, entity value, then specifications. Specifications are colon-sep-
arated. If any variable, such as an Organization name, has a space in
it, surround the name with single or double quotes.

sacctmgr dump/load must be run as a Slurm administrator or root. If us-
ing sacctmgr load on a database without any associations, it must be
run as root (because there aren't any users in the database yet).

dump
Dump cluster associations from the database into a file. If no file is
given then one will be generated, using the cluster name for the file
name. That file will be created in the current working directory.

To create a file with the association information you can run:

sacctmgr dump tux file=tux.cfg

Cluster=
Specify the cluster to dump the information for.

File= Specify a file to save flat file data to. If the file-
name is not specified it uses clustername.cfg filename by
default.

load
Load cluster associations into the database. The imported associations
will be reconciled with existing ones.

To load a previously created file you can run:

sacctmgr load file=tux.cfg

clean Delete what was already there and start from scratch with
this information. With no options this will only remove
the cluster along with it's associations. No accounts,
users, or QOS will be removed. This also accepts a
comma-separated list of other options to remove. Those
include 'account', 'qos' and 'user'. If you would like to
remove accounts, qos, and users along with the cluster
and associations give the input of clean=ac-
count,qos,user.

Cluster=
Specify a different name for the cluster than that which
is in the file.

File= Specify a flat file to load from.

SPECIFICATIONS FOR FLAT FILE
Since the associations in the system follow a hierarchy, so does the
file. Anything that is a parent needs to be defined before any chil-
dren. The only exception is the understood 'root' account. This is al-
ways a default for any cluster and does not need to be defined.

To edit/create a file start with a cluster line for the new cluster:

Cluster - cluster_name:MaxTRESPerJob=node=15

Anything included on this line will be the default for all associations
on this cluster. The options for the cluster are:

FairShare=
Allocated shares used for fairshare calculation.

GrpJobs=
Maximum number of running jobs in aggregate for this as-
sociation and its children.

GrpJobsAccrue=
Maximum number of pending jobs in aggregate able to ac-
crue age priority for this association and its children.

GrpNodes=
This option has been deprecated in favor of the more ver-
satile TRES. Equivalent limit definition is now Grp-
TRES=node=#.

GrpSubmitJobs=
Maximum number of jobs in a pending or running state at
any time in aggregate for this association and its chil-
dren.

GrpTRES=
Maximum number of TRES able to be allocated by running
jobs in aggregate for this association and its children.

GrpTRESMins=
Maximum number of TRES minutes that can possibly be used
by past, present and future jobs in this association and
its children.

GrpTRESRunMins=
Maximum number of TRES minutes able to be allocated by
running jobs in this association and its children. This
takes into consideration time limit of running jobs and
consumes it. If the limit is reached no new jobs are
started until other jobs finish to allow time to free up.

GrpWall=
Maximum wall clock time able to be allocated by running
jobs in aggregate in this association and its children.

MaxJobs=
Maximum number of running jobs per user in this associa-
tion.

MaxTRESPerJob=
Maximum number of TRES each job can use in this associa-
tion.

MaxWallDurationPerJob=
Maximum wall clock time each job can use in this associa-
tion.

QOS= Comma-separated list of Quality of Service names (Defined
in sacctmgr).

After the entry for the root account you will have entries for the
other accounts on the system. The entries will look similar to this ex-
ample:

Parent - root
Account - cs:MaxTRESPerJob=node=5:MaxJobs=4:FairShare=399:MaxWallDurationPerJob=40:Description='Computer Science':Organization='LC'
Parent - cs
Account - test:MaxTRESPerJob=node=1:MaxJobs=1:FairShare=1:MaxWallDurationPerJob=1:Description='Test Account':Organization='Test'

Any of the options after a ':' can be left out and they can be in any
order. If you want to add any sub accounts just list the Parent THAT
HAS ALREADY BEEN CREATED before the account you are adding.

Account options are:

Description=
A brief description of the account.

FairShare=
Number used in conjunction with other associations to de-
termine job priority.

GrpTRES=
Maximum number of TRES able to be allocated by running
jobs in aggregate for this association and its children.

GrpTRESMins=
Maximum number of TRES minutes that can possibly be used
by past, present, and future jobs in this association and
its children.

GrpJobs=
Maximum number of running jobs in aggregate for this as-
sociation and its children.

GrpJobsAccrue=
Maximum number of pending jobs in aggregate able to ac-
crue age priority for this association and its children.

GrpNodes=
This option has been deprecated in favor of the more ver-
satile TRES. Equivalent limit definition is now Grp-
TRES=node=#.

GrpSubmitJobs=
Maximum number of jobs in a pending or running state at
any time in aggregate for this association and its chil-
dren.

GrpWall=
Maximum wall clock time able to be allocated by running
jobs in aggregate in this association and its children.

MaxJobs=
Maximum number of running jobs per user in this associa-
tion.

MaxNodesPerJob=
Maximum number of nodes per job in this association.

MaxWallDurationPerJob=
Maximum wall clock time each job can use in this associa-
tion.

Organization=
Name of organization that owns this account.

QOS(=,+=,-=)
Comma-separated list of Quality of Service names (Defined
in sacctmgr).

To add users to an account add a line after the Parent line, similar to
this:

Parent - test
User - adam:MaxTRESPerJob=node:2:MaxJobs=3:FairShare=1:MaxWallDurationPerJob=1:AdminLevel=Operator:Coordinator='test'

User options are:

AdminLevel=
Type of admin this user is (Administrator, Operator)
Must be defined on the first occurrence of the user.

Coordinator=
Comma-separated list of accounts this user is coordinator
over
Must be defined on the first occurrence of the user.

DefaultAccount=
System wide default account name
Must be defined on the first occurrence of the user.

FairShare=
Number used in conjunction with other associations to de-
termine job priority.

MaxJobs=
Maximum number of running jobs from this user.

MaxTRESPerJob=
Maximum number of TRES each job from this user can use.

MaxWallDurationPerJob=
Maximum wall clock time each job from this user can use.

QOS(=,+=,-=)
Comma-separated list of Quality of Service names (Defined
in sacctmgr).

ARCHIVE FUNCTIONALITY
Sacctmgr has the capability to archive to a flatfile and or load that
data if needed later. The archiving is usually done by the slurmdbd and
it is highly recommended you only do it through sacctmgr if you com-
pletely understand what you are doing. For slurmdbd options see "man
slurmdbd" for more information. Loading data into the database can be
done from these files to either view old data or regenerate rolled up
data. For information about configuring an archive server see
<https://slurm.schedmd.com/accounting.html#archive>.

archive dump
Dump accounting data to file. Data will not be archived unless the cor-
responding purge option is included in this command or in slur-
mdbd.conf. This operation cannot be rolled back once executed. If one
of the following options is not specified when sacctmgr is called, the
value configured in slurmdbd.conf is used.

Directory=
Directory to store the archive data.

Events Archive Events. If not specified and PurgeEventAfter is
set all event data removed will be lost permanently.

Jobs Archive Jobs. If not specified and PurgeJobAfter is set
all job data removed will be lost permanently.

PurgeEventAfter=
Purge cluster event records older than time stated in
months. If you want to purge on a shorter time period you
can include hours, or days behind the numeric value to
get those more frequent purges. (e.g. a value of
'12hours' would purge everything older than 12 hours.)

PurgeJobAfter=
Purge job records older than time stated in months. If
you want to purge on a shorter time period you can in-
clude hours, or days behind the numeric value to get
those more frequent purges. (e.g. a value of '12hours'
would purge everything older than 12 hours.)

PurgeStepAfter=
Purge step records older than time stated in months. If
you want to purge on a shorter time period you can in-
clude hours, or days behind the numeric value to get
those more frequent purges. (e.g. a value of '12hours'
would purge everything older than 12 hours.)

PurgeSuspendAfter=
Purge job suspend records older than time stated in
months. If you want to purge on a shorter time period you
can include hours, or days behind the numeric value to
get those more frequent purges. (e.g. a value of
'12hours' would purge everything older than 12 hours.)

Script=
Run this script instead of the generic form of archive to
flat files.

Steps Archive Steps. If not specified and PurgeStepAfter is set
all step data removed will be lost permanently.

Suspend
Archive Suspend Data. If not specified and PurgeSus-
pendAfter is set all suspend data removed will be lost
permanently.

archive load
Load in to the database previously archived data. The archive file will
not be loaded if the records already exist in the database - therefore,
trying to load an archive file more than once will result in an error.
When this data is again archived and purged from the database, if the
old archive file is still in the directory ArchiveDir, a new archive
file will be created (see ArchiveDir in the slurmdbd.conf man page), so
the old file will not be overwritten and these files will have dupli-
cate records.

Archive files from the current or any prior Slurm release may be loaded
through archive load.

File= File to load into database. The specified file must exist
on the slurmdbd host, which is not necessarily the ma-
chine running the command.

Insert=
SQL to insert directly into the database. This should be
used very cautiously since this is writing your sql into
the database.

PERFORMANCE
Executing sacctmgr sends a remote procedure call to slurmdbd. If enough
calls from sacctmgr or other Slurm client commands that send remote
procedure calls to the slurmdbd daemon come in at once, it can result
in a degradation of performance of the slurmdbd daemon, possibly re-
sulting in a denial of service.

Do not run sacctmgr or other Slurm client commands that send remote
procedure calls to slurmdbd from loops in shell scripts or other pro-
grams. Ensure that programs limit calls to sacctmgr to the minimum
necessary for the information you are trying to gather.

ENVIRONMENT VARIABLES
Some sacctmgr options may be set via environment variables. These envi-
ronment variables, along with their corresponding options, are listed
below. (Note: Command line options will always override these set-
tings.)

SLURM_CONF The location of the Slurm configuration file.

SLURM_DEBUG_FLAGS Specify debug flags for sacctmgr to use. See De-
bugFlags in the slurm.conf(5) man page for a full
list of flags. The environment variable takes
precedence over the setting in the slurm.conf.

SLURM_JSON Control JSON serialization:

compact
Output JSON as compact as possible.

pretty Output JSON in pretty format to make it more
readable.

SLURM_YAML Control YAML serialization:

compact Output YAML as compact as possible.

pretty Output YAML in pretty format to make it more
readable.

EXAMPLES
NOTE: There is an order to set up accounting associations. You must
define clusters before you add accounts and you must add accounts be-
fore you can add users.

$ sacctmgr create cluster tux
$ sacctmgr create account name=science fairshare=50
$ sacctmgr create account name=chemistry parent=science fairshare=30
$ sacctmgr create account name=physics parent=science fairshare=20
$ sacctmgr create user name=adam cluster=tux account=physics fairshare=10
$ sacctmgr delete user name=adam cluster=tux account=physics
$ sacctmgr delete user name=adam cluster=tux account=science partition=\"\"
$ sacctmgr delete account name=physics cluster=tux
$ sacctmgr modify user where name=adam cluster=tux account=physics set maxjobs=2 maxwall=30:00
$ sacctmgr add user brian account=chemistry
$ sacctmgr list associations cluster=tux format=Account,Cluster,User,Fairshare tree withd
$ sacctmgr list transactions Action="Add Users" Start=11/03-10:30:00 format=Where,Time
$ sacctmgr dump cluster=tux file=tux_data_file
$ sacctmgr load tux_data_file

A user's account can not be changed directly. A new association needs
to be created for the user with the new account. Then the association
with the old account can be deleted.

When modifying an object placing the key words 'set' and the optional
'where' is critical to perform correctly below are examples to produce
correct results. As a rule of thumb anything you put in front of the
set will be used as a quantifier. If you want to put a quantifier after
the key word 'set' you should use the key word 'where'. The following
is wrong:

$ sacctmgr modify user name=adam set fairshare=10 cluster=tux

This will produce an error as the above line reads modify user adam set
fairshare=10 and cluster=tux. Either of the following is correct:

$ sacctmgr modify user name=adam cluster=tux set fairshare=10
$ sacctmgr modify user name=adam set fairshare=10 where cluster=tux

When changing qos for something only use the '=' operator when wanting
to explicitly set the qos to something. In most cases you will want to
use the '+=' or '-=' operator to either add to or remove from the ex-
isting qos already in place.

If a user already has qos of normal,standby for a parent or it was ex-
plicitly set you should use qos+=expedite to add this to the list in
this fashion.

If you are looking to only add the qos expedite to only a certain ac-
count and or cluster you can do that by specifying them in the sacctmgr
line.

$ sacctmgr modify user name=adam set qos+=expedite

$ sacctmgr modify user name=adam acct=this cluster=tux set qos+=expedite

Let's give an example how to add QOS to user accounts. List all avail-
able QOSs in the cluster.

$ sacctmgr show qos format=name
Name
---------
normal
expedite

List all the associations in the cluster.

$ sacctmgr show assoc format=cluster,account,qos
Cluster Account QOS
-------- ---------- --------------------
zebra root normal
zebra root normal
zebra g normal
zebra g1 normal

Add the QOS expedite to account G1 and display the result. Using the
operator += the QOS will be added together with the existing QOS to
this account.

$ sacctmgr modify account name=g1 set qos+=expedite
$ sacctmgr show assoc format=cluster,account,qos
Cluster Account QOS
-------- ---------- --------------------
zebra root normal
zebra root normal
zebra g normal
zebra g1 expedite,normal

Now set the QOS expedite as the only QOS for the account G and display
the result. Using the operator = that expedite is the only usable QOS
by account G

$ sacctmgr modify account name=G set qos=expedite
$ sacctmgr show assoc format=cluster,account,qos
Cluster Account QOS
-------- ---------- --------------------
zebra root normal
zebra root normal
zebra g expedite
zebra g1 expedite,normal

If a new account is added under the account G it will inherit the QOS
expedite and it will not have access to QOS normal.

$ sacctmgr add account banana parent=G
$ sacctmgr show assoc format=cluster,account,qos
Cluster Account QOS
-------- ---------- --------------------
zebra root normal
zebra root normal
zebra g expedite
zebra banana expedite
zebra g1 expedite,normal

An example of listing trackable resources:

$ sacctmgr show tres
Type Name ID
---------- ----------------- --------
cpu 1
mem 2
energy 3
node 4
billing 5
gres gpu:tesla 1001
license vcs 1002
bb cray 1003

COPYING
Copyright (C) 2008-2010 Lawrence Livermore National Security. Produced
at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
Copyright (C) 2010-2022 SchedMD LLC.

This file is part of Slurm, a resource management program. For de-
tails, see <https://slurm.schedmd.com/>.

Slurm is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your op-
tion) any later version.

Slurm is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.

Header And Logo

Peripheral Links

Site Navigation

FreeBSD Manual Pages

Header And Logo

Peripheral Links

Search

Site Navigation

FreeBSD Manual Pages