RELEASE NOTES FOR SLURM VERSION 23.11

IMPORTANT NOTES:
If using the slurmdbd (Slurm DataBase Daemon) you must update this first.

NOTE: If using a backup DBD you must start the primary first to do any
database conversion, the backup will not start until this has happened.

The 23.11 slurmdbd will work with Slurm daemons of version 22.05 and above.
You will not need to update all clusters at the same time, but it is very
important to update slurmdbd first and having it running before updating
any other clusters making use of it.

Slurm can be upgraded from version 22.05 or 23.02 to version 23.11 without loss
of jobs or other state information. Upgrading directly from an earlier version
of Slurm will result in loss of state information.

All SPANK plugins must be recompiled when upgrading from any Slurm version
prior to 23.11.

HIGHLIGHTS
==========
 -- Remove 'none' plugins for all but auth and cred. scontrol show config
    will report (null) now.
 -- Removed select/cons_res. Please update your configuration to
    select/cons_tres.
 -- Change TreeWidth default from 50 to 16.
 -- job_submit/throttle - improve reset of submitted job counts per user in
    order to better honor SchedulerParameters=jobs_per_user_per_hour=#.
 -- Allow SlurmUser/root to use reservations without specific permissions.
 -- Add TopologyParam=RoutePart to route communications based on partition node
    lists.
 -- Added ability for configless to push Prolog and Epilog scripts to slurmds.
 -- Added --external-launcher option to srun to allow different MPI
    implementations to run their launcher (orte, hydra, etc.) inside a special
    step with access to all the allocated resources in the node, and without
    consuming any of them, allowing for other steps to run concurrently now.
 -- Replace SRUN_CPUS_PER_TASK with SLURM_CPUS_PER_TASK and get back to the
    behavior before Slurm 22.05. Starting in Slurm 22.05, --cpus-per-task
    implies --exact which is why we needed to make srun not read
    SLURM_CPUS_PER_TASK. Since now we have the new external launcher step,
    (srun --external-launcher), srun can read this env variable from within
    an allocation again, so even if -c1 is set, mpirun will run and won't be
    bound to a single cpu.
 -- Enable streaming replication for Galera 4 during upgrades.
 -- Remove cloud_reg_addrs and make it default behavior. Slurm will
    automatically manage NodeHostName and NodeAddr for cloud nodes.
 -- Remove NoAddrCache CommunicationParameter.
 -- Add QOS flag 'Relative'. If set the QOS limits will be treated as
    percentages of a cluster/partition instead of absolutes.
 -- The warning printed when using configure --without-PACKAGE has been changed
    to a notice.
 -- Userspace governor will now *not* accept a frequency range of min and max,
    and will simply statically set the required frequency. If the frequency is
    out of range, the closest value to the cpu limits will be chosen.
 -- PMIx support is nolonger built by default.  Passing --with-pmix option is
    now required to build with PMIx.
 -- Update slurmstepd processes with current SlurmctldHost settings, allowing
    for controller changes without draining all compute jobs.
 -- sreport - cluster Utilization PlannedDown field now includes the time that
    all nodes were in the POWERED_DOWN state instead of just cloud nodes.
 -- Remove SLURM_NODE_ALIASES env variable. Client code now uses slurm_addr_t's
    passed from controller.
 -- Enable fanout for dynamic and unaddresable cloud nodes.
 -- Make it so reservations can reserve GRES.
 -- The rpmbuild "--with mysql" option has been removed. The rpm has long
    required sql development libraries to build and the existence of this option
    was confusing. The default behavior now is to always require one of the sql
    development libraries.
 -- The reference slurmctld and slurmdbd service files now run under User=slurm
    and Group=slurm. (These are installed automatically for RPMs.)
 -- Added support for Debian packaging.  Please note that this set of packages
    is new and subject to more change than the long-standing and more stable
    spec file.
 -- switch/hpe_slingshot - Add support for collectives.
 -- Rename topology/none plugin to topology/default.
 -- Add gpu/nrt plugin for nodes using Trainium/Inferentia devices.
 -- Disable sorting of dynamic nodes to avoid issues when restarting with
    heterogenous jobs that cause jobs to abort on restart.
 -- Don't allow deletion of non-dynamic nodes.
 -- cgroup/v2 does not return Virtual Memory metrics for accounting anymore.
    As the kernel cgroups interface did not provide any interface to gather
    these values, the returned value was an unreliable approximation based on
    other cgroup metrics. This has been corrected and from now on a value of 0
    should be expected in the accounting for AveVMSize, MaxVMSize,
    MaxVMSizeNode, MaxVMSizeTask and vmem in TRESUsageInTot if using
    jobacct_gather/cgroup and cgroup/v2.

CONFIGURATION FILE CHANGES (see appropriate man page for details)
=====================================================================
 -- Removed JobCredentialPrivateKey and JobCredentialPublicCertificate
    parameters.
 -- Added max_submit_line_size to SchedulerParameters.
 -- cgroup.conf - Removed deprecated parameters AllowedKmemSpace,
    ConstrainKmemSpace, MaxKmemPercent, and MinKmemSpace.
 -- proctrack/cgroup - Add "SignalChildrenProcesses=<yes|no>" option to
    cgroup.conf. This allows signals for cancelling, suspending, resuming, etc.
    to be sent to children processes in a step/job rather than just the parent.
 -- Add PreemptParameters=suspend_grace_time parameter to control amount of
    time between SIGTSTP and SIGSTOP signals when suspending jobs.
 -- Add SlurmctldParameters=no_quick_restart to avoid a new slurmctld taking
    over the old slurmctld on accident.
 -- Changed the default SelectType to select/cons_tres (from select/linear).
 -- Remove CgroupAutomount= option from cgroup.conf. Modern kernels mount the
    cgroup file system automatically. CgroupAutomount could cause a cgroup v2
    system to be configured in a hybrid v1 and v2 system. The cgroup/v1 plugin
    will now fail if the cgroup filesystem is not mounted.
 -- Prolog and Epilog do not have to be fully qualified pathnames.
 -- Changed default value of PriorityType from priority/basic to
    priority/multifactor.
 -- Allow for a shared suffix to be used with the hostlist format. E.g.,
    "node[0001-0010]-int".
 -- Add format_stderr to LogTimeFormat of slurm.conf and slurmdbd.conf.
 -- Add SelectTypeParameters=LL_SHARED_GRES.
 -- Add SwitchParameters=hwcoll_addrs_per_job, hwcoll_num_nodes, fm_url,
    fm_auth, and fm_authdir to support collectives.
 -- Deprecate the ExtSensorsType and ExtSensorsFreq options.
 -- Cray XC support has been deprecated. Use '--enable-deprecated' to allow the
    the build to continue. Sites are encouraged to contact SchedMD about the EOL
    date for Cray XC support.
 -- RoutePlugin=route/topology has been replaced with TopologyParam=RouteTree.
 -- Add SchedulerParameters=extra_constraints. This enables various node
    filtering options in the --extra flag of salloc, sbatch, and srun.

COMMAND CHANGES (see man pages for details)
===========================================
 -- scontrol show assoc_mgr will display Lineage instead of Lft for
    associations.
 -- sacctmgr list associations 'lft' column is removed.
 -- sacctmgr list associations 'lineage' has been added.
 -- Fix --cpus-per-gpu for step allocations, which was previously ignored for
    job steps. --cpus-per-gpu implies --exact.
 -- Fix mutual exclusivity of --cpus-per-gpu and --cpus-per-task: fatal if both
    options are requested in the commandline or both are requested in the
    environment. If one option is requested in the command line, it will
    override the other option in the environment.
 -- slurmrestd - new argument '-s' has been added to allow explicit loading of
    data_parser plugins or '-s list' to list possible plugins.
 -- All commands supporting '--yaml' and '--json' arguments will now use the
    data_parser/v0.0.40 plugin for formatting the output by default.
 -- torque/mpiexec - Propagate exit code from launched process.
 -- sbatch - removed --export-file option (used with defunct Moab integration).
 -- Define SPANK options environment variables when --export=[NIL|NONE] is
    specified.
 -- Reject reservation update if it will result in previously submitted
    jobs losing access to the reservation.
 -- scontrol/sview - Remove FIRST_CORES flag from reservations.
 -- scontrol/sview - Remove comma separated CoreCnt option from reservations.
 -- scontrol/sview - Remove comma separated NodeCnt option from reservations.
 -- slurmd - add "instance-id", "instance-type", and "extra" options to allow
    them to be set on startup.
 -- scontrol - add InstanceId and InstanceType to node records.
 -- sacctmgr - add 'show instance' for cloud instance accounting data
 -- salloc/sbatch/srun --mem-per-cpu and select/linear: Fix memory calculation
    with --threads-per-core or --hint=nomultithread and --mem-per-cpu:
    Previously, memory = mem-per-cpu * all cpus including unusable threads.
    Now, memory = mem-per-cpu * only usuable threads. This behavior matches
    the documentation and select/cons_tres.
 -- salloc/srun - Remove --uid/--gid options.
 -- scrontab - Add @fika and @teatime as valid repetition times.
 -- scontrol update partition now allows Nodes+=<node-list> and
    Nodes-=<node-list> to add/delete nodes from the existing partition node
    list. Nodes=+host1,-host2 is also allowed.
 -- salloc/sbatch/srun - Modify the '--constraint' option to require square
    brackets around requests with multiple features that include node counts.
 -- sdiag - Added statistics on why the main and backfill schedulers have
    stopped evaluation on each scheduling cycle.
 -- Rename sbcast --fanout to --treewidth.
 -- salloc/sbatch/srun - When requesting --tres-per-task alter incorrect
    request for TRES, it should be TRESType/TRESName not TRESType:TRESName.
 -- salloc/sbatch/srun - Add disable_rdzv_get option to --network to disable
    rendezvous gets when using the switch/hpe_slingshot plugin.
 -- Requesting --cpus-per-task will now set SLURM_TRES_PER_TASK=cpu:# in the
    environment.
 -- scontrol - Removed "abort" command.

API CHANGES
===========
 -- cli_filter/lua - return nil for unset time options rather than the string
    "2982616-04:14:00" (which is the internal macro "NO_VAL" represented as
    time string).
 -- "flags" argument was added to slurm_kill_job_step().
 -- Fixed typo on "initialized" for the description of ESLURM_PLUGIN_NOT_LOADED.
 -- SPANK - added new spank_prepend_task_argv() function.
 -- SPANK - Failures from most spank functions (not epilog or exit) will now
    cause the step to be marked as failed and the command (srun, salloc,
    sbatch --wait) to return 1.
 -- "node_list" argument was added to slurm_print_topo_info_msg().
 -- remove slurm_print_topo_record().
 -- submit filters should use new --tres-per-task format: TRESType/TRESName

SLURMRESTD CHANGES
==================
 -- openapi/dbv0.0.37 and openapi/v0.0.37 plugins have been removed.
 -- openapi/dbv0.0.38 and openapi/v0.0.38 plugins have been tagged as
    deprecated to warn of their removal in the next release.
 -- New openapi plugins will no longer be versioned. Existing versioned openapi
    plugins will follow normal deprecation and removal schedule. Data format
    versioning will now be handled by the data_parser plugins which will now be
    used by the openapi plugins.
 -- data_parser plugins will now generate all schemas related to object
    formatting and structure. The openapi.json files in the openapi/slurmctld
    and openapi/slurmdbd directories should be considered templates only. All
    openapi specifications should be queried from slurmsrestd directly as they
    change depending on the loaded plugins and settings.
 -- The version field in the info object of the OpenAPI specfication will now
    list the Slurm version running and list out the loaded openapi plugins at
    time of generation using '&' as a delimiter in loading order.
 -- OpenAPI specfication from openapi/slurmctld and openapi/slurmdbd plugins is
    known to be incompatible with OpenAPI Generator version 5 and below. Sites
    are advised to port to OpenAPI Generator version 6 or greater for generated
    clients.
 -- Path parameters fields in OpenAPI specifications will now only give type as
    strings for openapi/slurmctld and openapi/slurmdbd end points. The 'enum'
    will now be auto-populated when parameter has list of known valid values.
    Prior more detailed formatting information was found to conflict with
    generated OpenAPI clients forcing limitations on the possible values not
    present in Slurm's parsing capabilities.
 -- openapi/v0.0.40 - add /instance and /instances endpoints.
 -- slurmrestd - OperationIDs may have changed during conversion to v0.0.40
    from v0.0.39 paths to better match their paths.
 -- slurmrestd - Default to not query assocations or coordinators with
    'GET /slurmdb/v0.0.40/accounts'. To query account with assocations, query
    'GET /slurmdb/v0.0.40/accounts?with_assocs'. To query account with
    coordinators, query 'GET /slurmdb/v0.0.40/accounts?with_coords'. To query
    both assocations and coordinators with accounts, query
    'GET /slurmdb/v0.0.40/accounts?with_coords&with_assocs'.
 -- slurmrestd - Default to not query assocations, wckeys or coordinators with
    'GET /slurmdb/v0.0.40/user'. To query user with assocations, query
    'GET /slurmdb/v0.0.40/user?with_assocs'. To query user with
    coordinators, query 'GET /slurmdb/v0.0.40/user?with_coords'. To query user
    with wckeys, query 'GET /slurmdb/v0.0.40/user?with_wckeys'. To query
    both assocations, wckeys, and coordinators with user, query
    'GET /slurmdb/v0.0.40/user?with_coords&with_assocs&with_wckeys'.
 -- slurmrestd - 'POST /slurm/v0.0.40/job/submit' will return "step_id" as
    OpenAPI string instead of OpenAPI integer type to provide descriptive step
    names (batch, extern, interactive, TBD) for non-numeric steps.
 -- slurmrestd - Tagged "result" field from 'POST /slurm/v0.0.40/job/submit'
    as deprecated which may be removed in a future release. Field was added in
    v0.0.39 to unify response formats but prior fields were kept to avoid
    breaking existing clients. The additional benefit was found to be
    insufficent for the change.
 -- slurmrestd - Tagged "job_id", "step_id", and "job_submit_user_msg" fields
    from 'POST /slurm/v0.0.40/job/{job_id}' response as deprecated due their
    only being valid for the first entry in the "result" field array. The
    "result" field should be used instead to get the detailed result of the
    update request.
 -- openapi/v0.0.40 - add /{accounts,users}_association endpoints.
