############################################################################
# Copyright (C) 2002-2007 The Regents of the University of California.
# Copyright (C) 2008-2011 Lawrence Livermore National Security.
# Copyright (C) 2010-2017 SchedMD LLC
# Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
# Written by Morris Jette <jette1@llnl.gov>
# Additions by Joseph Donaghy <donaghy1@llnl.gov>
# CODE-OCEC-09-009. All rights reserved.
#
# This file is part of Slurm, a resource management program.
# For details, see <https://slurm.schedmd.com/>.
# Please also read the supplied file: DISCLAIMER.
#
# Slurm is free software; you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free
# Software Foundation; either version 2 of the License, or (at your option)
# any later version.
#
# Slurm is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
# details.
#
# You should have received a copy of the GNU General Public License along
# with Slurm; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301  USA.
############################################################################

This directory contains a battery of Slurm regression tests. The tests make
use of the "expect" scripting language.

The expect tests first look for the (unified) testsuite.conf configuration
file. By default, this is expected to be found in SRCDIR/testsuite. If placed
elsewhere, the SLURM_TESTSUITE_CONF environment variable should be set to the
full path of the testsuite.conf file. This file can be created from a copy of
the autogenerated sample found in BUILDDIR/testsuite/testsuite.conf.sample
and then edited as necessary.

The expect tests next look for an (expect-specific) globals.local configuration
file. Values in this file override values in the testsuite.conf file. The
globals.local file can be in the directory containing this README, or
the one configured in the environment variable SLURM_LOCAL_GLOBALS_FILE.

The globals.local file contains (expect) testsuite parameters, for example:

slurm_dir:                    slurm installation directory (prefix) [/usr]
build_dir:                    slurm build directory [../..]
testsuite_cleanup_on_failure: run cleanup and remove test_dir on failure [true]
testsuite_shared_dir:         cluster-wide shared directory [.]
testsuite_log_level:          logging threshold [$LOG_LEVEL_DEBUG]
testsuite_log_format:         specifies the tracing fields used in logging
testsuite_colorize:           boolean that turns colorization on or off
testsuite_subtest_fatal:      first failing subtest aborts the test [false]
testsuite_subtest_details:    add details of subtest in the summary [fail]
                              possible values: all, fail, fail_skip or none
testsuite_testproc_details:   add details of testprocs in the summary [fail]
                              possible values: all, fail, fail_skip or none
testsuite_testproc_log_calls: yes to always print the whole testproc call,
                              no to never print it, or [ignore_skips] to print
                              all but the skipped testprocs
testsuite_user:               a system username that is NOT on the Slurm DB
                              that may be used to sudo commands as him
                              instead of current user (see check_run_as_user)

The testsuite needs at least one cluster-wide shared file system.
By default, we assume that it's the testsuite/expect directory (i.e. ".").
Files created within this directory on one node in the cluster should be
visible to all other nodes in the cluster.

For the valid fields usable with testsuite_log_format, see the header
comments for the _log_format procedure. See also the default defintion of
testsuite_log_format in globals for an example of syntax.

Each test can be executed independently. Upon successful completion, the test
will print "SUCCESS" and terminate with an exit code of zero. Upon failure,
the test will print "FAILURE" and terminate with a positive exit code. If a
test fails its precondition checks, it will print "SKIPPED" and terminate with
a negative exit code. Use the fail, skip or pass procedures to end the test.

A test can expect to have some global variables defined that it can make use
of within the test. These include:

test_name:  the name of the test script, e.g. test1.1
test_id:    the numeric suffix for the test, e.g. 1.1
test_dir:   a temporary subdirectory created within the testsuite_shared_dir *

* The test_dir will get removed at the end of the test unless the test failed
  and testsuite_cleanup_on_failure was set to false. The test_dir will be
  (removed and) recreated the next time the test is run.

A test can be run as normal executable (uses expect) and can read by default
two main arguments:
	-i : a comma-separated list of testproc ids to run
	-e : a comma-separated list of testproc ids to skip
These arguments will be removed and not available on argv on the tests.
Test writers should use testproc to launch their testproc functions to
take advantage of those arguments (see testproc function).

The script "regression.py" will execute all of the tests and summarize the
results. Standard output contains detailed logging of all events, which is
quite verbose. Failure information is written to standard error. A good
way to run "regression.py" is to write its standard output to one file and
either write standard error to another file or print it to the terminal.
Execution time of the full test suite is roughly 80 minutes, but can vary
considerably with the architecture, configuration, and system load. Some
tests send e-mail, so check for four e-mail messages sent to the user
running the tests.

When failures do occur, check the standard output for details. Searching
for the keyword "FAILURE" will typically locate the failing test. Note
that some of the tests are architecture or configuration specific.  Also
note that most tests are designed to be run as a normal user. Tests 3.#
are designed to be run as user root or SlurmUser, but will be skipped
when the full test suite is executed as an unprivileged user. The full
test suite is typically executed many times by the Slurm developers on a
variety of systems before a Slurm release is made. This has resulted in
high system reliability. When Slurm bugs are found or features added,
this test suite is expanded.

