xref: /third_party/ltp/doc/shell-test-api.txt

LTP Shell Test API
==================

NOTE: See also
      https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[Test Writing Guidelines],
      https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API].

1 Writing a testcase in shell
-----------------------------

LTP supports testcases to be written in a portable shell too.

There is a shell library modeled closely to the C interface at
'testcases/lib/tst_test.sh'.

WARNING: All identifiers starting with 'TST_' or 'tst_' are reserved for the
         test library.

1.1 Basic test interface
~~~~~~~~~~~~~~~~~~~~~~~~

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# This is a basic test for true shell builtin

TST_TESTFUNC=do_test
. tst_test.sh

do_test()
{
	true
	ret=$?

	if [ $ret -eq 0 ]; then
		tst_res TPASS "true returned 0"
	else
		tst_res TFAIL "true returned $ret"
	fi
}

tst_run
-------------------------------------------------------------------------------

TIP: To execute this test the 'tst_test.sh' library must be in '$PATH'. If you
     are executing the test from a git checkout you can run it as
     'PATH="$PATH:../../lib" ./foo01.sh'

The shell library expects test setup, cleanup and the test function executing
the test in the '$TST_SETUP', '$TST_CLEANUP' and '$TST_TESTFUNC' variables.

Both '$TST_SETUP' and '$TST_CLEANUP' are optional.

The '$TST_TESTFUNC' may be called several times if more than one test
iteration was requested by passing right command line options to the test.

The '$TST_CLEANUP' may be called even in the middle of the setup and must be
able to clean up correctly even in this situation. The easiest solution for
this is to keep track of what was initialized and act accordingly in the
cleanup.

WARNING: Similar to the C library, calling 'tst_brk' in the $TST_CLEANUP does
         not exit the test and 'TBROK' is converted to 'TWARN'.

Notice also the 'tst_run' shell API function called at the end of the test that
actually starts the test.

WARNING: cleanup function is called only after 'tst_run' has been started.
Calling 'tst_brk' in shell libraries, e.g. 'tst_test.sh' or 'tst_net.sh' does
not trigger calling it.

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Example test with tests in separate functions

TST_TESTFUNC=test
TST_CNT=2
. tst_test.sh

test1()
{
	tst_res TPASS "Test $1 passed"
}

test2()
{
	tst_res TPASS "Test $1 passed"
}

tst_run
# output:
# foo 1 TPASS: Test 1 passed
# foo 2 TPASS: Test 2 passed
-------------------------------------------------------------------------------

If '$TST_CNT' is set, the test library looks if there are functions named
'$\{TST_TESTFUNC\}1', ..., '$\{TST_TESTFUNC\}$\{TST_CNT\}' and if these are
found they are executed one by one. The test number is passed to it in the '$1'.

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Example test with tests in a single function

TST_TESTFUNC=do_test
TST_CNT=2
. tst_test.sh

do_test()
{
	case $1 in
	1) tst_res TPASS "Test $1 passed";;
	2) tst_res TPASS "Test $1 passed";;
	esac
}

tst_run
# output:
# foo 1 TPASS: Test 1 passed
# foo 2 TPASS: Test 2 passed
-------------------------------------------------------------------------------

Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc.,
the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed
to it in the '$1'.

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Example test with tests in a single function, using $TST_TEST_DATA and
# $TST_TEST_DATA_IFS

TST_TESTFUNC=do_test
TST_TEST_DATA="foo:bar:d dd"
TST_TEST_DATA_IFS=":"
. tst_test.sh

do_test()
{
	tst_res TPASS "Test $1 passed with data '$2'"
}

tst_run
# output:
# foo 1 TPASS: Test 1 passed with data 'foo'
# foo 2 TPASS: Test 1 passed with data 'bar'
# foo 3 TPASS: Test 1 passed with data 'd dd'
-------------------------------------------------------------------------------

It's possible to pass data for function with '$TST_TEST_DATA'. Optional
'$TST_TEST_DATA_IFS' is used for splitting, default value is space.

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Example test with tests in a single function, using $TST_TEST_DATA and $TST_CNT

TST_TESTFUNC=do_test
TST_CNT=2
TST_TEST_DATA="foo bar"
. tst_test.sh

do_test()
{
	case $1 in
	1) tst_res TPASS "Test $1 passed with data '$2'";;
	2) tst_res TPASS "Test $1 passed with data '$2'";;
	esac
}

tst_run
# output:
# foo 1 TPASS: Test 1 passed with data 'foo'
# foo 2 TPASS: Test 2 passed with data 'foo'
# foo 3 TPASS: Test 1 passed with data 'bar'
# foo 4 TPASS: Test 2 passed with data 'bar'
-------------------------------------------------------------------------------

'$TST_TEST_DATA' can be used with '$TST_CNT'. If '$TST_TEST_DATA_IFS' not specified,
space as default value is used. Of course, it's possible to use separate functions.

1.2 Library environment variables and functions for shell
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Similarily to the C library various checks and preparations can be requested
simply by setting right '$TST_NEEDS_FOO'.

[options="header"]
|=============================================================================
| Variable name            | Action done
| 'TST_NEEDS_ROOT'         | Exit the test with 'TCONF' unless executed under root.
|                          | Alternatively the 'tst_require_root' command can be used.
| 'TST_NEEDS_TMPDIR'       | Create test temporary directory and cd into it.
| 'TST_NEEDS_DEVICE'       | Prepare test temporary device, the path to testing
                             device is stored in '$TST_DEVICE' variable.
                             The option implies 'TST_NEEDS_TMPDIR'.
| 'TST_NEEDS_CMDS'         | String with command names that has to be present for
                             the test (see below).
| 'TST_NEEDS_MODULE'       | Test module name needed for the test (see below).
| 'TST_NEEDS_DRIVERS'      | Checks kernel drivers support for the test.
| 'TST_NEEDS_KCONFIGS'     | Checks kernel kconfigs support for the test (see below).
| 'TST_NEEDS_KCONFIGS_IFS' | Used for splitting '$TST_NEEDS_KCONFIGS' variable,
                             default value is comma, it only supports single character.
| 'TST_TIMEOUT'            | Maximum timeout set for the test in sec. Must be int >= 1,
                             or -1 (special value to disable timeout), default is 300.
                             Variable is meant be set in tests, not by user.
                             It's an equivalent of `tst_test.timeout` in C, can be set
                             via 'tst_set_timeout(timeout)' after test has started.
|=============================================================================

[options="header"]
|=============================================================================
| Function name              | Action done
| 'tst_set_timeout(timeout)' | Maximum timeout set for the test in sec.
                               See 'TST_TIMEOUT' variable.
|=============================================================================

NOTE: Network tests (see testcases/network/README.md) use additional variables
and functions in 'tst_net.sh'.

Checking for presence of commands
+++++++++++++++++++++++++++++++++

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh

...

TST_NEEDS_CMDS="modinfo modprobe"
. tst_test.sh

...

-------------------------------------------------------------------------------

Setting '$TST_NEEDS_CMDS' to a string listing required commands will check for
existence each of them and exits the test with 'TCONF' on first missing.

Alternatively the 'tst_require_cmds()' function can be used to do the same on
runtime, since sometimes we need to the check at runtime too.

'tst_check_cmds()' can be used for requirements just for a particular test
as it doesn't exit (it issues 'tst_res TCONF'). Expected usage is:

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh

TST_TESTFUNC=do_test
. tst_test.sh

do_test()
{
	tst_check_cmds cmd || return
	cmd --foo
	...
}

tst_run
-------------------------------------------------------------------------------

Locating kernel modules
+++++++++++++++++++++++

The LTP build system can build kernel modules as well, setting
'$TST_NEEDS_MODULE' to module name will cause the library to look for the
module in a few possible paths.

If module was found the path to it will be stored into '$TST_MODPATH'
variable, if module wasn't found the test will exit with 'TCONF'.

Alternatively the 'tst_require_module()' function can be used to do the same
at runtime.

1.3 Optional command line parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Optional test command line parameters

TST_OPTS="af:"
TST_USAGE=usage
TST_PARSE_ARGS=parse_args
TST_TESTFUNC=do_test

. tst_test.sh

ALTERNATIVE=0
MODE="foo"

usage()
{
	cat << EOF
usage: $0 [-a] [-f <foo|bar>]

OPTIONS
-a     Enable support for alternative foo
-f     Specify foo or bar mode
EOF
}

parse_args()
{
	case $1 in
	a) ALTERNATIVE=1;;
	f) MODE="$2";;
	esac
}

do_test()
{
	...
}

tst_run
-------------------------------------------------------------------------------

The 'getopts' string for optional parameters is passed in the '$TST_OPTS'
variable. There are a few default parameters that cannot be used by a test,
these can be listed with passing help '-h' option to any test.

The function that prints the usage is passed in '$TST_USAGE', the help for
the options implemented in the library is appended when usage is printed.

Lastly the function '$PARSE_ARGS' is called with the option name in the '$1'
and, if option has argument, its value in the '$2'.

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Optional test positional parameters

TST_POS_ARGS=3
TST_USAGE=usage
TST_TESTFUNC=do_test

. tst_test.sh

usage()
{
	cat << EOF
usage: $0 [min] [max] [size]

EOF
}

min="$1"
max="$2"
size="$3"

do_test()
{
	...
}

tst_run
-------------------------------------------------------------------------------

You can also request a number of positional parameters by setting the
'$TST_POS_ARGS' variable. If you do, these will be available as they were
passed directly to the script in '$1', '$2', ..., '$n'.

1.4 Useful library functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Retrieving configuration variables
++++++++++++++++++++++++++++++++++

You may need to retrieve configuration values such as PAGESIZE, there is
'getconf' but as some system may not have it, you are advised to use
'tst_getconf' instead. Note that it implements subset of 'getconf'
system variables used by the testcases only.

[source,sh]
-------------------------------------------------------------------------------
# retrieve PAGESIZE
pagesize=`tst_getconf PAGESIZE`
-------------------------------------------------------------------------------

Sleeping for subsecond intervals
++++++++++++++++++++++++++++++++

Albeit there is a sleep command available basically everywhere not all
implementations can support sleeping for less than one second. And most of the
time sleeping for a second is too much. Therefore LTP includes 'tst_sleep'
that can sleep for defined amount of seconds, milliseconds or microseconds.

[source,sh]
-------------------------------------------------------------------------------
# sleep for 100 milliseconds
tst_sleep 100ms
-------------------------------------------------------------------------------

Retry a function call multiple times
++++++++++++++++++++++++++++++++++++

Sometimes an LTP test needs to retry a function call multiple times because
the system is not ready to process it successfully on the first try. The LTP
library has useful tools to handle the call retry automatically.
'TST_RETRY_FUNC()' will keep retrying for up to 1 second. If you want a custom
time limit use 'TST_RETRY_FN_EXP_BACKOFF()'. Both methods return the value
returned by the last 'FUNC' call.

The delay between retries starts at 1 microsecond and doubles after each call.
The retry loop ends when the function call succeeds or when the next delay
exceeds the specified time (1 second for 'TST_RETRY_FUNC()'). The maximum
delay is multiplied by TST_TIMEOUT_MUL. The total cumulative delay may be up
to twice as long as the adjusted maximum delay.

The C version of 'TST_RETRY_FUNC()' is a macro which takes two arguments:

* 'FUNC' is the complete function call with arguments which should be retried
  multiple times.
* 'SUCCESS_CHECK' is a macro or function which will validate 'FUNC' return
  value. 'FUNC' call was successful if 'SUCCESS_CHECK(ret)' evaluates to
  non-zero.

Both retry methods clear 'errno' before every 'FUNC' call so your
'SUCCESS_CHECK' can look for specific error codes as well. The LTP library
also includes predefined 'SUCCESS_CHECK' macros for the most common call
conventions:

* 'TST_RETVAL_EQ0()' - The call was successful if 'FUNC' returned 0 or NULL
* 'TST_RETVAL_NOTNULL()' - The call was successful if 'FUNC' returned any
  value other than 0 or NULL.
* 'TST_RETVAL_GE0()' - The call was successful if 'FUNC' returned value >= 0.

[source,c]
-------------------------------------------------------------------------------
/* Keep trying for 1 second */
TST_RETRY_FUNC(FUNC, SUCCESS_CHECK)

/* Keep trying for up to 2*N seconds */
TST_RETRY_FN_EXP_BACKOFF(FUNC, SUCCESS_CHECK, N)
-------------------------------------------------------------------------------

The shell version of 'TST_RETRY_FUNC()' is simpler and takes slightly
different arguments:

* 'FUNC' is a string containing the complete function or program call with
  arguments.
* 'EXPECTED_RET' is a single expected return value. 'FUNC' call was successful
  if the return value is equal to EXPECTED_RET.

[source,sh]
-------------------------------------------------------------------------------
# Keep trying for 1 second
TST_RETRY_FUNC "FUNC arg1 arg2 ..." "EXPECTED_RET"

# Keep trying for up to 2*N seconds
TST_RETRY_FN_EXP_BACKOFF "FUNC arg1 arg2 ..." "EXPECTED_RET" "N"
-------------------------------------------------------------------------------

Checking for integers
+++++++++++++++++++++

[source,sh]
-------------------------------------------------------------------------------
# returns zero if passed an integer parameter, non-zero otherwise
tst_is_int "$FOO"
-------------------------------------------------------------------------------

Checking for integers and floating point numbers
++++++++++++++++++++++++++++++++++++++++++++++++

[source,sh]
-------------------------------------------------------------------------------
# returns zero if passed an integer or floating point number parameter,
# non-zero otherwise
tst_is_num "$FOO"
-------------------------------------------------------------------------------

Obtaining random numbers
++++++++++++++++++++++++

There is no '$RANDOM' in portable shell, use 'tst_random' instead.

[source,sh]
-------------------------------------------------------------------------------
# get random integer between 0 and 1000 (including 0 and 1000)
tst_random 0 1000
-------------------------------------------------------------------------------

Formatting device with a filesystem
+++++++++++++++++++++++++++++++++++

The 'tst_mkfs' helper will format device with the filesystem.

[source,sh]
-------------------------------------------------------------------------------
# format test device with ext2
tst_mkfs ext2 $TST_DEVICE
# default params are $TST_FS_TYPE $TST_DEVICE
tst_mkfs
# optional parameters
tst_mkfs ext4 /dev/device -T largefile
-------------------------------------------------------------------------------

Mounting and unmounting filesystems
+++++++++++++++++++++++++++++++++++

The 'tst_mount' and 'tst_umount' helpers are a safe way to mount/umount
a filesystem.

The 'tst_mount' mounts '$TST_DEVICE' of '$TST_FS_TYPE' (optional) to
'$TST_MNTPOINT' (defaults to mntpoint), optionally using the
'$TST_MNT_PARAMS'. The '$TST_MNTPOINT' directory is created if it didn't
exist prior to the function call.

If the path passed (optional, must be absolute path, defaults to '$TST_MNTPOINT')
to the 'tst_umount' is not mounted (present in '/proc/mounts') it's noop.
Otherwise it retries to umount the filesystem a few times on failure.
This is a workaround since there are daemons dumb enough to probe all newly
mounted filesystems, and prevents them from being umounted shortly after they
were mounted.

ROD and ROD_SILENT
++++++++++++++++++

These functions supply the 'SAFE_MACROS' used in C although they work and are
named differently.

[source,sh]
-------------------------------------------------------------------------------
ROD_SILENT command arg1 arg2 ...

# is shorthand for:

command arg1 arg2 ... > /dev/null 2>&1
if [ $? -ne 0 ]; then
        tst_brk TBROK "..."
fi


ROD command arg1 arg2 ...

# is shorthand for:

ROD arg1 arg2 ...
if [ $? -ne 0 ]; then
        tst_brk TBROK "..."
fi
-------------------------------------------------------------------------------

WARNING: Keep in mind that output redirection (to a file) happens in the
         caller rather than in the ROD function and cannot be checked for
         write errors by the ROD function.

As a matter of a fact doing +ROD echo a > /proc/cpuinfo+ would work just fine
since the 'ROD' function will only get the +echo a+ part that will run just
fine.

[source,sh]
-------------------------------------------------------------------------------
# Redirect output to a file with ROD
ROD echo foo \> bar
-------------------------------------------------------------------------------

Note the '>' is escaped with '\', this causes that the '>' and filename are
passed to the 'ROD' function as parameters and the 'ROD' function contains
code to split '$@' on '>' and redirects the output to the file.

EXPECT_PASS{,_BRK} and EXPECT_FAIL{,_BRK}
+++++++++++++++++++++++++++++++++++++++++

[source,sh]
-------------------------------------------------------------------------------
EXPECT_PASS command arg1 arg2 ... [ \> file ]
EXPECT_FAIL command arg1 arg2 ... [ \> file ]
-------------------------------------------------------------------------------

'EXPECT_PASS' calls 'tst_res TPASS' if the command exited with 0 exit code,
and 'tst_res TFAIL' otherwise. 'EXPECT_FAIL' does vice versa.

Output redirection rules are the same as for the 'ROD' function. In addition
to that, 'EXPECT_FAIL' always redirects the command's stderr to '/dev/null'.

There are also 'EXPECT_PASS_BRK' and 'EXPECT_FAIL_BRK', which works the same way
except breaking a test when unexpected action happen.

It's possible to detect whether expected value happened:
[source,sh]
-------------------------------------------------------------------------------
if ! EXPECT_PASS command arg1 2\> /dev/null; then
	continue
fi
-------------------------------------------------------------------------------

tst_kvcmp
+++++++++

This command compares the currently running kernel version given conditions
with syntax similar to the shell test command.

[source,sh]
-------------------------------------------------------------------------------
# Exit the test if kernel version is older or equal to 2.6.8
if tst_kvcmp -le 2.6.8; then
	tst_brk TCONF "Kernel newer than 2.6.8 is needed"
fi

# Exit the test if kernel is newer than 3.8 and older than 4.0.1
if tst_kvcmp -gt 3.8 -a -lt 4.0.1; then
	tst_brk TCONF "Kernel must be older than 3.8 or newer than 4.0.1"
fi
-------------------------------------------------------------------------------

[options="header"]
|=======================================================================
| expression | description
| -eq kver   | Returns true if kernel version is equal
| -ne kver   | Returns true if kernel version is not equal
| -gt kver   | Returns true if kernel version is greater
| -ge kver   | Returns true if kernel version is greater or equal
| -lt kver   | Returns true if kernel version is lesser
| -le kver   | Returns true if kernel version is lesser or equal
| -a         | Does logical and between two expressions
| -o         | Does logical or between two expressions
|=======================================================================

The format for kernel version has to either be with one dot e.g. '2.6' or with
two dots e.g. '4.8.1'.

.tst_fs_has_free
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh

...

# whether current directory has 100MB free space at least.
if ! tst_fs_has_free . 100MB; then
	tst_brkm TCONF "Not enough free space"
fi

...
-------------------------------------------------------------------------------

The 'tst_fs_has_free' shell interface returns 0 if the specified free space is
satisfied, 1 if not, and 2 on error.

The second argument supports suffixes kB, MB and GB, the default unit is Byte.

.tst_retry
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh

...

# Retry ping command three times
tst_retry "ping -c 1 127.0.0.1"

if [ $? -ne 0 ]; then
	tst_resm TFAIL "Failed to ping 127.0.0.1"
else
	tst_resm TPASS "Successfully pinged 127.0.0.1"
fi

...
-------------------------------------------------------------------------------

The 'tst_retry' function allows you to retry a command after waiting small
amount of time until it succeeds or until given amount of retries has been
reached (default is three attempts).

1.5 Restarting daemons
~~~~~~~~~~~~~~~~~~~~~~

Restarting system daemons is a complicated task for two reasons.

* There are different init systems
  (SysV init, systemd, etc...)

* Daemon names are not unified between distributions
  (apache vs httpd, cron vs crond, various syslog variations)

To solve these problems LTP has 'testcases/lib/daemonlib.sh' library that
provides functions to start/stop/query daemons as well as variables that store
correct daemon name.

.Supported operations
|==============================================================================
| start_daemon()   | Starts daemon, name is passed as first parameter.
| stop_daemon()    | Stops daemon, name is passed as first parameter.
| restart_daemon() | Restarts daemon, name is passed as first parameter.
| status_daemon()  | Detect daemon status (exit code: 0: running, 1: not running).
|==============================================================================

.Variables with detected names
|==============================================================================
| CROND_DAEMON | Cron daemon name (cron, crond).
| SYSLOG_DAEMON | Syslog daemon name (syslog, syslog-ng, rsyslog).
|==============================================================================

.Cron daemon restart example
[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0-or-later
# Cron daemon restart example

TCID=cron01
TST_COUNT=1
. test.sh
. daemonlib.sh

...

restart_daemon $CROND_DAEMON

...

tst_exit
-------------------------------------------------------------------------------

1.6 Access to the checkpoint interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The shell library provides an implementation of the checkpoint interface
compatible with the C version. All 'TST_CHECKPOINT_*' functions are available.

In order to initialize checkpoints '$TST_NEEDS_CHECKPOINTS' must be set to '1'
before the inclusion of 'tst_test.sh':

[source,sh]
-------------------------------------------------------------------------------
#!/bin/sh

TST_NEEDS_CHECKPOINTS=1
. tst_test.sh
-------------------------------------------------------------------------------

Since both the implementations are compatible, it's also possible to start
a child binary process from a shell test and synchronize with it. This process
must have checkpoints initialized by calling 'tst_reinit()'.

1.7 Parsing kernel .config
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The shell library provides an implementation of the kconfig parsing interface
compatible with the C version.

It's possible to pass kernel kconfig list for tst_require_kconfigs API with
'$TST_NEEDS_KCONFIGS'.
Optional '$TST_NEEDS_KCONFIGS_IFS' is used for splitting, default value is comma.

-------------------------------------------------------------------------------
#!/bin/sh
TST_NEEDS_KCONFIGS="CONFIG_EXT4_FS, CONFIG_QUOTACTL=y"

. tst_test.sh
-------------------------------------------------------------------------------