Next Previous Contents

2. Introduction

TESS is the (Te)st (S)ystem for (S)-Lang, which aims at reducing the workload and ad-hoc nature of regression testing S-Lang software ( http://www.jedsoft.org/slang/), by collecting common testing elements into a single, easy-to-use framework. TESS provides the S-Lang developer nominal mechanisms for tailoring the S-Lang environment and invoking functions with arbitrary inputs, while transparently inspecting and cleaning the stack, gathering pass/fail statistics, and providing error recovery from exceptions.

TESS is primarily a development tool, so we assume the reader is familiar with scripting in S-Lang. Knowledge of how to create S-Lang modules is also helpful. Since TESS is intended to assist in automating and simplifying regression testing, it is most often utilized in conjunction with the make tool via a Makefile. If these are unfamiliar terms then you will benefit from learning about them first prior to reading further.

Please note that as of version 0.3.0 TESS no longer supports S-Lang 1. This enables TESS to be implemented as purely interpreted code (i.e. S-Lang and shell scripts, no binaries), and makes it smaller, easier to build & use, as well as easier to bundle in other packages (only 2 small text files are required).

2.1 Motivation

Suppose you had a function defined within some file add.sl as 

        define add()
        {
           variable i, j;
           if (_NARGS < 2) usage("add(i,j)");
           (i,j) = ();
           return i+j;
        }
and that you wanted to exercise it in automated fashion on a range of input conditions. To test it with insufficient arguments, for instance, one might write a script add.t 
        () = evalfile("./add");
        add(1);
which, when invoked from slsh as 
        slsh ./add.t
yields something like 
        Usage: add(i,j)
        called from line 2, file: ./add.t

Model: Multiple Tests per Script

Ok, so far things look reasonable. Now, suppose you appended the line 

        add(1,"two");
to the test and reran it in slsh. Curiously enough, the output generated would look no different from above. Why? Because the first add() call generates a usage exception, which causes the interpreter to unwind the S-Lang stack and then exit. In other words, the second add() call is never even executed! One way to address this is to modify the script by adding an ERROR_BLOCK, such as: 
        () = evalfile("./add");

        define test1()
        {
           ERROR_BLOCK { _clear_error(); return; }
           add(1);
        }

        test1;
        add(1, "two");
Now both tests will be exercised when the script is invoked, generating a usage exception in the first case and a type mismatch error in the second. Progress, for sure, but the script has grown longer, and we needed to introduce a wrapper for the first test case in order to use the error block. In this model, where one script contains multiple test cases, each test would need to be invoked within such a wrapper, which explains why TESS offers the tess_invoke() function.

Model: Single Test per Script

An alternative to the model used above would be to each test case within its own .t file and invoke slsh upon each. This approach avoids the need for error blocks, but introduces a number of other concerns which collectively steer the author towards a preference for the first model.

For example, the resultant file proliferation makes it more cumbersome to enumerate/name and effectively organize the test suite. Packages of only moderate size might conceivably contain scores or perhaps hundreds of relatively tiny files, needlessly wasting disk resources, increasing sizes of software distributions, and wasting CPU cycles by launching slsh once per test instead of only once per group of tests. Moreover, each test invocation would also need to load both TESS and the package being tested, resulting in yet more CPU waste and code duplication.

Identifying semantically related test cases would not be as easy, since groupings are now distinguished by like-named files within a directory, instead of by cohabitation of test cases within a single file.

Collecting useful failure statistics becomes more difficult, since in this model aggregate counts can be be obtained only by metascripts, e.g. invoked within Makefiles at the operating system level to keep track of the 1 or 0 returned from each test, instead of within the test scripts themselves. In contrast, the tess_invoke function mentioned above transparently tallies pass/fail statistics.

For small test suites these issues may be negligible, but as packages grow their cumulative effect may not be so easily ignored, making it better to cultivate the preferable habit of "starting clean," rather than one of "cleaning up later".

Consistency and Reuse

TESS emerged as the coalescence of scripts used in testing a number of existing S-Lang packages. In fact, development versions of packages such as SLgtk and SLxpa have already been migrated from their original testing scheme towards TESS, and as such they serve as the wealthiest source of supplemental examples to this documentation.

By distilling common patterns from existing test suites into a self-contained distribution TESS fosters reuse, reduces duplicative busy work, and hopefully serves to ease the burden of testing (perhaps one of the least loved aspects of writing and maintaining software).

Next Previous Contents