Developer/Tests

=Introduction= We use TextTest to test whether the software still behaves as expected. What TextTest does is to compare an application's output, including the output to stdout, stderr, and into generated files with predefined outputs from files.

There are some advantages and disadvantages of this approach. On the one hand, you can guarantee that the application is doing what is wanted by comparing all outputs with files you think are right. But this is also the problem with this approach: you do not have the assurance that the files you are comparing the current outputs to ARE right - you have to check these by hand.

=Setup=

Windows Setup
We use TextTest as our testing environment which is python based and has a quite complex setup under Windows. Following the installation instructions You will need the following software (at least for a setup with GUI)
 * Python 2.7
 * GTK for Windows
 * PyGTK and PyCairo
 * TKDiff (you can choose another compatible diff tool if you like)

python and diff should also appear in your PATH.

Linux setup
This is much easier because at least with recent distributions there is probably everything included. With openSUSE 10.2 I needed to install python-gtk and python-cairo as well as tkdiff before installing TextTest. If you don't need the GUI you can probably even skip these.

=Running the Tests= Within the  /tests - folder you can find batch-files which start TextTest with our test suites. "runAllTests.bat" starts TextTest for testing all apllications located in the folder, "runNetconvertTests.bat" will only show tests for NETCONVERT, "runDuarouterTests.bat" only those for DUAROUTER etc.

=Adding an Application Test Suite= To add a test suite for a new application, you have to perform the following steps. For the examples below we'll use "polyconvert" as the example application. call testEnv.bat texttest.py -a polyconvert -gx export POLYCONVERT_BINARY="$SUMO_BIN/polyconvert" export POLYCONVERT_BINARY="$SUMO_BIN/polyconvert" set POLYCONVERT_BINARY=%CD%\..\bin\polyconvert.exe
 * go to  {SUMO}/tests 
 * copy one of the run...Tests.bat-files and rename it properly (runPolyconvertTests.bat in our case); change the name of the application within it. In our case the resulting file will look as this:
 * add the application to the list of applications that are tested each night by
 * adding it to runTest.sh; in our case, the following line was added:
 * adding it to testDaily.sh; in our case, the following line was added:
 * adding it to testEnv.bat; in our case, the following line was added:
 * build a test folder for the application, named as the application itself (without the ".exe" extension), in our case the folder is named polyconvert
 * go the folder
 * build a configuration file; its name is "config", the extension is the application's to test name, so in our case it's config.polyconvert. Please consult TextTest documentation about the content, nonetheless, here are some notes
 * do not forget the import of the config file
 * name the binary correct
 * name the file name properly in output
 * In all files that are collated, Version shold be ignored - the tests should be working along all versions
 * The initial file looks as following:

import_config_file:../config_all binary:$POLYCONVERT_BINARY copy_test_path:input_net.net.xml [collate_file] config:config.cfg log:log.txt [run_dependent_text] output:polyconvert.exe{REPLACE sumo-polyconvert} net:Version
 * build a top-level testsuite file; its name is "testsuite", the extension is the application's to test name, so in our case it's testsuite.polyconvert
 * I suppose, it is a good idea to start with tests of meta-output; Simply copy them from another application and patch the file names...

=Regular tests= At the moment all our tests run each night on two linux machines of different age and on a windows server.