Testing

From Buzztard

Each good software should be tested, at any time during development, again and again.

We use the check testing framework for our unit-tests.

First of all we try to check every public method and every constructor for each object we provide.

Currently we have some tests for the core library. These tests should verify that the objects behave well for all kinds of arguments we pass onto them. Further we have some tests for the end-user applications. Here we test e.g. commandline args, exit codes and so on. Altogether that makes at the moment about 60 testcases in 3 suites.

We also would like to have testtools to check our coding guidlines, method access (private or public access), state of documentations and so on.

On Guadec (2005) we've did a unit testing tutorial session. You can download the slides (540 kb).

Contents 1 Song test file access 2 Test naming scheme 3 Log-output capturing 4 Testing failures 5 GObject Testing 5.1 Type sanity 5.2 Property checks 5.3 Chaining up 6 Test Coverage 7 Spell checking 7.1 Spell checking c-sources 7.2 Spell checking po files 7.3 Spell checking xml api documentation 7.4 Spell checking xml user documentation 8 Test UI Application 8.1 Invisible X Server 8.2 unfocused windows 8.3 event recorders 9 Future testing 9.1 Add Valgrind support to test cases 9.2 GLib features 9.3 Enabling/Disabling Tests 9.4 File I/O testing 9.5 Array bound checking 10 Buildbot 11 Documentation and Links

[edit] Song test file access

If you, as a developer, would like to use any song-files for your tests, access them under the following directory structure:

buzztard
+- test (FROM HERE!)
|  +- songs
:  |  +- simple1.xml
:  :  +- simple2.xml
:  :  :

This can be done in code with following example:

 // postive test
 START_TEST(test_play4) {
       BtCmdApplication *app;
       int ret=0;
 
       app=bt_cmd_application_new();
       ret = bt_cmd_application_play(app, check_get_test_song_path("simple1.xml"));
       if (!ret) {
          fail("play does not work with a good file name");
       }
 }
 END_TEST

The

check_get_test_song_path()

method is needed to make test also work in non-srcdir builds (e.g. when doing make distcheck).

[edit] Test naming scheme

At first there is one test binary for each module or component (library, application). These are prefixed with m- We use file prefixes here to distinguish from the source file names to be tested.

Each test binary is further structured into test suites, test cases and tests. Suites are prefixed by s-.

Inside the directory tests you'll find the files: m-bt-core.c, m-bt-cmd.c and m-bt-edit.c. The m-bt-cmd.c file is the main entry point to check all functions related to the buzztard command application, short bt-cmd. The m-bt-core.c file is the main entry point to check all functions related to the buzztard core lib and so on ...

If you plan to write your own application with buzztard, please generate a new testcase like m-bt-cmd. All the test suites and test cases go to an own directoy, which is alike to the folder structure in src.

Each new test suite which you plan to create should have the prefix s- and then the same name as the whole file which you try to test. For example:

You plan to test methods from the song.c in the core lib. Then you should create a file named s-song.c in which you define the suite and files like t-song.c or e-song.c each containing respectively a test case and tests. Here the prefix are:

• t- for test: try to break the component.
• e- for example: show how to use the component properly.

Other aspects that can be tested are constrains (for which we suggest using the prefix c-). Such tests would supervise the execution of the code regarding some limits like cpu usage, memory consumption, response times and so on.

[edit] Log-output capturing

The tests capture (most) log-output and write that all to a log named by the test, e.g. /tmp/buzztard.log. Each test-case is delimmited by a line of '=' chars and each test with a line of '-'. This is useful to get in-depth information about test-failure.

The test application further controls the log-level itself. Please don't change that without reason, as we use log-parsing functions to test against expected output in some situations (methods pre-conditions).

[edit] Testing failures

At any point in development state you would verify, that your checks of wrong arguments or other kind of wrong behavior, in your software will be correct. To do such tests you need to capture the log output.

For this kind of *fail* checks we have created a helper function called:

 check_init_error_trapp()

With this function you are able the check if the following statement in your testcase produces es log-output with the same message as you give it to the check_init_error_trapp function. Later in your code you can check the error trapping with:

 fail_unless(check_has_error_trapped(), NULL);

Following example shows you, how to use this. Let us create a testcase for the BtSequence:

 /* try to create a new sequence with NULL for song object */
 START_TEST(test_btsequence_obj1) {
   BtSequence *sequence=NULL;
   GST_INFO("--------------------------------------------------------------------------------");
   check_init_error_trapp("bt_sequence_new","BT_IS_SONG(song)");
   sequence=bt_sequence_new(NULL);
   fail_unless(sequence == NULL, NULL);
   fail_unless(check_has_error_trapped(), NULL);
 }
 END_TEST

In this example we try to check, if the constructor of the BtSequence class handles correct a given NULL pointer. In our code we use the

 return_val_if_fail

method to create a log output and returning a defined value. In our testcase we check the created log output from the return_val_if_fail method.

[edit] GObject Testing

Our project is heavily based on the GObject oo sytem. This allows us to do some generic tests:

[edit] Type sanity

We should make a common object type test helper like this:

 GTypeQuery query;
 g_type_query(BT_TYPE_MACHINE,&query);
 fail_if(query.type == 0, NULL);
 fail_if(query.class_size != sizeof(GtkMachineClass), NULL);
 fail_if(query.instance_size != sizeof(GtkMachine), NULL);

[edit] Property checks

In the common check module we have a test helper that applies some sanity checks to object properties.

 gboolean check_gobject_properties(GObject *to_check)

[edit] Chaining up

If classes override dispose() / finalize() methods they should also chain up.

  klass=BT_XXX_GET_CLASS(self);
  parent_class=g_type_class_peek_parent(klass);
 
  parent_finalze=G_OBJECT_CLASS(parent_class)->finalize;
  G_OBJECT_CLASS(parent_class)->finalize=intercept;
 
  g_object_unref(self);

[edit] Test Coverage

The idea is to find out which code is covered by tests and which code is not. On the base of that new tests can be added and dead code can be eliminated.

It needs three things:

1. Build all code with CFLAGS="-fprofile-arcs -ftest-coverage" (configure with --enable-coverage=yes")
2. Run make check
3. Analyse coverage (make coverage)

We now have included a make target that generates the coverage report. You need to have lcov installed. To see how it looks like, check our current coverage report.

Below are some detail on how to manually get coverage information.

To analyse coverage of a source file do e.g.:

  cd ./src/lib/core
  gcov -p -f -o.libs/ song.c

Repeat this for every source file. Afterwards look at the *.gcov files. Lines marked with "#####" indicate code that has not been executed.

For a little report use:

  for file in *.c; do
    gcov -p -f -o.libs/ $file;
  done | grep "in file" | grep -v "include" | sort -nr

For better reports, we have integrated lcov on top of that. It generates html pages that give a good overview and allow to view details. You need the latest lcov version with this patch applied. To generate the report use the following commands:

  mkdir ./coverage
  lcov --directory . --zerocounters
  -$(MAKE) check
  lcov --directory . --capture --output-file ./coverage/buzztard.info
  genhtml -o ./coverage --num-spaces 2 ./coverage/buzztard.info

[edit] Spell checking

The aspell tool can be quite helpful to fix spelling errors. It can even be applied to sources!

I'm still looking for a way to supply an extra word list to aspell, that contains symbol names from the sources. That would help checking sources and api docs.

[edit] Spell checking c-sources

We are trying to make a local dictionary to ease checking sources. To build it do:

 make tags dict

Then you can check sources

 aspell -c --mode=ccpp --lang=en -p=buzztard.aspell_dict ./src/lib/core/machine.c

[edit] Spell checking po files

There is no explicit po mode. Maybe we can use comment mode?

I don't think that comment mode would work good enough (remember that there are always both english and translated messages). acheck seems to be able to check .po files. --SvenHerzberg 16:08, 2 Apr 2006 (CEST)

[edit] Spell checking xml api documentation

 aspell -c --mode=sgml --lang=en ./docs/reference/bt-core/bt-core-docs.sgml

[edit] Spell checking xml user documentation

 aspell -c --mode=sgml --lang=en ./docs/help/bt-edit/C/bt-edit.xml.in

See a working implementation from SvenHerzberg.

[edit] Test UI Application

The major problem that remains is to make the test run invisible. Just hiding the widnow won't do the trick, as then probably thing like 'mapping widget to the screen' won't occur.

As a side effect of doing controlled GUI tests, we can use such a tests to invoke one dialog page after the other and then produce up-to-date screen-shots (see gnome-screen-shooter panel applet for how-to save png images).

[edit] Invisible X Server

The idea to avoid this is to use a virtual framebuffer based X server display. One can get such a display by running:

  /usr/X11R6/bin/Xvfb -ac :9 -screen 0 1024x786x16

Then gtk application needs to be redirected to this display. While one usualy would do this by:

  ./my-app  --display=:9.0

we need to do it programmatically. See gtk+/demos/gtk-demo/changedisplay.c for information about how to do it.

To find a free display_id look in /tmp/.X11-unix/X*. Then create and shut down the server as follows:

 GPid pid;
 gulong flags=G_SPAWN_SEARCH_PATH|G_SPAWN_STDOUT_TO_DEV_NULL|G_SPAWN_STDERR_TO_DEV_NULL;
 GError *error=NULL;
 gchar *argv[]={
   "Xvfb",
   "-ac",":9","-screen","0","1024x786x16",
   NULL
 }
 
 if(!(g_spawn_async(NULL,argv,NULL,flags,NULL,NULL,&pid,&error))) {
   GST_ERROR("error creating virtual x-server : \"%s\"", error->message);
   g_error_free(error);
 }
 // ...
 kill(pid, SIGBRK);
 g_spawn_close_pid(pid);

To use such a display with gtk we need to do:

 display_manager = gdk_display_manager_get();
 display = gdk_display_open(":9");
 gdk_display_manager_set_default_display(display_manager,display);
 
 gdk_display_close(display);

[edit] unfocused windows

As another idea, couldn't we set some window-manager hints, so that these windows don't grab the focus. Unfortunately:

 gtk_window_set_focus_on_map(GTK_WINDOW(main_window),FALSE);
 gtk_window_set_accept_focus(GTK_WINDOW(main_window),FALSE);

Does not work, as it seems too late for it (show_all()) has already been called.

[edit] event recorders

We need to check out a few new testing technology:

1. the Linux Desktop Testing Project
2. Gerd - a Gtk+ Event Recorder. The gtk+-2 version is in gnome cvs

Can we use these for gui-app testing?

[edit] Future testing

[edit] Add Valgrind support to test cases

See Valgrind API and Valgrind Manual.

This is how to run an uninstalled test binary via valgrind:

  libtool --mode=execute valgrind --tool=memcheck ./bt_cmd

Add a configure option "--with-valgrind={no|yes|path}". If option given is 'yes' search for valgrind. If valgrind support is enabled and valgrind has been found activate it in the tests. Therefore conditionally add macros to the unit-test code:

1. check if valgrind.h is found
2. include valgrind.h and memcheck.h
3. use VALGRIND_COUNT_ERRORS in start and end of unit-tests to see if there have been errors
4. use VALGRIND_DO_LEAK_CHECK and VALGRIND_COUNT_LEAKS for each unit-test

Open question is how to run unittests via

  valgrind --tool=memcheck --log-fd=-1 <i>testname</i>

A possible solution would be to have shellscripts as tests and a makefile target that generates the shellscripts for binary unit tests. Another solution to execlp() ourself via valgrind, if we don't run under valgrind yet.

Currently we have a valgrind target in the makefiles. So doing make valgrind runs the tests under valgrind.

[edit] GLib features

GType debugging:

 g_type_init_with_debug_flags()

GMem debugging:

 g_mem_set_vtable(glib_mem_profiler_table);
 ...
 g_mem_profile();

[edit] Enabling/Disabling Tests

We disable tests that we can't fix right now using an ifdef preprocessor directive using a standard form. This way we can use a script to collect them as a nice list (a testing TODO).

The format looks like below:

 /*
  * tests if ...
  *
  * needs this feature implemented first (see : http://www.xyx.org/bugid=2875)
  */
 #ifdef __CHECK_DISABLED__
 BT_START_TEST(test_xyz) {
 ... test here ....
 }
 BT_END_TEST
 #endif

The problem is to collect all lines above #ifdef __CHECK_DISABLED until BT_START_TEST(test_xyz) below. Sounds like we need a little perl-script for this. The shell one-liner I found so far is:

 grep -r -A6 -B1 -Hn --include="*.c" --color=auto "#ifdef __CHECK_DISABLED__" .

It still has the problem that the it just grabs the 6 lines above the #ifdef :(.

The list is now available via make todo in the tests subdir.

[edit] File I/O testing

It would be interesting to have a fuse based test-file system that simulates all kind of errors. This includes:

• read-only files
• files go away in the middle of reads
• files containing garbage (fuzzing)

There is petardfs project that looks like it fits the needs here.

[edit] Array bound checking

http://gcc.gnu.org/wiki/Mudflap_Pointer_Debugging

[edit] Buildbot

We've started to run a buildbot that does cvs builds of 'gst-buzztard' and 'buzztard' modules every 6 hours.

[edit] Documentation and Links

• buildbot - how to setup the buildmaster and buildslave for buzztard
• Unit test patterns - naming and organisation of tests
• Gnome memory reduction project - has some nice references to tools