fMBT FAQ

Questions on test generation and execution

Test generation:

• How can I generate a simple test case?
• How can I speed up test generation?

Debugging:

• How can I debug AAL Python code?
• How can I execute AAL Python inputs one-by-one?
• How can I handle exceptions from AAL Python adapter blocks?
• How can I log method and function calls, return values and exceptions?

Repeating test runs:

• How can I execute test steps from a test log?

How can I generate a simple test case?

You can define specific test steps to be tested with the usecase coverage. For example, generate and execute a test case that will first test stopping alarm then playing music.

$ fmbt -o 'heuristic = lookahead(5)' \
       -o 'coverage = usecase("i:stop alarm" then "i:play music")' \
       test.conf

The test generator will search for a minimal test case that includes these two steps. You do not need to list steps that are prerequisites for your steps, the test generator will figure that out.

See usecase coverage in test configuration documentation for more information and examples.

How can I speed up test generation?

• Use smaller lookahead depth. If any action in the model can be executed by taking at most n steps from any state, you can get optimal test by using n as a lookahead depth. Using depth greater than n only makes test generation slower.
• Instead of lookahead(d) use lookahead(1:d). This means that the lookahead heuristic goes through all sequences of length 1, then 2, ..., and finally d. But it immediately stops when it finds sequences of length x (1 <= x <= d) that increase coverage. This is likely to be a lot faster than constructing always all sequences of length d and choosing the best, but resulting test may not be as well optimized.
• Instead of lookahead(d) use lookahead(db). (Add letter "b" for "burst" right after the depth.) This makes lookahead stick with the planned sequence until it is completely consumed, and only after that plan next d steps. Again, faster test generation but possibly less optimal test.

How can I debug AAL Python code?

You can access variables and execute the code line-by-line with pdb as follows:

1. Add fmbt.debug() function call right before the line where you want to stop (like a breakpoint).
2. Run fmbt-debug in another terminal window. It will give the (Pdb) prompt when fmbt.debug() gets called.

Quick help for selected Pdb commands:

help   - help on commands
l      - list source code
n      - next line
s      - step in
p EXPR - print the value of expression

How can I handle exceptions from AAL Python adapter blocks?

Define function adapter_exception_handler(name, exc) in the global scope. If an exception is raised from the adapter block of an action, this function will be called with the action name and the exception object as parameters. Example:

language "python" {
    def adapter_exception_handler(name, exc):
        import traceback
        file("/tmp/adapter_exceptions.txt", "a").write(
            "exception %s from the adapter of %s\n%s\n" %
            (exc, name, traceback.format_exc()))
}

The return value of the handler defines how the test generator should update AAL variables or if the test should fail immediately. See AAL/Python reference for more information.

How can I log method and function calls, return values and exceptions?

The fmbtlogger Python module helps logging modules, classes and objects. fmbtlogger.text(obj, log_target) returns a wrapped obj. All calls through the wrapped obj are logged in text format to log_target. The log_target is a file or a function that takes a message (string) to be logged as a parameter.

Example: log function calls from subprocess and os modules to a file.

import os
import subprocess
import fmbtlogger

# wrap subprocess and os modules to log to my_log_file
my_log_file = open("/tmp/my_log_file.txt", "w")
subprocess = fmbtlogger.text(subprocess, my_log_file)
os = fmbtlogger.text(os, my_log_file)

# use them
subprocess.check_output("pwd", shell=True)
os.mkdir("/proc/definitely/an/error")

This works for user-defined objects as well. Example: my_wrapped_instance = fmbtlogger.text(MyClass(), sys.stderr).

See help(fmbtlogger.text) for more options.

How can I execute AAL Python inputs one-by-one?

Launch fMBT in the interactive mode (-i). For instance, fmbt -i test.conf.

Quick help for selected interactive mode commands:

         - print help (press ENTER)
s        - list actions whose guard is True in the current state
sNUM     - execute action NUM in the list printed by "s"
i:NAME   - execute input NAME
oea/oem  - get/set execution of adapter and body blocks when executing actions

How can I execute test steps from a test log?

• Method 1: Force execution of the same steps in the exactly same order.

  First, extract generated test steps from old test log (old.log). Give them as an input to interactive fMBT that uses the same test configuration (model and adapter) as the old test (old.conf).

  fmbt-log -f '$as' old.log | fmbt -i -l new.log old.conf

  This method will work if the system under test works in the same way in all test runs. But if the system is non-deterministic, then Method 2 attempts to repeat the same steps more gently.

• Method 2: Generate new test that includes the same steps in the same order.

  First, extract test steps from old test log (old.log). Create a usecase coverage expression of the form "i:step a" then "i:step b" then ... then "i:step z" from the steps. Generate new test with the lookahead algorithm.

  fmbt -o "coverage=usecase($(fmbt-log -f '"$as"' -S ' then ' old.log))" -o "heuristic=lookahead(3)" -l new.log old.conf

  If the system under test behaves differently (but acceptably), the lookahead algorithm will try to find the quickest way to get back on the route given in usecase.