Testing use SSH fails if not deactivation when a previous test reboots

[JoshuaWatt]

I came across a weird behavior that was counterintuitive when writing a test that cause the remote device to reboot. Given the following test file:

# strategy is "ShellStrategy"
import pytest


@pytest.fixture(scope="function")
def shell_command(target, strategy, capsys):
    with capsys.disabled():
        strategy.transition("shell")
    return target.get_active_driver("ShellDriver")


@pytest.fixture(scope="function")
def ssh_command(target, strategy, capsys):
    with capsys.disabled():
        strategy.transition("shell")
    try:
        return target.get_driver("SSHDriver")
    except NoDriverFoundError:
        pytest.skip("No SSHDriver found")


def test_network(ssh_command):
    ssh_command.run_check("true")


def test_reboot(shell_command, strategy):
    shell_command.run_check("true")
    strategy.transition("off")
    shell_command.target.deactivate(shell_command)
    strategy.transition("shell")
    shell_command.target.activate(shell_command)
    shell_command.run_check("true")


def test_post_network(ssh_command):
    # This test fails because the ssh_command fixture was not deactivated
    ssh_command.run_check("true")

The test_post_network test fails with the error labgrid.driver.exception.ExecutionError: Keepalive no longer running. This is because the ssh_command fixture was not deactivated in test_reboot like the shell_command fixture. It seems odd that I would need to add ssh_command as a fixture to test_reboot just to resolve this problem; what is the recommended way to resolve this? It seems brittle to make any test that reboots need to know about all fixtures that might need to be deactivated.

[Emantor]

The intention is to not deactivate every single driver, but let the strategy handle the this via invalidation.
As an example, the Graphstrategy implements invalidate like this

labgrid/labgrid/strategy/graphstrategy.py

Lines 101 to 110 in a6571a4

	def invalidate(self):
	"""
	Marks the path to the current state as out-of-date. Subsequent transition() calls will
	start from the root state.
	Will be called if exceptions in state methods occur.
	"""
	self.path = []
	# deactivate all drivers to restore initial state
	self.target.deactivate_all_drivers()

the important part being that a call to invalidate also deactivates all drivers.

The strategies are really only included within labgrid as a starting point for your own individual strategy which is device specific. We include some basic ones and the GraphStrategy, but the intention is to really implement your own strategy and than using the imports: key within the YAML configuration to load your strategy into the tests.

Invalidate for your strategy could look like this:

     def invalidate(self): 
         self.status = Status.off # off instead of unknown to skip the power cycle and use a real reboot.
         self.target.deactivate_all_drivers() 

Given your example above and an implementation for invalidate within your strategy and rebooting your system properly, this would look something like this:

# strategy based on "ShellStrategy" with invalidate implemented
import pytest


@pytest.fixture(scope="function")
def shell_command(target, strategy, capsys):
    with capsys.disabled():
        strategy.transition("shell")
    return target.get_active_driver("ShellDriver")


@pytest.fixture(scope="function")
def ssh_command(target, strategy, capsys):
    with capsys.disabled():
        strategy.transition("shell")
    try:
        return target.get_driver("SSHDriver")
    except NoDriverFoundError:
        pytest.skip("No SSHDriver found")


def test_network(ssh_command):
    ssh_command.run_check("true")


def test_reboot(shell_command, strategy):
    shell_command.run_check("true")
    strategy.console.sendline("reboot") # Since reboot won't return to a shell (could also be sent using SSH)
    strategy.invalidate()
    strategy.transition("shell")
    shell_command.run_check("true")


def test_post_network(ssh_command):
    # This test should no longer fail
    ssh_command.run_check("true")

[JoshuaWatt]

Thanks! I think the piece I was missing was that it's somewhat expected that you will need to write your own strategy to deal with this. For posterity, here is a little mix-in class I wrote that you can inherit from your strategy that will allow you to reboot a device using one of several methods:

import enum

from labgrid import target_factory, step
from pexpect import TIMEOUT


class RebootMethod(enum.Enum):
    immediate = 0  # Immediately cycle power
    halt = 1  # Run the halt command and cycle power when finished
    reboot = 2  # Run the reboot command
    none = 3  # Assume a reboot is already in progress and wait for it to finish


class RebootMixIn(object):
    @step(args=["method"])
    def reboot(self, *, method=RebootMethod.immediate, timeout=60):
        if not isinstance(method, RebootMethod):
            method = RebootMethod[method]

        if method == RebootMethod.immediate:
            self.transition("off")
        elif method == RebootMethod.halt:
            self.console.sendline("halt")
            index, _, _, _ = self.console.expect(
                [r"Starting[^\n\r]+Halt", TIMEOUT], timeout=timeout
            )
            self.transition("off")
        elif method == RebootMethod.reboot:
            self.console.sendline("reboot")
            self.force("off")
        else:
            self.force("off")

        self.invalidate()

        self.target.activate(self.power)
        self.transition("shell")

    def invalidate(self):
        self.target.deactivate_all_drivers()