[test][NFC] Change from nose to pytest (analyzer library)

-> Motivation, why pytest

Nose is no longer supported, it is not compatible with python3.9, and
will never be:
nose-devs/nose#1099 (comment)

As python3.10 is the oldest python version available from
Ubuntu22.04's package manager by default, it is time to migrate.

We never really relied on many nose features (as far as I can tell, we
never actually import anything from the nose package), this change isn't
particularly painful (don't let the size of the change fool you, its
pretty much just moving code around!). Pytest seems to be the industry
standard framework lately, and supports most nose constructs out of the box.

The aim of this change that from the make target side, nothing changed
-- you still analyzer tests with 'make analyzer', a specific test file
with (having 'functional/skip' as the example):

'TEST=tests/functional/skip make test_analyzer_feature'

This is basically how granular nose could get. If you wanted to run a
specific test case in a test file, you couldn't do that, but you can
with pytest:

pytest analyzer/tests/functional/skip -v -k test_analyze_header_with_file_option

Mind that there are many environmental variables that needs to be set
in addition to the above command, so we need to write new make targets
to make this user-friendly, but nevertheless, it is possible.

-> What changed

Broadly speaking, the change can be divided into 4 parts:

1. Replacing nose with pytest in the makefiles
2. Replacing nose config files with pytest files
3. Replace `setup_package`/`teardown_package` with
   `setup_class`/`teardown_class`, and `setup`/`teardown` to
   `setup_method` and `teardown_method`.
4. Fix up individual test environments under `analyzer/tools`.

For the Makefile and config changes, I hope they are self explanatory.
Pytest is a rather painless drop-in replacement on the invocation side.

On the conversion, there is a page that discusses how one can convert
nose to pytest:
https://docs.pytest.org/en/7.1.x/how-to/nose.html
It is stated (and is true) that pytest supports most, but not quite all
features in nose. `{setup, teardown}_package` is not supported, but it
turns out that we dedicate a package to every test class, so simply
switching to `{setup, teardown}_class` was sufficient. That accounts for
the vast majority of the code change. That is the only meaningful
structural change -- `setup`->`setup_method` and the teardown variant
was really just a simple rename.

At last, you can notice that some `__init__.py` files are copied over
from `analyzer/test/__init__.py`. Frankly, I'm not sure how the tests
worked previously without these files setting up these variables
properly, but this skeleton fell out of the closet now.

analyzer/pytest.ini

@@ -0,0 +1,13 @@
+[pytest]
+
+addopts =
+# increase verbosity level
+  --verbose
+
+# stop running tests on first error
+# FIXME: Pretty please comment this horrible setting out, I hate it with a
+# passion
+#  --maxfail=1
+
+# do not capture stdout
+  --capture=sys

analyzer/requirements_py/dev/requirements.txt

@@ -1,5 +1,5 @@
 lxml==4.9.2
-nose==1.3.7
+pytest==7.3.1
 pycodestyle==2.7.0
 psutil==5.8.0
 portalocker==2.2.1

analyzer/tests/Makefile

@@ -6,8 +6,8 @@ PYTHON_BIN ?= python3
 
 REPO_ROOT ?= REPO_ROOT=$(ROOT)
 
-# Nose test runner configuration options.
-NOSECFG = --config .noserc
+# pytest test runner configuration options.
+PYTESTCFG = -c pytest.ini
 
 test_in_env: pycodestyle_in_env pylint_in_env test_unit_in_env test_functional_in_env test_build_logger test_tu_collector_in_env
 
@@ -33,14 +33,14 @@ pylint_in_env: venv_dev
 
 run_test:
 	$(REPO_ROOT) $(TEST_PROJECT) \
-		nosetests $(NOSECFG) ${TEST} || exit 1
+		pytest $(PYTESTCFG) ${TEST} || exit 1
 
 run_test_in_env: venv_dev
 	$(ACTIVATE_DEV_VENV) && $(REPO_ROOT) $(TEST_PROJECT) \
-		nosetests $(NOSECFG) ${TEST} || exit 1
+		pytest $(PYTESTCFG) ${TEST} || exit 1
 
-UNIT_TEST_CMD = $(REPO_ROOT) nosetests $(NOSECFG) tests/unit
-UNIT_TEST_COV_CMD = $(REPO_ROOT) coverage run -m nose $(NOSECFG) tests/unit && coverage report && coverage html
+UNIT_TEST_CMD = $(REPO_ROOT) pytest $(PYTESTCFG) tests/unit
+UNIT_TEST_COV_CMD = $(REPO_ROOT) coverage run -m pytest $(PYTESTCFG) tests/unit && coverage report && coverage html
 
 test_unit:
 	$(UNIT_TEST_CMD)
@@ -55,7 +55,7 @@ test_unit_cov_in_env: venv_dev
 	$(ACTIVATE_DEV_VENV) && $(UNIT_TEST_COV_CMD)
 
 FUNCTIONAL_TEST_CMD = $(REPO_ROOT) $(TEST_PROJECT) \
-		nosetests $(NOSECFG) tests/functional || exit 1
+		pytest $(PYTESTCFG) tests/functional || exit 1
 
 test_functional:
 	${PYTHON_BIN} $(ROOT)/scripts/test/check_clang.py || exit 1;

analyzer/tests/functional/analyze/__init__.py

@@ -7,38 +7,5 @@
 #
 # -------------------------------------------------------------------------
 
-"""Setup for the test package analyze.
-"""
-
-
-import os
-import shutil
-
-from libtest import env
-
-
-# Test workspace should be initialized in this module.
-TEST_WORKSPACE = None
-
-
-def setup_package():
-    """Setup the environment for the tests."""
-
-    global TEST_WORKSPACE
-    TEST_WORKSPACE = env.get_workspace('analyze')
-
-    report_dir = os.path.join(TEST_WORKSPACE, 'reports')
-    os.makedirs(report_dir)
-
-    os.environ['TEST_WORKSPACE'] = TEST_WORKSPACE
-
-
-def teardown_package():
-    """Delete the workspace associated with this test"""
-
-    # TODO: If environment variable is set keep the workspace
-    # and print out the path.
-    global TEST_WORKSPACE
-
-    print("Removing: " + TEST_WORKSPACE)
-    shutil.rmtree(TEST_WORKSPACE)
+# This file is empty, and is only present so that this directory will form a
+# package.

analyzer/tests/functional/analyze/test_analyze.py

@@ -29,9 +29,26 @@
 class TestAnalyze(unittest.TestCase):
     _ccClient = None
 
-    def setUp(self):
+    def setup_class(self):
+        """Setup the environment for the tests."""
+
+        global TEST_WORKSPACE
+        TEST_WORKSPACE = env.get_workspace('analyze')
+
+        report_dir = os.path.join(TEST_WORKSPACE, 'reports')
+        os.makedirs(report_dir)
+
+        os.environ['TEST_WORKSPACE'] = TEST_WORKSPACE
+
+    def teardown_class(self):
+        """Delete the workspace associated with this test"""
+
+        print("Removing: " + TEST_WORKSPACE)
+        shutil.rmtree(TEST_WORKSPACE)
+
+    def setup_method(self, method):
+        """Setup the environment for the tests."""
 
-        # TEST_WORKSPACE is automatically set by test package __init__.py .
         self.test_workspace = os.environ['TEST_WORKSPACE']
 
         test_class = self.__class__.__name__
@@ -51,7 +68,7 @@ def setUp(self):
         self.disabling_modeling_checker_regex = re.compile(
             r"analyzer-disable-checker=.*unix.cstring.CStringModeling.*")
 
-    def tearDown(self):
+    def teardown_method(self, method):
         """Restore environment after tests have ran."""
         os.chdir(self.__old_pwd)
         if os.path.isdir(self.report_dir):
@@ -1105,11 +1122,9 @@ def test_analyzer_and_checker_config(self):
 
         # It's printed as a found report and in the checker statistics.
         # Note: If this test case fails, its pretty sure that something totally
-        # unrelated to the analysis broke in CodeChecker. Comment out the line
-        # starting with 'nocapture' in 'analyzer/.noserc', and print both the
+        # unrelated to the analysis broke in CodeChecker. Print both the
         # stdout and stderr streams from the above communicate() call (the
-        # latter of which is ignored with _ above)
-        # Put a + if the above instructions saved you: +
+        # latter of which is ignored with _ above).
         self.assertEqual(out.count('hicpp-use-nullptr'), 2)
 
         analyze_cmd = [self._codechecker_cmd, "check", "-l", build_json,

analyzer/tests/functional/analyze_and_parse/__init__.py

@@ -7,32 +7,5 @@
 #
 # -------------------------------------------------------------------------
 
-"""Setup for the test package 'analyze' and 'parse'."""
-
-
-import os
-import shutil
-
-from libtest import env
-
-TEST_WORKSPACE = None
-
-
-def setup_package():
-    """Setup the environment for the tests."""
-
-    global TEST_WORKSPACE
-    TEST_WORKSPACE = env.get_workspace('analyze_and_parse')
-
-    os.environ['TEST_WORKSPACE'] = TEST_WORKSPACE
-
-
-def teardown_package():
-    """Delete the workspace associated with this test"""
-
-    # TODO: If environment variable is set keep the workspace
-    # and print out the path.
-    global TEST_WORKSPACE
-
-    print("Removing: " + TEST_WORKSPACE)
-    shutil.rmtree(TEST_WORKSPACE)
+# This file is empty, and is only present so that this directory will form a
+# package.

analyzer/tests/functional/analyze_and_parse/test_analyze_and_parse.py

@@ -30,6 +30,7 @@
 
 
 class AnalyzeParseTestCaseMeta(type):
+
     def __new__(mcs, name, bases, test_dict):
 
         def gen_test(path, mode):
@@ -64,7 +65,11 @@ class AnalyzeParseTestCase(
     def setup_class(cls):
         """Setup the class."""
 
-        # TEST_WORKSPACE is automatically set by test package __init__.py
+        global TEST_WORKSPACE
+        TEST_WORKSPACE = env.get_workspace('analyze_and_parse')
+
+        os.environ['TEST_WORKSPACE'] = TEST_WORKSPACE
+
         test_workspace = os.environ['TEST_WORKSPACE']
         cls.test_workspaces = {'NORMAL': os.path.join(test_workspace,
                                                       'NORMAL'),
@@ -108,7 +113,14 @@ def teardown_class(cls):
         """Restore environment after tests have ran."""
         os.chdir(cls.__old_pwd)
 
-    def tearDown(self):
+        # TODO: If environment variable is set keep the workspace
+        # and print out the path.
+        global TEST_WORKSPACE
+
+        print("Removing: " + TEST_WORKSPACE)
+        shutil.rmtree(TEST_WORKSPACE)
+
+    def teardown_method(self, method):
         """Restore environment after a particular test has run."""
         output_dir = AnalyzeParseTestCase.test_workspaces['OUTPUT']
         if os.path.isdir(output_dir):

analyzer/tests/functional/cmdline/__init__.py

@@ -7,35 +7,5 @@
 #
 # -------------------------------------------------------------------------
 
-"""Setup for the test package analyze.
-"""
-
-
-import os
-import shutil
-
-from libtest import env
-
-
-# Test workspace should be initialized in this module.
-TEST_WORKSPACE = None
-
-
-def setup_package():
-    """Setup the environment for the tests."""
-
-    global TEST_WORKSPACE
-    TEST_WORKSPACE = env.get_workspace('cmdline')
-
-    os.environ['TEST_WORKSPACE'] = TEST_WORKSPACE
-
-
-def teardown_package():
-    """Delete the workspace associated with this test"""
-
-    # TODO: If environment variable is set keep the workspace
-    # and print out the path.
-    global TEST_WORKSPACE
-
-    print("Removing: " + TEST_WORKSPACE)
-    shutil.rmtree(TEST_WORKSPACE)
+# This file is empty, and is only present so that this directory will form a
+# package.

analyzer/tests/functional/cmdline/test_cmdline.py

@@ -12,6 +12,7 @@
 
 import json
 import os
+import shutil
 import subprocess
 import unittest
 
@@ -38,7 +39,25 @@ class TestCmdline(unittest.TestCase):
     Simple tests to check CodeChecker command line.
     """
 
-    def setUp(self):
+    def setup_class(self):
+        """Setup the environment for the tests."""
+
+        global TEST_WORKSPACE
+        TEST_WORKSPACE = env.get_workspace('cmdline')
+
+        os.environ['TEST_WORKSPACE'] = TEST_WORKSPACE
+
+    def teardown_class(self):
+        """Delete the workspace associated with this test"""
+
+        # TODO: If environment variable is set keep the workspace
+        # and print out the path.
+        global TEST_WORKSPACE
+
+        print("Removing: " + TEST_WORKSPACE)
+        shutil.rmtree(TEST_WORKSPACE)
+
+    def setup_method(self, method):
         # TEST_WORKSPACE is automatically set by test package __init__.py .
         self.test_workspace = os.environ['TEST_WORKSPACE']
 

analyzer/tests/functional/config/__init__.py

@@ -7,35 +7,5 @@
 #
 # -------------------------------------------------------------------------
 
-"""Setup for the config test for the analyze command.
-"""
-
-
-import os
-import shutil
-
-from libtest import env
-
-
-# Test workspace should be initialized in this module.
-TEST_WORKSPACE = None
-
-
-def setup_package():
-    """Setup the environment for the tests."""
-
-    global TEST_WORKSPACE
-    TEST_WORKSPACE = env.get_workspace('config')
-
-    os.environ['TEST_WORKSPACE'] = TEST_WORKSPACE
-
-
-def teardown_package():
-    """Delete the workspace associated with this test"""
-
-    # TODO: If environment variable is set keep the workspace
-    # and print out the path.
-    global TEST_WORKSPACE
-
-    print("Removing: " + TEST_WORKSPACE)
-    shutil.rmtree(TEST_WORKSPACE)
+# This file is empty, and is only present so that this directory will form a
+# package.

analyzer/tests/functional/config/test_config.py

@@ -14,6 +14,7 @@
 
 import json
 import os
+import shutil
 import subprocess
 import unittest
 
@@ -23,7 +24,25 @@
 class TestConfig(unittest.TestCase):
     _ccClient = None
 
-    def setUp(self):
+    def setup_class(self):
+        """Setup the environment for the tests."""
+
+        global TEST_WORKSPACE
+        TEST_WORKSPACE = env.get_workspace('config')
+
+        os.environ['TEST_WORKSPACE'] = TEST_WORKSPACE
+
+    def teardown_class(self):
+        """Delete the workspace associated with this test"""
+
+        # TODO: If environment variable is set keep the workspace
+        # and print out the path.
+        global TEST_WORKSPACE
+
+        print("Removing: " + TEST_WORKSPACE)
+        shutil.rmtree(TEST_WORKSPACE)
+
+    def setup_method(self, method):
 
         # TEST_WORKSPACE is automatically set by test package __init__.py .
         self.test_workspace = os.environ['TEST_WORKSPACE']