Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Tests are reported as completed when GTEST_SKIP is used inside an environment's SetUp #4653

Open
lighterowl opened this issue Nov 5, 2024 · 0 comments

Comments

@lighterowl
Copy link

Describe the issue

When adding a global test environment whose SetUp function calls GTEST_SKIP, the actual tests are still reported as executed in the test program's output, and as completed in the XML output.

Running the attached source results in the following output :

$ ./foobar --gtest_output=xml:output.xml
[==========] Running 2 tests from 2 test suites.
[----------] Global test environment set-up.
/home/daniel/zabawki/poligon5/main.cpp:8: Skipped
skipping everything

skipping everything

[----------] Global test environment tear-down
[==========] 2 tests from 2 test suites ran. (0 ms total)
[  PASSED  ] 2 tests.

Meanwhile, the XML reports that the test results were completed and that the execution of each test took place at "time zero", though the root test suite's timestamp is set correctly :

<testsuites tests="2" failures="0" disabled="0" errors="0" time="0." timestamp="2024-11-05T19:48:23.647" name="AllTests">
  <testsuite name="Foo" tests="1" failures="0" disabled="0" skipped="0" errors="0" time="0." timestamp="1970-01-01T01:00:00.000">
    <testcase name="Bar" file="/home/daniel/zabawki/poligon5/main.cpp" line="12" status="run" result="completed" time="0." timestamp="1970-01-01T01:00:00.000" classname="Foo" />
  </testsuite>
  <testsuite name="Bar" tests="1" failures="0" disabled="0" skipped="0" errors="0" time="0." timestamp="1970-01-01T01:00:00.000">
    <testcase name="Foo" file="/home/daniel/zabawki/poligon5/main.cpp" line="17" status="run" result="completed" time="0." timestamp="1970-01-01T01:00:00.000" classname="Bar" />
  </testsuite>
</testsuites>

I expected the usage of a GTEST_SKIP in a global test environment to be equivalent to a GTEST_SKIP in every single test case, and the output to be the following :

$ ./foobar 
[==========] Running 2 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 1 test from Foo
[ RUN      ] Foo.Bar
/home/daniel/zabawki/poligon5/main.cpp:13: Skipped
foobar

[  SKIPPED ] Foo.Bar (0 ms)
[----------] 1 test from Foo (0 ms total)

[----------] 1 test from Bar
[ RUN      ] Bar.Foo
/home/daniel/zabawki/poligon5/main.cpp:18: Skipped
barfoo

[  SKIPPED ] Bar.Foo (0 ms)
[----------] 1 test from Bar (0 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 2 test suites ran. (0 ms total)
[  PASSED  ] 0 tests.
[  SKIPPED ] 2 tests, listed below:
[  SKIPPED ] Foo.Bar
[  SKIPPED ] Bar.Foo

In which case, the XML output also includes appropriate elements to indicate that test execution was skipped (and the timestamps are correct, too) :

<testsuites tests="2" failures="0" disabled="0" errors="0" time="0." timestamp="2024-11-05T19:58:05.464" name="AllTests">
  <testsuite name="Foo" tests="1" failures="0" disabled="0" skipped="1" errors="0" time="0." timestamp="2024-11-05T19:58:05.464">
    <testcase name="Bar" file="/home/daniel/zabawki/poligon5/main.cpp" line="11" status="run" result="skipped" time="0." timestamp="2024-11-05T19:58:05.464" classname="Foo">
      <skipped message="/home/daniel/zabawki/poligon5/main.cpp:13&#x0A;foobar&#x0A;"><![CDATA[/home/daniel/zabawki/poligon5/main.cpp:13
foobar
]]></skipped>
    </testcase>
  </testsuite>
  <testsuite name="Bar" tests="1" failures="0" disabled="0" skipped="1" errors="0" time="0." timestamp="2024-11-05T19:58:05.464">
    <testcase name="Foo" file="/home/daniel/zabawki/poligon5/main.cpp" line="16" status="run" result="skipped" time="0." timestamp="2024-11-05T19:58:05.464" classname="Bar">
      <skipped message="/home/daniel/zabawki/poligon5/main.cpp:18&#x0A;barfoo&#x0A;"><![CDATA[/home/daniel/zabawki/poligon5/main.cpp:18
barfoo
]]></skipped>
    </testcase>
  </testsuite>
</testsuites>

Steps to reproduce the problem

#include <gtest/gtest.h>

struct Env : public ::testing::Environment
{
    void SetUp() override
    {
        GTEST_SKIP() << "skipping everything";
    }
};

TEST(Foo, Bar)
{
    ASSERT_EQ(1, 2);
}

TEST(Bar, Foo)
{
    ASSERT_EQ(2, 1);
}

int main(int argc, char **argv)
{
    ::testing::InitGoogleTest(&argc, argv);
    ::testing::AddGlobalTestEnvironment(new Env);
    return RUN_ALL_TESTS();
}

What version of GoogleTest are you using?

d144031940543e15423a25ae5a8a74141044862f

What operating system and version are you using?

Arch Linux

What compiler and version are you using?

gcc version 14.2.1 20240910

What build system are you using?

cmake version 3.30.5

Additional context

No response

@lighterowl lighterowl changed the title [Bug]: Tests are reported as completed when GTEST_SKIP is used inside a environment's SetUp [Bug]: Tests are reported as completed when GTEST_SKIP is used inside an environment's SetUp Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant