Generate a tests report

[le-yams]

Hello,

Having the possibility to generate a test report would be great (especially to publish them in CIs).
No sure which format to use. JUnit is the most commonly used and a lot of tools supports it but ... xml 😢.
The testing tools generating json reports seems to all use a custom format 😞.
Looking into this I came across this initiative that is very new but could be interesting.

What do you think about it?

Regards,
Yann

[homorozeanu]

Agree with @le-yams . We test our model using the openfga cli in azure devops pipeline and would love to be able to export it in a format that is understood by azure devops. Hence, +1.

[aaguiarz]

@homorozeanu @le-yams would using https://github.com/ctrf-io work for you?

[homorozeanu]

@aaguiarz Unfortunately from what I know neither Azure DevOps nor NUnit (the testing framework that we use for our dotnet tests) understand and can export/import test results in CTRF format.

[yann-disanto]

@aaguiarz it would be ok on our side (a github action to publish the test results is being written so it will fit our needs).

As I said, the downside of this choice is that, as it's pretty new, this format is not yet well supported by most of CI tools. But this downside would be also true if we use a "yet another home made test report fornat".
This downside could be mitigated by the possibility for people to write ctrf to supported format transformer.

I would be interested in others openfga cli core-team developers opinion, @rhamzeh any thought about this?

[ewanharris]

As you might have seen @aaguiarz mention, last week we had a hackathon at Okta. I decided to spend some time working on integrating Cucumber (via godog) into the fga model test command to enable testing to live outside the store files. What's fun about this is that, given it's relatively easy to generate the Cucumber tests from a yaml file, we can use Cucumber as our test runner for those too and benefit from the reporters it offers, such as junit.

If you're curious, the work is at https://github.com/openfga/cli/tree/feat/support-gherkin-syntax (I'll be splitting this into multiple PRs eventually) and you can install it using go install github.com/openfga/cli/cmd/fga@feat/support-gherkin-syntax. Running your existing store based tests using fga model test --tests store.fga.yaml --reporter junit will print a junit report to the console with the test results. I've tested it against the store files in https://github.com/openfga/sample-stores but would be great to get any feedback from yourselves around the functionality.

There's also an example of the Gherkin syntax here that demonstrates the FolderBox store and ABAC store tests converted into the gherkin syntax

[yann-disanto]

Hello @ewanharris,

Is it planned to go further with the gherkin syntax support?

[ewanharris]

Hey @yann-disanto, sorry for the delay! Yes I think it's something that we'd like to continue with, although I don't really have a timeline for when.

We'd most likely release the gherkin syntax work initially and then at a later date switch to using it to as the "backend" for the yaml tests once we're confident in the syntax, my focus has been elsewhere the last few weeks but I'll see if I can set some time aside soon to do that.