F# support

[KeterSCP]

Project description states that this is a

.NET testing framework

But .NET != C#
Are there any plans to support F# projects?

[thomhurst]

Hey @KeterSCP . In all honesty I've never used F# but if it all compiles down to the same IL code, is there any reason this doesnt work with F#?

Thats a genuine question. If changes or tweaks need to be made please let me know.

A reproduction of any issues would be helpful, and if you want to help solve any challenges I'm open to that as like I said I'm not too versed in F#

[KeterSCP]

Hi @thomhurst! Thanks for the quick reply (and also nice job with this project!)

So here are the problems I encountered while trying minimal example in F# (it doesn't even assert anything, just returns 0):

module Tests

open TUnit.Core

[<Test>]
let ``My test`` () = 0

1. The test is not visible in the Tests tab in Rider (Testing platform support is enabled)
2. After running dotnet test from the CLI, build fails:

dotnet test
Restore complete (0.2s)
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  TestProject2 succeeded (1.3s) → bin\Debug\net8.0\TestProject2.dll
  TestProject2 test failed with 1 error(s) (0.2s)
    ...\TestProject2\bin\Debug\net8.0\TestProject2.dll : error run failed: Tests failed: '...\TestProject2\bin\Debug\net8.0\TestResults\TestProject2_net8.0_x64.log' [net8.0|x64]

Test summary: total: 0, failed: 0, succeeded: 0, skipped: 0, duration: 0.2s
Build failed with 1 error(s) in 2.0s

Log file contains following output:

████████╗██╗   ██╗███╗   ██╗██╗████████╗
╚══██╔══╝██║   ██║████╗  ██║██║╚══██╔══╝
   ██║   ██║   ██║██╔██╗ ██║██║   ██║   
   ██║   ██║   ██║██║╚██╗██║██║   ██║   
   ██║   ╚██████╔╝██║ ╚████║██║   ██║   
   ╚═╝    ╚═════╝ ╚═╝  ╚═══╝╚═╝   ╚═╝   
   
   v0.1.783.0 | 64-bit | Microsoft Windows 10.0.22631 | win-x64 | .NET 8.0.8 | Microsoft Testing Platform v1.4.0
   

Test run summary: Zero tests ran - ...\TestProject2\bin\Debug\net8.0\TestProject2.dll (net8.0|x64)
  total: 0
  failed: 0
  succeeded: 0
  skipped: 0
  duration: 59ms

=== COMMAND LINE ===
C:\Program Files\dotnet\dotnet.exe exec ...\TestProject2\bin\Debug\net8.0\TestProject2.dll --internal-msbuild-node testingplatform.pipe.467200f8a0ac44d1a3488e611ee746e0 

[thomhurst]

Hmmm... Could you create a new repo with minimal repo please? Should help debug issues. I'll take a look but also might need your help if that's okay!

[KeterSCP]

Sure thing! Here is a separate repo, which demonstrates working example with XUnit and non-working example with TUnit:

https://github.com/KeterSCP/tunit-fsharp-repo

[bevanweiss]

My understanding is that TUnit relies on C# code generation.
That won't work for a F# project, because the F# project will not compile the C# generated code... it will just be random files in the F# project.

What 'should' work however, is having a dedicated C# project just for the tests that references the F# project with the logic to be tested.

I don't think that F# has support for source generation in the same way that C# has, so I don't think it's an option at this stage.
It does look like the F# source generator issue has been raised fsharp/fslang-suggestions#864, but nothing advanced on it.

[thomhurst]

My understanding is that TUnit relies on C# code generation. That won't work for a F# project, because the F# project will not compile the C# generated code... it will just be random files in the F# project.

What 'should' work however, is having a dedicated C# project just for the tests that references the F# project with the logic to be tested.

I don't think that F# has support for source generation in the same way that C# has, so I don't think it's an option at this stage. It does look like the F# source generator issue has been raised fsharp/fslang-suggestions#864, but nothing advanced on it.

Ah interesting. Does this essentially mean it's not a TUnit bug? I just (maybe naively) assumed that the nugrt packages I published would work for any .net language because it all compiles down to the same thing. If not I just need to see if I can tweak the source generator depending on theangjage

[bevanweiss]

Ah interesting. Does this essentially mean it's not a TUnit bug? I just (maybe naively) assumed that the nugrt packages I published would work for any .net language because it all compiles down to the same thing. If not I just need to see if I can tweak the source generator depending on theangjage

You potentially know more in this space than me ;)
My understanding was, the 'Core' stuff that you (@thomhurst) write in fixed C# is indeed compiled down on your side to .NET IL (unless you opt for AoT etc etc) during the nuget publishing, and is purely runnable under the .NET CLR, so is agnostic to C#/F#/VB.NET usage.
However... for the source generated aspects, the parts which are spat out C# language code files, these need to be compiled at the 'end-users' side. If they have an F# / VB.NET project environment, they might not even have the Roslyn C# compiler AT ALL... I also think that the C# source generators are only enabled in C# projects, since they kind of inject themselves into the compilation process. Although they don't have to restrict themselves to .cs files (they can access other files also... but their output is stuffed into the C# compilation chain).

So I'd say it's not a TUnit 'bug' at all. It's just one of the limitations of using the (C#) Source Generator concept.
There's potentially some ways to get around it, maybe having an MSBuild Task which separately launches the csc compiler against some boilerplate core code that then allows for the source generator to kick in and parse the F# files, to generate the final C# source which the csc invocation then builds... but it would also need to somehow link this csc result back into the F# project... so I reckon the 'bang for buck' on maintaining such a monstrosity wouldn't be there...

In an F# project, there is no C# compiler invocation, so I don't believe your source generator is even called.

https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/source-generators-overview

[thomhurst]

Thanks for the detailed description that's really helpful!

So would the solution basically be to determine whether we're in an F# solution and if so generate F# code instead of C#?

I haven't actually done any experimentation on source generators with F# but I assume it's possible to check the language. The TUnit source generation isn't too complicated, so if F# needs it's own F# generated code, then let's make that happen :)

[bevanweiss]

Thanks for the detailed description that's really helpful!

So would the solution basically be to determine whether we're in an F# solution and if so generate F# code instead of C#?

I haven't actually done any experimentation on source generators with F# but I assume it's possible to check the language. The TUnit source generation isn't too complicated, so if F# needs it's own F# generated code, then let's add that as an issue and make that happen. Would you like to create an issue for that?

Unfortunately the F# compiler doesn't support this.
In the C# Roslyn compiler they have broken the compilation process into two parts, an initial phase where it tokenizes and parses everything, and then the subsequent phase where it lowers and links everything to the IL.
This is what allows the Source Generator concept to work. The first part runs, then it hands this information (and other 'project context') across to the Source Generator, which can make additional source objects etc, which are then also tokenized and parsed. Then it proceeds to do the 'real' backend compilation bit.

I'm not well versed in F#, but I think it's also missing core language support for things like 'partial classes' that were introduced to the C# language to help support source generation.

I think the things lacking from F# in the source generator space make it problematic to support at all for something like TUnit.

Two possible options might be:

1. Use runtime reflection as a fallback, like how the other Test Frameworks do... runtime reflection obviously doesn't care about which source language was used. It's all about the pure IL at that stage. But it means all the goodness that comes with Source Generation disappears.. and it would require a whole runtime reflection engine.
2. Go the route of IL weaving instead. So rather than integrating into the compile process like with C#, have a pre/post compilation process (post is probably 'easier') that directly manipulates / emits IL to do what the C# Source Generation currently does. Which sounds a bit painful. It would use the MSBuild stuff I mentioned previously, wouldn't get inbuilt compiler support for the parsing of the F#, would need to generate IL for the parts that are currently C# source gen'd (but couldn't rely on a C# compiler), and would need to work around various restrictions from the F# compilation (i.e. you couldn't just add extra parameters to test classes, since the F# IL might have done alignment / optimisations which make them difficult to change).

Of the two options, the first one is probably 'nicer' (easier, with less maintenance hassle).

[bevanweiss]

FYI: Because both VB.NET and C# are based within the Roslyn compiler, it does appear that Source Generation might be possible for VB... dotnet/vblang#586

But F# isn't a Roslyn thing... so has some quite different interfaces.

I guess technically it would be possible to use something like these:
https://fsharp.github.io/fsharp-compiler-docs/fcs/
https://fsprojects.github.io/fantomas/docs/end-users/GeneratingCode.html

To do a pre-build trigger, parse the F# and do like a source generator for it... and then the Microsoft F# compiler would kick in and compile the 'proper' F# project (which would include all the source generated files).
I'm pretty sure it's a different source generator / parser interface than the Roslyn interface you've been using for C# though.

[thomhurst]

Oh damn this seems pretty complicated then. I guess for now I'll have to brand TUnit as C# only 😢