NUnit 3 Visual Studio Test Adapter

The adapter operates in four distinct scenarios depending on whether the caller is the command line (dotnet test) or an IDE (like Visual Studio), and whether a filter is applied.

Execution Scenarios

Key Concept for IDE Users: When using an IDE, Visual Studio handles the filtering itself and passes only the matching TestCase objects to the adapter. The adapter does not see the original VS filter expression in this path; it simply receives the resulting subset of tests.

Example (1 fixture, 3 tests: Test1, Test2, Test3):

- A: `dotnet test` -> no filter
- B: `dotnet test --filter "FullyQualifiedName=MyNs.MyFixture.Test1"` -> FQN filter, 1 test
- C: VS Test Explorer -> Run All Tests -> all 3 test cases passed to adapter
- D: VS Test Explorer -> right-click Test1 -> Run -> only Test1 passed to adapter

The adapter behaves differently depending on whether you are running via the command line (dotnet test) or through an IDE (Visual Studio).

dotnet test (Command Line)

• No filter provided: The adapter sends TestFilter.Empty, running all discovered tests.
• With --filter: The adapter parses the VS filter expression into NUnit XML. For example, FullyQualifiedName=MyNs.MyFixture.Test1 is converted to <filter><test>MyNs.MyFixture.Test1</test></filter>. Note that the filter may be rebuilt during execution to match only the tests actually discovered by the engine.

IDE (Visual Studio Test Explorer)

• Run All: The IDE passes all discovered TestCase objects to the adapter. The adapter converts these into an <or> list of <test> elements containing their Full Qualified Names.
• Filtered Run (Search/Selection): When you use the search box or select specific tests in the IDE, Visual Studio resolves those filters to a concrete set of TestCase objects before calling the adapter. Consequently, the adapter receives a list of specific test cases and converts them into a list of <test> elements. The adapter does not see the original category or search expressions in this mode; it only sees the resulting list of FQNs.

The adapter uses a pre-execution discovery phase (NUnitEngineAdapter.Explore) to determine which tests are runnable. The results of this discovery are used to refine the final filter passed to the NUnit engine.

Filter Rebuilding Logic

The adapter's CheckFilter method decides whether to pass the original filter through or rebuild it based on the filter type:

1. Category Filters (<cat>): Detected via IsCategoryFilter(). These are passed through to the engine unchanged. Discovery is only used to potentially add an explicit-exclusion clause if discovery.HasExplicitTests is true.
2. Partition Filters (<partition>): Detected via IsPartitionFilter(). These are passed through unchanged and discovery is not consulted.
3. FQN / Name / Property Filters: These do not contain <cat> or <partition> tags. They are always rebuilt as a FilterByList using the test cases found during the discovery phase (discovery.LoadedTestCases).

Summary Table

The NUnit 3 VS Test Adapter determines its execution mode using a RunType enum. This mode dictates which execution class is used and how filters are handled. The mode is primarily determined by whether the caller provides a list of test sources (like assembly paths) or specific TestCase objects.

Execution Mapping

Common Caller Scenarios

• dotnet test (with or without --filter): Uses RunType.CommandLineCurrentNUnit via the RunTests(sources) entry point.
• vstest.console assembly.dll: Uses RunType.CommandLineCurrentNUnit via the RunTests(sources) entry point.
• vstest.console /Tests:Name1,Name2: Uses RunType.Ide via the RunTests(testCases) entry point (after running discovery).
• VS Test Explorer (Run All / Selected): Uses RunType.Ide via the RunTests(testCases) entry point.

/* Summary of RunType mapping for developers debugging execution flow */
// RunType.Ide is used when specific TestCases are provided (e.g. by VS or vstest.console /Tests:)
// RunType.CommandLineCurrentNUnit is used when sources are provided (e.g. dotnet test)

If the TestNodeUpdateMessage constructor is inconsistent across different versions of the testing platform (e.g., varying between .NET 5, .NET 8, and .NET 9), use reflection to dynamically resolve and invoke the correct constructor.

This is particularly useful when the exact type name or constructor signature is unknown at compile time due to assembly version differences in the workspace.

Steps to implement reflection-based creation:

1. Expand Type Search: Search through a list of known potential type names (e.g., Microsoft.Testing.Platform.Messages.TestNodeUpdateMessage) and iterate through loaded assemblies if direct resolution fails.
2. Inspect Constructors: Use GetConstructors() to find available signatures.
3. Match Parameters: Identify constructors that accept SessionUid and either a TestNode or a TestNodeUpdate object.
4. Invoke and Publish: Use constructor.Invoke to create the message and _messageBus.PublishAsync to send it.

private async Task<bool> CreateTestNodeUpdateMessageWithReflection(IDataProducer dataProducer, TestNode testNode, string testName)
{
    var messageTypeNames = new[] 
    {
        "Microsoft.Testing.Platform.Extensions.Messages.TestNodeUpdateMessage",
        "Microsoft.Testing.Platform.Messages.TestNodeUpdateMessage", 
        "Microsoft.Testing.Extensions.Messages.TestNodeUpdateMessage",
        "Microsoft.Testing.Platform.Extensions.TestNodeUpdateMessage",
        "Microsoft.Testing.Platform.TestNodeUpdateMessage",
        "Microsoft.Testing.TestNodeUpdateMessage"
    };

    Type? messageType = null;
    foreach (var typeName in messageTypeNames)
    {
        messageType = Type.GetType(typeName);
        if (messageType != null) break;
    }

    if (messageType == null)
    {
        // Fallback: Search loaded assemblies
        foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies())
        {
            foreach (var typeName in messageTypeNames)
            {
                messageType = assembly.GetType(typeName.Split('.').Last());
                if (messageType != null) break;
            }
            if (messageType != null) break;
        }
    }

    if (messageType == null) return false;

    var constructors = messageType.GetConstructors();
    foreach (var constructor in constructors)
    {
        var parameters = constructor.GetParameters();
        try
        {
            object? message = null;
            if (parameters.Length == 2 && parameters[0].ParameterType.Name.Contains("SessionUid"))
            {
                if (parameters[1].ParameterType == typeof(TestNode))
                {
                    message = constructor.Invoke(new object[] { _sessionUid, testNode });
                }
                else if (parameters[1].ParameterType.Name.Contains("TestNodeUpdate"))
                {
                    var testNodeUpdate = CreateTestNodeUpdate(parameters[1].ParameterType, testNode);
                    if (testNodeUpdate != null)
                    {
                        message = constructor.Invoke(new object[] { _sessionUid, testNodeUpdate });
                    }
                }
            }

            if (message != null)
            {
                await _messageBus.PublishAsync(dataProducer, (IData)message);
                return true;
            }
        }
        catch { continue; }
    }
    return false;
}

To ensure compatibility across different .NET runtimes, use conditional compilation (#if) to select the appropriate message creation logic. Different frameworks (e.g., .NET Framework vs .NET 9) may have different constructor requirements for TestNodeUpdateMessage or require fallback to TestNodeStateChangedMessage.

Common patterns by framework:

• .NET 9+: Often supports an enhanced constructor accepting both SessionUid and a TestNodeUpdate object.
• .NET Framework (4.6.2+): May require a simpler TestNodeUpdateMessage constructor or a fallback to TestNodeStateChangedMessage with status set to "Cancelled" or "Skipped".

private async Task<bool> CreateTestCompletionMessage(IDataProducer dataProducer, TestNode testNode, string testName)
{
#if NET9_0
    return await CreateTestMessageForNet9(dataProducer, testNode, testName);
#elif NET8_0
    return await CreateTestMessageForNet8(dataProducer, testNode, testName);
#elif NET48 || NET472 || NET462
    return await CreateTestMessageForFramework(dataProducer, testNode, testName);
#elif NETSTANDARD2_0
    return await CreateTestMessageForStandard(dataProducer, testNode, testName);
#else
    return await CreateTestMessageFallback(dataProducer, testNode, testName);
#endif
}

// Example fallback for .NET Framework
private async Task<bool> CreateTestMessageForFramework(IDataProducer dataProducer, TestNode testNode, string testName)
{
    try
    {
        var message = new TestNodeUpdateMessage(_sessionUid, testNode);
        await _messageBus.PublishAsync(dataProducer, message);
        return true;
    }
    catch
    {
        try
        {
            var stateMessage = new TestNodeStateChangedMessage(_sessionUid, testNode, "Skipped", "Cancelled");
            await _messageBus.PublishAsync(dataProducer, stateMessage);
            return true;
        }
        catch { return false; }
    }
}

To prevent interference with child test processes, use an environment variable filtering pattern. This involves creating a clean ProcessStartInfo object and selectively copying environment variables from the host, while skipping specific diagnostic, telemetry, or hot-reload variables that might cause instability or performance issues.

Variables to skip:

• TESTINGPLATFORM_DIAGNOSTIC
• DOTNET_DbgEnableMiniDump
• DOTNET_CreateDumpDiagnostics
• DOTNET_CLI_TELEMETRY_OPTOUT
• TESTINGPLATFORM_TELEMETRY_OPTOUT
• TESTINGPLATFORM_HOTRELOAD_ENABLED

public static readonly string[] EnvironmentVariablesToSkip = 
[
    "TESTINGPLATFORM_DIAGNOSTIC",
    "DOTNET_DbgEnableMiniDump",
    "DOTNET_CreateDumpDiagnostics",
    "DOTNET_CLI_TELEMETRY_OPTOUT",
    "TESTINGPLATFORM_TELEMETRY_OPTOUT",
    "TESTINGPLATFORM_HOTRELOAD_ENABLED",
];

private static ProcessStartInfo CreateCleanProcessStartInfo(string executable, string arguments)
{
    var startInfo = new ProcessStartInfo(executable, arguments)
    {
        UseShellExecute = false,
        CreateNoWindow = true,
        EnableRaisingEvents = true
    };

    foreach (DictionaryEntry entry in Environment.GetEnvironmentVariables())
    {
        string? key = entry.Key?.ToString();
        if (key != null && !EnvironmentVariablesToSkip.Contains(key, StringComparer.OrdinalIgnoreCase))
        {
            startInfo.EnvironmentVariables[key] = entry.Value?.ToString() ?? string.Empty;
        }
    }

    return startInfo;
}

When implementing or extending the bridge, use an asynchronous disposal pattern that prioritizes session completion over resource disposal. The sequence should be:

1. Cancel the CancellationTokenSource.
2. Execute the forced shutdown logic (sending session end events and flushing the bus).
3. Dispose of the TestFramework resources.
4. Stop the underlying process if it is still running.
5. Finally, dispose of the process object.

protected override async ValueTask DisposeAsyncCore()
{
    try
    {
        // Cancel all running operations
        _cancellationTokenSource?.Cancel();
        
        // Critical: Complete test session before disposal
        await HandleForcedShutdownAsync(CancellationToken.None);
        
        // Clean up VSTest framework resources
        _testFramework?.Dispose();
        
        // Clean up process resources
        if (_process?.HasExited == false)
        {
            await StopAsync(TimeSpan.FromSeconds(5));
        }
    }
    catch (Exception ex)
    {
        _logger.LogError("Error during bridge disposal: {Exception}", ex);
    }
    finally
    {
        _process?.Dispose();
    }
}

When running NUnit tests in parallel via VSTestBridge, cancellation requests may cause the test process to hang if cancellation isn't properly propagated. To prevent hangs, implement a two-stage shutdown: first, attempt a graceful shutdown using a CancellationTokenSource with a timeout, and if the process fails to exit within that window, force termination of the entire process tree.

Implementation Strategy

1. Graceful Stage: Use RequestGracefulShutdown and wait for the process to exit using WaitForExitAsync (on .NET 5+) or a TaskCompletionSource (on older frameworks).
2. Force Stage: If the graceful stage fails or times out, use _process.Kill(entireProcessTree: true) on .NET 5+ to ensure all parallel test threads and child processes are terminated.

public async Task<int> StopAsync(CancellationToken cancellationToken = default)
{
    using var combinedCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
    combinedCts.CancelAfter(TimeSpan.FromSeconds(30)); // Graceful shutdown timeout

    try
    {
        // First: Try graceful cancellation
        await RequestGracefulShutdown(combinedCts.Token);
        
        // Wait for process to exit gracefully
#if NET5_0_OR_GREATER
        if (await _process.WaitForExitAsync(combinedCts.Token))
#else
        var tcs = new TaskCompletionSource<bool>();
        combinedCts.Token.Register(() => tcs.TrySetCanceled());
        
        _process.EnableRaisingEvents = true;
        _process.Exited += (s, e) => tcs.TrySetResult(true);
        
        if (!_process.HasExited && await tcs.Task)
#endif
        {
            return _process.ExitCode;
        }
    }
    catch (OperationCanceledException)
    {
        // Graceful shutdown timed out or was cancelled
    }

    // Second: Force termination if graceful shutdown failed
    try
    {
#if NET5_0_OR_GREATER
        _process.Kill(entireProcessTree: true); // Kill entire process tree for parallel scenarios
        await _process.WaitForExitAsync(CancellationToken.None);
#else
        _process.Kill();
        _process.WaitForExit();
#endif
        return _process.ExitCode;
    }
    catch (Exception ex)
    {
        // Log the force termination
        _logger.LogWarning("Had to force terminate test process: {Exception}", ex);
        return -1;
    }
}