Expecto

Expecto allows you to assert that one implementation is faster than another using statistical testing (Welch's t-test).

Usage Patterns

1. Simple Comparison: Use Expect.isFasterThan (unit -> 'a) (unit -> 'a) string to compare two functions.
2. Setup/Teardown Comparison: Use Expect.isFasterThanSub when you need to perform setup or teardown (like resetting a buffer) before the measurement. It accepts a function with the signature Performance.Measurer<unit,'a> -> 'a.
3. Sequenced Tests: Performance tests should be wrapped in testSequenced to ensure they are handled correctly by the runner.

Important Notes

• The functions being compared must return the same result for the same input.
• The test is relative and runs quickly; it increases sample sizes until the difference is statistically significant (rejecting the null hypothesis with < 0.01% probability).
• If performance is very close, it declares them equal with 99.99% confidence if they differ by less than 0.5%.

[<Tests>]
let performance =
  testSequenced <| testList "performance" [
    testCase "half is faster" <| fun () ->
      Expect.isFasterThan (fun () -> repeat10000 log 76.0)
                          (fun () -> repeat10000 log 76.0 |> ignore; repeat10000 log 76.0)
                          "half is faster"
  ]

Expecto's printing mechanism is based on the Logary Facade. You can extend this to send notifications (e.g., via email using Mailgun) whenever tests fail. This requires initializing the Logary Facade with an Expecto logger and configuring Logary targets in your main function.

Example setup with Mailgun:

open Logary
open Logary.Configuration
open Logary.Adapters.Facade
open Logary.Targets
open Hopac
open Mailgun
open System.Net.Mail

let main argv = 
  let mgc = MailgunLogaryConf.Create(
      MailAddress("travis@example.com"),
      [ MailAddress("Your.Mail.Here@example.com") ],
      { apiKey = "deadbeef-2345678" },
      "example.com", 
      Error)

  use logary = 
    withLogaryManager "MyTests" (
      withTargets [
        LiterateConsole.create LiterateConsole.empty "stdout"
        Mailgun.create mgc "mail"
      ]
      >> withRules [
        Rule.createForTarget "stdout"
        Rule.createForTarget "mail"
      ])
    |> run

  LogaryFacadeAdapter.initialise<Expecto.Logging.Logger> logary
  Tests.runTestsInAssemblyWithCLIArgs [] argv

When upgrading to version 11.0.0, note the following breaking changes:

• Test SDK Compatibility: Expecto 11.0.0-alpha5 breaks compatibility with YoloDev.Expecto.TestSdk <= 0.14.3. You must use YoloDev.Expecto.TestSdk >= 0.15 for VisualStudio, Rider, dotnet test, and other vstest adapter systems.
• FsCheck Replay Config: Usages of the replay config (or stdGen with etestProperty* functions) must be updated to use uint64 by appending UL to literals. They also now require a third item indicating the size.
  • Old format: (1865288075, 296281834)
  • New format: (1865288075UL, 296281834UL, Some 3)
  • FsCheck 2 note: The size is ignored in FsCheck 2; you can use None (e.g., (1865288075UL, 296281834UL, None)).
• FsCheck Version: Expecto now uses FsCheck 3 by default. If you require FsCheck 2, use the versioned package [11.0.0-fscheck2](https://www.nuget.org/packages/Expecto.FsCheck/11.0.0-alpha1-fscheck2).

To enable complete logging solutions, stacktrace highlighting, and the ability to ship build logs, add Logary.Adapters.Facade (prerelease) to your project.

In your test entry point, initialize the Logary facade adapter before invoking the Expecto test runner. This ensures that your application logs and Expecto's test output are unified and not interlaced during concurrent execution.

open Hopac
open Logary
open Logary.Configuration
open Logary.Adapters.Facade
open Logary.Targets

[<EntryPoint>]
let main argv =
  let logary = 
    Config.create "MyProject.Tests" "localhost"
    |> Config.targets [ LiterateConsole.create LiterateConsole.empty "console" ]
    |> Config.processing (Events.events |> Events.sink ["console";])
    |> Config.build
    |> run
  LogaryFacadeAdapter.initialise<Expecto.Logging.Logger> logary

  // Invoke Expecto:
  runTestsInAssemblyWithCLIArgs [] argv