DOCSEARCHbyAlgolia
Submit Docs

Flutter Customer Test Registry

repository·main·Indexed 43 days ago

https://github.com/flutter/tests

A collection of tests maintained by external developers to ensure Flutter framework changes do not introduce breaking changes in real-world applications. Includes guidelines for contributing tests, requirements for hermeticity and reliability, and tools for generating SKPs (Skia Pictures) for Skia regression and performance testing.

Tokens
1.8K
Snippets
3
Records
9
Agent score
5%

What's inside Flutter Customer Test Registry

  1. SKP Generators for Skia testing

    main
    The skp_generator directory contains a Flutter program and shell scripts used to generate test SKPs (Skia Picture) for the Skia team. Developers can contribute by providing stateless widgets that represent scenes from their applications.

  2. Contribute private or large-scale tests

    main

    If you have a large volume of tests or proprietary tests that cannot be added to the public registry, you can request integration into the Flutter CI via the Flutter Discord server.

    Conditions for private integration:

    • You must be highly responsive to discussions on Discord during your time zone's working hours.
    • You must take responsibility for helping contributors fix issues flagged by your tests (including debugging help).
    • You must fund and maintain any tooling required for the integration.
    • Integration will be disabled if it causes frequent false-positives, infrastructure failures, or project disruption.

  3. Generate SKPs using build.sh

    main

    Use the build.sh script to generate SKPs of Flutter scenes for Skia regression and performance testing. Successful execution will place all generated SKPs in the skps/ directory.

    Prerequisites:

    • git must be available on your PATH.
    • flutter must be available on your PATH.
    • You must be using a POSIX command line.

    WARNING: This script deletes all files in the directory that have not been committed to Git. Ensure all your work is committed before running the script.

    ./build.sh

  4. Run Flutter Customer Test Registry tests locally

    main

    You can execute the registry tests on your local machine using the provided verification script. To run all tests using the flutter binary currently in your PATH, use the --local-flutter flag.

    ./scripts/verify_tests_on_main.sh --local-flutter

  5. Set up a development environment for SKP Generator

    main

    To develop or debug scenes without the script deleting your build artifacts, use the --no-clean flag. This allows you to use standard Flutter tools like flutter run, the analyzer, and hot reload.

    Setup Command:

    ./build.sh --no-clean

    Development Tip: To bypass the standard registry logic and enable faster iteration with hot reload, you can modify void main() in lib/main.dart to call runApp directly with your specific test widget:

    void main() {
  runApp(MyTestWidget());
  return; // Bypass regular logic
}

  6. Add new tests to the Flutter Customer Test Registry

    main

    To contribute a new batch of tests to the registry, follow these steps:

    1. Copy registry/template.test to create a new file in the registry/ directory.
    2. Fill in the required fields and remove all comments.
    3. Submit a Pull Request with the new file.

    Test Requirements

    To be accepted, tests must meet these criteria:

    • Licensing: All code must be publicly available on GitHub under a license compatible with the Flutter project.
    • Hermeticity: Tests must be self-contained. They must not perform network activity, spawn processes, or access the local file system (except for files packaged with the test).
    • Concurrency: Tests must be resilient to running concurrently with other tests or multiple instances of themselves.
    • Reliability: Tests must be deterministic. Running a test multiple times must always yield the same result, and a test must never claim to pass if it is failing.
    • Output: Passing tests must produce no output.
    • Performance:
      • Tests must be as fast as possible and avoid real timers or dependency on the wall clock.
      • Most suites should complete in seconds. An upper limit of approximately five minutes per test suite is applied.
    • Tooling Compatibility: Tests must be compatible with automatic Flutter code update tools. They cannot rely on custom code generation unless that generation can hook into the automatic update mechanism.
    • Best Practices: Tests must follow Flutter's documentation (e.g., not using an object after calling dispose, and not accessing private State fields via dynamic casting).
    • Analysis: Tests must pass even if there are analysis 'info' level issues. Use flutter analyze --no-fatal-infos if performing static analysis within the test.
    • Upstream Readiness: The upstream repository hosting the tests must be able to support the main channel of Flutter. CI on the upstream repository should use the main channel Flutter SDK.
    • Dependency Management: Dependencies must be pinned (e.g., by checking in pubspec.lock).
      • Note: Avoid pinning packages that are also pinned by the Flutter SDK (like intl). For these, use any so they update automatically when Flutter updates.
    # Example workflow
cp registry/template.test registry/my_new_test.test
# ... edit file ...
git add registry/my_new_test.test
git commit -m "Add new customer test"
git push

  7. Contribute new scenes to the SKP Generator

    main

    To add a new scene for Skia testing, follow these steps:

    1. Create the Widget: Create a new stateless widget in its own file within the lib/tests directory.
    2. Register the Widget: Add a reference to your new widget in the array located near the top of lib/main.dart.
    3. Handle Variants (Optional): To show multiple frames (e.g., for animations), pass different arguments to the stateless widget within the lib/main.dart array. Keep the number of variants small.
    4. Optimization Guidelines:
      • Scenes should be significantly distinct from existing ones.
      • Focus on scenes that are expensive to rasterize or represent complex real-world applications.
      • Avoid unnecessary widgets like Navigators or InheritedWidgets for data models; keep the hierarchy static for a single build.
      • For animations, consider placing all frames side-by-side in a single image to simulate the rendering cost.

    Verification: You can verify your scene output at https://debugger.skia.org/. Click on EndDrawPicture at the end of the op list to view the final output. Note that the output size is 800x600 at 3dpr.

  9. Handle broken tests in the registry

    main

    If a change in the flutter/flutter repository causes a customer test to fail, you have three primary paths:

    1. Fix the PR (Preferred): Modify your changes so the test passes while maintaining a high-quality API. Follow the Flutter Style Guide.
    2. Breaking Change Process: If the change is necessary but breaking, follow the Tree Hygiene documentation. This involves contacting the test owners, helping them fix their code, updating the registry, and implementing a "soft-breaking" change (deprecation) so users have time to migrate.
    3. Remove the Test (Least Ideal): A test may only be removed if the contacts are unresponsive (no plan within 72 hours), the test is poorly written (e.g., contains race conditions), or the breaking change process cannot be successfully completed.