microsoft/win32metadata

Win32Metadata is an open-source project that generates machine-readable metadata (Windows.Win32.winmd in ECMA-335 format) for the entire Win32 API surface. This metadata is used by language projection tools to auto-generate idiomatic API bindings for various programming languages.

Key Output Packages

• Microsoft.Windows.SDK.Win32Metadata (NuGet): Contains the Windows.Win32.winmd file, which provides the ECMA-335 metadata for the Win32 API.
• Microsoft.Windows.WinmdGenerator (NuGet): An MSBuild SDK used to generate .winmd files from C/C++ headers.
• Microsoft.Windows.SDK.Win32Docs (NuGet): Contains API documentation in MessagePack binary format.

Supported Language Projections

• C#: via CsWin32
• Rust: via windows-rs
• Community: Python, Zig, D, Dart, Beef, and .NET Standard.

The project uses a sophisticated pipeline to transform C/C++ Win32 headers into a structured .winmd file. Understanding these core concepts is essential for working with the metadata:

Partition-based Namespaces

The Win32 API is organized into approximately 240 partitions. Each partition maps specific header files to a logical Windows.Win32.* namespace, providing a structured way to navigate thousands of APIs.

Multi-Architecture Scraping

To handle architecture-specific type differences, headers are scraped independently for x64, x86, and arm64. A CrossArchTreeMerger then combines these into a unified representation, applying [SupportedArchitecture] attributes to relevant types.

Layered Configuration

Configuration follows a hierarchy from general to specific:

1. baseSettings.rsp (SDK-provided)
2. scraper.settings.rsp (Project-wide)
3. Individual partition settings.rsp (Per-partition customization)

Manual Overrides

For complex APIs that automated tools like ClangSharp cannot handle correctly (such as certain COM interfaces or complex DirectX types), the project maintains a manual/ folder containing hand-written C# files to ensure accuracy.

The Emitter layer converts generated C# files into ECMA-335 compliant .winmd files. Because ClangSharp generates code intended to be compilable by the CLR, it often uses patterns (like representing COM objects as structs instead of interfaces) that are not ideal for language-agnostic metadata. The Emitter corrects this.

ClangSharpSourceCompilation

Orchestrates the manipulation and compilation of C# files using several specialized classes:

• NamesToCorrectNamespacesMover: Moves APIs to specific namespaces based on requiredNamespacesForNames.rsp.
• MetadataSyntaxTreeCleaner: Visits C# Abstract Syntax Tree (AST) nodes to apply remaps and custom attributes.
• CrossArchTreeMerger: Merges C# files from multiple architectures to identify architecture-specific APIs.

ClangSharpSourceWinmdGenerator

Walks the final C# AST and writes each node to the Windows.Win32.winmd file.

When migrating #define groups to enums, you will encounter three primary patterns:

1. Renamed Members: If the enum member name differs from the #define name, simply reference the #define directly as the value. No macro conflicts occur.
2. Same-Name Members: If the enum member name is identical to the existing #define, use #pragma push_macro, #undef, and #pragma pop_macro to suppress the macro expansion inside the enum declaration. This prevents the preprocessor from expanding the name before the enum can define it.
3. Signed/Unsigned Overflow: For values that require casting (e.g., 0xFFFFFFFF into a signed int), use a C cast. ClangSharp will automatically generate the necessary unchecked() block in the resulting metadata.

// Pattern 2: Same-name members
#pragma push_macro("FILE_ATTRIBUTE_READONLY")
#undef FILE_ATTRIBUTE_READONLY

enum class [[clang::flag_enum]] FILE_ATTRIBUTE_FLAGS : DWORD {
    FILE_ATTRIBUTE_READONLY = 0x00000001,
};
DEFINE_ENUM_FLAG_OPERATORS(FILE_ATTRIBUTE_FLAGS)

#pragma pop_macro("FILE_ATTRIBUTE_READONLY")

// Pattern 3: Signed/unsigned overflow
enum class OBJECT_IDENTIFIER : int {
    OBJID_WINDOW  = 0,
    OBJID_SYSMENU = (int)0xFFFFFFFF, // Generates unchecked() in metadata
};

The scraping architecture consists of three primary components:

Win32MetadataScraper

Located in sources/Win32MetadataScraper/Program.cs, this is the main entry point. It:

• Parses RSP files for PInvokeGenerator settings.
• Walks the AST using RemapDiscovery.WalkTranslationUnit.
• Merges auto-discovered remaps with configured ones.
• Runs PInvokeGenerator.GenerateBindings().
• Writes the results to a .remaps sidecar file.

RemapDiscovery

Located in sources/Win32MetadataScraper/RemapDiscovery.cs, this static class handles the AST logic:

• WalkTranslationUnit(): Finds tag→typedef and function pointer prototype→alias relationships.
• ResolveTagRemaps(): Uses heuristics (stripping _ or tag prefixes, case-insensitive matching) to disambiguate tags.
• FilterTagRemaps(): Removes identity remaps, built-in defaults, or conflicts.
• ResolveFunctionPointerFixups(): Classifies pairs by pointer prefixes (LP, PFN, P) to determine the correct remap direction.

ScrapeHeaders

Located in sources/GeneratorSdk/MetadataTasks/ScrapeHeaders.cs, this manages the build process:

• Spawns Win32MetadataScraper.dll in isolated dotnet processes per partition.
• Aggregates .remaps sidecar files into thread-safe dictionaries.
• Writes the final merged RSP files via WriteAutoRemapsRsp().

The win32metadata project derives Windows Metadata (.winmd) from Windows SDK C/C++ headers using a three-stage pipeline:

1. ClangSharp: Parses headers via Clang's AST to generate C# declarations for functions, structs, COM interfaces, and typedefs.
2. ConstantsScraper: Performs raw text/regex scanning of headers to extract #define constants and assemble enums that are not present in the AST.
3. The Emitter (ClangSharpSourceToWinmd): Compiles the generated C# sources, applies transformations (remaps, attribute additions), and writes the final Windows.Win32.winmd file.

SDK C/C++ Headers
      │
      ├──[ClangSharp]──────────> Generated C# (functions, structs, COM interfaces)
      │                               │
      ├──[ConstantsScraper]────> Generated C# (constants, enums)
      │                               │
      └──[Manual C# overrides]────────┤
                                      │
                              [Emitter (ClangSharpSourceToWinmd)]
                                      │
                                      ▼
                              Windows.Win32.winmd

To embed tool-readable metadata into C/C++ headers without affecting compilation, use a SAL-like annotation mechanism. This allows metadata to be embedded in functions, parameters, struct fields, and return values.

When using Clang (which ClangSharp utilizes), these annotations are implemented via __attribute__((annotate(...))). When using MSVC, they use __declspec(...). If no analysis tool is running (e.g., _PREFAST_ is not defined), these annotations expand to nothing, ensuring zero runtime or compilation cost.

// When Clang is the compiler (ClangSharp uses Clang):
#if __clang__
  #define _SA_annotes0(n)   __attribute__((annotate("Name=" #n)))
  #define _SA_annotes1(n,p) __attribute__((annotate("Name=" #n "; p1=" #p)))

// When MSVC is the compiler:
#else
  #define _SA_annotes0(n)   __declspec(#n)
  #define _SA_annotes1(n,p) __declspec(#n "(" #p ")")
#endif

// When no analysis tool is running (_PREFAST_ not defined):
// All of the above expand to nothing — zero cost