DOCSEARCHbyAlgolia
Submit Docs

SPDX tools-golang

repository·main·Indexed 41 days ago

https://github.com/spdx/tools-golang

A collection of Go packages for interacting with Software Package Data Exchange (SPDX) files, supporting specification versions 2.1, 2.2, and 2.3. It provides tools for reading, writing, and converting SPDX documents across tag-value, JSON, YAML, and RDF formats, as well as utilities for building documents from directories, searching for short-form IDs, and comparing concluded licenses.

Tokens
3.2K
Snippets
15
Records
27
Agent score
51%

What's inside SPDX tools-golang

  1. Overview of SPDX tools-golang packages

    main

    The tools-golang repository provides a collection of Go packages designed to facilitate working with SPDX® files. It currently supports SPDX specification versions 2.1, 2.2, and 2.3.

    Key packages include:

    • spdx: Provides an in-memory data model for SPDX document sections.
    • tagvalue: A reader and writer for tag-value format documents.
    • rdf: A reader for RDF format documents.
    • json: A reader and writer for JSON format documents.
    • yaml: A reader and writer for YAML format documents.
    • builder: Builds "empty" SPDX documents (including hashes) based on directory contents.
    • idsearcher: Searches for SPDX short-form IDs and constructs an SPDX document.
    • licensediff: Compares concluded licenses between files in two different packages.
    • reporter: Generates basic license count reports from an SPDX document.
    • spdxlib: A collection of utility functions for in-memory SPDX document manipulation.
    • utils: General utility functions supporting the other packages.

    Note: This library does not support SPDX versions prior to v2.1.

  2. How files and snippets are associated during JSON loading

    main

    The jsonloader uses specific schema properties to manage the relationships between packages, files, and snippets:

    Package-File Association

    Packages contain a hasFiles property (an array of SPDX Identifiers). The parser follows this logic:

    1. All files are initially parsed into the document's UnpackagedFiles map.
    2. When a Package is parsed, the parser identifies the files listed in its hasFiles property.
    3. Those files are moved from the UnpackagedFiles map into the Files map of the specific Package.

    Snippet-File Association

    Snippets use a snippetFromFile property containing the SPDX identifier of the related file. Because of this dependency, the parser must process files before snippets. Once files are parsed, snippets are inserted into their respective files based on this identifier.

  3. Understand scanning limitations in package idsearcher

    main
    The idsearcher package uses bufio.Scanner to process files line by line. If a file contains a line that exceeds the scanner's buffer capacity, the scanning process for that specific file will terminate immediately. Consequently, any short-form SPDX IDs located on or after the oversized line will not be detected.

  5. JSON saver: Package, File, and Snippet relationships

    main

    The JSON saver manages the relationships between packages, files, and snippets using the following logic:

    • Packages and Files: Packages include a hasFiles property, which is an array of SPDX Identifiers for the files contained within that package. The saver iterates through a package's files, populates the hasFiles array, and simultaneously adds each file to a temporary storage map to ensure all files for the entire document are centralized.
    • Ordering Requirement: To ensure packaged files are correctly added to the temporary storage before they are serialized, packages must be saved before their associated files.
    • Snippets: Snippets are saved after files. Each snippet includes a snippetFromFile property that identifies the specific file containing that snippet.

  6. How the JSON saver works

    main

    The JSON saver converts an SPDX document into a map[string]interface{} and then serializes the entire map into a JSON string using a single json.MarshalIndent call.

    To comply with the SPDX JSON schema, the saver uses temporary memory to aggregate all files (both Packaged and Unpackaged) into a single data structure before final serialization. The conversion flow is:

    v2_2.Document $\rightarrow$ map[string]interface{} $\rightarrow$ JSON.

  7. Understand assumptions of the `licensediff` tool

    main

    When using the licensediff package to generate license differences, be aware of the following structural assumption:

    • Unique Filenames per Package: The tool assumes that within any single Package, a specific filename will appear exactly once. This uniqueness is required for the tool to correctly index and create a diff based on filenames.

    If your input data contains duplicate filenames within the same package, the diffing logic may not behave as expected.

  8. How exceptions are handled in PackageLicenseInfoFromFiles and LicenseInfoInFile

    main

    When using PackageLicenseInfoFromFiles (in the Package package) or LicenseInfoInFile (in the File package), SPDX exceptions are treated as distinct licenses.

    For example, if a file contains the expression GPL-2.0-only WITH Classpath-exception-2.0, the searcher will return both GPL-2.0-only and Classpath-exception-2.0 as separate license entries.

  9. Annotation identifier assumptions in the JSON loader

    main

    Because the SPDX JSON schema does not explicitly define an SPDX Identifier property within the annotation object itself, the jsonloader uses a hierarchical assumption:

    When parsing an array of annotations, the parser assumes that the SPDX Identifier of the parent property currently being parsed is the identifier for all annotation objects within that array.

  10. How the JSON loader processes SPDX documents

    main

    The jsonloader package implements a custom unmarshalling process for the v2_2.Document struct. Instead of a direct mapping, it uses a multi-step transformation to ensure complex relationships (like file ownership and snippet associations) are correctly resolved:

    1. Unmarshal to Map: The JSON is first unmarshalled into a temporary map[string]interface{}.
    2. Structured Parsing: The map is then parsed into the v2_2.Document using specialized functions for each section of the SPDX schema.

    Data Flow: JSON $\rightarrow$ map[string]interface{} $\rightarrow$ v2_2.Document

    JSON => map[string]interface{} => v2_2.Document

  11. Understand `tvsaver` assumptions for Document Creation Info and Relationship fields

    main

    When using the tvsaver package to save tag-value files, be aware that the tool treats mandatory fields and optional fields identically. If a mandatory field (such as SPDXVersion in the Document Creation Info section) or an optional field (such as RelationshipComment in the Relationship section) is provided as an empty string, tvsaver will omit it from the output.

    Warning: Omitting mandatory fields will result in the generation of an invalid SPDX tag-value file. Ensure all required fields are populated with valid data before saving.

  12. Assumptions of the tvloader tag-value file loader

    main

    The tvloader package (tag-value file loader) expects SPDX tag-value files to follow a specific structural order and use specific tags to identify sections. If these assumptions are not met, the parser may fail to identify the SPDX specification version or misinterpret sections.

    Section Ordering and Tagging Requirements

    1. Document Creation Info: Must be the first section in the file. This is critical because the parser uses this section to determine the SPDX specification version. It uses SPDXID: tags.
    2. Relationship: Sections must begin with the Relationship tag.
    3. Annotation: Sections must begin with the Annotator tag.
    4. Other License Info: Sections must begin with the LicenseID tag. These sections must appear after the Document Creation Info section and after all Package, File, and Snippet sections.
    5. Review: Sections must begin with the Reviewer tag. These sections must appear after the Document Creation Info section and after all Package, File, Snippet, and Other License sections.