DOCSEARCHbyAlgolia
Submit Docs

Follow the Money (FtM)

repository·main·Indexed 43 days ago

https://github.com/alephdata/followthemoney

A pragmatic data model and toolkit for anti-corruption data modeling and analysis. It provides tools for data normalization, validation, and mapping tabular data into a structured ontology for entities like people, companies, and financial transactions. Includes a Python library (version 3.8.4), a CLI tool (ftm), and bindings for TypeScript/JavaScript and Java.

Tokens
34.2K
Snippets
141
Records
170
Agent score
54%

What's inside Follow the Money

  1. Overview of Java bindings for FollowTheMoney

    main

    The Java bindings for FollowTheMoney (FtM) provide a way to process existing FtM data within a JVM environment.

    Important Limitations:

    • These are not full data-generating bindings.
    • They lack type validation logic.
    • Their primary purpose is to facilitate the processing of existing FtM data rather than the creation of new data structures.

  2. Overview of the Follow the Money data model

    main
    Follow the Money provides a pragmatic data model designed for investigative reporting. It focuses on entities such as people, companies, assets, payments, and court cases. The project includes code for validating and normalizing data elements, as well as tools to map tabular data into the model structure.

  3. Overview of FollowTheMoney (FtM) data model

    main

    FollowTheMoney (FtM) is a specialized data model designed for anti-corruption investigations. It provides a standardized ontology for defining entities (such as people or companies) and the relationships between them, effectively creating a graph of investigative data.

    Key features include:

    • Entity Definitions: Standardized models for investigative entities and supporting documents used as evidence.
    • Graph Structure: Support for entities to reference each other, enabling relationship mapping.
    • Tooling: Built-in capabilities to generate, validate, and export FtM-compliant data.
    • Aleph Integration: All data within the Aleph search engine is expressed using the FtM model, which adds layers for searching, viewing, and managing datasets and access controls.

  5. Use Entity Streams for data transfer

    main

    Entity streams are the standard format for transferring or storing information across the FtM ecosystem (CLI, Aleph API, and ingestors).

    Format Requirements:

    • An entity stream is a sequence of JSON entity objects.
    • Each entity must be serialized as a single line of JSON with no indentation.
    • Each entity must be separated by a newline.

    File Extensions: When saving entity streams to disk, use the following extensions:

    • .ftm
    • .ijson

  6. Understand the FollowTheMoney entity-only data model

    main

    FollowTheMoney (FtM) has transitioned from a traditional property graph model (nodes and edges) to an entity-only model.

    In the previous Aleph 2.x system, data was modeled as edges and nodes, which required opinionated decisions about whether certain events (like ownership or communications) were nodes or edges. In the current version, FtM models data as entities only. This approach is more descriptive and avoids the semantic ambiguity of trying to force complex domain data (like payments or customs declarations involving multiple parties) into a simple two-party edge structure.

    While property graphs are often optimized for specific analytical queries, the entity-only model aims to remain descriptive of the actual domain complexity.

  7. Understand Entity Fragmentation and Aggregation

    main

    Follow the Money (FtM) uses a stream-based approach to handle large datasets that cannot fit in memory. Instead of building complete graph nodes and edges incrementally, the toolkit generates entity fragments.

    Because a single real-world entity (like a person or company) might appear across multiple rows in a source dataset with different or incomplete information, these fragments must be aggregated (merged).

    Aggregation is typically achieved by sorting fragments by their ID so that all pieces of a single entity are grouped together and can be merged as they are read. This prevents data loss (e.g., losing a birth date provided in one row but missing in another) and avoids the performance penalty of fetching and updating entities in an index one by one.

  8. Ways to use FollowTheMoney

    main

    FollowTheMoney is available in three primary interfaces depending on your development environment:

    1. Command-line utility: For direct interaction and data processing via the CLI.
    2. Python library: For integrating FtM logic into Python-based data science or investigative workflows.
    3. TypeScript/JavaScript library: For web-based applications or Node.js environments.

  9. Understand the core Follow the Money (FtM) data model

    main

    The Follow the Money (FtM) data model is designed for storing complex object graphs using three primary concepts: Entities, References, and Streams. Understanding how these interact is essential for modeling data correctly within the ecosystem.

    1. Entities

    An entity is a JSON object representing a specific object in the graph. Every entity must contain three standard fields:

    • id: A unique identifier for the entity.
    • schema: A string specifying the type of the entity (e.g., Person, Company).
    • properties: A dictionary where keys are property names and values are arrays of strings. Even single-valued properties must be represented as an array.

    2. References

    Entities link to one another using a special property type called entity. Instead of nesting the target object, the property stores the id of the referenced entity.

    3. Interstitial Entities

    When a relationship between two entities has its own metadata (e.g., a person's ownership percentage in a company), do not attempt to model this as a simple edge. Instead, create an interstitial entity. This is a dedicated entity that holds references to both participants and contains the relationship-specific properties.

    Warning: Do not assume that entities derived from schema:Thing are nodes and schema:Interval are edges. This is a false assumption that leads to bugs; always use interstitial entities for attributed relationships.

    {
  "id": "1b38214f88d139897bbd13eabde464043d84bbf9",
  "schema": "Person",
  "properties": {
    "name": ["John Doe"],
    "nationality": ["us", "au"],
    "birthDate": ["1982"]
  }
}

  10. How entities are translated to a graph

    main

    When converting FollowTheMoney (FtM) data to a property graph model (such as Neo4J), the following semantic mappings apply:

    1. Standard Links: Most links in FtM are property values. In a property graph, both the source and target (e.g., schema:Passport and schema:Person) become nodes, and the property (e.g., holder) becomes an edge with no attributes.
    2. Inverse Relationships: The holder property has an inverse called passports added to the schema:Person schema. This is primarily used to store labels for passports linked to a person, though it functions as a stub in the FtM model.
    3. Metadata Workaround (Contracted Edges): Because standard FtM edges do not support properties, metadata (like time intervals via schema:Interval) is handled via a workaround. Certain schemas like schema:Membership or schema:Ownership are designed to 'contract' during graph generation, turning into an edge with properties rather than a separate node. This is considered a special shortcut rather than the standard FtM pattern.
    4. Stub Property Ambiguity: Due to the contraction workaround, it may be conceptually ambiguous whether stub properties refer to the original edge (e.g., Person.directorshipsDirector → Directorship.director) or the contracted edge (e.g., Person → directorOf → Organization). In practice, metadata is retrieved from the edge annotation of the contracted schema.

  11. Understand Entity ID namespacing and signatures

    main

    Entity ID namespacing is a security mechanism used to prevent ID conflicts in the Aleph search index when users or external processes generate arbitrary entity IDs.

    Instead of using a raw ID, Aleph uses a two-part ID consisting of the user-provided entity_id and a system-generated signature. The full ID follows the format entity_id.signature.

    To ensure isolation and prevent collisions across different datasets, the signature is calculated using an HMAC of the entity_id and the dataset_id. This allows the combined ID to be unique to a specific dataset without requiring an expensive index lookup. Because it is an HMAC, the signature can be safely generated on the client or server while maintaining security and isolation.

    entity_id.a40a29300ac6bb79dd2f911e77bbda7a3b502126

  12. Create relationships between entities

    main

    Relationships in FollowTheMoney are modeled as entities themselves (often called 'links' or 'events'). To create a relationship between two existing entities, define a third entity where the properties use the entity keyword to reference the internal names of the other entities defined in the mapping.

    Example: Creating a Membership entity that links a Person (member) to an Organization (party).

    gb_parliament_57:
  queries:
    - csv_url: http://bit.ly/uk-mps-csv
      entities:
        member:
          schema: Person
          keys:
            - id
          properties:
            name:
              column: name
        party:
          schema: Organization
          keys:
            - group_id
          properties:
            name:
              column: group
        membership:
          schema: Membership
          keys:
            - id
            - group_id
          properties:
            organization:
              entity: party
            member:
              entity: member