DOCSEARCHbyAlgolia
Submit Docs

Synapse Documentation

repository·master·Indexed 19 days ago

https://github.com/vertexproject/synapse

A central intelligence system designed to assist analyst teams across the intelligence life cycle. Documentation covers the deployment of core services including Cortex, Axon (binary blob storage), and JSONStor, as well as the management of the Synapse Data Model, local JSON schema caching via getrefs.py, and the development of Advanced Power-Ups through Storm Services.

Tokens
73.2K
Snippets
162
Records
468
Agent score
65%

What's inside Synapse

  2. Overview of Advanced Storm features

    master

    While basic Storm operations are sufficient for most analytical workflows, developers and power users can leverage advanced constructs for automation, custom Power-Ups, and complex logic.

    Advanced features include:

    • Variables
    • Methods
    • Control Flow
    • Functions
    • Storm Type Libraries
    • Storm Primitive Types

    Rapid Power-Ups are a specific implementation of these advanced features, where extensions are written entirely in Storm and exposed as Storm commands, allowing them to be deployed or updated on the fly without restarting Synapse.

  3. Introduction to Synapse

    master

    Synapse is a central intelligence and analysis system designed to support analysts through the intelligence life cycle. It is built to fuse large, disparate datasets from multiple disciplines to answer complex questions.

    Core Architecture

    • Cortex: The Synapse data store, organized as a hypergraph. It is designed to scale vertically to tens of billions of nodes.
    • Note on Usage: Synapse is not a replacement for big-data/data-lake storage. It is designed to connect to your existing data sources on demand and ingest relevant data for specific analyses.
    • Storm: A powerful and intuitive query language used to interact with data within a Cortex.
    • Data Model: An extensible model capable of representing real-world objects, relationships, and events.
    • Power-Ups: A modular system used to extend functionality, integrate third-party data, or connect to external databases.

  4. Understand the Synapse Library Structure

    master

    The Synapse library is organized into several functional submodules. Understanding these helps locate specific primitives, data models, or tools:

    • synapse.cmds: Command implementations for the Cmdr CLI tool.
    • synapse.data: Internal data files.
    • synapse.lib: Core primitives used to implement applications.
    • synapse.lookup: Lookup definitions.
    • synapse.models: Core Synapse data model definitions.
    • synapse.servers: Modules used to start and run Synapse applications.
    • synapse.tools: Tools for interacting with the Synapse ecosystem.
    • synapse.vendor: Third-party code (internal use only; no API stability guaranteed).
    • synapse.tests.utils: Helper utilities for writing tests.

  5. Access Storm via Telepath and HTTP APIs

    master

    Storm queries can be executed through two primary interfaces: Telepath (for message streams and RPC-style calls) and HTTP API (for web-based access). Both interfaces accept a Storm query string and optional opts arguments.

    Telepath API

    • storm(text, opts=None): Returns a continuous stream of messages.
    • callStorm(text, opts=None): Returns a single message specified by the Storm return() syntax.
    • count(text, opts=None): Returns the number of nodes that would be emitted by the query.
    # Example Telepath usage (conceptual)
# storm(text, opts=None) -> returns stream
# callStorm(text, opts=None) -> returns single message via return()
# count(text, opts=None) -> returns node count

  6. What is a Dmon (Daemon)?

    master
    A Dmon (short for daemon) is a long-running or recurring query or process that runs continuously in the background. It is typically implemented by a Storm service and is used for non-blocking background tasks, such as processing elements from a queue. Dmons are persistent and will restart if they exit.

  7. What is a Constructor (Ctor)?

    master

    A constructor (or ctor) is code that defines how a property value of a specific type is constructed to ensure it is well-formed. Constructors support type normalization (type-norm) and type enforcement (type-enforce).

    Constructor, Guid: A specific constructor used to create guid values. Often referred to as a "gutor", it uses JSON dictionary syntax to deconflict guid nodes and generate predictable values.

  8. What is a Storm Package?

    master

    A Storm Package is a Rapid Power-Up used to extend the capabilities of the Storm query language. It consists of a YAML configuration file that defines commands, modules, documentation, and workflows. These packages can be loaded directly into your Cortex to provide custom functionality, implement use-case specific commands, and embed documentation.

    Best Practices:

    • Namespacing: Always use a unique namespace (e.g., company.functionality) to avoid collisions with other packages.
    • File Structure: The YAML file acts as the root. Modules and commands are typically stored in subdirectories (storm/modules/, storm/commands/) relative to the YAML file location.
    name: acme-hello
version: 0.0.1
synapse_version: '>=2.144.0,<3.0.0'

genopts:
    dotstorm: true

author:
  url: https://acme.newp
  name: ACME Explosives and Anvils

desc: Acme-Hello is a minimal example of a Rapid Power-Up.

modules:
  - name: acme.hello

commands:
  - name: acme.hello.sayhi
    descr: Print the hello message.

  10. What is an Advanced Power-Up?

    master

    An Advanced Power-Up is a standalone application designed to extend the capabilities of the Cortex. It achieves this by implementing a Storm Service.

    A typical use case for an Advanced Power-Up is to introduce a new Storm command that executes custom Python logic. This logic can be used to parse files, translate the parsed data into the Synapse datamodel, and subsequently ingest that data into the hypergraph.

  12. What is Storm?

    master
    Storm is the query language used to interact with data in Synapse. It is designed as a "data language" rather than a programming language, allowing users to ask questions about, retrieve, annotate, add, modify, and delete data within a Synapse Cortex. It uses intuitive syntax, such as the -> arrow for pivot operations, to make querying feel like natural language.