Grapher

Hypernova is Grapher's high-performance query engine designed to minimize database requests by aggregating filters at each level of a nested query.

Instead of performing an $N+1$ style fetch (where you fetch a parent collection and then perform individual requests for every child/related document), Hypernova calculates the required data graph and executes a predictable number of requests equal to the number of collection nodes in the graph. It fetches all necessary documents for a specific collection level in bulk and then reassembles them into the requested object structure.

Key Benefits:

• Predictable Performance: The number of database requests is tied to the number of collection nodes, not the number of documents returned.
• Bandwidth Efficiency: It smartly re-uses shared entities (like authors or categories) at each collection node to reduce data transfer, often outperforming SQL JOIN operations in terms of bandwidth consumption.

Important Limitation:

• Hypernova is only used for static queries. For reactive queries, Grapher continues to rely on recursive fetching.

// Example of a nested query that Hypernova optimizes
createQuery({
    posts: {
        categories: {
            name: 1,
        },
        author: {
            name: 1,
        },
        comments: {
            $options: {limit: 10},
            author: {
                name: 1,
            }
        }
    }
})

While denormalization improves performance, be aware of the following constraints and trade-offs:

1. Deep Filters

If you attempt to use $filters inside the body of a denormalized link, Grapher will treat it as a foreign field and fall back to a standard linked fetch, bypassing the cache:

// This will NOT hit the cache
{
    users: {
        avatar: {
            $filters: {}
        }
    }
}

For deep filtering, use the $postFilter() method instead.

2. Meta Links

Currently, you cannot store $metadata inside a nested object for denormalized meta links. This is because querying from the inversed side would require additional fetching of the link storage.

3. Performance vs. Storage

• Computation Cost: Denormalization adds database hooks to keep caches updated. Changes to a linked document trigger additional computation.
• Storage Cost: Large caches can significantly increase document size. Use denormalization for data that rarely changes (e.g., avatars) or for fields required for frequent, powerful searches.

For highly scalable reactive data graphs, it is recommended to move from tailing the MongoDB oplog to using redis-oplog. Grapher is fully compatible with redis-oplog.

To optimize performance, you can use $options.namespace inside a $filter() to create namespaced watchers. This prevents unnecessary reactivity across the entire collection by limiting updates to specific subsets of data.

Using $filter with namespaces

You can define the namespace directly in the query. This is useful for grouping updates (e.g., by a specific thread ID).

export default Messages.createQuery('messagesForThread', {
    $filter({filters, options, params}) {
        filters.threadId = params.threadId;
        options.namespace = `thread::${params.threadId}`;
    },
    text: 1,
})

Embodying $filter server-side

To prevent clients from seeing or manipulating the filter logic, use embody in your .expose.js file. This encapsulates the filtering and namespacing logic on the server.

// query.expose.js
query.expose({
    embody: {
        $filter({options, filters, params}) {
            filters.threadId = params.threadId;
            options.namespace = `thread::${params.threadId}`;
        }
    }
})

Alternatively, embody can be a function that allows you to modify the query body directly:

// query.expose.js
query.expose({
    embody(body, params) {
        // Modify body here
    }
})

To implement a custom cacher (for example, to use Redis for distributed caching), you must extend BaseResultCacher and implement the following interface:

• generateQueryId(queryName, params): Returns a unique string for the query and its parameters.
• fetch(cacheId, fetchables): Handles the retrieval and storage logic. The fetchables object contains:
  • query: An object with a fetch() method to retrieve the data.
  • countCursor: An object with a count() method (this is not present when caching Resolver Queries).

If you need to invalidate the cache, you must implement that logic within your custom cacher, as automatic invalidation is not provided by Grapher.

import {BaseResultCacher} from 'meteor/cultofcoders:grapher';
import {EJSON} from 'meteor/ejson';

/**
 * Redis Cacher
 */
export default class RedisCacher extends BaseResultCacher {
    // the constructor accepts a config object, which stores it in this.config
    
    // if not specified, it uses the implementation from BaseResultCacher
    generateQueryId(queryName, params) {
        return `${queryName}::${EJSON.stringify(params)}`;    
    }
    
    // in case of a count cursor cacheId gets prefixed with 'count::'
    fetch(cacheId, fetchables) {
        const {client, ttl} = this.config;
        
        const cacheData = client.get(cacheId);
        if (cacheData) {
            return EJSON.parse(cacheData);
        }

        const data = BaseResultCacher.fetchData(fetchables);
        client.set(cacheId, EJSON.stringify(data), 'PX', ttl);

        return data;
    }
}

The $body parameter allows you to request a subset of the data defined in a Named Query. Grapher will intersect your requested $body with the query's allowed data graph, removing any fields that are not permitted by the original query definition.

This is useful for 'expanding' data (e.g., fetching more details when a user clicks a list item) without creating multiple redundant queries.

Important: If you use validation for parameters, you must include $body in your validation schema.

const fullPostList = Posts.createQuery('fullPostList', {
    title: 1,
    author: { firstName: 1, lastName: 1 },
    comments: { text: 1, createdAt: 1 }
});

// Requesting a subset via $body
const subset = fullPostList.clone({
    $body: {
        title: 1,
        author: { firstName: 1 },
        unauthorizedField: 1 // This will be removed after intersection
    }
}).fetch();

// Resulting shape: { title: 1, author: { firstName: 1 } }

Post-filtering allows you to manipulate or filter data after it has been retrieved from the database and assembled. This is useful for complex logic that cannot be expressed in a standard database query.

Available post-processing tools (only for top-level nodes):

• $postFilters: Uses the sift library to apply MongoDB-like filters to the assembled result set. For metadata links, use the path field.$metadata.key.
• $postOptions: Supports limit, sort, and skip applied to the assembled results.
• $postFilter(results, params): A function that receives the assembled results and must return the new set of results.

Warning: Post-filtering can be performance-intensive because the database fetches all potential records before the filtering is applied.

// Example using $postFilters with metadata
const users = Meteor.users.createQuery({
    $postFilters: {
        'groups.$metadata.roles': {$in: 'ADMIN'},
    },
    name: 1,
    groups: {
        name :1,
    }
}).fetch();

// Example using $postFilter function
const users = Meteor.users.createQuery({
    $postFilter(results, params) {
        if (params.mustHaveGroupsAsAdmin) {
            return results.filter(r => {
                // custom logic
            });
        }
        return results;
    },
    name: 1,
    groups: {
        name: 1,
    },
}, {
    params: {
        mustHaveGroupsAsAdmin: true
    },
}).fetch();