Terascope File Asset APIs

The package provides several interfaces for system and job configuration:

• APIConfig: Configuration for loading APIs and Observers into an ExecutionContext.
• OpConfig: Configuration for an Operation; requires the _op property to identify the operation.
• Slice: Represents a trackable set of work to be performed by a Worker.
• SliceRequest: Metadata created by the Slicer and processed through a job pipeline.
• Other configuration interfaces include ConnectionConfig, ExecutionConfig, JobConfig, JobConfigParams, SysConfig, TerafoundationConfig, TerasliceConfig, and ValidatedJobConfig.

The xlucene-parser package uses a set of interfaces to represent the Abstract Syntax Tree (AST) for parsed queries. Key node types include:

• Node and EmptyNode: Base interfaces for all AST nodes.
• Term: Represents a simple term query.
• TermLikeNode: Base interface for searchable terms.
• LogicalGroup: Represents logical groupings with AND/OR operations.
• Conjunction: Represents an implicit AND operation.
• Negation: Represents a negated term or group.
• Range: Represents a range query, with RangeNode configuring the range boundaries.
• Wildcard and Regexp: Represent pattern-based queries.
• Exists: Represents a check for field existence.
• FieldGroup: Represents multiple operations targeting the same field.
• TermList: Represents a list of terms.
• FunctionNode: Represents a function call.

The @terascope/elasticsearch-asset-apis package provides a set of tools for interacting with Elasticsearch, specifically focusing on bulk data sending and reader API functionality. Key exported entities include:

• Core APIs: createElasticsearchBulkSender, createElasticsearchReaderAPI, and createSpacesReaderAPI.
• Classes: ElasticsearchBulkSender, ElasticsearchReaderAPI, and ElasticsearchReaderClient.
• Date Utilities: dateSlicer, dateFormat, dateFormatSeconds, divideRange, and various DateSlicer configuration/result interfaces.
• ID & Chunking Utilities: Chunker, base64, base64url, and determineIDSlicerRanges.
• Configuration Interfaces: ElasticsearchSenderConfig, DateConfig, and ElasticsearchAPIArgs.

The spaces_reader is used to fetch data from an endpoint on a spaces server. It is a wrapper around the elasticsearch_reader and shares its behaviors and requirements.

Key Requirements:

• Date Field: The data must contain a date field with sufficient time resolution to slice data into chunks and be spread out over time. If the data lacks a date field or dates are identical, use the id_reader instead.
• Authentication: A valid space endpoint URL and user token are required.
• Recoverability: This reader is recoverable; jobs can be stopped and resumed from where they left off.
• Metadata: Fetched records include associated metadata, such as the _key field.