Apache NiFi

Apache NiFi is a platform for managing directed graphs of data routing, transformation, and system mediation logic. It provides a web-based user interface for designing, controlling, and monitoring dataflows.

Key features include:

• Configurability: Supports tuning for loss tolerance vs. guaranteed delivery, low latency vs. high throughput, dynamic prioritization, runtime flow modifications, and back pressure.
• Data Provenance: Provides the ability to track dataflow from beginning to end.
• Extensibility: Designed to allow developers to build custom processors and other extensions.
• Security: Supports SSL, SSH, HTTPS, and encrypted content, along with multi-tenant and internal authorization/policy management.

Apache NiFi MiNiFi is a subproject of Apache NiFi designed as a lightweight data collection agent. It is intended to be deployed at the source of data creation (e.g., on sensors, systems, or servers) to supplement core NiFi dataflow management.

Key characteristics include:

• Small and lightweight footprint: Optimized for resource-constrained environments.
• Central management: Designed to allow for the central management of agents.
• Data Provenance: Generates provenance information for data tracking.
• NiFi Integration: Integrates with Apache NiFi for follow-on dataflow management and to maintain a full chain of custody for information.

The MergeRecord processor combines multiple FlowFiles containing record-oriented data (using any available Record Reader) into a single, larger FlowFile.

Key Benefits:

• Downstream Efficiency: Prepares data for systems that prefer large batches (e.g., HDFS).
• Performance Optimization: Reduces the total number of FlowFiles in the system, decreasing contention on the FlowFile, Provenance, and Content Repositories, as well as FlowFile Queues.

The QueryNiFiReportingTask allows you to execute SQL queries against NiFi metrics, which are modeled as virtual tables. The results of these queries are then transmitted to a destination via a configured Record Sink service (e.g., SiteToSiteReportingRecordSink or DatabaseRecordSink).

Key features:

• Apache Calcite Integration: Supports standard SQL features including JOINs and aggregate functions.
• Non-materialized Views: The tables are not persistent; they are re-queried from their sources at every execution.
• Unique Processing: For BULLETINS and PROVENANCE tables, you can ensure items are only processed once by using special SQL placeholders to define a time window. This prevents duplicate reporting of the same event across different task executions.

The GetSNMP processor polls an SNMP agent to retrieve information using two strategies:

1. GET Strategy: Retrieves information for a specific OID or a set of OIDs.
2. WALK Strategy: Retrieves all information for the entire subtree associated with a given OID.

This processor supports SNMPv1, SNMPv2c, and SNMPv3 and is built on the SNMP4J library.

Output Behavior

• FlowFile Content: The outgoing FlowFile content is always empty.
• FlowFile Attributes: All retrieved information is written to FlowFile attributes.
• GET Request Output: For a single GET request, the properties from the received PDU are transferred directly into FlowFile attributes.
• WALK Request Output: For a WALK request, only "OID/value" pairs are transferred into FlowFile attributes.

The YamlTreeReader Controller Service converts YAML objects into NiFi Record objects.

Behavioral Rules:

• Root Element as Array: If the YAML root is an array, each object in the array becomes a separate Record.
• Root Element as Object: If the YAML root is an object, the entire YAML structure is treated as a single Record.
• Schema Matching: The service must be configured with a schema.
  • Fields in YAML but not in the schema are skipped.
  • Fields in the schema but not in the YAML are populated with null (or the schema's default value).
• Subparts: You can use 'Starting Field Strategies' (detailed in other documentation) to read only specific sub-sections of the YAML tree as records.

The NiFi frontend uses Angular Material Design 3 (MD3) with a custom teal-based color palette and a specialized canvas environment for the flow designer.

Key Implementation Details:

• Theming: Uses Angular Material's --md-ref-palette-* system mapped to NiFi's palette in libs/shared/src/assets/themes/material.scss.
• Custom Tokens: Uses --nf-* CSS custom properties for canvas and status semantics that fall outside standard MD3.
• Stack: Angular (NgModule and standalone), Angular Material (MD3), Tailwind CSS v4, and Font Awesome 4.7.
• Layout: Uses Tailwind CSS for spacing and flex/grid utilities. Avoid custom spacing SCSS.
• Icons: Uses Font Awesome 4.7 via <i class="fa fa-*"> syntax.

The ConsumeJMS processor consumes messages from any JMS-compliant messaging system and converts them into NiFi FlowFiles.

When a message is consumed:

• The message body is written to the FlowFile content.
• Standard JMS Headers and Properties are extracted and set as FlowFile attributes.

The JsonTreeReader Controller Service converts JSON objects into NiFi Record objects.

Behavioral Rules:

• Root Element as Array: If the JSON root is an array, each object in the array is treated as a separate Record.
• Root Element as Object: If the JSON root is an object, the entire JSON is treated as a single Record.
• Schema Matching: The service uses a configured schema to define the Record structure.
  • Fields in JSON but not in the schema are skipped.
  • Fields in the schema but not in the JSON are populated with null (or the schema's default value).
• Subpart Parsing: You can configure the reader to create a Record for the entire JSON tree or just a specific subpart (refer to 'Starting Field Strategies' in the full documentation).