Amundsen

The Amundsen Frontend Service is a metadata-driven application designed to improve productivity for data analysts, scientists, and engineers. It acts as a 'Google search for data' by indexing data resources like tables, dashboards, and streams, and providing search results based on usage patterns (e.g., ranking highly queried tables higher).

Technically, it is a Flask application with a React frontend that interacts with two other core services:

1. Search Service: Enables users to search for data resources.
2. Metadata Service: Allows users to view and edit metadata for specific resources.

Amundsen ingests query metadata and query composition metadata using four distinct components. To provide a complete view of query usage and structure, you must ingest these four parts:

1. QueryMetadata: Represents the actual query text/definition.
2. QueryExecutionsMetadata: An aggregation representing the frequency of a query's execution over specific time windows (e.g., per hour, day, or week).
3. QueryJoinMetadata: Represents a join operation between two specific columns.
4. QueryWhereMetadata: Represents a WHERE clause used in a query, which may be associated with one or more columns and tables.

Amundsen relies on QueryExecutionsMetadata to manage time-sensitive relevance. As new execution metadata is added and old records are removed, Amundsen automatically updates which queries, joins, and where-clauses remain relevant in the system.

Amundsen uses specific models to represent feature metadata. The core model is FeatureMetadata, which serves as the baseline and must be ingested before any supplemental models that build relations to it.

Key feature-related models include:

• FeatureMetadata: Encapsulates basic feature details.
• Feature_Generation_Code: Stores the text of the generation code (SQL or otherwise) used to create a feature.
• Feature_Watermark: Stores the high and low data range of a feature. This is a general model that does not require specific storage implementation details (unlike the table-specific Watermark model).
• Supplemental models: Tag, Badge, Owner, and Programmatic_Description also work in conjunction with features.

The Amundsen Frontend Strategy is a technical roadmap designed to address stagnation in the UI codebase. It outlines the transition from the current state (characterized by inconsistent UX, sub-par codebase quality, and developer friction) to a target state focused on consistency, high-quality testing, and improved developer experience.

Key areas of focus include:

• User Experience: Moving towards A11y WAG 2.1 compliance, consistent typography, and a tailored homepage.
• Codebase Quality: Upgrading to React v18, implementing React Testing Library, achieving 80%+ test coverage, and cleaning up Redux logic and API endpoints.
• Developer Experience: Improving documentation, updating Docker images for OSS contributors, and moving toward a Next.js framework that is independent of the Flask API.

Amundsen's architecture consists of four primary components:

1. Frontend: A Flask-based web application with a React/Redux UI for user interaction.
2. Search: A proxy service providing a RESTful API for search requests. It leverages Elasticsearch (or Apache Atlas) and currently supports indexing and searching of table resources.
3. Metadata: A service that uses a Neo4j proxy to interact with a Neo4j graph database, serving metadata represented as a graph model.
4. Databuilder: A data ingestion library used to build and populate the metadata.

Amundsen uses Python classes located in databuilder/models/ to represent data structures stored in Neo4j or Elasticsearch.

Models that extend Neo4jSerializable include built-in methods to create both nodes and relationships. This abstraction allows developers to build pipelines that load Python objects into Neo4j or Elasticsearch without needing to manage the underlying Neo4j schema manually.

Dashboard models in Amundsen are designed using a normalized approach. This means models are separated (e.g., DashboardMetadata, DashboardOwner, DashboardTable) to decouple the model from the extraction logic.

While this can be less efficient than a denormalized approach (where one extraction job pulls all data), it provides maximum flexibility for various metadata sources.

To allow users to search Amundsen directly from their web browser's address bar, Amundsen implements the OpenSearch standard using an opensearch.xml file. This file provides a template for search URLs that browsers can use to add Amundsen as a custom search engine.

To enable this feature, ensure that the FRONTEND_BASE configuration variable is set. When FRONTEND_BASE is configured, the index.html file includes a link to the opensearch.xml file. Browsers will then automatically detect this and offer to add Amundsen as a search engine (e.g., in Chrome, these can be managed at chrome://settings/searchEngines).