DOCSEARCHbyAlgolia
Submit Docs

Amorphic Documentation v2.6

website·Indexed 31 days ago

https://www.docs.amorphicdata.io/

Documentation for Amorphic v2.6, covering data ingestion via Kinesis and AWS App Flow, infrastructure management, Tag-Based Access Control (TBAC), SAML and Google SSO integrations, and AmorphicAI agent orchestration.

Tokens
215K
Snippets
281
Records
1.6K
Agent score
53%

What's inside Amorphic

  1. Overview of Amorphic ML Notebook properties

    Amorphic hosts Jupyter/IPython notebooks for interactive coding, visualization, and documentation. Each notebook instance is defined by several key properties:

    • Notebook Name: Unique identifier for the notebook.
    • Status: Current state (e.g., Creating, InService, Stopping, Stopped).
    • Instance Type: The ML compute instance type used.
    • Volume Size: Size of the attached ML storage volume in GB.
    • URLs: Provides specific endpoints for both the Jupyter server and Jupyter Lab server.
    • Glue Sessions: Indicates if Glue sessions are enabled.
    • Auto Stop: Status (Enabled/Disabled), remaining time, and the scheduled stop time to manage resource costs.
    • Internet Access: Determines if the instance can access the internet or is restricted to the VPC.
    • Lifecycle Configuration: The name of the attached lifecycle configuration.

  2. Overview of Lakeformation Datasets

    v2.6
    Lakeformation datasets extend S3-Athena datasets with enhanced security and support for structured files including CSV, TSV, XLSX, JSON, and Parquet. Key features include data integrity checks, ACID transactions, data compaction, and time-travel queries. Data validation is enabled by default, checking each column against the uploaded schema to detect invalid data, though this may increase processing time and cost.

  3. Overview of Amorphic Data Cloud orchestration

    v2.6
    Amorphic Data Cloud is a SaaS or managed software platform designed to orchestrate AWS advanced analytics and machine learning. It provides self-service capabilities for data ingestion, management, and storage across structured, semi-structured, and unstructured data types. The platform integrates traditional data pipelines with compute job orchestration, allowing users to apply machine learning models for predictive analytics and connect data to visualization dashboards and reports within a single cloud environment.

  4. Overview of Amorphic Athena Datasets

    v2.6
    Athena Datasets allow users to store structured data files (CSV, TSV, XLSX, JSON, and Parquet) in Amazon S3 and run SQL queries on them without maintaining a full data warehousing solution. Supported data types for validation include Strings/Varchar, Integers, Double, Boolean, Date, and Timestamp. For complex data structures, it is recommended to enclose them in quote characters and register the column as String/Varchar, then perform ETL casting after loading.

  5. Overview of Delta Lake support in Amorphic

    v2.6

    Amorphic supports Delta Lake datasets when using the Lakeformation target location. Delta Lake extends Parquet files with a transaction log to provide ACID transactions and scalable metadata handling. Supported file types include CSV, JSON, and Parquet.

    Key supported features include:

    • ACID transaction guarantees for consistent snapshots during simultaneous reads and writes.
    • Schema enforcement to prevent bad records during ingestion.
    • Time travel for data versioning and historical audits.
    • Support for upserts and deletes (merge, update, delete operations).

  6. Overview of Amorphic Data Cloud capabilities

    Amorphic Data Cloud is a cloud orchestration SaaS or managed software platform designed to simplify access to AWS advanced analytics and machine learning. It provides self-service capabilities for data ingestion, management, and storage across structured, semi-structured, and unstructured data types. Key capabilities include:

    • Automated Data Processing: Uses native machine learning models to transcribe PDF or audio files into text.
    • Metadata Management: Creates a global, secure metadata catalog to support search, sharing, and analytics.
    • Governance and Security: Implements role-based access controls (RBAC) for users and domains to ensure compliance.
    • Integration: Allows users to search files stored on S3 or Redshift and connect data to visualization dashboards, reports, and AI/ML tools.

  8. Overview of Lakeformation Datasets

    Lakeformation extends S3-Athena datasets with enhanced security and supports structured data files including CSV, TSV, XLSX, JSON, and Parquet. It provides data integrity checks, ACID transactions, data compaction, and time-travel queries. Data validation is enabled by default, checking columns against the uploaded schema to detect invalid data, though this may increase processing time and cost.

  9. Overview of Amorphic Shared ETL Libraries

    v2.6

    Shared ETL Libraries are central repositories of organization-approved packages used across multiple ETL jobs. Unlike External Libraries, which are scoped to a single job and deleted with it, Shared Libraries have a universal scope and persist independently of the jobs they are attached to.

    Key capabilities include:

    • Attaching multiple packages to a single library.
    • Attaching a single library to multiple jobs.
    • Granular customization of job dependencies.

    Supported file extensions by job type:

    • Python Shell applications: .py, .egg, .whl
    • PySpark applications: .py, .zip, .jar

  10. Overview of Amorphic Multi-tenancy

    v2.6
    Amorphic supports multi-tenancy to segregate organizational data and logically isolate users and resources. Each tenant's data is stored in separate databases, ensuring that users in one tenant cannot access resources in another unless explicitly granted. Multi-tenancy is currently only supported for Redshift type data lakes. Key capabilities include complete resource isolation, tenant-specific resource management, and support for cross-tenant querying.

  11. Overview of Amorphic Jobs

    v2.6
    Jobs in Amorphic allow users to execute Python, PySpark, and drag-and-drop built code to perform operations on datasets. Starting from version 2.2, encryption (both in-flight and at-rest) is enabled by default for all new and existing jobs and catalogs.

  12. Overview of Apache Hudi support in Amorphic

    v2.6

    Amorphic supports Apache Hudi datasets created with a Lakeformation target location. Key supported features include:

    • ACID transactions with snapshot isolation between writers, table services, and readers.
    • Support for Copy on Write (CoW) and Merge on Read (MoR) table types.
    • Supported file formats: CSV, JSON, and Parquet.
    • Ability to set/unset table properties (e.g., Hudi Write Operation Type).