TM1py

repository·master·Indexed 42 days ago

https://github.com/cubewise-code/tm1py

A Python module version 2.4.0 for interacting with IBM Planning Analytics (TM1). It provides high-level abstractions for data manipulation, metadata management, and process execution, including CRUD operations for cubes, dimensions, and subsets, as well as support for MDX queries and TurboIntegrator (TI) processes. Requires Python 3.7+ and TM1/Planning Analytics version 11 or higher.

Tokens
7.9K
Snippets
20
Records
35
Agent score
52%

What's inside TM1py

  1. Overview of TM1py features

    master

    TM1py is a Python library designed to interact with IBM Planning Analytics (TM1). It provides a high-level interface for common TM1 tasks, including:

    • Data Retrieval: Read data from cubes using cube views or MDX queries (e.g., tm1.cells.execute_mdx).
    • Data Writing: Write data to cubes (e.g., tm1.cells.write) or use asynchronous functions to parallelize operations (e.g., tm1.cells.write_async).
    • Metadata Management: Update dimensions and hierarchies (e.g., tm1.hierarchies.get) and manage elements using pandas DataFrames (e.g., tm1.elements.get_elements_dataframe).
    • Process Execution: Execute TI (TurboIntegrator) processes or loose TI statements (e.g., tm1.processes.execute_with_return).
    • CRUD Operations: Full Create, Read, Update, and Delete capabilities for all TM1 objects, including cubes, dimensions, and subsets.
  2. Core features of TM1py

    master

    TM1py provides a comprehensive suite of tools for interacting with IBM Planning Analytics (TM1):

    • Data Operations: Read data via cube views or MDX queries (tm1.cells.execute_mdx) and write data to cubes (tm1.cells.write).
    • Metadata Management: Update dimensions, hierarchies (tm1.hierarchies.get), and elements. It supports CRUD operations for all TM1 objects (cubes, dimensions, subsets, etc.).
    • Pandas Integration: Update metadata or write/clear cubes directly using pandas DataFrames (e.g., tm1.elements.get_elements_dataframe).
    • Process Execution: Execute TI (TurboIntegrator) processes or loose TI statements (tm1.processes.execute_with_return).
    • Concurrency: Use async functions to parallelize read/write operations (e.g., tm1.cells.write_async).
    with TM1Service(address='localhost', port=8001, user='admin', password='apple', ssl=True) as tm1:
        subset = Subset(dimension_name='Month', subset_name='Q1', elements=['Jan', 'Feb', 'Mar'])
        tm1.subsets.create(subset, private=True)
  3. Explore TM1py extensions and tools

    master

    Several community and professional tools extend the functionality of TM1py or work alongside it:

    • OptimusPy: An optimization tool to find the ideal dimension order for your cubes (GitHub).
    • tm1cli: A command-line interface for TM1 (GitHub).
    • RushTI: A tool that uses TM1py to execute TI processes in parallel using only a single connection (GitHub).
    • TM1 Data Integration and Automation Toolkit: A toolkit for data integration and automation in TM1 (GitHub).
  4. Understand the documentation deployment workflow

    master

    The TM1py documentation is automatically deployed to the gh-pages branch via GitHub Actions (defined in .github/workflows/docs.yml). Deployment is triggered by:

    • Pushing to master: Deploys the changes as the latest version.
    • Tagging a commit: Using a version tag like x.x or x.x.x (e.g., 2.0 or 2.0.1) deploys the documentation as a specific versioned release.
    • Manual Trigger: Can be initiated via the GitHub Actions interface.

    The deployment uses Mike to manage versioned documentation and handles the CNAME file for custom domains.

  5. Understand the TM1Py release process

    master

    TM1Py uses automated nightly releases based on semantic versioning.

    For Contributors

    You do not need to manage versioning, branch naming, or release notes. Simply submit your PR and wait for review.

    For Maintainers (Release Labels)

    To trigger a version bump during the nightly automated workflow, maintainers must add one of the following labels to a merged PR:

    LabelVersion BumpUse Case
    release:patch2.1.5 → 2.1.6Bug fixes, small improvements
    release:minor2.1.5 → 2.2.0New features, enhancements
    release:major2.1.5 → 3.0.0Breaking changes, major updates
    skip-release (or no label)No bumpDocs, tests, or CI changes

    Note: If no release label is added, no release will be created. This is the default behavior.

    Release Lifecycle

    1. Merge to master: PR is merged after validation passes.
    2. Nightly Workflow (4 AM CET): An automated process runs integration tests, determines the version bump from labels, creates a GitHub Release, and publishes to PyPI.
    3. Availability: New versions are available via pip install --upgrade TM1py the following morning.
  6. Set up the TM1py documentation environment

    master

    To modify the TM1py documentation, you need to clone the repository and install the necessary development and documentation dependencies.

    1. Clone the repository:
      git clone https://github.com/cubewise-code/tm1py.git
      cd tm1py
    2. Install dependencies: Install the package in editable mode with development extras, then install the documentation-specific requirements:
      pip install -e .[dev]
      pip install -r docs/requirements-docs.txt
    git clone https://github.com/cubewise-code/tm1py.git
    cd tm1py
    pip install -e .[dev]
    pip install -r docs/requirements-docs.txt
  7. Make changes and run tests

    master

    When making changes to the codebase, follow this workflow:

    1. Create a feature branch:

      git checkout -b your-feature-name
    2. Format and Lint your code: Use black for formatting and ruff for linting and fixing:

      black .
      ruff check --fix .
    3. Run Tests: If you have access to a TM1 instance, run the test suite using pytest:

      pytest Tests/

      You can also run specific files or use verbose output:

      pytest Tests/test_cube_service.py
      pytest Tests/ -v
    git checkout -b your-feature-name
    black .
    ruff check --fix .
    pytest Tests/
  8. Connect to TM1 using TM1Service

    master

    The primary way to interact with TM1 is through the TM1Service class, which is used as a context manager. The connection parameters depend on your TM1 deployment type (On-premise, IBM Cloud, or PAaaS).

    TM1 11 On-premise

    Use address, port, user, password, and ssl.

    TM1 11 on IBM Cloud

    Use base_url, user, namespace, password, ssl, verify, and async_requests_mode.

    TM1 12 PAaaS

    Use base_url, user, password, async_requests_mode, ssl, and verify.

    TM1 12 On-premise & Cloud Pak For Data

    Use address, instance, database, application_client_id, application_client_secret, user, and ssl.

    TM1 12 On-premise with Access Token

    Use base_url, user, access_token, async_requests_mode, ssl, and verify.

    from TM1py.Services import TM1Service
    
    # Example: TM1 11 On-premise
    with TM1Service(address='localhost', port=8001, user='admin', password='apple', ssl=True) as tm1:
        print(tm1.server.get_product_version())
  9. Install TM1py

    master

    You can install TM1py using pip. Depending on whether you need pandas integration, choose one of the following commands:

    • Standard installation (without pandas): pip install tm1py

    • Installation with pandas support (recommended for data manipulation): pip install "tm1py[pandas]"

    pip install "tm1py[pandas]"
  10. Implement the TM1Py Integration Test Workflow

    master

    Create a workflow file at .github/workflows/tm1py-integration-tests.yml to run tests in parallel across a matrix of environments. The workflow uses workflow_dispatch to allow manual triggers with a JSON array of environments.

    Key steps in the workflow:

    1. Checkout code using actions/checkout@v3.
    2. Set up Python using actions/setup-python@v4.
    3. Install dependencies using pip install -e .[unit_test].
    4. Generate config.ini by running the Tests/resources/generate_config.py script, which consumes TM1_CONNECTION and TM1_CONNECTION_SECRET.
    5. Run tests using pytest Tests/.
    6. Upload artifacts containing test reports.
    name: TM1Py Integration Tests
    
    on:
      workflow_dispatch:
        inputs:
          environments:
            description: 'JSON array of environments to test (e.g., ["tm1-11-onprem", "tm1-11-cloud"])"
            required: true
            default: '["tm1-11-onprem","tm1-11-cloud","tm1-12-mcsp"]'
    
    jobs:
      test:
        runs-on: ubuntu-latest
        strategy:
          matrix:
            environment: ${{ fromJson(github.event.inputs.environments) }}
        environment: ${{ matrix.environment }}
        steps:
          - name: Checkout code
            uses: actions/checkout@v3
    
          - name: Set up Python
            uses: actions/setup-python@v4
            with:
              python-version: '3.x'
    
          - name: Install dependencies
            run: |
              pip install -e .[unit_test]
    
          - name: Retrieve TM1 Connection Details
            id: get-connection
            run: echo "Retrieving TM1 connection details"
            env:
              TM1_CONNECTION: ${{ vars.TM1_CONNECTION }}
              TM1_CONNECTION_SECRET: ${{ secrets.TM1_CONNECTION_SECRET }}
    
          - name: Generate config.ini
            run: |
              python Tests/resources/generate_config.py
            env:
              TM1_CONNECTION: ${{ vars.TM1_CONNECTION }}
              TM1_CONNECTION_SECRET: ${{ secrets.TM1_CONNECTION_SECRET }}
    
          - name: Run tests
            run: pytest Tests/
    
          - name: Upload test results
            if: always()
            uses: actions/upload-artifact@v3
            with:
              name: test-results-${{ matrix.environment }}
              path: Tests/test-reports/
    
          # Optional: Post results to PR
          - name: Post comment to PR
            if: always() && github.event_name == 'workflow_dispatch' && github.event.pull_request
            uses: actions/github-script@v6
            with:
              script: |
                github.rest.issues.createComment({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  issue_number: context.payload.pull_request.number,
                  body: 'Tests completed for environment: ${{ matrix.environment }}. Check artifacts for details.'
                })