Open Charge Map (OCM) System

repository·master·Indexed 40 days ago

https://github.com/openchargemap/ocm-system

Backend systems for Open Charge Map, a global public registry of electric vehicle charging locations. Includes the .NET 6.x API, the OCM API Mirror Service for high-performance read operations via MongoDB, server-side import processing for OCPI providers, and localization tools.

Tokens
10K
Snippets
29
Records
50
Agent score
100%

What's inside Open Charge Map

  1. How failed import tracking and retries work

    master

    The worker manages failed imports using an in-memory cache to prevent resource wasting and cascading failures:

    • Failure Trigger: An import is marked as failed only when an actual exception occurs. If an import completes but importedOK = false (indicating no changes were needed), it is not treated as a failure.
    • Quarantine: When a failure occurs, the provider is added to the failed imports cache with a timestamp and is skipped for the next 24 hours (controlled by IMPORT_DUE_THRESHOLD_HOURS).
    • Recovery:
      • If an import succeeds, the provider is immediately removed from the failed imports cache.
      • If the 24-hour threshold expires, the provider is automatically removed from the cache and becomes eligible for retry in the next cycle if it is still due.
    • Failure Count: The worker tracks how many times a provider has failed for diagnostic purposes.
  2. How the OCM Import Worker scheduling works

    master

    The OCM Import Worker uses an intelligent, API-driven scheduling system to process data providers. The scheduling logic follows these rules:

    1. Cycle Execution: Every ImportRunFrequencyMinutes, the worker fetches the DataProvider list from the OCM API.
    2. Filtering: It filters providers based on the EnabledImports configuration and identifies those due for import (where DateLastImported is NULL or older than the IMPORT_DUE_THRESHOLD_HOURS, which defaults to 24 hours).
    3. Failure Exclusion: Providers that have failed an import within the last 24 hours are excluded from the current cycle.
    4. Priority Queue: Providers are processed in order of oldest to newest DateLastImported. The provider with the oldest timestamp has the highest priority.
    5. Wait Behavior: If no providers meet the criteria, the worker logs the status and waits for the next cycle without performing unnecessary imports.
  3. Install and build the OCM API on Linux

    master

    Follow these steps to set up the environment and build the API from source:

    1. Install .NET 6.0 SDK:
      sudo snap install dotnet-sdk --classic --channel=6.0
       Verify with `dotnet --version`.
    
    2. **Install MongoDB**: Install the latest version for your system and ensure the service is set to run on startup.
    
    3. **Clone and Build**:
       ```bash
    git clone https://github.com/openchargemap/ocm-system
    cd ocm-system/API/OCM.Net/OCM.API.Web
    dotnet build
    sudo snap install dotnet-sdk --classic --channel=6.0
    
    git clone https://github.com/openchargemap/ocm-system
    cd ocm-system/API/OCM.Net/OCM.API.Web
    dotnet build
  4. Add an OCPI Provider via Code (Legacy)

    master

    Use this approach only if the provider requires custom processing logic. This involves creating a new class that inherits from ImportProvider_OCPI.

    Steps:

    1. Add DataProvider definition in database.
    2. Add/get operator id for the target network.
    3. Store credential in keyvault.
    4. Implement the OCPI import provider in OCM.Import.Common.Providers.OCPI.
    5. Add import provider to OCM.Import.Worker:
      • Update appsettings.json under EnabledImports.
      • Add the provider to OCM.Import.Common.ImportManager.

    Implementation Details:

    • Download and verify the OCPI feed manually to check credentials and data format.
    • Add a test with a copy of the JSON feed in OCM.API.Tests - ImportTests to ensure data consistency.
    • Ensure the provider maps to its own network operator and any other relevant operators via GetOperatorMappings().
    using System.Collections.Generic;
    
    namespace OCM.Import.Providers.OCPI
    {
        public class ImportProvider_MyProvider : ImportProvider_OCPI, IImportProvider
        {
            public ImportProvider_MyProvider() : base()
            {
                ProviderName = "myprovider";
                OutputNamePrefix = "myprovider";
    
                IsAutoRefreshed = true;
                IsProductionReady = true;
    
                CredentialKey = "OCPI-MYPROVIDER"; // or null if no auth needed
    
                DefaultOperatorID = 1234;
    
                Init(dataProviderId: 99, "https://api.example.com/ocpi/2.2/locations");
            }
    
            public override Dictionary<string, int> GetOperatorMappings()
            {
                return new Dictionary<string, int>()
                {
                    { "My Operator", 1234 }
                };
            }
        }
    }
  5. Verify core application workflows

    master

    The following test plan outlines the critical user journeys required to validate the Open Charge Map application functionality:

    Search and Discovery

    1. Search Near Me: Verify that selecting 'Search Near Me' displays search progress and correctly renders search results.
    2. POI Details: Selecting a search result must display the Point of Interest (POI) details.
    3. Map Integration: Returning to the Map view via the Menu Icon must render markers for all current search results. Selecting a marker must display the POI details.
    4. Data Refresh: After adding comments or photos, refreshing search results must ensure new content appears in the POI details.

    User Authentication and Contributions

    1. Authentication Flow: Attempting to 'Add Comment' should prompt a Sign In. Completing the Sign In from the main menu should return the user to the app in a 'Signed in' state.
    2. Commenting: Once signed in, navigate to POI details, select 'Add Comment', enter details, and verify submission.
    3. Photo Uploads: Navigate to POI details, select 'Add Photo', enter details, and verify submission.

    Location and Profile Management

    1. Location Management: Verify the ability to edit an existing location and add a new location.
    2. Favorites: Verify the ability to add and remove a favorite location.
  6. Build the OCM Import Worker image

    master

    You can build the OCM Import Worker image using either standard Docker commands from the repository root or via Docker Compose.

    Using Docker: Run the build command from the ocm-system root directory. You can specify a build configuration (e.g., Release) or perform a clean build using --no-cache.

    Using Docker Compose: Navigate to the Import/OCM.Import.Worker directory and run docker-compose build.

    # Build using Docker from repository root
    docker build -f Import/OCM.Import.Worker/Dockerfile -t ocm-import-worker:latest .
    
    # Build using Docker Compose
    cd Import/OCM.Import.Worker
    docker-compose build
    
    # Build with specific configuration
    docker build \
      -f Import/OCM.Import.Worker/Dockerfile \
      --build-arg BUILD_CONFIGURATION=Release \
      -t ocm-import-worker:v1.0.0 \
      .
    
    # Build with no cache (clean build)
    docker build --no-cache \
      -f Import/OCM.Import.Worker/Dockerfile \
      -t ocm-import-worker:latest \
      .
  7. Prerequisites for running the OCM API Mirror

    master

    To run the API Mirror Service, your host must meet the following requirements:

    • Runtime: A host capable of running the latest .NET Core runtime (specifically .NET 6.x for current builds).
    • Database: MongoDB installed on the host with read/write permissions enabled for local connections.
    • Hardware (Production): At least 2GB of main RAM and adequate swap space configured. Lack of swap space on Linux can cause the service to hang.
  8. Debug the OCM Import Worker container

    master

    If the container is running or failing, use these methods to inspect the internal state:

    • Interactive Shell: Start a new container with a bash entrypoint or execute bash in a running container.
    • File Inspection: Use ls to check paths or cat to read appsettings.json.
    • Environment Check: Run env to verify loaded variables.
    • Resource Monitoring: Use docker stats for real-time usage or df -h for disk space.
    # Execute shell in running container
    docker exec -it ocm-import-worker /bin/bash
    
    # Check application files
    docker exec ocm-import-worker ls -la /app
    docker exec ocm-import-worker cat /app/appsettings.json
    docker exec ocm-import-worker env
    
    # Monitor performance
    docker stats ocm-import-worker
    docker exec ocm-import-worker df -h