artifactory-cleanup

repository·master·Indexed 41 days ago

https://github.com/devopshq/artifactory-cleanup

An extended and flexible cleanup tool for JFrog Artifactory that automates the removal of old or unused artifacts and Docker images. It supports cleanup policies defined via YAML or custom Python rules, featuring a dry-run mode by default and a --destroy flag for actual deletion. The tool can be installed via Docker or as a Python CLI and includes specialized rules for Docker images and Conan repositories.

Tokens
7K
Snippets
16
Records
17
Agent score
1%

What's inside artifactory-cleanup

  1. Install artifactory-cleanup via Docker or Python

    master

    You can install and run artifactory-cleanup using either a Docker container or as a Python CLI tool.

    Docker

    Pull the image and run the help command to verify:

    docker pull devopshq/artifactory-cleanup
    docker run --rm devopshq/artifactory-cleanup artifactory-cleanup --help

    Python (CLI)

    Install via pip:

    python3 -mpip install artifactory-cleanup
    artifactory-cleanup --help
    # docker
    docker pull devopshq/artifactory-cleanup
    docker run --rm devopshq/artifactory-cleanup artifactory-cleanup --help
    
    # python (later we call it 'cli')
    python3 -mpip install artifactory-cleanup
    artifactory-cleanup --help
  2. Configure and load custom rules

    master

    Once a custom rule is defined and registered in a Python file, you must reference it in your artifactory-cleanup.yaml and tell the tool where to find the implementation using the --load-rules flag.

    Configuration Example:

    - rule: MySimpleRule
      my_param: "Hello, world!"
      value: 42

    Execution via CLI:

    artifactory-cleanup --load-rules=myrule.py

    Execution via Docker:

    docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py
    # cli
    artifactory-cleanup --load-rules=myrule.py
  3. Run artifactory-cleanup in dry-run and destroy modes

    master

    By default, artifactory-cleanup runs in dry-run mode, which only prints the artifacts that would be deleted without actually removing them. To perform the actual deletion, you must explicitly use the --destroy flag or set the ARTIFACTORY_CLEANUP_DESTROY environment variable to True.

    # Dry run (default) - shows what will be deleted
    artifactory-cleanup
    
    # Actual removal
    artifactory-cleanup --destroy
    
    # Or via environment variable
    export ARTIFACTORY_CLEANUP_DESTROY=True
    artifactory-cleanup
  4. Connect self-signed certificates in Docker

    master

    If your Artifactory instance uses self-signed certificates, you must provide the certificate chain of trust to the Docker container.

    1. Place all certificates into a folder named certificates in your working directory.
    2. Mount this folder to /mnt/self-signed-certs/ inside the container using a volume flag.
    docker run -v "$(pwd)":/app -v "$(pwd)/certificates":/mnt/self-signed-certs/ devopshq/artifactory-cleanup artifactory-cleanup
  5. Create a custom Rule

    master

    You can extend the tool by implementing your own logic via the Rule class. A custom rule follows a specific lifecycle:

    1. Rule.check(*args, **kwargs): Verifies the rule configuration and can call other services.
    2. Rule.aql_add_filter(filters): Adds Artifactory Query Language (AQL) expressions to the query.
    3. Rule.aql_add_text(aql): Adds raw text to the resulting AQL query.
    4. Rule.filter(artifacts): Processes the list of artifacts returned by Artifactory. Crucially, this method must return the list of artifacts that are intended to be REMOVED.
      • To prevent an artifact from being deleted, use the artifacts.keep(artifact) method.

    To use a custom rule, create a Python file (e.g., myrule.py) in the same directory as your artifactory-cleanup.yaml, define your class inheriting from Rule, and call register(YourClassName) at the end of the file.

    # myrule.py
    from typing import List
    from artifactory_cleanup import register
    from artifactory_cleanup.rules import Rule, ArtifactsList
    
    class MySimpleRule(Rule):
        """
        This doc string is used as rule title
        """
        def __init__(self, my_param: str, value: int):
            self.my_param = my_param
            self.value = value
    
        def aql_add_filter(self, filters: List) -> List:
            return filters
    
        def filter(self, artifacts: ArtifactsList) -> ArtifactsList:
            # You can make requests to artifactory by using self.session:
            # url = f"/api/storage/{self.repo}"
            # r = self.session.get(url)
            # r.raise_for_status()
            return artifacts
    
    # Register your rule in the system
    register(MySimpleRule)
  6. Configure artifactory-cleanup using a YAML file

    master

    The tool uses a YAML configuration file (defaulting to artifactory-cleanup.yaml) to define the Artifactory server connection and cleanup policies.

    Security Note: Do not save credentials directly in the configuration file. Use environment variables (e.g., $ARTIFACTORY_USERNAME) which are automatically populated by the tool.

    # artifactory-cleanup.yaml
    artifactory-cleanup:
      server: https://repo.example.com/artifactory
      # $VAR is auto populated from environment variables
      user: $ARTIFACTORY_USERNAME
      password: $ARTIFACTORY_PASSWORD
    
      policies:
        - name: Remove all files from repo-name-here older than 7 days
          rules:
            - rule: Repo
              name: "reponame"
            - rule: DeleteOlderThan
              days: 7
  7. Use the Artifactory Cleanup CLI

    master

    The ArtifactoryCleanupCLI is the primary command-line interface for managing Artifactory artifact cleanup. It can run in two modes:

    1. Verbose Mode (Default): Simulates the cleanup process and reports how much space would be reclaimed without actually deleting anything.
    2. Destroy Mode: Uses the --destroy flag to actually remove the artifacts from Artifactory.

    By default, the tool uses a configuration file named artifactory-cleanup.yaml unless specified otherwise via --config or the ARTIFACTORY_CLEANUP_CONFIG_FILE environment variable.

    # Example: Simulate cleanup using a specific config
    artifactory-cleanup --config my-config.yaml
    
    # Example: Actually delete artifacts
    artifactory-cleanup --destroy
    
    # Example: Simulate cleanup for a specific policy
    artifactory-cleanup --policy "cleanup-old-docker-images"
  8. Keep the latest N Docker images

    master

    You can combine Docker-specific rules with standard file rules to maintain a specific number of recent images. For example, to delete old images but always keep the 3 most recent tags, use DeleteDockerImagesOlderThan, ExcludeDockerImages (to protect latest or release tags), and KeepLatestNFilesInFolder.

    artifactory-cleanup:
      server: https://repo.example.com/artifactory
      user: $ARTIFACTORY_USERNAME
      password: $ARTIFACTORY_PASSWORD
    
      policies:
        - name: Remove docker images, but keep last 3
          rules:
            - rule: Repo
              name: docker-demo
            - rule: DeleteDockerImagesOlderThan
              days: 30
            - rule: ExcludeDockerImages
              masks:
                - "*:latest"
                - "*:release*"
            - rule: ExcludePath
              masks: "*base-tools*"
            - rule: KeepLatestNFilesInFolder
              count: 3
  9. Clean up Conan repositories

    master

    To safely clean up Conan repositories without breaking metadata, use a two-policy approach:

    1. Policy 1: Removes actual files but preserves metadata files (using ExcludeFilename for patterns like .timestamp or index.json).
    2. Policy 2: Identifies folders that contain only metadata files and removes the empty folders to fix the index.

    Example Configuration:

    artifactory-cleanup:
      server: https://repo.example.com/artifactory
      user: $ARTIFACTORY_USERNAME
      password: $ARTIFACTORY_PASSWORD
    
      policies:
        - name: Conan - delete files older than 60 days
          rules:
            - rule: Repo
              name: "conan-testing"
            - rule: DeleteNotUsedSince
              days: 60
            - rule: ExcludeFilename
              masks:
                - ".timestamp"
                - "index.json"
        - name: Conan - delete empty folders (to fix the index)
          rules:
            - rule: Repo
              name: "conan-testing"
            - rule: DeleteEmptyFolders
            - rule: ExcludeFilename
              masks:
                - ".timestamp"
                - "index.json"
  10. Configure the CLI via environment variables

    master

    The following environment variables can be used to configure the CLI behavior without passing command-line arguments:

    VariableDescription
    ARTIFACTORY_CLEANUP_CONFIG_FILEPath to the configuration file (default: artifactory-cleanup.yaml)
    ARTIFACTORY_CLEANUP_DESTROYSet to enable artifact removal
    ARTIFACTORY_CLEANUP_IGNORE_NOT_FOUNDSet to ignore 404 errors during deletion
    ARTIFACTORY_CLEANUP_WORKER_COUNTNumber of parallel workers
    ARTIFACTORY_CLEANUP_CONFIG_FILEPath to the configuration file (default: artifactory-cleanup.yaml)
  11. Reference: Common cleanup rules

    master

    Common rules used to target specific repositories or artifacts based on properties.

    # Common Rules
    
    - `Repo` - Apply the rule to one repository. If no name is specified, it is taken from the rule name (in `CleanupPolicy` definition)
    
    ```yaml
    - rule: Repo
      name: reponame

    OR - if you have a single policy for the repo - you can name the policy as reponame

    Both configurations are equal

    policies:

    • name: reponame rules:
      • rule: Repo
    
    - `RepoList` - Apply the policy to list of repositories.
    
    ```yaml
    - rule: RepoList
      repos:
        - repo1
        - repo2
        - repo3
    • RepoByMask - Apply rule to repositories matching by mask
    - rule: RepoByMask
      mask: "*.banned"
    • PropertyEq - Delete repository artifacts only with a specific property value (property_key is the name of the parameter, property_value is the value)
    - rule: PropertyEq
      property_key: key-name
      property_value: 1
    • PropertyNeq - Delete repository artifacts only if the value != specified. If there is no value, delete it anyway. Allows you to specify the deletion flag do_not_delete = 1.
    - rule: PropertyNeq
      property_key: key-name
      property_value: 1
  12. Reference: Docker rules

    master

    Rules specifically designed for cleaning up Docker images in Artifactory.

    # Docker Rules
    
    - `DeleteDockerImagesOlderThan` - Delete docker images that are older than N days
    
    ```yaml
    - rule: DeleteDockerImagesOlderThan
      days: 1
    • DeleteDockerImagesOlderThanNDaysWithoutDownloads - Deletes docker images that are older than N days and have not been downloaded
    - rule: DeleteDockerImagesOlderThanNDaysWithoutDownloads
      days: 1
    • DeleteDockerImagesNotUsed - Removes Docker image not downloaded since N days
    - rule: DeleteDockerImagesNotUsed
      days: 1
    • IncludeDockerImages - Apply to docker images with the specified names and tags
    - rule: IncludeDockerImages
      masks: "*singlemask*"
    - rule: IncludeDockerImages
      masks:
        - "*production*"
        - "*release*"
    • ExcludeDockerImages - Exclude Docker images by name and tags.
    - rule: ExcludeDockerImages
      masks:
        - "*production*"
        - "*release*"
    • KeepLatestNVersionImagesByProperty(count=N, custom_regexp='some-regexp', number_of_digits_in_version=X) - Leaves N Docker images with the same major. (^\d+\.\d+\.\d+$) is the default regexp how to determine version which matches semver 1.1.1. If you need to add minor then set number_of_digits_in_version to 2 or if patch then set to 3 (by default we match major, which 1). Semver tags prefixed with v are supported by updating the regexp to include an optional v in the expression (e.g., (^v?\d+\.\d+\.\d+$)).
    - rule: KeepLatestNVersionImagesByProperty
      count: 1
      custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"
    • KeepLatestNDockerImages(count=N) - Leaves N most recently updated Docker image digests. This ensures all tags matching the same digest is kept.
    - rule: KeepLatestNDockerImages
      count: 1
    • DeleteDockerImageIfNotContainedInProperties(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None) - Remove Docker image, if it is not found in the properties of the artifact repository.

    • DeleteDockerImageIfNotContainedInPropertiesValue(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None) - Remove Docker image, if it is not found in the properties of the artifact repository.