textdistance Documentation

You can use the EntropyNCD distance metric from textdistance to compare contents of multiple files. EntropyNCD can be instantiated and then called with two strings to return the distance value. In the example below, it is used within a nested loop to build a list of distance tuples containing the names of the compared items and their calculated distance.

from textdistance import EntropyNCD

distances = []
for name1, content1 in licenses.items():
    for name2, content2 in licenses.items():
        # EntropyNCD(qval=None) initializes the metric
        distances.append((name1, name2, EntropyNCD(qval=None)(content1, content2)))

To visualize a distance matrix, convert the distance results into a pandas.DataFrame and use plotnine to create a heatmap. The geom_tile layer is used to map the names to axes and the distance to the fill color.

import plotnine as gg
import pandas as pd

# Assuming 'distances' is a list of (name1, name2, distance) tuples
df = pd.DataFrame(distances, columns=['name1', 'name2', 'distance'])

(
    gg.ggplot(df)
    + gg.geom_tile(gg.aes(x='name1', y='name2', fill='distance'))
    + gg.scale_fill_continuous(palette=lambda *args: gg.scale_fill_continuous().palette(*args)[::-1])
    + gg.theme(
        figure_size=(12, 8), 
        axis_text_x=gg.element_text(angle=90),
    )
)