repository·master·Indexed 42 days ago
https://github.com/lipoja/urlextractA Python library and command-line tool for extracting URLs from unstructured text by locating Top-Level Domains (TLDs). It provides the URLExtract class for programmatic use and a CLI for terminal-based extraction, featuring DNS validation, uniqueness filtering, and TLD list updates from iana.org.
urlextract is a Python package designed to extract URLs from provided text. It provides two primary ways to use its functionality:
.name is a valid TLD, so p.bold.name might be extracted as a URL).urlextract project, use the tox command-line tool. This will run the automated tests in their configured environments.$ toxYou can install the urlextract package directly from PyPI using pip.
pip install urlextractThe urlextract command-line program scans an input text file and prints all discovered URLs to stdout.
Basic usage:
$ urlextract <input_file>If you are installing dependencies manually, ensure you have the following required packages installed:
idna: For converting links to IDNA format.uritools: For domain name validation.platformdirs: For determining the user's cache directory.dnspython: To cache DNS results.You can also install them via a requirements.txt file.
pip install idna
pip install uritools
platformdirs
dnspython
# Or via requirements.txt
pip install -r requirements.txtTo process URLs one by one using a generator, use the gen_urls method.
from urlextract import URLExtract
extractor = URLExtract()
example_text = "Text with URLs. Let's have URL janlipovsky.cz as an example."
for url in extractor.gen_urls(example_text):
print(url)URLExtract downloads its list of TLDs from iana.org. You can manually trigger an update or set a threshold for automatic updates to ensure you have the latest TLDs.
from urlextract import URLExtract
extractor = URLExtract()
# Force an immediate update
extractor.update()
# Update only if the current list is older than X days
extractor.update_when_older(7)The URLExtract class provides several methods to find URLs in a given string. It works by locating Top-Level Domains (TLDs) and expanding the boundaries to find the full URL.
from urlextract import URLExtract
extractor = URLExtract()
urls = extractor.find_urls("Text with URLs. Let's have URL janlipovsky.cz as an example.")
print(urls) # prints: ['janlipovsky.cz']urlextract.URLExtract class is the primary interface for extracting URLs from text. It provides methods to identify and retrieve URL patterns within a given string. Use this class to automate the discovery of web links in unstructured text data.If you only need to know if a string contains at least one URL, use the has_urls method.
from urlextract import URLExtract
extractor = URLExtract()
example_text = "Text with URLs. Let's have URL janlipovsky.cz as an example."
if extractor.has_urls(example_text):
print("Given text contains some URL")The urlextract CLI provides several flags to filter, validate, and limit the URL extraction process:
-u, --unique: Only print unique URLs found in the file.-dl, --disable-localhost: Prevents the extraction of localhost as a URL.-i <ignore_file>, --ignore-file <ignore_file>: Provide a text file containing URLs to be excluded from the results.-p <permit_file>, --permit-file <permit_file>: Provide a text file containing only the URLs that are allowed to be processed.-c, --check-dns: Only prints URLs that have existing domain names (performs DNS checks).-l LIMIT, --limit LIMIT: Sets a maximum count of URLs to process. Set to 0 to disable the limit. The default limit is 10000.-h, --help: Show help message.-v, --version: Show version number.Usage: $ urlextract [-h] [-v] [-u] [-dl] [-c] [-i <ignore_file>] [-p <permit_file>] [-l LIMIT] [<input_file>]
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
-u, --unique print out only unique URLs found in file
-dl, --disable-localhost
disable extracting "localhost" as URL
-c, --check-dns print out only URLs for existing domain names
-i <ignore_file>, --ignore-file <ignore_file>
input text file with URLs to exclude from extraction
-p <permit_file>, --permit-file <permit_file>
input text file with URLs that can be processed
-l LIMIT, --limit LIMIT
Maximum count of URLs that can be processed. Set 0 to
disable the limit. Default: 10000