pySmartDL

pySmartDL is a smart download manager for Python that provides several advanced features:

• Download Acceleration: Uses multipart downloading techniques to increase speed.
• Mirrors Support: Can utilize multiple mirror URLs.
• Control Features: Supports pausing, unpausing, and speed limiting.
• Integrity: Includes hash checking.
• User Interface: Non-blocking execution with a progress bar, download speed, and ETA.
• Customization: Full support for custom HTTP headers and methods.
• Compatibility: Built for Python 3.

When implementing a counter to track downloaded bytes across multiple threads in pySmartDL, you must use a mechanism that ensures data integrity.

Based on performance testing, using multiprocessing.Value (referred to as shared3 in the implementation notes) is the recommended approach. It provides the highest accuracy for large file downloads compared to global variables or custom Python objects with thread locks.

Comparison of approaches for a 64MB file:

• multiprocessing.Value: Most accurate (approx. 0.06% error rate).
• Global variable: Low error rate (approx. 0.07%) but potentially unsafe for complex concurrency.
• SharedObject() (Python object with locks): High error rate (approx. 3.72%) and not recommended.

import multiprocessing

# Recommended approach for thread-safe byte counting
shared3 = multiprocessing.Value('L', 0) # 'L' for unsigned long

To install from source, download the source distribution, unpack the zip file, and run the installation script using distutils to install the package site-wide.

python setup.py install

The recommended way to install pySmartDL is using pip. Ensure python-pip is installed on your system. If you are using a virtualenv, pip is already included in your environment.

pip install pySmartDL

Instead of saving to a file, you can fetch the downloaded content directly into memory using .get_data(). This is useful for processing text or JSON content without disk I/O.

from pySmartDL import SmartDL
 
url = "http://wiki.python.org/moin/Python2orPython3"
obj = SmartDL(url, progress_bar=False)
obj.start()

data = obj.get_data()  # Returns the content (e.g., HTML) as data

If you do not provide a destination path to the SmartDL constructor, the library will choose a temporary location. You can disable the progress bar by setting progress_bar=False in the constructor.

from pySmartDL import SmartDL
 
url = "https://github.com/iTaybb/pySmartDL/raw/master/test/7za920.zip"
obj = SmartDL(url, progress_bar=False)
obj.start()

# Use obj.get_dest() to find where the file was saved

To download a file to a specific directory, pass the URL and the destination path to the SmartDL constructor. Call .start() to begin the download. You can retrieve the final file path using .get_dest().

from pySmartDL import SmartDL

url = "https://github.com/iTaybb/pySmartDL/raw/master/test/7za920.zip"
dest = "C:\\Downloads\" # or '~/Downloads/' on linux

obj = SmartDL(url, dest)
obj.start()

path = obj.get_dest()

You can pass a list of URLs to SmartDL. The library will attempt to download from them sequentially. If one URL fails, it will try the next. .isSuccessful() will return True if at least one mirror worked.

from pySmartDL import SmartDL
 
urls = ["http://totally_fake_website/7za.zip" ,"https://github.com/iTaybb/pySmartDL/raw/master/test/7za920.zip"]
obj = SmartDL(urls, progress_bar=False)
obj.start()

print(obj.isSuccessful()) # Returns True if any mirror succeeded

Use the request_args parameter in the SmartDL constructor to pass custom arguments (like headers) to the underlying urllib.request.Request object. This is useful for setting custom User-Agent strings or other HTTP headers.

from pySmartDL import SmartDL

request_args = {"headers": {"User-Agent": "pySmartDL/1.3.2"}}
obj = SmartDL("http://httpbin.org/headers", request_args=request_args, progress_bar=False)
obj.start()

data = obj.get_json()
print(data)

To download from a site with invalid or self-signed SSL certificates, pass verify=False to the SmartDL constructor.

from pySmartDL import SmartDL

obj = SmartDL("https://github.com/iTaybb/pySmartDL/raw/master/test/7za920.zip", verify=False)
obj.start()

You can ensure file integrity by adding a hash verification requirement using .add_hash_verification(algorithm, hash_value). Supported algorithms include 'md5', 'sha1', and 'sha256'. If the downloaded file does not match the hash, a HashFailedException is raised (in blocking mode) or .isSuccessful() will return False (in non-blocking mode).

from pySmartDL import SmartDL, HashFailedException
 
urls = ["https://github.com/iTaybb/pySmartDL/raw/master/test/7za920.zip"]
obj = SmartDL(urls, progress_bar=False)
obj.add_hash_verification('sha256' ,'2a3afe19c180f8373fa02ff00254d5394fec0349f5804e0ad2f6067854ff28ac')

try:
    obj.start()
except HashFailedException:
    print("Hash check failed!")

To download a file, import the SmartDL class, create an instance by providing the target URL and the destination directory, and then call the .start() method. You can retrieve the final file path using .get_dest() after the download is complete.

from pySmartDL import SmartDL

url = "https://github.com/iTaybb/pySmartDL/raw/master/test/7za920.zip"
dest = "C:\\Downloads\" # or '~/Downloads/' on linux

obj = SmartDL(url, dest)
obj.start()
# Progress bar is shown during execution

path = obj.get_dest()