DataDeps.jl

DataDeps.jl is designed for static data and does not provide built-in versioning for the data itself. If your data changes, DataDeps.jl cannot force users to download new copies automatically.

If you need to manage dynamic data, consider these workarounds:

• Manual Refresh: Use a deps/build.jl script to run rm(datadep"MyData", recursive=true, force=true) during package updates to force a re-download.
• Versioned Names: Treat each version of the data as a unique DataDep with a different name.
• Git Integration: If the data is small and plain-text, store it in your repository's deps/data/ folder and use a ManualDataDep to reference it. This avoids the need for manual installation when the package is installed via Git.

If multiple packages use the same DataDep name, DataDeps.jl may load the wrong one. To prevent this, you can move a package's specific data dependency into its own package directory under deps/data/.

For example, a dependency located at ~/.julia/v0.6/EXAMPLEPKG/deps/data/EXAMPLEDATADEP/ will be visible only to EXAMPLEPKG and will not conflict with other packages using the name EXAMPLEDATADEP.

Note on file-level loading: If you use datadep"Name/subfolder/file.txt" and the file is not in the first folder found with that Name, DataDeps.jl will present a recovery dialog. The easiest fix is to select the option to delete the folder and retry, which forces DataDeps to check the next folder in the load path.

While most use cases are covered by ManualDataDep or standard automatic DataDep types, you can extend DataDeps.jl by defining your own AbstractDataDep types. This is useful for complex workflows that require:

1. Additional Validation: Performing checks after the download is complete and the post_fetch_method has run.
2. Data Synthesis: Steps that require working with multiple files simultaneously (which exceeds the capabilities of the standard post_fetch_method).
3. Hybrid Workflows: Implementing a SemiManualDataDep pattern where some steps are user-driven and others are automated.

To implement a custom type, you can model it after the existing ManualDataDep implementation or encapsulate an existing DataDep as an element within your new type.

DataDeps.jl simplifies getting static files from a remote server onto a local machine and providing a consistent way for programs to locate them.

To use data in your project:

1. Host your data anywhere that provides an open HTTP/HTTPS link.
2. Write a DataDep registration block in your package.
3. Access the data using the datadep"Dataname/file.csv" syntax. DataDeps.jl handles the download and path resolution automatically.
4. For Continuous Integration (CI) environments, set the DATADEPS_ALWAYS_ACCEPT environment variable to bypass interactive prompts.

To avoid downloading data unless it is actually needed, ensure the datadep"Name" macro is only evaluated when required. DataDeps only triggers a download when the macro is evaluated.

Common patterns for lazy installation:

1. Conditional assignment: Only evaluate the macro if a certain condition (like training_mode) is met.
2. Function default arguments: Use the macro as a default value for a function parameter. The macro is only evaluated if the user calls the function without providing that argument.

# Pattern 1: Conditional assignment
data_folder = training_mode ? datadep"SecurityFootage" : "/srv/webcam/today"

# Pattern 2: Function default arguments
function predict(path=datadep"SecurityFootage")
    # ...
end

Install DataDeps.jl using the standard Julia package manager to ensure it is added to your Project.toml and automatically available to end-users.

using Pkg
Pkg.add("DataDeps")
# Or in the REPL:
pkg> add DataDeps

DataDeps 1.0 refactored the download/fetch API to use Downloads.download() instead of HTTP.jl. If you are upgrading from a version prior to 1.0, you must update your function calls as several wrappers were removed. The default fetch_method is now fetch.

# Replace removed wrappers with the new fetch API:

# 1. Replace fetch_default(remote, dir)
fetch(remote, dir)

# 2. Replace fetch_base(remote, dir)
fetch(remote, dir; update_period=Inf)

# 3. Replace fetch_http(remote, dir; update_period=...)
fetch(remote, dir; update_period=...)

# 4. If overloading download for custom types:
# Overload Downloads.download(::YourType) instead of Base.download(::YourType)

To ensure data is installed as soon as the package is loaded, place the datadep"Name" macro inside the package's __init__ function (immediately after the registration block).

Warning: Do not place the macro at the global scope, as it will run before __init__ and cause an error.

If you need to install data at Pkg.build time via deps/build.jl, you must set the environment variable ENV["DATADEPS_ALWAYS_ACCEPT"]="true" because user I/O (like download prompts) is not possible during the build process.

function MyPackage.__init__()
    # Registration code goes here
    datadep"MyData" # Eagerly trigger download
end