CDC File Transfer

CDC File Transfer provides tools for syncing and streaming files from Windows to Windows or Linux using Content Defined Chunking (CDC), specifically the FastCDC algorithm. It is designed to overcome the limitations of scp and standard rsync when transferring large amounts of data or many small files over sub-par internet connections.

Key tools include:

• cdc_rsync: A sync tool optimized for cases where an old version of files already exists on the target. It uses fast compression and only transfers the differences (deltas) when files change.
• cdc_stream: A tool for streaming files and directories from Windows to Linux, optimized for read speed. It caches data on the Linux device and only re-streams differences if a file is re-read after changing on Windows. Note that the Linux directory is read-only.

To use the tools without building from source:

1. Download the precompiled binaries from the latest release to a Windows device.
2. Unzip the files.
3. The Windows tools will automatically deploy the necessary Linux binaries to ~/.cache/cdc-file-transfer on the Linux device. No manual deployment is required.

To build cdc_stream, you must build components on both Linux and Windows and then combine them.

1. On Linux, build the FUSE filesystem component:

   bazel build --config linux --compilation_mode=opt --linkopt=-Wl,--strip-all --copt=-fdata-sections --copt=-ffunction-sections --linkopt=-Wl,--gc-sections //cdc_fuse_fs

2. On Windows, build the client component:

   bazel build --config windows --compilation_mode=opt --copt=/GL //cdc_stream

3. Combine: Copy the Linux build output files cdc_fuse_fs and libfuse.so from bazel-bin/cdc_fuse_fs to the Windows machine's bazel-bin\cdc_stream directory.

The tools require passwordless SSH and SFTP access from the Windows machine to the Linux device (e.g., via key-based authentication).

By default, the tools look for ssh.exe and sftp.exe in your system PATH. You can verify your setup by running:

ssh user@linux.device.com
sftp user@linux.device.com

If you need to provide additional arguments or specific paths, you can use an SSH config file at %USERPROFILE%\.ssh\config or specify the commands via environment variables or CLI flags.

set CDC_SSH_COMMAND="C:\path with space\to\ssh.exe"
set CDC_SFTP_COMMAND="C:\path with space\to\sftp.exe"

To build cdc_rsync, you must build components on both Linux and Windows and then combine them.

1. On Linux, build the server component:

   bazel build --config linux --compilation_mode=opt --linkopt=-Wl,--strip-all --copt=-fdata-sections --copt=-ffunction-sections --linkopt=-Wl,--gc-sections //cdc_rsync_server

2. On Windows, build the client component:

   bazel build --config windows --compilation_mode=opt --copt=/GL //cdc_rsync

3. Combine: Copy the Linux build output cdc_rsync_server from bazel-bin/cdc_rsync_server to the Windows machine's bazel-bin\cdc_rsync directory.

The CDC algorithm can be tuned for experimentation using preprocessor macros defined in indexer.h. You can pass these constants during the build process using the --copt flag in Bazel. For example, you can set CDC_GEAR_BITS to a specific value.

bazel build -c opt --copt=-DCDC_GEAR_BITS=32 //cdc_indexer

You can run the CDC indexer sample using Bazel. Use the --inputs flag to specify the path to the files you wish to process.

bazel run -c opt //cdc_indexer -- --inputs '/path/to/files'

To build from source, perform these steps on both Windows and Linux:

1. Install Bazel.
2. Clone the repository:

   git clone https://github.com/google/cdc-file-transfer

3. Initialize submodules:

   cd cdc-file-transfer
   git submodule update --init --recursive

Windows specific requirement: Ensure an SSH client is installed on the Windows machine. The tools require ssh.exe and sftp.exe to function.

If ssh.exe or sftp.exe are not in your PATH, or if you need to pass specific flags (like identity files or ports), set the CDC_SSH_COMMAND and CDC_SFTP_COMMAND environment variables.

Note: Use lowercase -p for ssh.exe and uppercase -P for sftp.exe when passing port arguments.

set CDC_SSH_COMMAND=C:\path\to\ssh.exe -p 12345 -i C:\path\to\id_rsa -oUserKnownHostsFile=C:\path\to\known_hosts
set CDC_SFTP_COMMAND=C:\path\to\sftp.exe -P 12345 -i C:\path\to\id_rsa -oUserKnownHostsFile=C:\path\to\known_hosts

cdc_stream operates via a background service.

• Default Log Location: %APPDATA%\cdc-file-transfer\logs
• Custom Service Configuration: Create a JSON file at %APPDATA%\cdc-file-transfer\cdc_stream.json to pass flags to the service (e.g., {"verbosity":3}).
• Manual Service Execution: Run cdc_stream start-service directly. Use the --log-to-stdout flag to see logs in the console.
• Debugging SSH/SFTP: Both tools include SSH/SFTP commands in debug logs. If a command fails, copy it and run it manually with -vv or -vvv to investigate.