ZBox File Syncronization Tool

Premise:

This project, pronounced with a german accent as in "Zee-Box" or "The box," is my personalized imitation of dropbox. This project started many moons ago when Dropbox first dropped support for the 1TB tier and forced people to either be on the 2TB or 20Gb tiers. Of course there are other implementations out there, but "how hard can file syncing be!"

Sync algorithm:

The core of this entire program is a simple set of rules which I came up with the first night. They work fantastically and are based on the differences between three datasets:

The list of all files in a remote cache
The list of all files that were present last time the folder was synced
The list of all files that are currently present

These datasets give rise to the following rules:

Program Features:

Easy install with simple install script
Syncs to S3 (could be good or bad)
Ability to sync subfolders. This is one of the coolest ideas that I had while building this project. If my desktop has a large folder of 50Gb that I want to sync with tens of thousands of files, my laptop can specify that it only wants to sync a small subfolder of that large folder (e.g. Documents)
Fairly strong CLI interface
Strong focus on consistency. This program can be stopped at any point during execution (including a sync) with no loss of data or state (might have to retransmit a few files)

CLI Arguments:

Help for ZBox: ping: say pong remove: Removes box from local system Format: remove [options] Example: remove -b test1box [options] Options: (One of two first required) -p, -path [relative file path]: specifies root folder of box -b, -box [box name]: specifies name of box -f, -files: Removes all files present in the box create: Link a pre-existing box to a folder or create a new box if none exist. Format: create [box name] [folder path] Example: create test1box /home/joe/Documents Expected outcome: Link the public box test1box to /home/joe/Documents and upload all current files while downloading those that are already there. delete: Deletes box from S3. Format: delete [box name] This DOES NOT remove the box from the local filesystem Warning: This will require a password and should be done carefully ls: List files of a certain type. Format: ls [box name] [options] Example: ls test1box -a -h If no box is specified, all boxes will be listed instead Options: -s, -synced: lists all synced files -a, -all: lists all files in the box -d, -deleted: lists all recently deleted files getOutput: Hooks up an output-stream of current log data. Format: getOutput Warning: This must be exited with ctrl+C timingInfo: Prints timing info for current session Format: timingInfo sync: Force a sync of a box or of all boxes. Format: sync [options] OR Format: sync If no argument is provided, all boxes are synced. Options: (One of two required for single sync) -p, -path [relative file path]: specifies root folder of box -b, -box [box name]: specifies name of box start: shutdown: Shutdown the box: Options: -f, -force: force shutdown now. (Otherwise will finish sync in progress if any then shutdown): -t, -thread: only shut down the sync thread

Current Status:

This project is something that I have worked off and on for a while now. I have only just recently (2020) added it to github. I currently use it to syncronize school documents between my laptop and my desktop. It has a few performance issues and possible crashes with some large files that I haven't figured out yet. (While the simple sync algorithm is easy as pie, all the infastructure around it, and consistency requirements are remarkably difficult.)

If you're interested, the source code is available here (also linked above)