Rclone & MergerFS Explainer

This guide explains the concepts of Rclone and MergerFS to empower you to manage and support your tailored setups. It is not intended to be a detailed “How To” guide. We have a selection of those here.

Overview

You can mount and interact with your cloud storage as if it is a directory (folder) within your home directory. For example, you can create a directory called Gdrive on your server that contains all the files stored on your Google Drive.

To achieve this, we use an open-source tool called Rclone to connect to cloud storage and create mounted directories. The basic steps are:

  1. Download the latest version of Rclone. Your ultra.cc service comes with Rclone pre-installed, but it is only updated occasionally, so it’s better to have your own up-to-date version.
  2. Create an Rclone remote. This means configuring Rclone with details about the cloud storage provider (or server) you want to connect to. Once this step is completed, you can interact with your cloud storage using rclone <command> <remotename:>.
  3. Use the Rclone remote created in step 2 to mount your cloud storage to a directory using the rclone mount <remotename:> command.
  4. Set up MergerFS to union your cloud files and local files into a single directory.

Rclone

Installation (Part 1)

curl https://scripts.usbx.me/main/MergerFS-Rclone/Installer%20Scripts/rclone-install-stable.sh | bash

*If you are installing rclone to use at home or elsewhere, download rclone from the official site and manually install it.

rclone version
rclone selfupdate

Configuring a Rclone Remote (Part 2)

Now that you have the latest version of Rclone, you can configure a connection to any supported storage provider. The process is similar for all storage providers, but each has specific options that need to be considered.

To create a new Rclone remote:

rclone config
  1. Select n for New Remote
user@server:~$ rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
  1. Select a name of your choice. It makes sense to select a meaningful name; for example, if you’re creating a connection to Dropbox, select something like dropbox or dropbox1 if you plan on having more than one!
Enter name for new remote.
name> dropbox
  1. Type the number next to the storage provider you want to connect to. When Rclone adds new providers, all the numbers can change, so ensure you select the correct one. At the time of writing, if connecting to Dropbox, I would type 13 and press ENTER.
....
13 / Dropbox
   \ (dropbox)
....
Storage> 13
  1. From this point forward, the options you will be prompted to supply will differ for each storage provider. Jump to https://rclone.org and select the required option from the Storage Systems dropdown menu. This will guide you through the specifics of each provider.

On your local PC:

  1. Install Rclone locally. You can find the installation files here: https://rclone.org/downloads/
  2. Drop into the command shell (Command Prompt on Windows, Terminal on Mac).
  3. Change to the directory (using the cd command) that you installed Rclone in to, for example: cd \rclone.
  4. Type rclone version to ensure it works. The version number should be displayed. If it isn’t, review steps 5 to 7 above.
Microsoft Windows [Version 10.0.22621.2715]
(c) Microsoft Corporation. All rights reserved.

C:\Users\yourname>cd\rclone

C:\rclone>rclone version
rclone v1.64.2
- os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
- os/kernel: 10.0.22621.2715 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: cmount
  1. Now jump back to your Ultra.cc ssh session and copy the Rclone command from there into your Command Prompt/Terminal. For example, for Dropbox, you would copy this line to your local device:
rclone authorize "dropbox"
  1. A web browser session will open. Log in to the cloud storage provider using the credentials you signed up with. If you have switched on Two Factor Authentication with your cloud storage provider, you will need to supply those details too.
  2. You will then be asked to Approve/Accept Rclone. Ensure you are happy to do so, and click on Approve/Accept (or similar).
  3. Now go back to the Command Prompt/Terminal on your local PC/Mac and you will see a token displayed. Copy this to your clipboard (everything between the ---> and <---)
C:\Users\username>cd\rclone

C:\rclone>rclone authorize "dropbox"
2023/11/30 16:01:20 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=m5bBA0FfH8nMZyHmGkP0Tg
2023/11/30 16:01:20 NOTICE: Log in and authorize rclone for access
2023/11/30 16:01:20 NOTICE: Waiting for code...
2023/11/30 16:06:29 NOTICE: Got code
Paste the following into your remote machine --->
{"access_token":"sl.Bq2irrOkj8l6-ngKY6apc6Gkhbxb0tRnJECnNG1iw6PAF9s6TzPuDG4Baegb9da3yBeDCzP1D9ZtQ4_LYuDAJLmOjPuFyFPiw2lnlP_TpUS26oNMX_SZPGSrWHFJHHorvaYe46jVvFFBiaU","token_type":"bearer","refresh_token":"jGekUl1W1UIAAAAAAAAAAd1zHanzbZlMaSMlC8ogY79hyvV1mNUVbFOt6nPJnCUM","expiry":"2023-11-30T20:06:30.642061Z"}
<---End paste

On your Ultra.cc ssh session:

  1. Switch back to your Ultra.cc ssh session and paste the token.
Use web browser to automatically authenticate rclone with remote?
 * Say Y if the machine running rclone has a web browser you can use
 * Say N if running rclone on a (remote) machine without web browser access
If not sure try Y. If Y failed, try N.

y) Yes (default)
n) No
y/n> n

Option config_token.
For this to work, you will need rclone available on a machine that has
a web browser available.
For more help and alternate methods see: https://rclone.org/remote_setup/
Execute the following on the machine with the web browser (same rclone
version recommended):
        rclone authorize "dropbox"
Then paste the result.
Enter a value.
config_token> {"access_token":"sl.Bq2irrOkj8l6-ngKY6apc6Gkhbxb0tRnJECnNG1iw6PAF9s6TzPuDG4Baegb9da3yBeDCzP1D9ZtQ4_LYuDAJLmOjPuFyFPiw2lnlP_TpUS26oNMX_SZPGSrWHFJHHorvaYe46jVvFFBiaU","token_type":"bearer","refresh_token":"jGekUl1W1UIAAAAAAAAAAd1zHanzbZlMaSMlC8ogY79hyvV1mNUVbFOt6nPJnCUM","expiry":"2023-11-30T20:06:30.642061Z"}
  1. Finish the remaining configuration options.
rclone lsd remotename:
rclone copy yourlocalfile remotename:folder-path
rclone copy remotename:full-path-and-filename localpath

Creating a Mount (Part 3)

  1. Create a new directory that will be used as access your cloud files - this can be called anything you chose and doesn’t need to be the same as your rclone remote name you created above (step 2) but it can be the same if you find that easier.
mkdir DropBoxFiles
  1. Now we want to mount our Rclone remote to that directory:
rclone mount remotename: directoryname
rclone remote dropbox: DropBoxFiles
  1. This command will continue to run until you press CTRL+C. So how do you test? Open up another ssh session, leaving the rclone mount running.
ls -al directoryname
ls -al DropBoxFiles

You now have the knowledge to create a mount to any cloud storage provider that Rclone supports! Next, you should consider MergerFS.

MergerFS (Step 4)

While Ultra.cc services come pre-installed with MergerFS, it is better to download your own version so you can keep it up to date. See the Install MergerFS part of this guide.

MergerFS = Directory_A UNION Directory_B (UNION Directory_C...)

The following visualizes how the union works:

mergerfs.png

mergerfs ~/Directory_A:~/Directory_B ~/MergerFS
mergerfs ~/Directory_A:~/Directory_B:~/Directory_C:~/Directory_D ~/MergerFS
mergerfs ~/Stuff/Local:~/Stuff/Gdrive:~/Stuff/Dropbox ~/MergerFS

where:

[Unit]
Description = MergerFS Service
After=rclone-vfs.service
RequiresMountsFor=%h/Stuff/Local
RequiresMountsFor=%h/Stuff/Mount

[Service]
Type=forking
TimeoutStopSec=60
ExecStart=%h/bin/mergerfs \
    -o use_ino,func.getattr=newest,category.action=all \
    -o category.create=ff,cache.files=auto-full,threads=8 \
    %h/Stuff/Local:%h/Stuff/Mount %h/MergerFS

StandardOutput=file:%h/scripts/mergerfs_mount.log
ExecStop=/bin/fusermount -uz %h/MergerFS
Restart=on-failure

[Install]
WantedBy=default.target
mkdir -p ~/Stuff/Local  for your local files
mkdir -p ~/Stuff/Mount  for your rclone mount
mkdir ~/MergerFS        for your mergerfs unioned mount
mksir ~/scripts         for the log file & upload script
systemctl --user enable --now mergerfs.service
%h/Stuff/Local:%h/Stuff/Mount %h/MergerFS

Multiple Cloud Providers

Using the information you have learned here should mean you can create Rclone mounts for all your cloud providers and add them to your MergerFS mount. Just keep in mind:

  1. Each connection to a cloud provider needs its own Rclone remote (rclone config)
  2. Each cloud provider needs its own directory to display the files (rclone mount)
  3. Each cloud provider needs its own service file to perform the mount (nano ~/.config/systemd/user/rclone-mountname.service)
  4. Each cloud provider mount directory needs to be added to your MergerFS service file (nano ~/.config/systemd/user/mergerfs.service)
  5. Remember, any time you edit a service file, run systemctl --user daemon-reload or you will get an error telling you to do so.
  6. You need to restart your service files for them to take effect after your edits (systemctl —user restart servicefilename)

Troubleshooting

fusermount -uz path-to-mount
	ps ux | grep -i mergerfs         for MergerFS issues

	ps us | grep -i rclone           for Rclone issue 
username@servername:~$ ps ux | grep -i rclone
username   36384  0.1  0.0 1364856 65748 pts/140 Sl+  Sep25 105:16 rclone mount Gdrive: /home/username/Stuff/Mount --cache-db-purge --dir-cache-time 5000h -vvv --log-file /home/username/.config/rclone/rclone.log --poll-interval 15s --vfs-cache-mode write 
username  128007  0.0  0.0 218480   824  pts/77   S+   15:32   0:00 grep -i rclone
kill -9 36384

We hope you find this guide useful and that you now feel confident to experiment with your own set-up.


If you are having trouble with this, you can open a support ticket here!
Feedback and suggestions regarding this article are welcome in our Discord server


Revision #13
Created 7 December 2023 04:41:47 by Excel_USB
Updated 14 April 2024 11:39:52 by warp