Purpose
Backup your files is important and often neglect even with a dedicated day now (the 31th March of each year).
Especially if you use services like Nextcloud to store all your files from your devices (desktop, laptop, smartphone…) : documents, photos, contacts, calendars… The disk space requires can be huge since it’s almost a 1:1 ratio. And storage is often the most expansive part when you look for a online server.
A possible alternative is to use Object Storage storage (like AWS S3) with cold option. The backups are not design to be used every day and often.
Account setup
I will be using Scaleway C14 Cold Storage (https://www.scaleway.com/en/object-storage/). The datastores are located in France with a lower cost than AWS. Moreover, you have first 75GB of storage and outgoing data transfer. It’s enough if you want to test it before send all your data. After that, it’s about €0.002/GB/month for data storage and €0.01/GB for outgoing data. So, even for 1TB of data, your monthly cost will be around 2€.
Installation
rclone is a power tools and allow you to perform rsync-like operations using a very large option as destination. You can refer here to view the list: https://rclone.org/#providers.
It can installed from system repository in almost any case: apt/dnf install rclone.
rclone onfiguration
Once installed, you will need to run once the setup configuration for storage part and one more time if you want to use encryption.
Object storage configuration
Note: not all the lines are displayed to be more clear.
$ rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> scaleway
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[...]
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
\ "s3"
[...]
Storage> s3
provider> Scaleway
env_auth> false
access_key_id> <ACCESS_KEY>
secret_access_key> <SECRET_KEY>
region> fr-par
endpoint> https://s3.fr-par.scw.cloud
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a string value. Press Enter for the default ("").
location_constraint> fr-par
acl> 1
y/n> n
Remote config
--------------------
[scaleway]
type = s3
provider = other
env_auth = false
access_key_id = <ACCESS_KEY>
secret_access_key = <SECRET_KEY>
endpoint = https://s3.fr-par.scw.cloud
location_constraint = fr-par
acl = private
region = fr-par
--------------------
y/e/d> y
Current remotes:
Name Type
==== ====
scaleway s3
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
Encryption configuration
Encryption is not required and can be skip. BUT, since you will upload all your data to a third party service, it’s strongly advised to encrypt the data before sending them.
It will cost a little more CPU but nothing to worry about. The data size is not really increased too, so the cost will not be much more.
Note: not all the lines are displayed to be more clear.
$ rclone config
Current remotes:
Name Type
==== ====
scaleway s3
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
name> scaleway_crypt
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[...]
10 / Encrypt/Decrypt a remote
\ "crypt"
[...]
Storage> crypt
** See help for crypt backend at: https://rclone.org/crypt/ **
Remote to encrypt/decrypt.
Normally should contain a ':' and a path, eg "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a string value. Press Enter for the default ("").
remote> scaleway:rclone
filename_encryption> standard
directory_name_encryption> true
y/g> g
Bits> 1024
Your password is: <YOUR_PASSWORD>
y/n> y
y/g/n> g
Bits> 1024
Your password is: <YOUR_PASSWORD>
y/n> y
Remote config
--------------------
[secret]
type = crypt
remote = scaleway:rclone
filename_encryption = standard
directory_name_encryption = true
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***
--------------------
y/e/d> y
Current remotes:
Name Type
==== ====
scaleway s3
scaleway_crypt crypt
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
At the end after configuring the Object storage and encryption, you should get this output:
$ rclone config
Current remotes:
Name Type
==== ====
scaleway s3
scaleway_crypt crypt
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
SystemD configuration
Perfect! We can now create the script to sync all desired folders (here Nextcloud user files) with rclone:
for dir in /data/nextcloud/*/ ; do
rclone sync --progress --s3-chunk-size=20M /data/nextcloud/$(basename $dir)/files/ scaleway_crypt:/$(basename $dir)
done
# not mandatory but can be used to check is the stript is correctly executed in time
curl -m 10 --retry 5 https://hc-ping.com/TOKEN
Because almost all modern GNU/Linux distribution use SystemD, we can use it instead of a crontab to execute the backup at a daily basis.
First, the service using /etc/systemd/system/rclone.service file:
[Unit]
Description=rclone nextcloud backup
[Service]
Type=simple
ExecStart=/root/rclone.sh
And the timer /etc/systemd/system/rclone.timer:
[Unit]
Description=Run rclone backup
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
Once it’s done, we can reload SystemD and start the timer:
systemctl reload
systemctl start rclone.timers
# If you want to run the backup now, you can manually start the service
systemctl start rclone.service
The first run can take from few minutes to hours depending of the data amount to send your, of course, your upload speed. But the next time, it will takes much less since it will sync only the delta.
References
https://www.scaleway.com/en/docs/tutorials/encrypt-s3-data-rclone/