StorageBox

You can purchase a StorageBox on Hetzner, which can be used for storing application data and backups. Dedicated root servers usually have a 100GB storagebox for free, and you can get a 1TB storagebox for 3-4 EUR/month.

1. Accessing

From the web console of Hetzner we can enable SSH support and reset the password of the storagebox (which means set a new password). Then we can access it via ssh on the port 23:

ssh -p23 u478741@u478741.your-storagebox.de

Note: For more details see the docs.

For automating data storage and backup operations, we need to access the storagebox without a password, so let's setup access with SSH keys, (by following these instructions):

1. Generate an SSH key pair:

   cd ~
   mkdir -p storagebox
   cd storagebox/
   
   ssh-keygen -q -N '' -f key1
   ls key1*

2. Upload the public key to the Storage Box server:

   cat key1.pub \
       | ssh -p23 u478741@u478741.your-storagebox.de install-ssh-key
   
   ### test it
   ssh u478741@u478741.your-storagebox.de -p23 -i $(pwd)/key1

3. Create an SSH config entry:

   cat << _EOF_ >> ~/.ssh/config
   Host storagebox
       HostName u478741.your-storagebox.de
       User u478741
       Port 23
       IdentityFile /root/storagebox/key1
   _EOF_
   
   chmod 600 ~/.ssh/config
   ls -l ~/.ssh/config
   cat ~/.ssh/config
   
   ### test it
   ssh storagebox

2. NextCloud

We can keep the data of the NextCloud on the StorageBox, by mounting it with SSHFS.

1. First let's make sure that sshfs is installed:

   apt install sshfs

2. Let's create on the StorageBox the directory storage/cloud.user1.fs.al/data:

   ssh storagebox "mkdir -p storage/cloud.user1.fs.al/data"
   ssh storagebox "tree storage"

3. Find out the uid and gid of the data directory (which has owner and group www-data):

   cd /var/ds/cloud.user1.fs.al/
   ds exec id www-data

   They are both 33:

   uid=33(www-data) gid=33(www-data) groups=33(www-data),106(redis)

4. Add an entry on /etc/fstab for mounting the storagebox directory to /var/ds/cloud.user1.fs.al/www/data:

   STORAGEBOX_HOST=u478741@u478741.your-storagebox.de
   STORAGEBOX_DIR=storage/cloud.user1.fs.al/data
   LOCAL_MOUNT_POINT=/var/ds/cloud.user1.fs.al/www/data
   STORAGEBOX_SSH_KEY=/root/storagebox/key1
   MAP_UID=33
   MAP_GID=33
   SSHFS_OPTIONS=x-systemd.automount,x-systemd.requires=network-online.target,_netdev,user,idmap=user,transform_symlinks,port=23,identityfile=$STORAGEBOX_SSH_KEY,allow_other,default_permissions,uid=$MAP_UID,gid=$MAP_GID

   cat <<EOF >> /etc/fstab
   $STORAGEBOX_HOST:$STORAGEBOX_DIR  $LOCAL_MOUNT_POINT  fuse.sshfs  $SSHFS_OPTIONS   0  0
   EOF
   cat /etc/fstab

   We are appending a single line, but if we break it for readability, it looks like this:

   u478741@u478741.your-storagebox.de:storage/cloud.user1.fs.al/data \
    /var/ds/cloud.user1.fs.al/www/data \
    fuse.sshfs \
    x-systemd.automount,\
        x-systemd.requires=network-online.target,\
        _netdev,\
        user,\
        idmap=user,\
        transform_symlinks,\
        port=23,\
        identityfile=/root/storagebox/key1,\
        allow_other,\
        default_permissions,\
        uid=33,\
        gid=33\
    0 \
    0

5. Finally we have to mount it and move the data to it:

   cd /var/ds/cloud.user1.fs.al/
   ds occ maintenance:mode --on
   
   mv www/data www/data-1
   mkdir -p www/data
   
   mount $(pwd)/www/data
   systemctl daemon-reload
   
   rsync -a www/data-1/ www/data
   
   ds occ maintenance:mode --off

Now the data is on a storagebox directory, that is mounted through SSHFS.

caution

When the server is rebooted, the data directory is mounted automatically (from /etc/fstab). However, depending on when and how this happens, we may need to restart NextCloud, so that it can access the data properly:

ds @cloud.user1.fs.al restart

### or:
cd /var/ds/cloud.user1.fs.al/
ds restart

tip

It is also possible to automate this restart command with a systemd service. We have to make sure that this service is executed on server reboot, but after the SSHFS data directory is mounted.

1. Each mount is handled by a (generated) systemd unit. We can find out the name of the data mount service with a command like this:

   systemctl list-units --type=mount

   It should be something like this: var-ds-cloud.user1.fs.al-www-data.mount

2. Let's create a systemd service that will be executed after the service above is finished:

   cat <<EOF > /etc/systemd/system/restart-nextcloud.service
   [Unit]
   Description=Restart Nextcloud after the storagebox is mounted to the data directory
   After=network.target var-ds-cloud.user1.fs.al-www-data.mount
   Requires=var-ds-cloud.user1.fs.al-www-data.mount
   
   [Service]
   Type=simple
   ExecStart=/usr/local/bin/ds @cloud.user1.fs.al restart
   
   [Install]
   WantedBy=default.target
   EOF

3. Enable this service:

   systemctl daemon-reload
   systemctl enable restart-nextcloud.service

Now, whenever the server is rebooted, nextcloud will be restarted as well after the SSHFS data storage is mounted.

3. BigBlueButton

The recorded sessions of BBB are kept inside the directory /var/bigbluebutton/ (inside the container). We can mount a directory from the host to this directory inside the container, and then mount a directory from the storage to the host directory (using SSHFS).

3.1 Make a backup of the BBB data

Let's start by making a backup of the directory /var/bigbluebutton/, inside the container:

incus shell bbb
bbb-conf --stop

mv /var/bigbluebutton /var/bigbluebutton-1
mkdir -p /var/bigbluebutton
chown bigbluebutton: /var/bigbluebutton
ls -al /var/bigbluebutton

exit

We stopped the BBB services and moved the data directory to /var/bigbluebutton-1.

3.2 Mount a host dir to the BBB container

We want to mount the directory /mnt/bbb/ from the host, to the directory /var/bigbluebutton/ inside the container.

1. Let's add the directory /mnt/bbb as a disk device to the container:

   mkdir -p /mnt/bbb
   
   incus config device add bbb var_bigbluebutton disk \
       source=/mnt/bbb \
       path=/var/bigbluebutton
   
   incus config device show bbb

   We have named this device var_bigbluebutton, and it is of type disk.

2. By default, the directory is mounted read-only inside the container, so we cannot write in it from inside the container. We have to make it writable (this article explains more about how to do it):

   cat /etc/subuid
   cat /etc/subgid
   id
   
   echo "root:0:1" | tee -a /etc/subuid /etc/subgid
   cat /etc/{subuid,subgid}
   
   incus config set bbb raw.idmap "both 0 0"
   incus config get bbb raw.idmap
   
   incus restart bbb    # may take a while, due to remapping
   incus exec bbb -- bbb-conf --stop

3. Let's also give the right ownership to the mounted directory:

   incus exec bbb -- ls -al /var/bigbluebutton
   incus exec bbb -- chown bigbluebutton: /var/bigbluebutton
   incus exec bbb -- ls -al /var/bigbluebutton

3.3 Use SSHFS to mount the storage box to /mnt/bbb

1. First let's create a directory on the storagebox:

   ssh storagebox "mkdir -p storage/bbb.user1.fs.al/data"
   ssh storagebox "tree storage -L 3"

2. Check out the uid and gid of the mounted directory /mnt/bbb:

   ls -al /mnt/bbb/

3. Add an entry on /etc/fstab for mounting the storagebox directory to /mnt/bbb:

   STORAGEBOX_HOST=u478741@u478741.your-storagebox.de
   STORAGEBOX_DIR=storage/bbb.user1.fs.al/data
   LOCAL_MOUNT_POINT=/mnt/bbb
   STORAGEBOX_SSH_KEY=/root/storagebox/key1
   MAP_UID=1000999
   MAP_GID=1000998
   SSHFS_OPTIONS=x-systemd.automount,x-systemd.requires=network-online.target,_netdev,user,idmap=user,transform_symlinks,port=23,identityfile=$STORAGEBOX_SSH_KEY,allow_other,default_permissions,uid=$MAP_UID,gid=$MAP_GID

   cat <<EOF >> /etc/fstab
   $STORAGEBOX_HOST:$STORAGEBOX_DIR  $LOCAL_MOUNT_POINT  fuse.sshfs  $SSHFS_OPTIONS   0  0
   EOF
   cat /etc/fstab

   We are appending a single line, but if we break it for readability, it looks like this:

   u478741@u478741.your-storagebox.de:storage/bbb.user1.fs.al/data \
       /mnt/bbb \
       fuse.sshfs \
       x-systemd.automount,\
           x-systemd.requires=network-online.target,\
           _netdev,\
           user,\
           idmap=user,\
           transform_symlinks,\
           port=23,\
           identityfile=/root/storagebox/key1,\
           allow_other,\
           default_permissions,\
           uid=1000999,\
           gid=1000998\
       0 \
       0

4. Mount the directory:

   mount /mnt/bbb
   systemctl daemon-reload
   umount /mnt/bbb
   mount /mnt/bbb

3.4 Restore the BBB data

incus restart bbb
incus shell bbb
bbb-conf --stop

rsync -a /var/bigbluebutton-1/ /var/bigbluebutton
chown bigbluebutton: -R /var/bigbluebutton
rm -rf /var/bigbluebutton-1/

bbb-conf --start
exit

4. Backup

On the maintenance section, at backup scripts we used directories like /mnt/storage/mirror/, /mnt/storage/borg/, /mnt/storage/incus-backup/, /mnt/storage/incus-export/, etc.

The directory /mnt/storage/ is local, but it is easy to use the StorageBox instead of it. For example we can mount the directory backup/ from the StorageBox to the local directory /mnt/storage/ like this:

ssh storagebox mkdir -p backup
sshfs storagebox:backup /mnt/storage

Or, in the script borg.sh, instead of defining BORG_REPO like this:

export BORG_REPO='/mnt/storage/borg/mycloud'

we can define it like this:

export BORG_REPO='storagebox:borg/mycloud'

note

But first we should make sure that the directory borg/mycloud/ on the StorageBox exists:

ssh storagebox mkdir -p borg/mycloud/