New rsync Remote, Incremental Backup Script – n2nBackup

With my Pi I took the opportunity to re-write my rsync backup scripts.

This new setup does everything my first shot did, especially incremental backups via rsync’s “–link-dest” command, but I also believe is more modular, even though I have not had the need to use all its (perceived) capabilities… Or completed them all.

Some Basic Capabilities

  • Does incremental local or remote rsync’ing
  • Able to use private certs for authentication
  • Accounts for labeling backups daily or monthly
  • Uses rsync profile directories
  • (future) Allows pre & post command executions

Setup Structure

n2nBackup:
total 20
drwxrwxr-- 2 4096 May 7 22:10 logs
-rwxrwxr-- 1 6857 May 7 22:10 n2nRsync.sh
-rwxrwxr-- 1 245 May 7 22:10 n2nWrapper.sh
drwxrwxr-- 3 4096 May 7 22:11 profiles

n2nBackup/logs:
total 0

n2nBackup/profiles:
total 4
drwxr-xr-- 2 4096 May 7 22:10 template

n2nBackup/profiles/template:
total 24
-rw-r--r-- 1 367 May 7 22:10 dest.conf
-rw-r--r-- 1 78 May 7 22:10 excludes.conf
-rwxr-xr-- 1 21 May 7 22:10 n2nPost.sh
-rwxr-xr-- 1 21 May 7 22:10 n2nPre.sh
-rw-r--r-- 1 585 May 7 22:10 rsync.conf
-rw-r--r-- 1 126 May 7 22:10 src_dirs.conf

Script Usage

Usage: n2nRsync.sh [-p profile] [-c] [-l] [-t] [-h]

 -p profile              rsync profile to execute
 -c                      use when running via cron
                         when used, will output to log file
                         otherwise, defaults to stdout
 -l                      list profiles available
 -t                      runs with --dry-run enabled
 -h                      shows this usage info

Sample crontab Usage

00 00 * * * /dir/n2nBackup/n2nRsync.sh -p profName -c

Some Details

Right now I’m running everything with the n2nRsync.sh.  I have not implemented the n2nWrapper or pre & post command execution stuff.  In my previous backup script, that was run directly on my Synology NAS, I had a need for some pre-backup commands, but for whatever reason… Past bad coding… Ignorance… Synology quirks… Accessing the data via NFS now… The issues I had to work around are no longer being experienced.

I still need to create cleanup scripts that will age-off data based on specified criteria.  My plan right now, since this backup scheme relies on hard links and thus, takes up far less space than independent daily full backups would, is to keep a minimum of 30 daily backups… And since this new setup also labels a single backup as “monthly”… The last 6 monthly backups.  Which are not any different than just a different named daily backup.

I may post the actual script text in the future, but for now I’ll just provide a tgz for download.

n2nBackup.tar.gz

Raspbian – Misc To-Done

Misc things I’ve done to configure my Pi for my personal usage.

Most have to be done via “sudo”

  • Create a new user, separate from “pi”
    • Command:  ‘adduser’
    • Appears to be a Debian specific command, different than the usual Linux ‘useradd’
  • Make new user’s primary group be “users”
    • Since I’m connecting to my Synology NAS over NFS, this allows any files I create as the new user to be part of a common group between the Pi and NAS
    • Command: ‘usermod -g users <newuser>’
  • As “pi” user, give new user ‘sudo’ access
    • Command: ‘visudo’
  • Create RSA key for authentication
    • Command: ‘ssh-keygen’
    • Be sure to keep your key safe and retrievable so that access is not lost… Don’t lose your key!
  • Add pub key to “~/.ssh/authorized_keys” file for new user
  • After achieving access via key authentication, disable SSH password authentication
    • Edit /etc/ssh/sshd_config
    • “PasswordAuthentication no”
  • Optional, specify SSH access for accounts
    • Edit /etc/ssh/sshd_config
    • At bottom of the file, add:
      • AllowUsers newUser1 newUser2
    • Good way to leave default “pi” user “active”, but not directly accessible via SSH
  • Changed hostname
    • nano /etc/hosts
    • nano /etc/hostname
    • reboot
  • Install rsync
    • aptitude install rsync
  • rsync backup script caused an error
    • Error: Too many open files
    • Testing solution: edit /etc/security/limits.conf
      • @users     hard     nofile     32768
  • Configure NTP to sync with NAS
    • Edit /etc/ntp.conf
    • Comment out existing lines starting with “server” that look like “server 0.debian.pool.ntp.org”
    • Add line like “server <nas IP>”
    • Save
    • service ntp restart

To be continued…

Synology My Old Backup Cleanup Script

Since I created my own rsync backup solution for my Synology DS’s, I knew I’d need a script to cleanup and remove the older incremental backups.

The good thing is that since I’m using rsync’s “–link-dest” option, it should be pretty straight forward to cleanup old backups.  Every incremental backup is really a full backup, so just delete the oldest directories.

At least, that’s my theory…

I now have over 30 days of backups, so I was able to create and test a script that follows my theory.

I’ve only run this script manually at this point.  Since I’m actually deleting entire directories, I’m somewhat cautious in creating a new cron job to run this automatically.

#!/bin/sh
#
# Updated:
#
#
# Changes
#
# -
#

# Set to the max number of backups to keep
vMaxBackups=30

# Base directory of backups
dirBackups="/volume1/<directory containing all your backups>"

# Command to sort backups
# sorts based on creation time, oldest
# at the top of the list
cmdLs='ls -drc1'

# Count the number of total current backups
# All my backups start with "In"
cntBackups=`$cmdLs ${dirBackups}/In* | wc -l`

# Create the list of all backups
# All backups start with "In"
vBackupDirs=`$cmdLs ${dirBackups}/In*`

vCnt=0

for myD in $vBackupDirs
do
# Meant to be a safety mechanism
# that will hopefully kick in if the other
# test fails for some reason
tmpCnt=`$cmdLs ${dirBackups}/In* | wc -l`

if [ $tmpCnt -le 14 ]; then
exit
fi

# Main removal code
# If wanting to test first
# comment the "rm" line and uncomment the "echo" line
# echo $myD
rm -rf $myD

# Track how many directories have been deleted
vCnt=$((vCnt+1))

# Check to see if the script should exit
if [ $((cntBackups-vCnt)) -le $vMaxBackups ]; then
exit
fi
done

Synology Custom Remote Backup Solution

Created a personalized rsync incremental remote backup solution.

  • Tested with DSM 4.1 (and a little with 4.0)
  • Uses default rsync
  • Does not require any additional packages (e.g. ipkg)
  • Utilizes SSH transport and ssl keys for secure transport
Still To Do
  • Age-off script
  • Combine Full and Incremental backup scripts
  • Create age-off script
Some details
The below scripts utilize rsync’s “–link-dest” option, so that each incremental update takes up very little space compared to the first.  It does require a full backup initially, which is why I currently have 2 scripts.  I believe they can easily be combined, but this is the initial solution.
Hard links are pretty nifty.  Google them and how the rsync “–link-dest” option works.
I do not consider myself an advanced linux user, so there are probably a number of best practices with this solution that were not simply ignored, but totally unknown because of my ignorance.

This system should work in both directions between 2 Synology boxes.  I’ve only implemented in a single direction thus far, but reversing should be pretty simple.

Full backup Script
  • Needs to be run the first time to do the initial full backup
  • If able, recommend doing this locally across a LAN and then doing incremental backups over the WAN
#!/bin/sh

#
# Updated:
#

#
# Changes
#
#
# Future Ideas
# - If there's an rsync error, automatic retry
#

RSYNC=/usr/syno/bin/rsync

#
# Need the directory where the script runs
#
shDir="$( cd "$( dirname "$0" )" && pwd )"

#
# Config files of interest
#
confRemHost="rem_host.sh"
confSrcDirs="src_dirs.conf"
confLastFile="last_backup.conf"

#
# Read in remote host info
#
. ${shDir}/${confRemHost}

#
# Misc Variables
#
vDate=`date +%Y-%m-%dT%H%M`
dirLogBackup="/volume1/<your directory path>/backup_logs"
dirBckupName="Initial_${vDate}"
vRsyncOpts="--archive --partial"
vLogLvl="--verbose"
dirLogs="$shDir"
vLogF=$dirLogs/rsync_${vDate}.log
vErr="no"
vMaxRetries=5

if [ -f "${shDir}/is.test" ]; then
dirLogBackup=${dirLogBackup}/test
rDir_base=${rDir_base}/test
fi

if [ ! -d "$dirLogBackup" ]; then
mkdir -p $dirLogBackup
chmod 777 $dirLogBackup
fi

exec > $vLogF 2>&1

if [ ! -f "${shDir}/${confSrcDirs}" ]; then
echo ---
echo --- $shDir/$confSrcDirs not found. Exiting...
echo ---
exit 1
fi

#
# Loop through each directory to backup
# (There may be a better way to do this, but this works)
#
for myD in `cat $shDir/$confSrcDirs`
do
echo ---
echo "--- Starting directory backup: $myD"
echo ---

if [ -d "$myD" ]; then
$RSYNC $vRsyncOpts $vLogLvl -e "ssh -i ${rUserKey}" \
$myD $rUser@$rHost:$rDir_base/$dirBckupName

if [ "${?}" -ne "0" ]; then
vErr="yes"
echo "ERR ($?) : $myD" >> $dirLogs/rsync_${vDate}.err

# Put a test here for exit code 23, Partial Transfer Error
# And then retry?
# a while loop could work
fi
else
echo
echo "--- WARN: Directory $myD does not exist"
echo
fi

echo ---
echo --- Completed directory backup: $myD
echo ---
echo
done

#
# Some cleanup / completion stuff
#
if [ $vErr = "no" ]; then
echo $dirBckupName > $shDir/$confLastFile #track last backup dir
else
chmod 733 $shDir/*.err
mv $shDir/rsync_*.err $dirLogBackup #save off err file
fi

#
# Want to move log file to new location
#
chmod 733 $shDir/*.log
mv $shDir/rsync_*.log $dirLogBackup
Incremental Backup Scripts
  • Added the “–temp-dir” & “–link-dest” options for rsync when compared to the Full backup script
#!/bin/sh

#
# Updated:
#

#
# Changes
#
# - Added ulimit change to 30720
#

#
# Future Ideas
# - If there's an rsync error, automatic retry
#

RSYNC=/usr/syno/bin/rsync

#
# Need the directory where the script runs
#
shDir="$( cd "$( dirname "$0" )" && pwd )"

#
# Config files of interest
#
confRemHost="rem_host.sh"
confSrcDirs="src_dirs.conf"
confLastFile="last_backup.conf"

#
# Read in remote host info
#
. ${shDir}/${confRemHost}

#
# Misc Variables
#
vDate=`date +%Y-%m-%dT%H%M`
dirLogBackup="/volume1/<your directory path>/backup_logs"
dirBckupName="Incremental_${vDate}"
vRsyncOpts="--archive --partial --delete"
vLogLvl="--verbose"
dirLogs="$shDir"
vLogF=$dirLogs/rsync_${vDate}.log
vErr="no"
vMaxRetries=5
dirTemp="/volume1/meister_backup/backup_temp"

if [ -f "${shDir}/is.test" ]; then
dirLogBackup=${dirLogBackup}/test
rDir_base=${rDir_base}/test
fi

if [ ! -d "$dirLogBackup" ]; then
mkdir -p $dirLogBackup
chmod 777 $dirLogBackup
fi

exec > $vLogF 2>&1

if [ ! -f ${shDir}/${confLastFile} ]; then
echo ---
echo --- $confLastFile not found. Exiting...
echo ---
exit 42
fi

if [ ! -f "${shDir}/${confSrcDirs}" ]; then
echo ---
echo --- $shDir/$confSrcDirs not found. Exiting...
echo ---
exit 1
fi

vLastDir=`cat $shDir/$confLastFile`

#
# Loop through each directory to backup
# (There may be a better way to do this, but this works)
#
for myD in `cat $shDir/$confSrcDirs`
do
echo ---
echo "--- Starting directory backup: $myD"
echo ---

if [ -d "$myD" ]; then
$RSYNC $vRsyncOpts $vLogLvl -e "ssh -i ${rUserKey}" \
--temp-dir=$dirTemp \
--link-dest=$rDir_base/$vLastDir \
$myD $rUser@$rHost:$rDir_base/$dirBckupName

if [ "${?}" -ne "0" ]; then
vErr="yes"
echo "ERR ($?) : $myD" >> $dirLogs/rsync_${vDate}.err

# Put a test here for exit code 23, Partial Transfer Error
# And then retry?
# a while loop could work
fi
else
echo
echo "--- WARN: Directory $myD does not exist"
echo
fi

echo ---
echo --- Completed directory backup: $myD
echo ---
echo
done

#
# Some cleanup / completion stuff
#
if [ $vErr = "no" ]; then
echo $dirBckupName > $shDir/$confLastFile #track last backup dir
else
chmod 733 $shDir/*.err
mv $shDir/rsync_*.err $dirLogBackup #save off err file
fi

#
# Want to move log file to new location
#
chmod 733 $shDir/*.log
mv $shDir/rsync_*.log $dirLogBackup

Additional Required Files

  • src_dirs.conf
    • Located in the same directory as both scripts
    • Used by both scripts
    • Simple text file
    • Separate directory on each line
      • These are the directories that you want backed up
      • Sync’d individually via the “for” loop in each script
  • last_backup.conf
    • Located in the same directory as both scripts
    • Used by both scripts
    • Simple text file
    • Should only have 1 line
      • Name of the last directory where your last backup was placed
      • Only updated if rsync completes with zero / no errors
    • Used primarily by the Incremental backup script to do the incremental backup with rsync that’s implemented by using the “–link-dest” option
  • rem_host.sh
##
#
# Variables Defining the remote host / destination server
#
##

rHost=<ip/hostname>
rUser=<remote user, needs to exist on remote box, should not be root/admin>
rDir_base=/volume1/<base backup directory> # target base directory
rUserKey="/<directory tree to private key>/.ssh/<private-key>"

 

Additional Optional Files

  • There is a little test / debug code in each script.
  • File:  is.test
    • If exists, then backup scripts put backups and logs into separate “test” directories relative to the normal directories
    • If used, need to make sure both “test” locations exist first or else an error will happen
    • Made it easier for me to develop and test the scripts in one directory and then just do a “cp” to a “live” directory.
    • All I had to do in the dev directory was “touch is.test”

FYI

  • Private key authentication
    • Used openssl to create private key.  Nothing special in the creation of the key, so a quick google on the process should work fine
    • When setting up, I had some problems getting things to work for the user I created to do the rsync’s
      • Its totally possible that in my troubleshooting and Synology / DSM ignorance I did (or did not) do something I was (or was not) supposed to do
    • The key thing to remember though, is to make sure that the backup user’s home directory is where you place the “.ssh/backup-key” file
      • In my case, for some reason the user’s home in “/etc/passwd” was not pointing to where I was expecting.  So I changed it and restarted SSH
        • Should be able to use “synoservice” to restart SSH
    • Once I figured this out, everything else with configuring this went smoothly.  Luckily I’d had some experience in this area at work.  So that may have helped.
  • This entire solution works for me and my system setup.  I imagine there are a number of configurations where it will not work “out of the box”.
    • This may have problems if trying to rsync directories that are located on external devices or mounted from remote hosts or …. other stuff.