Subject: Longstanding rsync script no longer excludes

From: Clive Menzies <[email-masked]>
Date: Sat, 17 Sep 2016 11:42:46 +0100

Hi

One of the issues we've encountered following a recent Jessie upgrade, 
(around 5th September) is an rsync script seems to have changed behaviour.

We've been using rsync for a daily automated incremental backup and have 
three scripts to maintain an archive of daily snapshots for a week; 
weekly snapshots for a month; and 6 monthly snapshots (files are 
irretrievable 6months after they been deleted from the backed up server).

We've two pairs of FileServer and BackupServer, rsyncd runs on 
FileServer and the following daily script on BackupServer

#!/bin/bash

# This script is called daily from cron to perform overnight backups
# The full paths of the programs used in this script
     rm=/bin/rm
     mv=/bin/mv
     cp=/bin/cp
     rsync=/usr/bin/rsync

# Good rsync options for FileServer_backups.
     rsync_opts="-av --delete --delete-excluded"

# The name of the file containing the rsync connection password
     password="--password-file=/etc/.rs_pass"

# A list of files and directories that do not need to be backed up
      exclude_list="music funstuff cma/TransitStuff lost+found/"
      excludes=""
      for exclude in $exclude_list; do
      excludes="$excludes --exclude=$exclude"
  done

# define NOW for adding date to rsync log filename eg. FileServer.date
     NOW=$(date +"%F.%H:%M")

# Move all other backups up a level. Copy previous backup to /backup/daily.
# Backup FileServer according to the [rsync] sections of the rsyncd.conf 
files
# on FileServer. Use the password given in /etc/.rs_pass. Dump any 
output and
# error messages to /var/rsync/FileServer
     mv /home/backup/snapshot/day.5 /home/backup/snapshot/day.6
     mv /home/backup/snapshot/day.4 /home/backup/snapshot/day.5
     mv /home/backup/snapshot/day.3 /home/backup/snapshot/day.4
     mv /home/backup/snapshot/day.2 /home/backup/snapshot/day.3
     mv /home/backup/snapshot/day.1 /home/backup/snapshot/day.2
     cp -al /home/backup/home /home/backup/snapshot/day.1
     $rsync $rsync_opts $password FileServer::rsync \
/home/backup/home/ > /var/rsync/FileServer.$NOW.log

The incremental backups have worked seamlessly for over 10 years but 
recently, possibly related to the upgrade of Debian Jessie, the exclude 
options have ceased to have effect.

I've dug around the docs but haven't found anything relevant. Any 
suggestions as to what may be wrong?

Thanks

Clive

-- 
Clive Menzies
http://freecriticalthinking.org

---

From: Clive Menzies <[email-masked]>
Date: Sat, 17 Sep 2016 17:12:09 +0100

On 17/09/16 13:32, Mike Bird wrote

>> I don't see where the $excludes you built is actually used.
>>
>

On 17/09/16 13:44, Clive Menzies wrote:
> The script wasn't changed and I've always assumed the $rsync-opts took 
> care of this. The script was copied from someone else and adapted.
>

Thanks to prompting by Mike, I've changed the way excludes are handled 
and it works :-)

OLD version:
# Good rsync options for uhuru_backups.
     rsync_opts="-av --delete --delete-excluded"

<snip>

WORKING version:
# Good rsync options for uhuru_backups.
     rsync_opts="-av --exclude-from=exclude_list --delete --delete-excluded"

The exclude_list is a file with one excluded directory per line with an 
asterisk at the end directory name.

Regards

Clive

---

From: Clive Menzies <[email-masked]>
Date: Mon, 19 Sep 2016 12:11:14 +0100

On 17/09/16 17:12, Clive Menzies wrote:
> The exclude_list is a file with one excluded directory per line with 
> an asterisk at the end directory name.

      $rsync $rsync_opts $password Uhuru::rsync \
       /root/uhuru_backup/uhuru/home/ > /var/rsync/uhuru.$NOW.log

The daily backup script works fine when run manually as sudo but doesn't 
complete when run as a root cronjob. I know the script starts because it 
creates a log. When successful it gives a list of files copied but when 
run as a cronjob it's blank (0 bytes)

$ sudo crontab -l
20 21 1 * * /root/uhuru_backup/uhuru.monthly.sh > /dev/null
25 21 * * 7 /root/uhuru_backup/uhuru.weekly.sh > /dev/null
30 21 * * * /root/uhuru_backup/uhuru.daily.sh > /dev/null

This should be impossible but I've got the same issue on two backup servers.

Regards

Clive

---

From: "Thomas Schmitt" <[email-masked]>
Date: Mon, 19 Sep 2016 13:31:15 +0200

Hi,

> complete when run as a root cronjob.

The classic reason for this is difference in environment variables.
I.e. the cron job could die from a non set variable.

The classic remedy would be to set the missing variables inside the
cronjob script.

> /root/uhuru_backup/uhuru.daily.sh

Consider to put some "echo" commands into the script and direct them to
a log file:

   log_file=/tmp/uhuru_daily_sh.log
   ...
   echo "Currently at: ..." >>$log_file 
   ...

This should give you an idea what part of the script causes the premature
end.

Have a nice day :)

Thomas

---

From: Clive Menzies <[email-masked]>
Date: Mon, 19 Sep 2016 22:10:04 +0100

> This should give you an idea what part of the script causes the premature
Thanks Thomas

I confess, I've not originated scripts before and so am on a steep 
learning curve. Your suggestion was very useful but my implementation is 
poor. It did yield useful information though.

It didn't find the exclude_list I created. I modified the script to 
point to it but clearly the syntax is not right

# exclude list location:
         exclude_list=/root/uhuru_backup/exclude_list

echo "Currently at line 17">>$log_file

The log reported thus:
/root/uhuru_backup/uhuru.daily.sh: line 17: $log_file: ambiguous redirect
rsync: failed to open exclude file exclude_list: No such file or 
directory (2)
rsync error: error in file IO (code 11) at exclude.c(1179) [client=3.1.1]

Regards

Clive

---

From: "Thomas Schmitt" <[email-masked]>
Date: Tue, 20 Sep 2016 00:04:08 +0200

Hi,

Clive Menzies wrote:
> rsync: failed to open exclude file exclude_list: No such file or directory (2)

The decisive difference between dialog and cron could be the
current working directory. 

Try with an absolute path as parameter to --exclude-from= :

  rsync_opts="-av --exclude-from=/root/uhuru_backup/exclude_list --delete --delete-excluded"

Or if you want to use the variable exclude_list, set it first.
Then evaluate it by $exclude_list in the definition of variable rsync_opts:

  exclude_list=/root/uhuru_backup/exclude_list
  rsync_opts="-av --exclude-from=$exclude_list --delete --delete-excluded"

This should yield the same rsync_opts content as above line.

> /root/uhuru_backup/uhuru.daily.sh: line 17: $log_file: ambiguous redirect

Hm. To what value did you set the variable log_file ?

If i set it to "/tmp/log_file", then the echo works in dash and bash.
(We have to expect that cron uses the shell dash, whereas the dialog
 shell is probably bash.)
 
If i set it to empty text, then dash complains
  dash: 2: Syntax error: newline unexpected
bash says
  -bash: syntax error near unexpected token `newline'

A blank would do it in bash but not in dash:

  $ log_file="/tmp/log file"
  $ echo "Currently at line 17">>$log_file
  -bash: $log_file: ambiguous redirect

dash is brutal enough to really create a file "/tmp/log file".

Have a nice day :)

Thomas

---

From: Clive Menzies <[email-masked]>
Date: Tue, 20 Sep 2016 00:09:17 +0100

Hi Thomas

The absolute path was the answer.

Your explanation has added to my limited knowledge :-)

Thanks

Clive

---

From: Clive Menzies <[email-masked]>
Date: Mon, 19 Sep 2016 22:51:47 +0100

> rsync error: error in file IO (code 11) at exclude.c(1179) [client=3.1.1]
D'oh! I searched around a bit and realised I'd over-complicated it.

This works :-)

# Good rsync options for uhuru_backups.
     rsync_opts="-av --exclude-from=/root/uhuru_backup/exclude_list \
          --delete --delete-excluded"

Thanks Thomas, I'd still be going round in circles.

Regards

Clive

---

From: Eduardo M KALINOWSKI <[email-masked]>
Date: Sat, 17 Sep 2016 09:10:06 -0300

On 17-09-2016 07:42, Clive Menzies wrote:
> We've been using rsync for a daily automated incremental backup and
> have three scripts to maintain an archive of daily snapshots for a
> week; weekly snapshots for a month; and 6 monthly snapshots (files are
> # Backup FileServer according to the [rsync] sections of the
> rsyncd.conf files
> recently, possibly related to the upgrade of Debian Jessie, the
> exclude options have ceased to have effect.
>

You define $excludes, but you don't seem to use it anywhere.

BTW, your rotating backups are basically what rsnapshot does, you might
want to look at that.

-- 
Eduardo M KALINOWSKI
[email-masked]