LINUX - RSYNC Self Restarting Backup

What is this?

This is how you make a backup job with linux or windows (with cygwin - make sure to install rsync utility) that runs until its complete. Lots of times jobs fail for no reason, and should get restarted but they have to wait for manual input to restart, well not this. This runs rsync, if rsync fails it produces an error, this detects an errored exit status and repeats the command. If the job is successful the exit status is success and thus it stops. 


NOTE: In CYGWIN a taskkill(CYGWINS version of kill/killall) makes RSYNC quit with a success status, so if you ever taskill rsync while this is running it will not restart, LINUX on the other hand with its killall and kill command produces an error so the job might restart, so with linux you will need to kill the script as well


LINUX VERSION:

Note for all the variables you need to change I bolded and underlined them. Pay attention to the Logfile LF variable,  you just need to change the LF variable to match the filename of the log you want to write, I appended `date ...` to it so that it gets new name with every run. The other entry you need to change the rsync source and rsync destination. And finally in starter.sh you need to changed the cd folder, cd where the log is if you. Do not run the command from the source or destination, and dont write the log to source or destination. Always have a 3rd party for the log.

Recap of what you need to change for the linux and cygwin version.

1. LF variable name you can leave the -`date..` part as that just appends the date

2. Rsync commands source in this case its /cygdrive/g/

3. Rsync commands destination in this case its /cygdrive/s/cases/12345678-aiplanes-store/

4. In the Green starter script change the cd directory, to match the directory where the LF variabl e points to, this is also the directory from where the command will start, this is the directory where nohup.out will save to. We can tail -f the nohup.out to view the progress.


FIRST MAKE BACKUP LOCATION

mkdir -p /cygdrive/s/cases/12345678-airplanes-store/


SIMPLE FORM 

THIS PUTS /cygdrive/g/* TO /cygdrive/s/cases/12345678-airplanes-store/<here> SO NO EXTRA FOLDERS ONLY WHATS SUPPOSED TO HAPPEN

rsync -avh --progress --stats /cygdrive/g/ /cygdrive/s/cases/12345678-airplanes-store/

 

CYGWIN SELFRESTARTING RSYNC WITH LOG SAVING TO SPACED DESTINATION = s.sh:

#!/bin/bash

echo "##########################"

echo "NEW JOB STARTED- ignore kill rsync error"

date

echo NOTE: Every restarted job creates a new log

echo "##########################"

echo

while [ 1 ]

do

echo "CONTROL+C TWICE TO CANCEL, FIRST ONE STOPS RSYNC, BUT WILL RESTART WHEN TIMER TIMES OUT"

killall -9 rsync

LF=/cygdrive/c/0Kostia/Dropbox/Work/Docs/0L3-Cases/12345678\ -\ readynas\ backup\ -\ bob\ c/after\ fix/log-`date +%D-%T |  tr /: --`

rsync -avh --progress --stats --log-file="$LF" /cygdrive/g//cygdrive/s/cases/12345678-airplanes-store/

if [ $? = 0 ] ; then

echo

echo

echo "#########################"

echo "RSYNC SUCCESSFULL"

date

echo "#########################"

exit

else

echo

echo "#########################"

echo "RSYNC FAILED RESTARTING IN 10 SECONDS"

date

echo "#########################"

echo

sleep 10

fi

done

 

WRAPPER FOR STARTING RSYNC SCRIPT = starter.sh:

#!/bin/bash

echo TO QUIT - RUN: taskkill /F /IM rsync* - CONTROL-C doesnt do the job

cd /cygdrive/c/0Kostia/Dropbox/Work/Docs/0L3-Cases/12345678\ -\ readynas\ backup\ -\ bob\ c/after\ fix/

echo TO VIEW PROGRESS: tail -f nohup.out

echo WHERE IS NOHUP? `pwd`

echo --- Hit enter right now to get back to the shell, while the program runs in the background ---

nohup ./s.sh &

 

MAKE ALL SCRIPTS EXECUTABLE:

chmod +x *.sh

 

RUN THE STARTER SCRIPT, SO IF SHELL CLOSES IT STILL RUNS:

./starter.sh

 

TO VIEW BACKGROUNDED JOB:

cd /cygdrive/c/0Kostia/Dropbox/Work/Docs/0L3-Cases/12345678\ -\ readynas\ backup\ -\ bob\ c/after\ fix/

tail -f nohup

 

IN MY HONEST OPIONION THIS IS BEST WAY TO RUN IT IF YOU CAN GUARANTEE THE SHELL TO NOT CLOSE - GOOD 4 Screen & tmux & dtach:

./s.sh

 

FOR CYGWIN

 

FIRST MAKE BACKUP LOCATION

mkdir -p /cygdrive/s/cases/12345678-airplanes-store/

 

SIMPLE FORM 

THIS PUTS /cygdrive/g/* TO /cygdrive/s/cases/12345678-airplanes-store/<here> SO NO EXTRA FOLDERS ONLY WHATS SUPPOSED TO HAPPEN

rsync -avh --progress --stats /cygdrive/g/ /cygdrive/s/cases/12345678-airplanes-store/

 

CYGWIN SELFRESTARTING RSYNC WITH LOG SAVING TO SPACED DESTINATION = s.sh:

#!/bin/bash

echo "##########################"

echo "NEW JOB STARTED- ignore kill rsync error"

date

echo NOTE: Every restarted job creates a new log

echo "##########################"

echo

while [ 1 ]

do

echo "CONTROL+C TWICE TO CANCEL, FIRST ONE STOPS RSYNC, BUT WILL RESTART WHEN TIMER TIMES OUT"

taskkill /F /IM rsync*

LF=/cygdrive/c/0Kostia/Dropbox/Work/Docs/0L3-Cases/12345678\ -\ readynas\ backup\ -\ bob\ c/after\ fix/log-`date +%D-%T |  tr /: --`

rsync -avh --progress --stats --log-file="$LF" /cygdrive/g//cygdrive/s/cases/12345678-airplanes-store/

if [ $? = 0 ] ; then

echo

echo

echo "#########################"

echo "RSYNC SUCCESSFULL"

date

echo "#########################"

exit

else

echo

echo "#########################"

echo "RSYNC FAILED RESTARTING IN 10 SECONDS"

date

echo "#########################"

echo

sleep 10

fi

done

 

WRAPPER FOR STARTING RSYNC SCRIPT = starter.sh:

#!/bin/bash

echo TO QUIT - RUN: taskkill /F /IM rsync* - CONTROL-C doesnt do the job

cd /cygdrive/c/0Kostia/Dropbox/Work/Docs/0L3-Cases/12345678\ -\ readynas\ backup\ -\ bob\ c/after\ fix/

echo TO VIEW PROGRESS: tail -f nohup.out

echo WHERE IS NOHUP? `pwd`

echo --- Hit enter right now to get back to the shell, while the program runs in the background ---

nohup ./s.sh &

 

MAKE ALL SCRIPTS EXECUTABLE:

chmod +x *.sh

 

RUN THE STARTER SCRIPT, SO IF SHELL CLOSES IT STILL RUNS:

./starter.sh

 

TO VIEW BACKGROUNDED JOB:

cd /cygdrive/c/0Kostia/Dropbox/Work/Docs/0L3-Cases/12345678\ -\ readynas\ backup\ -\ bob\ c/after\ fix/

tail -f nohup

 

IN MY HONEST OPIONION THIS IS BEST WAY TO RUN IT IF YOU CAN GUARANTEE THE SHELL TO NOT CLOSE - GOOD 4 Screen & tmux & dtach:

./s.sh

 

==============================================================

 

OLD ARTICLE:

OVERVIEW

Given that the Rsync command tends to fail a lot so the following scripts purpose is to run rsync copy and if it fails the rsync job will restart - it will continue to backup from where it left of. On big copies it might take a moment for it to find out where it left from.

 

LEGEND:

Yellow - is the actual rsync backup script

Green - is the script that should be used to start the start1.sh script - its a wrapper script

Blue - is just a progress script you can run

Red Text - needs to be altered per job because your source and destination folders will differ.

 

HOW TO:

 

First Make the three files for the scripts:

 

# cd /tmp

# touch start1.sh; chmod +x start1.sh; vim start1.sh

     Then look at yellow script below to get the self restarting rsync script.

# touch startbest.sh; chmod +x startbest.sh; vim startbest.sh

    Then look at green script below to get the start script

# touch prog.sh; chmod +x prog.sh; vim prog.sh

    Then look at blue script to get the progress script

 

 

START1.SH SCRIPT - the restart rsync script - the guts of the operation

 

#!/bin/bash

while [ 1 ]

do

      killall rsync

     rsync -av --progress --stats --human-readable /c /mnt/dest/nasbackup/

     if [ $? = 0 ] ; then

        echo

        echo "#########################"

        echo

        echo "RSYNC SUCCESSFULL"

        exit

     else

        echo

        echo "#########################"

        echo

        echo "RSYNC FAILED RESTARTING IN 180 SECONDS"

        echo

        sleep 180

     fi

done

 

STARTBEST.SH SCRIPT - the script to run that will run START1 for us

#!/bin/bash

nohup /tmp/start1.sh &

 

2 WAYS TO RUN THE SCRIPTS

  1. Manually run script

cd /

/tmp/start1.sh

(Or)

cd /

nohup /tmp/start1.sh &

Then hit enter again after running that command

Remember nohup and & will run the job in the background and in different process tree so if you get out of shell it will still run

  1. Start START_BEST.SH which will start everything for you

 cd /

/tmp/startbest.sh

 

==============================================================================

 

PROG.SH SCRIPT -  the script to view progress

#!/bin/bash

date

echo

echo ---LOAD:---

cat /proc/loadavg

echo

echo ---MEMORY:---

free -m

echo

echo "==============="

echo

tail -n10 /nohup.out

echo

echo "=============="

echo

echo --HUMAN READABLE--

df -hP | egrep -i "/c|file|/mnt/dest"

echo

echo --KILOBYTES:--

df -P | egrep -i "/c|file|/mnt/dest"

echo

echo "=============="

 

 

TO VIEW PROGRESS

cd /

/tmp/prog.sh

Or

watch -n2 /tmp/prog.sh

    NOTE: The progress script might not report correct numbers. It only reports correctly/understanbly if your copying an entire filesystem into another empty         filesystem.

    NOTE: If your not copying to an empty filesystem, or your only copying from a subset of a filesyetem instead of an entire filesystem. You will need to alter your progress script to view reliable information.

 

Pasted from <http://netshare/Departments/techsupport/L3%20Wiki/Pages/RSYNC%20copy%20that%20restarts%20when%20fails%20until%20success.aspx>

 

Comments