Automated backup using Rsync

[johannes]

This is a small guide i wrote a while ago, while setting up my own backup system. I'm also just a beginner on these things, so please correct me if I did something stupid. All I know is that it works, not that it is the best solution.

Backup using rsync

Overview

Backing up your critical data is essential. This is one way to do it, describing a way to automize backups from one Linux machine to another (in my example two Bubba servers, but it could be any two Linux machines).

I use rsync, which contains a clever algorithm identifying what's changed since the last backup, and only copying the changed files. This means that a complete file copy doesn't have to be transferred on every backup occation, it only transfers the files that have changed.

The rsync command is initiated with a bash script, that in this case also does some copying and emails the backup status to the a user.

Finally, the script is added as a cron entry, making it run regularly, in my case, once every night.

I need to backup the email handled by my main Bubba, to a secondary Bubba acting (among other things) as a backup server. Files are copied from my main Bubba (used as email and web server) to my backup-Bubba. The script is run at the backup-Bubba, thus, the files are "pulled" to that machine. In the following guide, the machines are denoted as main-Bubba and backup-Bubba. I have called the users on each machine "main-user" and "backup-user".


Note: I give no warranties that this actually works, or even does what I intend. Use this guide at your own risk, for educational purposes only. Test it thoroughly before trusting it as a backup solution. Excito is not responsible for the content herein.


Now, to the guide:
Setting up "backup-Bubba"

Log on as backup-user, then do 'su' to become root. Uncomment the debian sources from /etc/apt/sources.list, then do:

Code: Select all

apt-get install rsync

Also install cron and open ssh packages, if not already installed (as they are if you are using Bubba).

Type exit to return as your normal user, "backup-user".

Now, create a directory holding your script:

Code: Select all

mkdir /home/backup-user/cron

Also create a backup directory, holding your backed up files:

Code: Select all

mkdir /home/backup-user/backup

To test the rsync connection (note: rsync needs to be installed on both systems for this to work):

Code: Select all

rsync -av -e ssh main-user@main-Bubba:/home/main-user/something_to_backup /home/backup-user/backup/

(type your remote password upon request).

Since we want this to be automated, typing your password every time isn't an option. This can be bypassed by depositing ssh keys on main-Bubba (google for 'ssh-keygen' if you wan't to dig deeper). To generate ssh keys:

Code: Select all

ssh-keygen -t dsa -b 1024 -f /home/backup-user/cron/backup-Bubba-rsync-key

(Note that this will take a minuter or two. You may wish to use a longer key than 1024 bits, but be prepared to wait for a while.).

Copy the public key file from backup-Bubba to main-Bubba, using scp:

Code: Select all

scp /home/backup-user/cron/backup-Bubba-rsync-key.pub main-user@main-Bubba:/home/main-user/.ssh

(If the .ssh directory isn't already created, you may have to do so manually, on main-Bubba).

Setting up "main-Bubba"

Log on as main-user, then do 'su' to become root. Uncomment the debian sources as before, then do:

Code: Select all

apt-get install rsync

Also install cron and open ssh packages, if not already installed (as they are if you are using Bubba). Type 'exit' to return as a normal user (main-user).

Add the key to the authorized_keys file:

Code: Select all

cat bubba-01-rsync-key.pub >> .ssh/authorized_keys

Note that this makes anyone at your LAN able to log in using the user 'main-user', using that key. If you consider this being a security risc, it is possible to prohibit ssh logons from other machines than backup-Bubba, but this isn't covered by this guide.

Now, back to backup-Bubba for further testing:
Testing the ssh and rsync connection

To test the ssh-connection, without having to use your password:

Code: Select all

ssh -i /home/backup-user/cron/backup-Bubba-rsync-key main-user@main-Bubba

Just type exit to return to your local session.

And to test an rsync transfer in the same manner:

Code: Select all

rsync -av -e "ssh -i /home/backup-user/cron/backup-Bubba-rsync-key" main-user@main-Bubba:/home/main-user/something_to_backup backup

Check that the files located in your /something_to_backup/ folder ended up in /backup/ on backup-Bubba.

Setting up a cron job to do this regularly

Create the script file:

Code: Select all

nano /home/backup-user/cron/backup-script

And put something like this in there:

Code: Select all

#!/bin/sh

rsync -av -e "ssh -i /home/backup-user/cron/backup-Bubba-rsync-key" main-user@main-Bubba:/home/main-user/something_to_backup/ 
 /home/backup-user/backup/

Make this script executable:

Code: Select all

chmod +x backup-script

And test it:

Code: Select all

./backup-script

Now put this in the cron-tab:

Code: Select all

crontab -e

And put something like:

Code: Select all

0 23 * * * /home/backup-user/cron/backup-script

in there. This example means that your script will run every night at 11:00 PM (23:00), every day of the week all year around. Using:

Code: Select all

42 07 19 * * /home/backup-user/cron/backup-script

means that it will run once every month, the 19:th at 07:42 AM. To read more about how to configure cron, read here.

Other functionality in your backup script

You can add just about anything there, for example emailing of the backup status, using a script looking something like this:

Code: Select all

#!/bin/sh

MESSAGE="/tmp/emailmessage.txt" #A temp file storing the contents of your email message
echo "Your backup started at " > $MESSAGE
DATE=`date +%y%m%d-%T`
echo $DATE >> $MESSAGE
echo "..with the following log information:" >> $MESSAGE 

#Do the actual file transfer 
#(The --delete option means that files deleted at main-Bubba also will be deleted at backup-Bubba, something that fits me better
#since I'm backing up email.)
rsync -avz --delete -e "ssh -i /home/backup-user/cron/backup-Bubba-rsync-key" main-user@main-Bubba:/home/main-user/Mail/ current 
  >> $MESSAGE

#Create a new folder with todays date and time
DATE=`date +%y%m%d-%T`
mkdir $DATE >> $MESSAGE 

#Copy the contents of your current backup to todays folder
cp -r current/ $DATE/ >> $MESSAGE 

echo "..and ended at" >> $MESSAGE
echo $DATE >> $MESSAGE

mail -s "Backup done" you@yourdomain.com  < $MESSAGE

In this case, you would probably want to delete some of the older backup folders, keeping a cyclic repository of backups, but I'll leave this for your to solve.


Also, don't forget to restore /etc/apt/sources.list when you are done!

[johannes]

that works between from a Linux machine such as Bubba to any other network attached storage.

[url]http://www.howtoforge.com/linux_backuppc[/url]

[mountaindude]

that works between from a Linux machine such as Bubba to any other network attached storage.

http://www.howtoforge.com/linux_backuppc

BackupPC looks like a good system, but is it small/lean enough to run on the Bubba? It is described as "a high-performance, enterprise-grade system for backing up Linux, WinXX and MacOSX PCs and laptops".

More info at http://backuppc.sourceforge.net

Has anyone tried it on the Bubba?

[johannes]

You have a point there. Since it's written in perl it will probably be quite slow on Bubba, although I suppose it should work. The web administration interface may of course be painfully slow, but apart from that, I think the backups should run fairly decent.

I just found it when googling, didn't read up on it at all.

Anyone with first-hand experience?

Thanks,

[marcolino]

I did install backuppc, test it but uninstall for the very raisons you mentions:
it works, but it's very slow, especially if using samba to backup a windows machine. Better do all the heavy work on the client...

[eeeuser1]

I've tested this in in't basic form and am blown away, once the first sync is done its blindingly quick.


Just to confirm for me, the depositing keys bit, this only makes each PC/server trust each other, it wont affect ssh from outside my firewall on the net will it?

Thanks
Jeff

[Kramer]

Hi

I've been trying to setup a backup system from one computer (WinXP) to bubba but no luck so far.

I want to backup my Music achive to bubba and keep it updated so that if I add music or change tags on files the changes exists on bubba also.

But I have problem with this because my music is on a second drive (D:/iTunes Music)

Is here anyone that have experience with this?

I've tryed using rsync on bubba and Cygwin/rsync ( http://www.gaztronics.net/rsync.php ) and cwRsync ( http://www.itefix.no/phpws/index.php?mo ... tion=23:23 ) but no luck yet.

Hope someone can help me.

/Kramer

[Ubi]

I'm using allwaysync (http://www.allwaysync.com, free for home use) to sync from the windows clients to bubba, then use Rsync scripts together with hard linking to make differential backups.
AllwaySync can give you all the tweeking you need and keeps the cpu intensive issues on the client. Personaly I found hardlinking more elegant then copying but that's just me.

Here's some other solutions (aimed for unix-unix backups though):
http://www.mikerubel.org/computers/rsync_snapshots/
http://www.howtoforge.com/mirroring_with_rsync
http://www.howtoforge.com/rsync_increme ... ot_backups

[Kramer]

Hi

Thanks for the reply Ubi but I still cant get it to work. Now I've moved over from Bubba fetching the files to WinXP pushing the files to Bubba.

This is what I've done.

Installed Cygwin with rsync on WinXP
Installed Rsync on Bubba with
/etc/rsync.conf set to chmod 644 and chown root
/etc/rsync.secrets set to chmod 600 and chown root

rsync.conf looks like this
------------------------
use chroot = no
hosts allow = 192.168.0.9

[music]
path = /storage/music
read only = false
comment = music

[backup]
path = /home/kramer/test (chmod 777) chown (kramer)
read only = false
comment = test
--------------------------

rsync.secrets looks like this

--------------------------
kramer:password
--------------------------

Rsync is running on Bubba
ps -ef |grep rsync looks like this

root 1283 1 0 00:04 ? 00:00:00 /usr/bin/rsync --no-detach --daemon --config /etc/rsync.conf

From WinXP I do i command window

1.] C:\Program\plink.exe -v -t -l kramer 192.168.0.3 rsync --daemon --port=1873 --config=$HOME/rsyncd.conf

2.] C:\Program\plink.exe -v N -L 873:localhost:1873 -l kramer 192.168.0.3

3.] C:\cygwin\bin\rsync.exe -av /cygdrive/d/Nerladdat 127.0.0.1::backup

The error I'm getting is

(In the window where I did 2.] )
Forward connection to server refused: Connection failed [Connection refused]

(In Window where I did 3.] )
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at /home/lapo/packaging/tmp/rsync-2.6.9/io.c(604) [sender=2.6.9]

I've followed (almost) http://aplawrence.com/Unixart/backup_rsync.html

I'm starting to get frustrated about not getting the synking to work

Hope that someone can help me.

Ps. Happy New Year!!!! .ds

/Kramer

[Ubi]

3.] C:\cygwin\bin\rsync.exe -av /cygdrive/d/Nerladdat 127.0.0.1::backup

why the double :: ?
Is that required in the windows version?

[Kramer]

I think I got it now.

Found a differnent page.

It seems to work with the command

c:\cygwin\bin\rsync.exe -av "/cygwin/d/Nerladdat" kramer@192.168.0.3::backup

Now I try to backup my music Hope I dont mess it up.

/Kramer

[Kramer]

Hi

When I try to sync my music archive to Bubba with Rsync my PC (running WinXP) crashes usally after 30-40 min. When I synced my picture & MVID archive everything works fine. These syncs took 3-5 hours.

Anybody who has experienced this to?

Does rsync have limits (file size, filename lenght, number of files etc) that can crash the PC?

/Kramer