Rsync and Inotify

Rsync

1. rsync intruction

1.1 DESCRIPTION

rsync is a fast, versatile, remote (and local) file-copy tools.

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for back-ups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a quick check algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the files data does

not need to be updated.

Some of the additional features of rsync are:

o support for copying links, devices, owners, groups, and permissions

o exclude and exclude-from options similar to GNU tar

o a CVS exclude mode for ignoring the same files that CVS would ignore

o can use any transparent remote shell, including ssh or rsh

o does not require super-user privileges

o pipelining of file transfers to minimize latency costs

o support for anonymous or authenticated rsync daemons (ideal for mirroring)

1.2 SYNOPSIS

Local: rsync [OPTION...] SRC... [DEST]

Access via remote shell:

Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]

Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Access via rsync daemon:

Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]

rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]

Push: rsync [OPTION...] SRC... [USER@]HOST::DEST

rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST

Usages with just one SRC arg and no DEST arg will list the source files instead of copying.

In a word, rsync is tools that can move data from one driver to another driver

2 setting and using

2.1 check and use it in local

use this command “rpm -qa rsync” then you will see

rsync-3.0.6-9.el6_4.1.x86_64

rsync is default install when preparing Linux system.

so we use it to copy file in local

The common usage is:

rsync [OPTION...] SRC... [DEST]

examples:

rsync /etc/hosts /tmp/

if you want copy directory, you can use “-r” options

rsync -r /tmp /root/

this usage is same as the “copy”

2.2 using based on remote shell

We also can use it by SSH pipelining, so first you should set SSH pipelining

The common usage is:

Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]

examples:

rsync -r -e 'ssh -p 52113' ygh_ssh@192.168.98.132:~ /backup/

-e --rsh=COMMAND specify the remote shell to use

Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

examples:

rsync -r -e 'ssh -p 52113' /backup/demo.txt ygh_ssh@192.168.98.132:~

2.3 using based on a rsync daemon(this is a very important way)

2.3.1 preparation works

Before use this usage, we need know some important options

-v --verbose increase verbosity

-z --compress compress file date during the transfer

-a --archive archive mode;equals -rlptgoD (no -H,-A,-X)

-r --recursive recurse into directories

-t --times preserve modification times

-o --owner preserve owner(super user only)

-p --perms preserve permissions

-P --progress show progress during transfer

-D --devices special files

-l --links copy symlinks as symlinks

-a is contain above options

-e --rsh=COMMAND specify the remote shell to use

--exclude=PATTERN exclude files matching PATTERN

--exclude-from=FILE read exclude patterns from FILE

--bwlimit=KBPS limit I/O bandwidth; KBytes per second

--delete delete extraneous files from dest dirs

the options “avz” is usually used

If you want more, you can reference this command “man rsync”

Now new need two machines, one as the client, another is as service to run rsync

daemon

The details information is following:

Hostname	Network card	Gateway	Functions
NFS_SERVICE	192.168.98.132	255.255.255.0	Run rsync daemon
NFS_CLIENT_1	192.168.98.135	255.255.255.0	Pull or push data
NFS_CLIENT_2	192.168.98.134	255.255.255.0	Pull or push data

2.3.2 Setting daemon in service

Firstly, create “/etc/rsyncd.conf” file, the file is not existing default, you need to create it..

use this command “vim /etc/rsyncd.conf”,then writing following content

#rsync configuration

#user and group

uid = rsync

gid = rsync

use chroot = no

max connections = 200

timeout = 300

#the file put pid of rsync

pid file = /var/run/rsyncd.pid

#the file put lock information of rsync

lock file = /var/run/rsync.lock

#the logs file

log file = /var/log/rsyncd.log

ignore errors

read only = false

list = false

hosts allow = 255.255.255.0/24

hosts deny = 0.0.0.0/32

#the vitual user

auth users = rsync_backup

##the password is put in this

secrets file = /etc/rsync.password

#exclude file

#exclude= a b

#this is mode

[oldboy]

path = /oldboy/

you can get more information by “man rsyncd.conf”

Secondly, create password file “/etc/rsync.password” then write following content,

then set it’s mod is 600,othwise rsync will give exception.

#username:password

rsync_backup:oldboy

Thirdly, run rsync daemon

use this command “[root@nfs_service ~]# rsync --daemon” to run rsync daemon

then use following command to view rsync daemon status

[root@nfs_service ~]# ps -ef|grep rsync

root 7700 1 0 15:43 ? 00:00:00 rsync --daemon

root 7702 1472 0 15:43 pts/0 00:00:00 egrep --color=auto rsync

Above status indicate you rsync daemon is successfully running

now you can use COMMAND in client to transfer data

2.3.3 Test rsync daemon in client

In client, we need to set a password file for rsync COMMAND

echo “oldboy” >>/etc/rsync.password

the “oldboy” is same as the password in service

The USE is “/etc/rsyncd.conf” auth users = rsync_backup and the shared directory is the path = /oldboy/ by oldboy to build connection from auth_users to shared directory

The common usage is following:

Access via rsync daemon:

Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]

example:

rsync -avz rsync_backup@192.168.98.132::oldboy /rysnc --password-file=/etc/rsync.password

这里的oldboy不是目录，是服务器rsync配置文件的一个模块，通过模块建立共享目录和虚拟用户直接的联系

rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]

examples:

rsync -avz rsync://rsync_backup@192.168.98.132/oldboy /rsync --password-file=/etc/rsync.password

Push: rsync [OPTION...] SRC... [USER@]HOST::DEST

examples:

rsync -avz /tmp rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST

examples:

rsync -avz /etc/ rsync://rsync_backup@192.168.98.132/oldboy --password-file=/etc/rsync.password

2.3.4 some other method

exclude copy：exclude some file you want not to copy

--exclude=PATTERN exclude files matching PATTERN

--exclude-from=FILE read exclude patterns from FILE

Examples:

one file exclude:

rsync -avz --exclude=a /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

more files exclude:

rsync -avz --exclude={a,b} /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

more files exclude sortly:

rsync -avz --exclude={a..g} /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

rsync -avz --exclude={1..10} /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

more files names are put in exclude.log

seq 10 > exclude.log

cat exclude.log

rsync -avz --exclude-from=/rsync/exclude.log /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/

the same copy keep client and service is same, the file is more in service will be delete：

rsync -avz --delete --exclude=a /rsync/ rsync_backup@192.168.98.132::oldboy --password-file=/etc/rsync.password

2.4 set more shared directory

If you want to create more shared directory, just set in “/etc/rsyncd.conf”

just like:

#rsync configuration

#user and group

uid = rsync

gid = rsync

use chroot = no

max connections = 200

timeout = 300

#the file put pid of rsync

pid file = /var/run/rsyncd.pid

#the file put lock information of rsync

lock file = /var/run/rsync.lock

#the logs file

log file = /var/log/rsyncd.log

ignore errors

read only = false

list = false

hosts allow = 255.255.255.0/24

hosts deny = 0.0.0.0/32

#the vitual user

auth users = rsync_backup

##the password is put in this

secrets file = /etc/rsync.password

#exclude file

#exclude= a b

[oldboy]

path = /oldboy/

[ygh]

path = /ygh/

#self setting

read only = true

[backup]

path = /backup/

If the set information is public, you can set it at top

if you need special setting at your mode, you can setting in self mode

3 Example

某公司有一台Web服务器，里面数据很重要，但是如果硬盘坏了，数据就会丢失，

现在领导要求你对数据做一下备份这样的Web服务器数据丢失就可以恢复，要求如下：

每天晚上00:00在web服务A上打包备份系统配置文件，网站程序目录以及访问日志并通过rsync命令推送到B

服务器上进行备份（备份的思路是:现在本地按日期打包，然后在推到B服务器上）

具体要求如下：

1）web服务器A和备份服务器B的备份目录都为/backup

2)系统的配置文件包括，但是不限于如下

a.定时任务服务的配置文件(/var/spool/cron/root)

b.开机自启的配置文件(/ect/rc.local)

c.日常的脚本目录(/server/scripts)

d.防火墙iptables的配置文件(/etc/sysconfig/iptables)

3)Web服务器的站点目的假定为(/var/html/www)

4)Web服务器A的访问日志的假定路径为(/app/log)

5)Web服务器保留打包7天的备份数据即可(本地留存不能多余7天，因为太多硬盘会满，准备服务器上要保留6个月以上的数据副本)

6)备份服务器上要按照备份数据服务的IP为目录保存，打包的文件按照时间名字保存

we solve it by a shell script and a crontab in every you can use this scripts

The shell scripts is following:

#!/bin/sh

#backup scripts

dir=/backup

ip=$(ifconfig|awk -F '[ :]+' 'NR==2{print $4}')

#if [ ! -d $dir/$ip ]

#then

mkdir -p /backup/$ip

#fi

cd /backup && cd $ip

tar -zcf backup_$(date +%F).tar.gz /var/www/html/ /app/log/ /server/scripts/ /var/spool/cron/ /etc/

rsync -avz /backup/ rsync_backup@192.168.98.132::backup --password-file=/etc/rsync.password && touch ${ip}-flag-$(date +%F)

rsync -avz /backup/ rsync_backup@192.168.98.132::backup --password-file=/etc/rsync.password

find /backup/ -type f -name "backup*.tar.gz" -mtime +7 |xargs rm -f

then set it in crontab

#backup data by rsync

00 00 * * * /bin/sh /server/scripts/backup/backup.sh > /dev/null 2>&1

Inotify

1 inotify instruction

inotify is a monitoring file system events, In a directory, if a file or directory is created or updated or deleted, it will tell you what file is change, then you can use rsync to deal with this file or directory.

Default, out machine don’t install it, so we need to install it at client

The steps of install is following:

install inotify:

1、dowmload inotify-tools-3.14.tar.gz

2、tar -zxvf inotify-tools-3.14.tar.gz

3、cd inotify-tools-3.14

4、./configure --prefix=/usr/local/inotify-tools-3.14

5、make && make install

6、cd ../

7、ln -s /usr/local/inotify-tools-3.14/ /usr/local/inotify

then, you can use this command “”

2 Testing and Using at Client

view inotify help

1、cd /usr/local/inotify

2、./bin/inotifywait --help

inotifywait 3.14

Wait for a particular event on a file or set of files.

Usage: inotifywait [ options ] file1 [ file2 ] [ file3 ] [ ... ]

Options:

-h|--help Show this help text.

@<file> Exclude the specified file from being watched.

--exclude <pattern>

Exclude all events on files matching the

extended regular expression <pattern>.

--excludei <pattern>

Like --exclude but case insensitive.

-m|--monitor Keep listening for events forever. Without

this option, inotifywait will exit after one

event is received.

-d|--daemon Same as --monitor, except run in the background

logging events to a file specified by --outfile.

Implies --syslog.

-r|--recursive Watch directories recursively.

--fromfile <file>

Read files to watch from <file> or `-' for stdin.

-o|--outfile <file>

Print events to <file> rather than stdout.

-s|--syslog Send errors to syslog rather than stderr.

-q|--quiet Print less (only print events).

-qq Print nothing (not even events).

--format <fmt> Print using a specified printf-like format

string; read the man page for more details.

--timefmt <fmt> strftime-compatible format string for use with

%T in --format string.

-c|--csv Print events in CSV format.

-t|--timeout <seconds>

When listening for a single event, time out after

waiting for an event for <seconds> seconds.

If <seconds> is 0, inotifywait will never time out.

-e|--event <event1> [ -e|--event <event2> ... ]

Listen for specific event(s). If omitted, all events are

listened for.

Exit status:

0 - An event you asked to watch for was received.

1 - An event you did not ask to watch for was received

(usually delete_self or unmount), or some error occurred.

2 - The --timeout option was given and no events occurred

in the specified interval of time.

Events:

access file or directory contents were read

modify file or directory contents were written

attrib file or directory attributes changed

close_write file or directory closed, after being opened in

writeable mode

close_nowrite file or directory closed, after being opened in

read-only mode

close file or directory closed, regardless of read/write mode

open file or directory opened

moved_to file or directory moved to watched directory

moved_from file or directory moved from watched directory

move file or directory moved to or from watched directory

create file or directory created within watched directory

delete file or directory deleted within watched directory

delete_self file or directory was deleted

unmount file system containing file or directory unmounted

some example:

Number 1

inotify listening command:

/usr/local/inotify/bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%T %w%f' -e create /backup

-m -r and -q and -e you can retrieve above

execute following commands:

touch b.txt

listening result:

17-01-17 09:43 /backup/b.txt

Number 2

inotify listening command:

/usr/local/inotify/bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%T %w%f' -e create,delete /backup

if want to listen more evenet, you can use "," as the seperator,%T is to print time %w is to print directory name

%f is to print filename

execute following commands:

mkdir t1

touch t2

rm t2

rm -rf t1

the listening results:

17-01-17 09:46 /backup/t1

17-01-17 09:46 /backup/t2

17-01-17 09:48 /backup/t2

17-01-17 09:48 /backup/t1

Number 3:

inotify listening command:

/usr/local/inotify/bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%T %w%f' -e create,delete,close_write /backup

execute following commands:

echo "sdsada

touch c.txt

the listening results:

17-01-17 09:54 /backup/a.txt

17-01-17 09:55 /backup/c.txt

the touch.txt will be listened by create and close_write

3 Examples

In some important situation, you should to backup data constantly, so you will use

inotify and rsync

you can use following shell scripts to solve it.

#!/bin/sh

#use inotify and rsync to backup data all time

host=192.168.98.132

src=/backup

dst=oldboy

user=rsync_backup

rsync_passfile=/etc/rsync.password

inotify_home=/usr/local/inotify-tools/

${inotify_home}bin/inotifywait -mrq --timefmt '%d-%m-%y %H:%M' --format '%w%f' -e close_write,delete,create,attrib $src

|while read line

rsync -avz $line $user@$host::$dst --password-file=${rsync_passfile} > /dev/null 2>&1

done

exit 0

then you can run this script at behind

/bin/sh /server/scripts/inotify/inotify1.sh &