* btrfs issue with mariadb incremental backup @ 2017-08-09 4:32 siranee.ja 2017-08-09 5:46 ` Chris Murphy 0 siblings, 1 reply; 31+ messages in thread From: siranee.ja @ 2017-08-09 4:32 UTC (permalink / raw) To: linux-btrfs; +Cc: voravat Hi btrfs support team, My name is siranee jaraswachirakul. I tested btrfs incremental send and receive and I found something incorrect. I posted detail in the url http://www.linuxquestions.org/questions/showthread.php?p=5746238#post5746238 Could you please help me to find the reason? Best Regards, Siranee Jaraswachirakul. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-09 4:32 btrfs issue with mariadb incremental backup siranee.ja @ 2017-08-09 5:46 ` Chris Murphy 2017-08-09 6:36 ` siranee.ja 0 siblings, 1 reply; 31+ messages in thread From: Chris Murphy @ 2017-08-09 5:46 UTC (permalink / raw) To: siranee.ja; +Cc: Btrfs BTRFS, voravat On Tue, Aug 8, 2017 at 10:32 PM, <siranee.ja@tpc.co.th> wrote: > Hi btrfs support team, > > My name is siranee jaraswachirakul. I tested btrfs incremental send and receive and > I found something incorrect. > > I posted detail in the url > http://www.linuxquestions.org/questions/showthread.php?p=5746238#post5746238 > > Could you please help me to find the reason? The command "btrfs sub snap mysql_201707210830 mysql" is unexpected for two reasons: 1. This is a rw snapshot. It needs to be ro using -r flag to make it ro, and only ro snapshots can be used with btrfs send receive. 2. The naming convention seems reversed. The subvolume you're snapshotting should be first, and the resulting snapshot is second, and it is the second one that you'll send. So I think really what you'll end up wanting is: Machine A btrfs sub snap -r mysql mysql_20170721 btrfs send mysql_20170721 Machine B btrfs sub snap mysql_20170721 mysql That will make a rw snapshot from the ro snapshot. The next problem with the post is that you say "incremental" but I do not see the exact send receive command you're using. An incremental send requires -p option. Machine A btrfs sub snap -r mysql mysql_20170721 btrfs send mysql_20170721 Machine B btrfs sub snap mysql_20170721 mysql Test mysql btrfs sub del mysql Machine A (next day) Machine A btrfs sub snap -r mysql mysql_20170722 btrfs send -p mysql_20170721 mysql_20170722 Machine B btrfs sub snap mysql_20170722 mysql Test mysql btrfs sub del mysql And so on. You must have the -p snapshot on both file systems for the incremental send/receive to work. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-09 5:46 ` Chris Murphy @ 2017-08-09 6:36 ` siranee.ja 2017-08-09 17:59 ` Chris Murphy 0 siblings, 1 reply; 31+ messages in thread From: siranee.ja @ 2017-08-09 6:36 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS, voravat Hi Chris, Sorry for the incompleted command that I used in the post. This is the real command from Machine A backup step : in crontab 30 08 * * * root /root/script/backup/backupsnap.sh root password /var/lib/mariadb/mysql [root@backuplogC7 ~]# cat /root/script/backup/backupsnap.sh #Backup # user=$1 password=$2 basepath=$3 datet=$(date +%Y%m%d%H%M) snappath=${basepath}_${datet} echo "Locking databases" mysql -u$user -p$password << EOF FLUSH TABLES WITH READ LOCK; system btrfs sub snap -r $basepath $snappath UNLOCK TABLES; quit EOF echo "Databases unlocked" The send incremental step : 487 history|grep send 488 btrfs send /var/lib/mariadb/mysql_201707210830 | ssh 192.168.10.29 btrfs receive /var/lib/mariadb 489 btrfs send -p /var/lib/mariadb/mysql_201707210830 /var/lib/mariadb/mysql_201707220830 |ssh 192.168.10.29 btrfs receive /var/lib/mariadb 490 btrfs send -p /var/lib/mariadb/mysql_201707220830 /var/lib/mariadb/mysql_201707230830 |pv| ssh 192.168.10.29 btrfs receive /var/lib/mariadb This is the real command from Machine B 463 systemctl stop mariadb 464 cd /var/lib/mariadb 465 ls -l 466 btrfs sub del mysql* 467 btrfs sub sync . 468 ip addr 469 btrfs sub list 470 btrfs sub list . 471 btrfs sub snap mysql_201707210830 mysql 472 systemctl start mariadb 473 mysql -uroot -ppassword 474 btrfs sub list 475 btrfs sub list . 476 mysql -uroot -ppassword 477 systemctl stop mariadb 478 btrfs sub list . 479 btrfs sub del mysql 480 btrfs sub sync . 481 btrfs sub snap mysql_201707220830 mysql 482 systemctl start mariadb 483 mysql -uroot -ppassword 484 systemctl stop mariadb 485 btrfs sub del mysql 486 btrfs sub sync . 487 btrfs sub list . 488 btrfs sub snap mysql_201707230830 mysql 489 systemctl start mariadb 490 btrfs sub list . 491 cat /var/log/mariadb/mariadb.log PS: No any error in btrfs command or in /var/log/messages. Best Regards Siranee Jaraswachirakul. > On Tue, Aug 8, 2017 at 10:32 PM, <siranee.ja@tpc.co.th> wrote: >> Hi btrfs support team, >> >> My name is siranee jaraswachirakul. I tested btrfs incremental send and receive >> and >> I found something incorrect. >> >> I posted detail in the url >> http://www.linuxquestions.org/questions/showthread.php?p=5746238#post5746238 >> >> Could you please help me to find the reason? > > The command "btrfs sub snap mysql_201707210830 mysql" is unexpected > for two reasons: > > 1. This is a rw snapshot. It needs to be ro using -r flag to make it > ro, and only ro snapshots can be used with btrfs send receive. > 2. The naming convention seems reversed. The subvolume you're > snapshotting should be first, and the resulting snapshot is second, > and it is the second one that you'll send. So I think really what > you'll end up wanting is: > > Machine A > btrfs sub snap -r mysql mysql_20170721 > btrfs send mysql_20170721 > > Machine B > btrfs sub snap mysql_20170721 mysql > > That will make a rw snapshot from the ro snapshot. > > The next problem with the post is that you say "incremental" but I do > not see the exact send receive command you're using. An incremental > send requires -p option. > > > > Machine A > btrfs sub snap -r mysql mysql_20170721 > btrfs send mysql_20170721 > > Machine B > btrfs sub snap mysql_20170721 mysql > Test mysql > btrfs sub del mysql > > Machine A (next day) > > Machine A > btrfs sub snap -r mysql mysql_20170722 > btrfs send -p mysql_20170721 mysql_20170722 > > Machine B > btrfs sub snap mysql_20170722 mysql > Test mysql > btrfs sub del mysql > > And so on. You must have the -p snapshot on both file systems for the > incremental send/receive to work. > > > > -- > Chris Murphy > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-09 6:36 ` siranee.ja @ 2017-08-09 17:59 ` Chris Murphy 2017-08-10 2:03 ` siranee.ja 2017-08-11 4:40 ` siranee.ja 0 siblings, 2 replies; 31+ messages in thread From: Chris Murphy @ 2017-08-09 17:59 UTC (permalink / raw) To: siranee.ja; +Cc: Chris Murphy, Btrfs BTRFS, voravat On Wed, Aug 9, 2017 at 12:36 AM, <siranee.ja@tpc.co.th> wrote: > 488 btrfs sub snap mysql_201707230830 mysql > 489 systemctl start mariadb > 490 btrfs sub list . > 491 cat /var/log/mariadb/mariadb.log OK so mysql_201707230830 once on machine B is inconsistent somehow. So the questions I have are: Is mysql_201707230830 on machine A really identical to mysql_201707230830 on machine B? You can do an rsync -anc (double check those options) which should independently check whether those two subvolumes are in fact identical. The -n is a no op, which doesn't really matter much because as read only subvolumes any attempt to sync will just result in noisy messages. The -c causes rsync to do its own checksum verification on both sides. If the subvolumes are different, we need to find out why. If the subvolumes are the same, then I wonder if you can reproduce the mariadb complaint on machine A merely by making a rw snapshot of mysql_201707230830 and trying to start it. If so, then it's not a send receive problem, it sounds like the snapshot itself is inconsistent, maybe mariadb hasn't actually completely closed out the database at the time the read only snapshot was taken? I'm not sure. If the subvolumes are different, I'm going to recommend updating at least the btrfs-progs because 4.4 is kinda old at this point. The kernel code is what's mainly responsible for the send stream, and the user space code is mainly responsible for receiving. And I don't off hand know or want to look up all the send receive changes between 4.4 and 4.12 to speculate on whether this is has already been fixed. What's the kernel version? -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-09 17:59 ` Chris Murphy @ 2017-08-10 2:03 ` siranee.ja 2017-08-11 4:40 ` siranee.ja 1 sibling, 0 replies; 31+ messages in thread From: siranee.ja @ 2017-08-10 2:03 UTC (permalink / raw) To: Chris Murphy; +Cc: Chris Murphy, Btrfs BTRFS, voravat I'll test and response the result to you soon. I have another question about the combination within Host lxd ubuntu and lxd container centos Host Machine X (16.04.3 LTS with lxd 2.0.10 and btrfs-progs v4.4 kernel 4.4 ) set 1 lvm raw partitions and format to btrfs filesystem (on host) Partition A mount to /opt/partA Container A (CentOS Linux release 7.3.1611 (Core) ) is in Host Machine X mount /opt/partA to /var/lib/mariadb in container A Container A making local readonly snapshot everyday The Options are (online "btrfs send ||ssh btrfs receive" or batch "btrfs send > file.btrfs" and scp from local to remote then "btrfs receive -f file.btrfs") 1. use btrfs send from Host Machine X to Host Machine Y 2. use btrfs send from Container A to Host Machine Y 3. use btrfs send from Container A to Container B 4. use btrfs send incremental from Host Machine X to Host Machine Y 5. use btrfs send incremental from Container A to Host Machine Y 6. use btrfs send incremental from Container A to Container B Host Machine Y (16.04.3 LTS with lxd 2.0.10 and btrfs-progs v4.4 kernel 4.4 ) set 1 lvm raw partitions and format to btrfs filesystem (on host) Partition B mount to /opt/partB Container B (CentOS Linux release 7.3.1611 (Core) ) is in Host Machine Y mount /opt/partB to /var/lib/mariadb in container B The Options are (online "btrfs send ||ssh btrfs receive" or batch "btrfs send > file.btrfs" and scp from local to remote then "btrfs receive -f file.btrfs") 1. use btrfs receive from Host Machine X to Host Machine Y 2. use btrfs receive from Container A to Host Machine Y 3. use btrfs receive from Container A to Container B 4. use btrfs receive incremental from Host Machine X to Host Machine Y 5. use btrfs receive incremental from Container A to Host Machine Y 6. use btrfs receive incremental from Container A to Container B Could you please suggest me what is the combination that should work properly with btrfs send and receive? I plan to setup 2 Site (Production Site and DR site) each site has 1 box (Host Machine X and Y) and want to test btrfs to send incremental. Best Regards, Siranee Jarwachirakul. > On Wed, Aug 9, 2017 at 12:36 AM, <siranee.ja@tpc.co.th> wrote: > >> 488 btrfs sub snap mysql_201707230830 mysql >> 489 systemctl start mariadb >> 490 btrfs sub list . >> 491 cat /var/log/mariadb/mariadb.log > > OK so mysql_201707230830 once on machine B is inconsistent somehow. So > the questions I have are: > > Is mysql_201707230830 on machine A really identical to > mysql_201707230830 on machine B? You can do an rsync -anc (double > check those options) which should independently check whether those > two subvolumes are in fact identical. The -n is a no op, which doesn't > really matter much because as read only subvolumes any attempt to sync > will just result in noisy messages. The -c causes rsync to do its own > checksum verification on both sides. > > If the subvolumes are different, we need to find out why. > > If the subvolumes are the same, then I wonder if you can reproduce the > mariadb complaint on machine A merely by making a rw snapshot of > mysql_201707230830 and trying to start it. If so, then it's not a send > receive problem, it sounds like the snapshot itself is inconsistent, > maybe mariadb hasn't actually completely closed out the database at > the time the read only snapshot was taken? I'm not sure. > > If the subvolumes are different, I'm going to recommend updating at > least the btrfs-progs because 4.4 is kinda old at this point. The > kernel code is what's mainly responsible for the send stream, and the > user space code is mainly responsible for receiving. And I don't off > hand know or want to look up all the send receive changes between 4.4 > and 4.12 to speculate on whether this is has already been fixed. > > What's the kernel version? > > -- > Chris Murphy > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-09 17:59 ` Chris Murphy 2017-08-10 2:03 ` siranee.ja @ 2017-08-11 4:40 ` siranee.ja 2017-08-11 6:00 ` siranee.ja 2017-08-11 15:06 ` Chris Murphy 1 sibling, 2 replies; 31+ messages in thread From: siranee.ja @ 2017-08-11 4:40 UTC (permalink / raw) To: Chris Murphy; +Cc: Chris Murphy, Btrfs BTRFS, voravat Hi Chris, The kernel version that I test is "4.4.0-89-generic" as I tested on ubuntu lxd If I want to change the kernel version I have to upgrade the host box. As you suggest the rsync to compare the subvolumes. I found the point. the subvolumes are different only after I start to del old subvolumes on machine A the steps are 30 08 * * * root /root/script/backup/backupsnap.sh root password /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log 05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166 /var/lib/mariadb >> /var/log/btrfs_send.log 30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >> /var/log/btrfs_del.log The following script maintain snapshot to currently only 7 snapshots. [root@backuplogC7 ~]# cat /root/script/delete_btrfs_sub_snap_volume.sh basepath=$1 keepcount=$2 havecount=`btrfs sub list -s ${basepath}|cut -d' ' -f14|wc -l` delcount=$[$keepcount-$havecount]; datet=$(date +%Y%m%d%H%M) echo "Start Delete ${datet}" if [ $delcount -lt 0 ]; then # list only snapshot subvolume for i in `btrfs sub list -s ${basepath}|cut -d' ' -f14|head ${delcount}` do echo "btrfs sub delete ${basepath}/$i" btrfs sub delete ${basepath}/$i btrfs sub sync ${basepath} done else echo "$delcount -gt 0 nothing to delete" fi echo "Stop Delete ${datet}" Does it mean my delete script is not the properly way of the btrfs purge old snapshot on source? Best Regards, Siranee Jarwachirakul. > On Wed, Aug 9, 2017 at 12:36 AM, <siranee.ja@tpc.co.th> wrote: > >> 488 btrfs sub snap mysql_201707230830 mysql >> 489 systemctl start mariadb >> 490 btrfs sub list . >> 491 cat /var/log/mariadb/mariadb.log > > OK so mysql_201707230830 once on machine B is inconsistent somehow. So > the questions I have are: > > Is mysql_201707230830 on machine A really identical to > mysql_201707230830 on machine B? You can do an rsync -anc (double > check those options) which should independently check whether those > two subvolumes are in fact identical. The -n is a no op, which doesn't > really matter much because as read only subvolumes any attempt to sync > will just result in noisy messages. The -c causes rsync to do its own > checksum verification on both sides. > > If the subvolumes are different, we need to find out why. > > If the subvolumes are the same, then I wonder if you can reproduce the > mariadb complaint on machine A merely by making a rw snapshot of > mysql_201707230830 and trying to start it. If so, then it's not a send > receive problem, it sounds like the snapshot itself is inconsistent, > maybe mariadb hasn't actually completely closed out the database at > the time the read only snapshot was taken? I'm not sure. > > If the subvolumes are different, I'm going to recommend updating at > least the btrfs-progs because 4.4 is kinda old at this point. The > kernel code is what's mainly responsible for the send stream, and the > user space code is mainly responsible for receiving. And I don't off > hand know or want to look up all the send receive changes between 4.4 > and 4.12 to speculate on whether this is has already been fixed. > > What's the kernel version? > > -- > Chris Murphy > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-11 4:40 ` siranee.ja @ 2017-08-11 6:00 ` siranee.ja 2017-08-11 15:35 ` Chris Murphy 2017-08-11 15:06 ` Chris Murphy 1 sibling, 1 reply; 31+ messages in thread From: siranee.ja @ 2017-08-11 6:00 UTC (permalink / raw) To: siranee.ja; +Cc: Chris Murphy, Btrfs BTRFS, voravat Sorry Chris, I forgot to send the diff from rsync -anc result the source container A start data as data on snapshot mysql_201708040830 [root@backuplogC7 tmp]# ls -l /var/lib/mariadb total 0 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql drwxrwxr-x+ 1 mysql mysql 260 Jul 12 08:29 mysql_201708040830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708050830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708060830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708070830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708080830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708090830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708100830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708110830 the dest container B start data as data on snapshot mysql_201708070830 [root@joytest tmp]# ls -l /var/lib/mariadb total 0 drwxrwxr-x+ 1 mysql mysql 260 Aug 11 10:24 mysql drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708070830 drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708080830 drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708090830 drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708100830 drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708110830 tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt dest_mysql_201708110830.txt 1c1 < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708110830 --- > drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708110830 5c5 < -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1 --- > -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1 Best Regards, Siranee Jaraswachirakul. > Hi Chris, > The kernel version that I test is "4.4.0-89-generic" as I tested on ubuntu lxd If I > want to change the kernel version I have to upgrade the host box. > > As you suggest the rsync to compare the subvolumes. I found the point. > the subvolumes are different only after I start to del old subvolumes on machine A > the steps are > > 30 08 * * * root /root/script/backup/backupsnap.sh root password > /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log > 05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166 > /var/lib/mariadb >> /var/log/btrfs_send.log > 30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >> > /var/log/btrfs_del.log > > > The following script maintain snapshot to currently only 7 snapshots. > [root@backuplogC7 ~]# cat /root/script/delete_btrfs_sub_snap_volume.sh > basepath=$1 > keepcount=$2 > havecount=`btrfs sub list -s ${basepath}|cut -d' ' -f14|wc -l` > delcount=$[$keepcount-$havecount]; > datet=$(date +%Y%m%d%H%M) > echo "Start Delete ${datet}" > if [ $delcount -lt 0 ]; then > # list only snapshot subvolume > for i in `btrfs sub list -s ${basepath}|cut -d' ' -f14|head ${delcount}` > do > echo "btrfs sub delete ${basepath}/$i" > btrfs sub delete ${basepath}/$i > btrfs sub sync ${basepath} > done > else > echo "$delcount -gt 0 nothing to delete" > fi > echo "Stop Delete ${datet}" > > Does it mean my delete script is not the properly way of the btrfs purge old > snapshot on source? > > Best Regards, > > Siranee Jarwachirakul. > >> On Wed, Aug 9, 2017 at 12:36 AM, <siranee.ja@tpc.co.th> wrote: >> >>> 488 btrfs sub snap mysql_201707230830 mysql >>> 489 systemctl start mariadb >>> 490 btrfs sub list . >>> 491 cat /var/log/mariadb/mariadb.log >> >> OK so mysql_201707230830 once on machine B is inconsistent somehow. So >> the questions I have are: >> >> Is mysql_201707230830 on machine A really identical to >> mysql_201707230830 on machine B? You can do an rsync -anc (double >> check those options) which should independently check whether those >> two subvolumes are in fact identical. The -n is a no op, which doesn't >> really matter much because as read only subvolumes any attempt to sync >> will just result in noisy messages. The -c causes rsync to do its own >> checksum verification on both sides. >> >> If the subvolumes are different, we need to find out why. >> >> If the subvolumes are the same, then I wonder if you can reproduce the >> mariadb complaint on machine A merely by making a rw snapshot of >> mysql_201707230830 and trying to start it. If so, then it's not a send >> receive problem, it sounds like the snapshot itself is inconsistent, >> maybe mariadb hasn't actually completely closed out the database at >> the time the read only snapshot was taken? I'm not sure. >> >> If the subvolumes are different, I'm going to recommend updating at >> least the btrfs-progs because 4.4 is kinda old at this point. The >> kernel code is what's mainly responsible for the send stream, and the >> user space code is mainly responsible for receiving. And I don't off >> hand know or want to look up all the send receive changes between 4.4 >> and 4.12 to speculate on whether this is has already been fixed. >> >> What's the kernel version? >> >> -- >> Chris Murphy >> > > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-11 6:00 ` siranee.ja @ 2017-08-11 15:35 ` Chris Murphy 2017-08-12 2:38 ` siranee.ja 0 siblings, 1 reply; 31+ messages in thread From: Chris Murphy @ 2017-08-11 15:35 UTC (permalink / raw) To: siranee.ja; +Cc: Chris Murphy, Btrfs BTRFS, voravat On Fri, Aug 11, 2017 at 12:00 AM, <siranee.ja@tpc.co.th> wrote: > Sorry Chris, > > I forgot to send the diff from rsync -anc result > > the source container A start data as data on snapshot mysql_201708040830 > > [root@backuplogC7 tmp]# ls -l /var/lib/mariadb > total 0 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql > drwxrwxr-x+ 1 mysql mysql 260 Jul 12 08:29 mysql_201708040830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708050830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708060830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708070830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708080830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708090830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708100830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708110830 > > the dest container B start data as data on snapshot mysql_201708070830 > > [root@joytest tmp]# ls -l /var/lib/mariadb > total 0 > drwxrwxr-x+ 1 mysql mysql 260 Aug 11 10:24 mysql > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708070830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708080830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708090830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708100830 > drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708110830 > > > > tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt > dest_mysql_201708110830.txt > 1c1 > < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708110830 > --- >> drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708110830 > 5c5 > < -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1 > --- >> -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1 > I don't really understand this. I don't see the actual rsync -anc command, what I see is a diff of two text files whose contents I also don't understand. So I have to guess that ib_logfile1 is a file inside of a snapshot on machine A, that was btrfs send -p / receive to machine B and is now different for some reason? If the problem is that ib_logfile1 is wrong only on machine B after Btrfs send receive, that suggests it might be a network problem. The Btrfs send receive stream only checksums btrfs metadata (the internal commands in the stream). The data is not checksummed so it is possible an uncaught network error can inject silent data corruption which Btrfs will not catch - it's just the normal TCP/IP network checksumming happening. Anyway, I'm still confused whether the problem is a change only during send/receive, or if there's a change happening on a machine in isolation just when you delete other snapshots. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-11 15:35 ` Chris Murphy @ 2017-08-12 2:38 ` siranee.ja 2017-08-12 4:31 ` Chris Murphy 0 siblings, 1 reply; 31+ messages in thread From: siranee.ja @ 2017-08-12 2:38 UTC (permalink / raw) To: Chris Murphy; +Cc: Chris Murphy, Btrfs BTRFS, voravat [-- Attachment #1: Type: text/plain, Size: 3251 bytes --] Hi Chris, I explain what I have done in the attached file. Please suggest me what I should do next? I plan to reproduce the data mannually for the daily inserted data to check the detail step by step to know exactly which steps make the btrfs send /receive result diff on the Second box (Machine Y). I will start with the first btrfs snapshot that I already had (mysql_201707210830). Do you think I should do others? Best Regards, Siranee Jaraswachirakul. > On Fri, Aug 11, 2017 at 12:00 AM, <siranee.ja@tpc.co.th> wrote: >> Sorry Chris, >> >> I forgot to send the diff from rsync -anc result >> >> the source container A start data as data on snapshot mysql_201708040830 >> >> [root@backuplogC7 tmp]# ls -l /var/lib/mariadb >> total 0 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql >> drwxrwxr-x+ 1 mysql mysql 260 Jul 12 08:29 mysql_201708040830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708050830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708060830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708070830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708080830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708090830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708100830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708110830 >> >> the dest container B start data as data on snapshot mysql_201708070830 >> >> [root@joytest tmp]# ls -l /var/lib/mariadb >> total 0 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 11 10:24 mysql >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708070830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708080830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708090830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708100830 >> drwxrwxr-x+ 1 mysql mysql 260 Aug 7 09:26 mysql_201708110830 >> >> >> >> tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt >> dest_mysql_201708110830.txt >> 1c1 >> < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708110830 >> --- >>> drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708110830 >> 5c5 >> < -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1 >> --- >>> -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1 >> > > I don't really understand this. I don't see the actual rsync -anc > command, what I see is a diff of two text files whose contents I also > don't understand. So I have to guess that ib_logfile1 is a file inside > of a snapshot on machine A, that was btrfs send -p / receive to > machine B and is now different for some reason? > > If the problem is that ib_logfile1 is wrong only on machine B after > Btrfs send receive, that suggests it might be a network problem. The > Btrfs send receive stream only checksums btrfs metadata (the internal > commands in the stream). The data is not checksummed so it is possible > an uncaught network error can inject silent data corruption which > Btrfs will not catch - it's just the normal TCP/IP network > checksumming happening. > > Anyway, I'm still confused whether the problem is a change only during > send/receive, or if there's a change happening on a machine in > isolation just when you delete other snapshots. > > > -- > Chris Murphy > [-- Attachment #2: detail_explain_more.txt --] [-- Type: text/plain, Size: 53585 bytes --] Hi Chris, I would like to describe what I have done. Exactly I have 3 boxes of ubuntu 16.04 LTS in mycompany environment and the other on hosting provider. Overall steps First -> started first box in company environment and send btrfs offline (batch) to receive with scp to remote box on hosting provider. After incorrect result of incremental I had to send the new base to start over on remote box around 3 times I changed to the Second step. Second -> Setup the second box in company environment on the different network segment of the first box and still send btrfs offline (batch) to receive with scp from first box to second box Yes the result still the same as the First even different date but the incremental unusable vary (not only on the second incremental send) Third -> Change from test with 2 boxes to be only one box with 2 containers and use online send and receive btrfs as you see the current result of btrfs sub volumes. The result still incorrect some day which I had to start send the new base btrfs snapshot whenever it started to be incorrect. I suspect the send/recieve machanism then I uses the snapshot on mysql_201708040830 to be the start base on the source btrfs in container A (backuplogC7) (As I said the local without send/receive it work) until now. Since mysql_201708040830 the incremental still work until mysql_201708110830 it start to incorrect. 1. I started with the first box (Machine X called "virtualtrust3") tpcorp@virtualtrust3:~$ uname -a Linux virtualtrust3 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux tpcorp@virtualtrust3:~$ cat /etc/*release* DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS" NAME="Ubuntu" VERSION="16.04.2 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.2 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial 2. I have a container for the mariadb databases with have data inserted every day at 7:30 AM. tpcorp@virtualtrust3:~$ lxc list backuplog +-------------+---------+-----------------------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +-------------+---------+-----------------------+------+------------+-----------+ | backuplogC7 | RUNNING | 192.168.47.185 (eth0) | | PERSISTENT | 0 | +-------------+---------+-----------------------+------+------------+-----------+ inside container backuplogC7 (Container A) tpcorp@virtualtrust3:~$ lxc exec backuplogC7 -- bash [root@backuplogC7 ~]# uname -a Linux backuplogC7 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux [root@backuplogC7 ~]# cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) [root@backuplogC7 ~]# [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb ID 257 gen 532 top level 5 path mysql ID 315 gen 491 top level 5 path mysql_201708050830 ID 316 gen 498 top level 5 path mysql_201708060830 ID 317 gen 503 top level 5 path mysql_201708070830 ID 318 gen 507 top level 5 path mysql_201708080830 ID 319 gen 514 top level 5 path mysql_201708090830 ID 320 gen 524 top level 5 path mysql_201708100830 ID 321 gen 529 top level 5 path mysql_201708110830 [root@backuplogC7 ~]# ls -l /var/lib/mariadb total 0 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708050830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708060830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708070830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708080830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708090830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708100830 drwxrwxr-x+ 1 mysql mysql 260 Aug 4 13:10 mysql_201708110830 [root@backuplogC7 ~]# 3. First time I created the container A I took the first snapshot at mysql_201707210830 [root@backuplogC7 ~]# cat /etc/crontab SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # For details see man 4 crontabs # Example of job definition: # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | .---------- day of month (1 - 31) # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # | | | | | # * * * * * user-name command to be executed 30 08 * * * root /root/script/backup/backupsnap.sh root password /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log 05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166 /var/lib/mariadb >> /var/log/btrfs_send.log 0 19 * * * root /root/script/backup/doBackup_file_backuplog.sh 30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >> /var/log/btrfs_del.log # temp ##00 09 * * * root /root/script/backupbtrfs_all.sh /var/lib/mariadb /backupbtrfs mysql_201707210830 You will see the /root/script/backupbtrfs_all.sh because I stated with batch send and receive to the box on hosting provider. Daily Routine Steps to test still be the same except change from offline (batch) to online 3.1 I had application to insert data to mariadb everyday around 7:30 AM 3.2 the snapshot has been took every day around 8:30 AM with the backupsnap.sh script/backup/backupsnap 3.3 First test I have to use the 09:00 AM /root/script/backupbtrfs_all.sh to btrfs send to files on /backupbtrfs with the starting snapshot mysql_201707210830 3.4 at 19:00 root/script/backup/doBackup_file_backuplog.sh for the ordinary online backup of mariadb just mysqldump 3.5 after backup ordinary I delete the old snapshot to keep only 7 snapshots Now the offline btrfs send is commented out and the Daily Routine Steps are 3.1 I had application to insert data to mariadb everyday around 7:30 AM 3.2 the snapshot has been took every day around 8:30 AM with the backupsnap.sh script/backup/backupsnap 3.3 09:05 AM /root/script/backupbtrfs_inc.sh to btrfs send to receive at destination online 3.4 at 19:00 root/script/backup/doBackup_file_backuplog.sh for the ordinary online backup of mariadb just mysqldump 3.5 after backup ordinary I delete the old snapshot to keep only 7 snapshots As you would like to see the rsync command [root@backuplogC7 ~]# history|grep rsync 436 rsync 437 yum install rsync 444 rsync -anc mysql_201708110830 > /tmp/source_mysql_201708110830.txt 453 history|grep rsync 454 rsync -anc mysql_201708100830 > /tmp/source_mysql_201708100830.txt 455 rsync -anc /var/lib/mariadb/mysql_201708100830 > /tmp/source_mysql_201708100830.txt 462 history|grep rsync 463 rsync -anc /var/lib/mariadb/mysql_201708090830 > /tmp/source_mysql_201708090830.txt 464 rsync -anc /var/lib/mariadb/mysql_201708080830 > /tmp/source_mysql_201708080830.txt 465 rsync -anc /var/lib/mariadb/mysql_201708070830 > /tmp/source_mysql_201708070830.txt [root@joytest ~]# history|grep rsync 423 rsync -anc /var/lib/mariadb/mysql_201708090830 424 yum install rsync 425 rsync -anc /var/lib/mariadb/mysql_201708090830 439 rsync -anc mysql_201708110830 > /tmp/dest_mysql_201708110830.txt 448 hsitory|grep rsync 449 history|grep rsync 450 rsync -anc /var/lib/mariadb/mysql_201708100830 > /tmp/dest_mysql_201708100830.txt 451 rsync -anc /var/lib/mariadb/mysql_201708090830 > /tmp/dest_mysql_201708090830.txt 452 rsync -anc /var/lib/mariadb/mysql_201708080830 > /tmp/dest_mysql_201708080830.txt 453 rsync -anc /var/lib/mariadb/mysql_201708070830 > /tmp/dest_mysql_201708070830.txt 461 rsync -anc mysql_201708110830 > /tmp/dest_mysql_201708110830.txt2 462 rsync -anc /var/lib/mariadb/mysql_201708110830 > /tmp/dest_mysql_201708110830.txt2 485 history|grep rsync 486 rsync -anc /var/lib/mariadb/mysql_201708110830 > /tmp/dest_mysql_201708110830.txt3 501 history|grep rsync tpcorp@virtualtrust3:~$ history|grep pull 1966 lxc file pull backuplogC7/tmp/source_/tmp/dest_mysql_201708110830.txt 1967 lxc file pull backuplogC7/tmp/source_mysql_201708110830.txt . 1968 lxc file pull joytest/tmp/dest_mysql_201708110830.txt . 1975 history|grep pull 1976 lxc file pull backuplogC7/tmp/source_mysql_201708100830.txt . 1977 lxc file pull joytest/tmp/dest_mysql_201708100830.txt . 1982 lxc file pull joytest/tmp/dest_mysql_*.txt . 1983 history|grep pull 1984 lxc file pull joytest/tmp/dest_mysql_201708090830.txt . 1985 lxc file pull joytest/tmp/dest_mysql_201708080830.txt . 1986 lxc file pull joytest/tmp/dest_mysql_201708070830.txt . 1987 lxc file pull backuplogC7/tmp/source_mysql_201708090830.txt . 1988 lxc file pull backuplogC7/tmp/source_mysql_201708080830.txt . 1989 lxc file pull backuplogC7/tmp/source_mysql_201708070830.txt . tpcorp@virtualtrust3:~$ history|grep diff 1970 diff source_mysql_201708110830.txt dest_mysql_201708110830.txt 1972 diff source_mysql_201708110830.txt dest_mysql_201708110830.txt 1978 diff source_mysql_201708100830.txt dest_mysql_201708100830.txt 1991 diff source_mysql_201708070830.txt dest_mysql_201708070830.txt 1992 diff source_mysql_201708080830.txt dest_mysql_201708080830.txt 1993 diff source_mysql_201708090830.txt dest_mysql_201708090830.txt 1994 diff source_mysql_201708100830.txt dest_mysql_201708100830.txt 1995 diff source_mysql_201708110830.txt dest_mysql_201708110830.txt "The diff command just display only the difference between source and destination" "The following means the source differs from destination not only subvolume "mysql_201708110830" diff at datetime but also file "ib_logfile1" which diff only one second. The mariadb can not start. tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708110830.txt dest_mysql_201708110830.txt 1c1 < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708110830 --- > drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708110830 5c5 < -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1 --- > -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1 "The following means the source differs from destination only diff at datetime tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708100830.txt dest_mysql_201708100830.txt 1c1 < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708100830 --- > drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708100830 tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708090830.txt dest_mysql_201708090830.txt 1c1 < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708090830 --- > drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708090830 tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708080830.txt dest_mysql_201708080830.txt 1c1 < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708080830 --- > drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708080830 tpcorp@virtualtrust3:/tmp$ diff source_mysql_201708070830.txt dest_mysql_201708070830.txt 1c1 < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708070830 --- > drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708070830 The Last snapshot send /receive tpcorp@virtualtrust3:/tmp$ lxc file pull backuplogC7/tmp/source_mysql_201708120830.txt . tpcorp@virtualtrust3:/tmp$ lxc file pull joytest/tmp/dest_mysql_201708120830.txt . tpcorp@virtualtrust3:/tmp$ diff /tmp/source_mysql_201708120830.txt /tmp/dest_mysql_201708120830.txt 1c1 < drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708120830 --- > drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708120830 5c5 < -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708120830/ib_logfile1 --- > -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708120830/ib_logfile1 tpcorp@virtualtrust3:/tmp$ ----------------- As you request for the real output on screen of source (container A) rsync ----------------- [root@backuplogC7 ~]# rsync -anc /var/lib/mariadb/mysql_201708110830 drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708110830 -rw-rwx--- 16384 2017/08/04 13:08:08 mysql_201708110830/aria_log.00000001 -rw-rwx--- 52 2017/08/04 13:08:08 mysql_201708110830/aria_log_control -rw-rwx--- 5242880 2017/08/11 08:29:07 mysql_201708110830/ib_logfile0 -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708110830/ib_logfile1 -rw-rwx--- 706740224 2017/08/11 08:29:07 mysql_201708110830/ibdata1 -rw-rw---- 264 2017/03/30 08:40:47 mysql_201708110830/mysql-bin.000001 -rw-rw---- 19 2017/03/30 08:38:29 mysql_201708110830/mysql-bin.index drwx------ 644 2017/03/31 09:39:55 mysql_201708110830/backups_db -rw-rw---- 61 2017/03/28 15:14:28 mysql_201708110830/backups_db/db.opt -rw-rw---- 8554 2017/03/28 15:17:54 mysql_201708110830/backups_db/ints.frm -rw-rw---- 8766 2017/03/31 09:14:26 mysql_201708110830/backups_db/tpc_backup_detail.frm -rw-rw---- 8836 2017/03/31 09:25:41 mysql_201708110830/backups_db/tpc_backup_header.frm -rw-rw---- 9020 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_history.frm -rw-rw---- 8730 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_schedule_level_master.frm -rw-rw---- 8774 2017/03/31 09:39:55 mysql_201708110830/backups_db/tpc_backup_schedule_master.frm -rw-rw---- 8774 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_schedule_master_extra.frm -rw-rw---- 8774 2017/03/28 15:18:59 mysql_201708110830/backups_db/tpc_backup_schedule_old.frm -rw-rw---- 8672 2017/03/28 15:18:59 mysql_201708110830/backups_db/tpc_backup_server.frm -rw-rw---- 8888 2017/03/31 09:13:41 mysql_201708110830/backups_db/tpc_backup_step.frm -rw-rw---- 8722 2017/03/28 15:19:10 mysql_201708110830/backups_db/tpc_backup_user.frm -rw-rw---- 1851 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_backup_header_status.frm -rw-rw---- 716 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_calendar.frm -rw-rw---- 1611 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_dup_backup.frm drwxrwx--- 2094 2017/03/28 14:54:08 mysql_201708110830/mysql -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.MYI -rw-rw---- 8820 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.frm -rw-rw---- 440 2017/08/01 17:00:06 mysql_201708110830/mysql/db.MYD -rw-rw---- 5120 2017/08/02 08:30:01 mysql_201708110830/mysql/db.MYI -rw-rw---- 9582 2017/03/28 14:54:06 mysql_201708110830/mysql/db.frm -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708110830/mysql/event.MYD -rw-rw---- 2048 2017/03/28 14:54:06 mysql_201708110830/mysql/event.MYI -rw-rw---- 10239 2017/03/28 14:54:06 mysql_201708110830/mysql/event.frm -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708110830/mysql/func.MYD -rw-rw---- 1024 2017/03/28 14:54:06 mysql_201708110830/mysql/func.MYI -rw-rw---- 8665 2017/03/28 14:54:06 mysql_201708110830/mysql/func.frm -rw-rwx--- 35 2017/08/11 08:30:01 mysql_201708110830/mysql/general_log.CSM -rw-rwx--- 0 2017/03/22 14:30:05 mysql_201708110830/mysql/general_log.CSV -rw-rwx--- 8776 2017/03/22 14:30:05 mysql_201708110830/mysql/general_log.frm -rw-rw---- 1092 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.MYD -rw-rw---- 3072 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.MYI -rw-rw---- 8700 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.frm -rw-rw---- 91408 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.MYD -rw-rw---- 16384 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.MYI -rw-rw---- 8612 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.frm -rw-rw---- 9252 2017/03/28 14:54:07 mysql_201708110830/mysql/help_relation.MYD -rw-rw---- 17408 2017/03/28 14:54:07 mysql_201708110830/mysql/help_relation.MYI -rw-rw---- 8630 2017/03/28 14:54:06 mysql_201708110830/mysql/help_relation.frm -rw-rw---- 450388 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.MYD -rw-rw---- 20480 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.MYI -rw-rw---- 8770 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/host.MYD -rw-rw---- 2048 2017/03/28 14:54:07 mysql_201708110830/mysql/host.MYI -rw-rw---- 9510 2017/03/28 14:54:07 mysql_201708110830/mysql/host.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.MYI -rw-rw---- 8778 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.MYI -rw-rw---- 8586 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.MYD -rw-rw---- 2048 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.MYI -rw-rw---- 10012 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.MYI -rw-rw---- 8875 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.frm -rw-rw---- 1386 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.MYD -rw-rw---- 5120 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.MYI -rw-rw---- 8800 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.MYI -rw-rw---- 8838 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.frm -rw-rwx--- 35 2017/08/11 08:30:01 mysql_201708110830/mysql/slow_log.CSM -rw-rwx--- 0 2017/03/22 14:30:05 mysql_201708110830/mysql/slow_log.CSV -rw-rwx--- 8976 2017/03/22 14:30:05 mysql_201708110830/mysql/slow_log.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.MYI -rw-rw---- 8955 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.MYI -rw-rw---- 8636 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.MYI -rw-rw---- 8624 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.MYI -rw-rw---- 8606 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.MYI -rw-rw---- 8686 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.MYI -rw-rw---- 8748 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.frm -rw-rw---- 668 2017/08/01 17:00:06 mysql_201708110830/mysql/user.MYD -rw-rw---- 2048 2017/08/02 08:30:01 mysql_201708110830/mysql/user.MYI -rw-rw---- 10630 2017/03/28 14:54:07 mysql_201708110830/mysql/user.frm drwxrwx--- 868 2017/03/22 14:30:05 mysql_201708110830/performance_schema -rw-rwx--- 8624 2017/03/22 14:30:05 mysql_201708110830/performance_schema/cond_instances.frm -rw-rwx--- 61 2017/03/22 14:30:05 mysql_201708110830/performance_schema/db.opt -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_current.frm -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_history.frm -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_history_long.frm -rw-rwx--- 8878 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_by_instance.frm -rw-rwx--- 8854 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_by_thread_by_event_name.frm -rw-rwx--- 8814 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_global_by_event_name.frm -rw-rwx--- 8654 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_instances.frm -rw-rwx--- 8800 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_summary_by_event_name.frm -rw-rwx--- 8840 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_summary_by_instance.frm -rw-rwx--- 8684 2017/03/22 14:30:05 mysql_201708110830/performance_schema/mutex_instances.frm -rw-rwx--- 8776 2017/03/22 14:30:05 mysql_201708110830/performance_schema/performance_timers.frm -rw-rwx--- 8758 2017/03/22 14:30:05 mysql_201708110830/performance_schema/rwlock_instances.frm -rw-rwx--- 8605 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_consumers.frm -rw-rwx--- 8637 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_instruments.frm -rw-rwx--- 8650 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_timers.frm -rw-rwx--- 8650 2017/03/22 14:30:05 mysql_201708110830/performance_schema/threads.frm drwx------ 12 2017/03/29 13:29:07 mysql_201708110830/test -rw-rw---- 61 2017/03/29 13:11:26 mysql_201708110830/test/db.opt [root@backuplogC7 ~]# rsync -anc /var/lib/mariadb/mysql_201708120830 drwxrwxr-x 260 2017/08/04 13:10:56 mysql_201708120830 -rw-rwx--- 16384 2017/08/04 13:08:08 mysql_201708120830/aria_log.00000001 -rw-rwx--- 52 2017/08/04 13:08:08 mysql_201708120830/aria_log_control -rw-rwx--- 5242880 2017/08/12 07:30:46 mysql_201708120830/ib_logfile0 -rw-rwx--- 5242880 2017/08/10 07:30:35 mysql_201708120830/ib_logfile1 -rw-rwx--- 706740224 2017/08/12 07:30:46 mysql_201708120830/ibdata1 -rw-rw---- 264 2017/03/30 08:40:47 mysql_201708120830/mysql-bin.000001 -rw-rw---- 19 2017/03/30 08:38:29 mysql_201708120830/mysql-bin.index drwx------ 644 2017/03/31 09:39:55 mysql_201708120830/backups_db -rw-rw---- 61 2017/03/28 15:14:28 mysql_201708120830/backups_db/db.opt -rw-rw---- 8554 2017/03/28 15:17:54 mysql_201708120830/backups_db/ints.frm -rw-rw---- 8766 2017/03/31 09:14:26 mysql_201708120830/backups_db/tpc_backup_detail.frm -rw-rw---- 8836 2017/03/31 09:25:41 mysql_201708120830/backups_db/tpc_backup_header.frm -rw-rw---- 9020 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_history.frm -rw-rw---- 8730 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_schedule_level_master.frm -rw-rw---- 8774 2017/03/31 09:39:55 mysql_201708120830/backups_db/tpc_backup_schedule_master.frm -rw-rw---- 8774 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_schedule_master_extra.frm -rw-rw---- 8774 2017/03/28 15:18:59 mysql_201708120830/backups_db/tpc_backup_schedule_old.frm -rw-rw---- 8672 2017/03/28 15:18:59 mysql_201708120830/backups_db/tpc_backup_server.frm -rw-rw---- 8888 2017/03/31 09:13:41 mysql_201708120830/backups_db/tpc_backup_step.frm -rw-rw---- 8722 2017/03/28 15:19:10 mysql_201708120830/backups_db/tpc_backup_user.frm -rw-rw---- 1851 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_backup_header_status.frm -rw-rw---- 716 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_calendar.frm -rw-rw---- 1611 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_dup_backup.frm drwxrwx--- 2094 2017/03/28 14:54:08 mysql_201708120830/mysql -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.MYI -rw-rw---- 8820 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.frm -rw-rw---- 440 2017/08/01 17:00:06 mysql_201708120830/mysql/db.MYD -rw-rw---- 5120 2017/08/02 08:30:01 mysql_201708120830/mysql/db.MYI -rw-rw---- 9582 2017/03/28 14:54:06 mysql_201708120830/mysql/db.frm -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708120830/mysql/event.MYD -rw-rw---- 2048 2017/03/28 14:54:06 mysql_201708120830/mysql/event.MYI -rw-rw---- 10239 2017/03/28 14:54:06 mysql_201708120830/mysql/event.frm -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708120830/mysql/func.MYD -rw-rw---- 1024 2017/03/28 14:54:06 mysql_201708120830/mysql/func.MYI -rw-rw---- 8665 2017/03/28 14:54:06 mysql_201708120830/mysql/func.frm -rw-rwx--- 35 2017/08/12 08:30:01 mysql_201708120830/mysql/general_log.CSM -rw-rwx--- 0 2017/03/22 14:30:05 mysql_201708120830/mysql/general_log.CSV -rw-rwx--- 8776 2017/03/22 14:30:05 mysql_201708120830/mysql/general_log.frm -rw-rw---- 1092 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.MYD -rw-rw---- 3072 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.MYI -rw-rw---- 8700 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.frm -rw-rw---- 91408 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.MYD -rw-rw---- 16384 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.MYI -rw-rw---- 8612 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.frm -rw-rw---- 9252 2017/03/28 14:54:07 mysql_201708120830/mysql/help_relation.MYD -rw-rw---- 17408 2017/03/28 14:54:07 mysql_201708120830/mysql/help_relation.MYI -rw-rw---- 8630 2017/03/28 14:54:06 mysql_201708120830/mysql/help_relation.frm -rw-rw---- 450388 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.MYD -rw-rw---- 20480 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.MYI -rw-rw---- 8770 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/host.MYD -rw-rw---- 2048 2017/03/28 14:54:07 mysql_201708120830/mysql/host.MYI -rw-rw---- 9510 2017/03/28 14:54:07 mysql_201708120830/mysql/host.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.MYI -rw-rw---- 8778 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.MYI -rw-rw---- 8586 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.MYD -rw-rw---- 2048 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.MYI -rw-rw---- 10012 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.MYI -rw-rw---- 8875 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.frm -rw-rw---- 1386 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.MYD -rw-rw---- 5120 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.MYI -rw-rw---- 8800 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.MYI -rw-rw---- 8838 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.frm -rw-rwx--- 35 2017/08/12 08:30:01 mysql_201708120830/mysql/slow_log.CSM -rw-rwx--- 0 2017/03/22 14:30:05 mysql_201708120830/mysql/slow_log.CSV -rw-rwx--- 8976 2017/03/22 14:30:05 mysql_201708120830/mysql/slow_log.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.MYI -rw-rw---- 8955 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.MYI -rw-rw---- 8636 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.MYI -rw-rw---- 8624 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.MYI -rw-rw---- 8606 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.MYI -rw-rw---- 8686 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.MYI -rw-rw---- 8748 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.frm -rw-rw---- 668 2017/08/01 17:00:06 mysql_201708120830/mysql/user.MYD -rw-rw---- 2048 2017/08/02 08:30:01 mysql_201708120830/mysql/user.MYI -rw-rw---- 10630 2017/03/28 14:54:07 mysql_201708120830/mysql/user.frm drwxrwx--- 868 2017/03/22 14:30:05 mysql_201708120830/performance_schema -rw-rwx--- 8624 2017/03/22 14:30:05 mysql_201708120830/performance_schema/cond_instances.frm -rw-rwx--- 61 2017/03/22 14:30:05 mysql_201708120830/performance_schema/db.opt -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_current.frm -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_history.frm -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_history_long.frm -rw-rwx--- 8878 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_by_instance.frm -rw-rwx--- 8854 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_by_thread_by_event_name.frm -rw-rwx--- 8814 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_global_by_event_name.frm -rw-rwx--- 8654 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_instances.frm -rw-rwx--- 8800 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_summary_by_event_name.frm -rw-rwx--- 8840 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_summary_by_instance.frm -rw-rwx--- 8684 2017/03/22 14:30:05 mysql_201708120830/performance_schema/mutex_instances.frm -rw-rwx--- 8776 2017/03/22 14:30:05 mysql_201708120830/performance_schema/performance_timers.frm -rw-rwx--- 8758 2017/03/22 14:30:05 mysql_201708120830/performance_schema/rwlock_instances.frm -rw-rwx--- 8605 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_consumers.frm -rw-rwx--- 8637 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_instruments.frm -rw-rwx--- 8650 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_timers.frm -rw-rwx--- 8650 2017/03/22 14:30:05 mysql_201708120830/performance_schema/threads.frm drwx------ 12 2017/03/29 13:29:07 mysql_201708120830/test -rw-rw---- 61 2017/03/29 13:11:26 mysql_201708120830/test/db.opt ----------------- As you request for the real output on screen of source (container B) rsync ----------------- tpcorp@virtualtrust3:/tmp$ lxc exec joytest -- bash [root@joytest ~]# rsync -anc /var/lib/mariadb/mysql_201708110830 drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708110830 -rw-rwx--- 16384 2017/08/04 13:08:08 mysql_201708110830/aria_log.00000001 -rw-rwx--- 52 2017/08/04 13:08:08 mysql_201708110830/aria_log_control -rw-rwx--- 5242880 2017/08/11 08:29:07 mysql_201708110830/ib_logfile0 -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708110830/ib_logfile1 -rw-rwx--- 706740224 2017/08/11 08:29:07 mysql_201708110830/ibdata1 -rw-rw---- 264 2017/03/30 08:40:47 mysql_201708110830/mysql-bin.000001 -rw-rw---- 19 2017/03/30 08:38:29 mysql_201708110830/mysql-bin.index drwx------ 644 2017/03/31 09:39:55 mysql_201708110830/backups_db -rw-rw---- 61 2017/03/28 15:14:28 mysql_201708110830/backups_db/db.opt -rw-rw---- 8554 2017/03/28 15:17:54 mysql_201708110830/backups_db/ints.frm -rw-rw---- 8766 2017/03/31 09:14:26 mysql_201708110830/backups_db/tpc_backup_detail.frm -rw-rw---- 8836 2017/03/31 09:25:41 mysql_201708110830/backups_db/tpc_backup_header.frm -rw-rw---- 9020 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_history.frm -rw-rw---- 8730 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_schedule_level_master.frm -rw-rw---- 8774 2017/03/31 09:39:55 mysql_201708110830/backups_db/tpc_backup_schedule_master.frm -rw-rw---- 8774 2017/03/28 15:18:58 mysql_201708110830/backups_db/tpc_backup_schedule_master_extra.frm -rw-rw---- 8774 2017/03/28 15:18:59 mysql_201708110830/backups_db/tpc_backup_schedule_old.frm -rw-rw---- 8672 2017/03/28 15:18:59 mysql_201708110830/backups_db/tpc_backup_server.frm -rw-rw---- 8888 2017/03/31 09:13:41 mysql_201708110830/backups_db/tpc_backup_step.frm -rw-rw---- 8722 2017/03/28 15:19:10 mysql_201708110830/backups_db/tpc_backup_user.frm -rw-rw---- 1851 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_backup_header_status.frm -rw-rw---- 716 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_calendar.frm -rw-rw---- 1611 2017/03/28 15:19:11 mysql_201708110830/backups_db/v_dup_backup.frm drwxrwx--- 2094 2017/03/28 14:54:08 mysql_201708110830/mysql -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.MYI -rw-rw---- 8820 2017/03/28 14:54:06 mysql_201708110830/mysql/columns_priv.frm -rw-rw---- 440 2017/08/01 17:00:06 mysql_201708110830/mysql/db.MYD -rw-rw---- 5120 2017/08/02 08:30:01 mysql_201708110830/mysql/db.MYI -rw-rw---- 9582 2017/03/28 14:54:06 mysql_201708110830/mysql/db.frm -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708110830/mysql/event.MYD -rw-rw---- 2048 2017/03/28 14:54:06 mysql_201708110830/mysql/event.MYI -rw-rw---- 10239 2017/03/28 14:54:06 mysql_201708110830/mysql/event.frm -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708110830/mysql/func.MYD -rw-rw---- 1024 2017/03/28 14:54:06 mysql_201708110830/mysql/func.MYI -rw-rw---- 8665 2017/03/28 14:54:06 mysql_201708110830/mysql/func.frm -rw-rwx--- 35 2017/08/11 08:30:01 mysql_201708110830/mysql/general_log.CSM -rw-rwx--- 0 2017/03/22 14:30:05 mysql_201708110830/mysql/general_log.CSV -rw-rwx--- 8776 2017/03/22 14:30:05 mysql_201708110830/mysql/general_log.frm -rw-rw---- 1092 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.MYD -rw-rw---- 3072 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.MYI -rw-rw---- 8700 2017/03/28 14:54:06 mysql_201708110830/mysql/help_category.frm -rw-rw---- 91408 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.MYD -rw-rw---- 16384 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.MYI -rw-rw---- 8612 2017/03/28 14:54:06 mysql_201708110830/mysql/help_keyword.frm -rw-rw---- 9252 2017/03/28 14:54:07 mysql_201708110830/mysql/help_relation.MYD -rw-rw---- 17408 2017/03/28 14:54:07 mysql_201708110830/mysql/help_relation.MYI -rw-rw---- 8630 2017/03/28 14:54:06 mysql_201708110830/mysql/help_relation.frm -rw-rw---- 450388 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.MYD -rw-rw---- 20480 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.MYI -rw-rw---- 8770 2017/03/28 14:54:07 mysql_201708110830/mysql/help_topic.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/host.MYD -rw-rw---- 2048 2017/03/28 14:54:07 mysql_201708110830/mysql/host.MYI -rw-rw---- 9510 2017/03/28 14:54:07 mysql_201708110830/mysql/host.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.MYI -rw-rw---- 8778 2017/03/28 14:54:07 mysql_201708110830/mysql/ndb_binlog_index.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.MYI -rw-rw---- 8586 2017/03/28 14:54:07 mysql_201708110830/mysql/plugin.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.MYD -rw-rw---- 2048 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.MYI -rw-rw---- 10012 2017/03/28 14:54:07 mysql_201708110830/mysql/proc.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.MYI -rw-rw---- 8875 2017/03/28 14:54:07 mysql_201708110830/mysql/procs_priv.frm -rw-rw---- 1386 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.MYD -rw-rw---- 5120 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.MYI -rw-rw---- 8800 2017/03/28 14:54:07 mysql_201708110830/mysql/proxies_priv.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.MYI -rw-rw---- 8838 2017/03/28 14:54:07 mysql_201708110830/mysql/servers.frm -rw-rwx--- 35 2017/08/11 08:30:01 mysql_201708110830/mysql/slow_log.CSM -rw-rwx--- 0 2017/03/22 14:30:05 mysql_201708110830/mysql/slow_log.CSV -rw-rwx--- 8976 2017/03/22 14:30:05 mysql_201708110830/mysql/slow_log.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.MYI -rw-rw---- 8955 2017/03/28 14:54:07 mysql_201708110830/mysql/tables_priv.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.MYI -rw-rw---- 8636 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.MYI -rw-rw---- 8624 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_leap_second.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.MYI -rw-rw---- 8606 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_name.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.MYI -rw-rw---- 8686 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.MYI -rw-rw---- 8748 2017/03/28 14:54:07 mysql_201708110830/mysql/time_zone_transition_type.frm -rw-rw---- 668 2017/08/01 17:00:06 mysql_201708110830/mysql/user.MYD -rw-rw---- 2048 2017/08/02 08:30:01 mysql_201708110830/mysql/user.MYI -rw-rw---- 10630 2017/03/28 14:54:07 mysql_201708110830/mysql/user.frm drwxrwx--- 868 2017/03/22 14:30:05 mysql_201708110830/performance_schema -rw-rwx--- 8624 2017/03/22 14:30:05 mysql_201708110830/performance_schema/cond_instances.frm -rw-rwx--- 61 2017/03/22 14:30:05 mysql_201708110830/performance_schema/db.opt -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_current.frm -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_history.frm -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_history_long.frm -rw-rwx--- 8878 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_by_instance.frm -rw-rwx--- 8854 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_by_thread_by_event_name.frm -rw-rwx--- 8814 2017/03/22 14:30:05 mysql_201708110830/performance_schema/events_waits_summary_global_by_event_name.frm -rw-rwx--- 8654 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_instances.frm -rw-rwx--- 8800 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_summary_by_event_name.frm -rw-rwx--- 8840 2017/03/22 14:30:05 mysql_201708110830/performance_schema/file_summary_by_instance.frm -rw-rwx--- 8684 2017/03/22 14:30:05 mysql_201708110830/performance_schema/mutex_instances.frm -rw-rwx--- 8776 2017/03/22 14:30:05 mysql_201708110830/performance_schema/performance_timers.frm -rw-rwx--- 8758 2017/03/22 14:30:05 mysql_201708110830/performance_schema/rwlock_instances.frm -rw-rwx--- 8605 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_consumers.frm -rw-rwx--- 8637 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_instruments.frm -rw-rwx--- 8650 2017/03/22 14:30:05 mysql_201708110830/performance_schema/setup_timers.frm -rw-rwx--- 8650 2017/03/22 14:30:05 mysql_201708110830/performance_schema/threads.frm drwx------ 12 2017/03/29 13:29:07 mysql_201708110830/test -rw-rw---- 61 2017/03/29 13:11:26 mysql_201708110830/test/db.opt [root@joytest ~]# rsync -anc /var/lib/mariadb/mysql_201708120830 drwxrwxr-x 260 2017/08/07 09:26:32 mysql_201708120830 -rw-rwx--- 16384 2017/08/04 13:08:08 mysql_201708120830/aria_log.00000001 -rw-rwx--- 52 2017/08/04 13:08:08 mysql_201708120830/aria_log_control -rw-rwx--- 5242880 2017/08/12 07:30:46 mysql_201708120830/ib_logfile0 -rw-rwx--- 5242880 2017/08/07 07:30:36 mysql_201708120830/ib_logfile1 -rw-rwx--- 706740224 2017/08/12 07:30:46 mysql_201708120830/ibdata1 -rw-rw---- 264 2017/03/30 08:40:47 mysql_201708120830/mysql-bin.000001 -rw-rw---- 19 2017/03/30 08:38:29 mysql_201708120830/mysql-bin.index drwx------ 644 2017/03/31 09:39:55 mysql_201708120830/backups_db -rw-rw---- 61 2017/03/28 15:14:28 mysql_201708120830/backups_db/db.opt -rw-rw---- 8554 2017/03/28 15:17:54 mysql_201708120830/backups_db/ints.frm -rw-rw---- 8766 2017/03/31 09:14:26 mysql_201708120830/backups_db/tpc_backup_detail.frm -rw-rw---- 8836 2017/03/31 09:25:41 mysql_201708120830/backups_db/tpc_backup_header.frm -rw-rw---- 9020 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_history.frm -rw-rw---- 8730 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_schedule_level_master.frm -rw-rw---- 8774 2017/03/31 09:39:55 mysql_201708120830/backups_db/tpc_backup_schedule_master.frm -rw-rw---- 8774 2017/03/28 15:18:58 mysql_201708120830/backups_db/tpc_backup_schedule_master_extra.frm -rw-rw---- 8774 2017/03/28 15:18:59 mysql_201708120830/backups_db/tpc_backup_schedule_old.frm -rw-rw---- 8672 2017/03/28 15:18:59 mysql_201708120830/backups_db/tpc_backup_server.frm -rw-rw---- 8888 2017/03/31 09:13:41 mysql_201708120830/backups_db/tpc_backup_step.frm -rw-rw---- 8722 2017/03/28 15:19:10 mysql_201708120830/backups_db/tpc_backup_user.frm -rw-rw---- 1851 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_backup_header_status.frm -rw-rw---- 716 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_calendar.frm -rw-rw---- 1611 2017/03/28 15:19:11 mysql_201708120830/backups_db/v_dup_backup.frm drwxrwx--- 2094 2017/03/28 14:54:08 mysql_201708120830/mysql -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.MYI -rw-rw---- 8820 2017/03/28 14:54:06 mysql_201708120830/mysql/columns_priv.frm -rw-rw---- 440 2017/08/01 17:00:06 mysql_201708120830/mysql/db.MYD -rw-rw---- 5120 2017/08/02 08:30:01 mysql_201708120830/mysql/db.MYI -rw-rw---- 9582 2017/03/28 14:54:06 mysql_201708120830/mysql/db.frm -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708120830/mysql/event.MYD -rw-rw---- 2048 2017/03/28 14:54:06 mysql_201708120830/mysql/event.MYI -rw-rw---- 10239 2017/03/28 14:54:06 mysql_201708120830/mysql/event.frm -rw-rw---- 0 2017/03/28 14:54:06 mysql_201708120830/mysql/func.MYD -rw-rw---- 1024 2017/03/28 14:54:06 mysql_201708120830/mysql/func.MYI -rw-rw---- 8665 2017/03/28 14:54:06 mysql_201708120830/mysql/func.frm -rw-rwx--- 35 2017/08/12 08:30:01 mysql_201708120830/mysql/general_log.CSM -rw-rwx--- 0 2017/03/22 14:30:05 mysql_201708120830/mysql/general_log.CSV -rw-rwx--- 8776 2017/03/22 14:30:05 mysql_201708120830/mysql/general_log.frm -rw-rw---- 1092 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.MYD -rw-rw---- 3072 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.MYI -rw-rw---- 8700 2017/03/28 14:54:06 mysql_201708120830/mysql/help_category.frm -rw-rw---- 91408 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.MYD -rw-rw---- 16384 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.MYI -rw-rw---- 8612 2017/03/28 14:54:06 mysql_201708120830/mysql/help_keyword.frm -rw-rw---- 9252 2017/03/28 14:54:07 mysql_201708120830/mysql/help_relation.MYD -rw-rw---- 17408 2017/03/28 14:54:07 mysql_201708120830/mysql/help_relation.MYI -rw-rw---- 8630 2017/03/28 14:54:06 mysql_201708120830/mysql/help_relation.frm -rw-rw---- 450388 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.MYD -rw-rw---- 20480 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.MYI -rw-rw---- 8770 2017/03/28 14:54:07 mysql_201708120830/mysql/help_topic.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/host.MYD -rw-rw---- 2048 2017/03/28 14:54:07 mysql_201708120830/mysql/host.MYI -rw-rw---- 9510 2017/03/28 14:54:07 mysql_201708120830/mysql/host.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.MYI -rw-rw---- 8778 2017/03/28 14:54:07 mysql_201708120830/mysql/ndb_binlog_index.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.MYI -rw-rw---- 8586 2017/03/28 14:54:07 mysql_201708120830/mysql/plugin.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.MYD -rw-rw---- 2048 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.MYI -rw-rw---- 10012 2017/03/28 14:54:07 mysql_201708120830/mysql/proc.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.MYI -rw-rw---- 8875 2017/03/28 14:54:07 mysql_201708120830/mysql/procs_priv.frm -rw-rw---- 1386 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.MYD -rw-rw---- 5120 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.MYI -rw-rw---- 8800 2017/03/28 14:54:07 mysql_201708120830/mysql/proxies_priv.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.MYI -rw-rw---- 8838 2017/03/28 14:54:07 mysql_201708120830/mysql/servers.frm -rw-rwx--- 35 2017/08/12 08:30:01 mysql_201708120830/mysql/slow_log.CSM -rw-rwx--- 0 2017/03/22 14:30:05 mysql_201708120830/mysql/slow_log.CSV -rw-rwx--- 8976 2017/03/22 14:30:05 mysql_201708120830/mysql/slow_log.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.MYD -rw-rw---- 4096 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.MYI -rw-rw---- 8955 2017/03/28 14:54:07 mysql_201708120830/mysql/tables_priv.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.MYI -rw-rw---- 8636 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.MYI -rw-rw---- 8624 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_leap_second.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.MYI -rw-rw---- 8606 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_name.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.MYI -rw-rw---- 8686 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition.frm -rw-rw---- 0 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.MYD -rw-rw---- 1024 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.MYI -rw-rw---- 8748 2017/03/28 14:54:07 mysql_201708120830/mysql/time_zone_transition_type.frm -rw-rw---- 668 2017/08/01 17:00:06 mysql_201708120830/mysql/user.MYD -rw-rw---- 2048 2017/08/02 08:30:01 mysql_201708120830/mysql/user.MYI -rw-rw---- 10630 2017/03/28 14:54:07 mysql_201708120830/mysql/user.frm drwxrwx--- 868 2017/03/22 14:30:05 mysql_201708120830/performance_schema -rw-rwx--- 8624 2017/03/22 14:30:05 mysql_201708120830/performance_schema/cond_instances.frm -rw-rwx--- 61 2017/03/22 14:30:05 mysql_201708120830/performance_schema/db.opt -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_current.frm -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_history.frm -rw-rwx--- 9220 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_history_long.frm -rw-rwx--- 8878 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_by_instance.frm -rw-rwx--- 8854 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_by_thread_by_event_name.frm -rw-rwx--- 8814 2017/03/22 14:30:05 mysql_201708120830/performance_schema/events_waits_summary_global_by_event_name.frm -rw-rwx--- 8654 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_instances.frm -rw-rwx--- 8800 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_summary_by_event_name.frm -rw-rwx--- 8840 2017/03/22 14:30:05 mysql_201708120830/performance_schema/file_summary_by_instance.frm -rw-rwx--- 8684 2017/03/22 14:30:05 mysql_201708120830/performance_schema/mutex_instances.frm -rw-rwx--- 8776 2017/03/22 14:30:05 mysql_201708120830/performance_schema/performance_timers.frm -rw-rwx--- 8758 2017/03/22 14:30:05 mysql_201708120830/performance_schema/rwlock_instances.frm -rw-rwx--- 8605 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_consumers.frm -rw-rwx--- 8637 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_instruments.frm -rw-rwx--- 8650 2017/03/22 14:30:05 mysql_201708120830/performance_schema/setup_timers.frm -rw-rwx--- 8650 2017/03/22 14:30:05 mysql_201708120830/performance_schema/threads.frm drwx------ 12 2017/03/29 13:29:07 mysql_201708120830/test -rw-rw---- 61 2017/03/29 13:11:26 mysql_201708120830/test/db.opt ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-12 2:38 ` siranee.ja @ 2017-08-12 4:31 ` Chris Murphy 2017-08-12 5:08 ` siranee.ja 0 siblings, 1 reply; 31+ messages in thread From: Chris Murphy @ 2017-08-12 4:31 UTC (permalink / raw) To: siranee.ja; +Cc: Chris Murphy, Btrfs BTRFS, voravat On Fri, Aug 11, 2017 at 8:38 PM, <siranee.ja@tpc.co.th> wrote: > Hi Chris, > > I explain what I have done in the attached file. That is way too long. And please don't attach files directed at me, that's not appropriate. Just message the list normally, so it can be searched by others down the road. There is a problem with your rsync command, you really don't understand what I was asking. The whole point of rsync is to compare the origin and destination subvolumes to see if they are different from each other. You ran the command on a single subvolume on a single machine which is pointless. On machine A: $ rsync -avnc /mnt/snapshot6/ chris@192.168.1.116:/mnt/snapshot6/ chris@192.168.1.116's password: sending incremental file list sent 125 bytes received 12 bytes 39.14 bytes/sec total size is 30,686 speedup is 223.99 (DRY RUN) [chris@f26h ~]$ Because no files are listed, all are confirmed to be identical. If files are listed, there's a difference. > Please suggest me what I should do next? Check this. I wonder if you're running into this obscure bug where maybe your read only snapshot is not yet synced before it's being sent. Hence it *is* changing on origin machine, but not because of other snapshots being deleted. I have no idea if your scripting handles errors like the bogus stale NFS handle error, maybe it's possible it happens but you're not seeing it? Anyway, you might try just adding a single sync right after taking the snapshot, change nothing else and see if the problem reproduces itself. https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Initial_Bootstrapping I'm pretty sure this is fixed in newer kernels, I haven't run into it in a long time myself. But I don't know when it was fixed. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-12 4:31 ` Chris Murphy @ 2017-08-12 5:08 ` siranee.ja 2017-08-12 21:34 ` Chris Murphy 0 siblings, 1 reply; 31+ messages in thread From: siranee.ja @ 2017-08-12 5:08 UTC (permalink / raw) To: Chris Murphy; +Cc: Chris Murphy, Btrfs BTRFS, voravat Hi Chris, Sorry for the misunderstanding and inappropriate attached the file (I thought that too long text in the mail message so I used the attached file). The backup script has the btrfs sync command since Aug 3 I sent the following rsync command as you said (sorry for the misunderstanding with the rsync command that I'd never used before) The snapshot has been different since snapshot mysql_201708090830 but the mariadb can recovery and start until mysql_201708110830 seems too much differences that can not start. [root@backuplogC7 ~]# more /root/script/backup/backupsnap.sh #Backup # user=$1 password=$2 basepath=$3 datet=$(date +%Y%m%d%H%M) snappath=${basepath}_${datet} echo "Locking databases ${datet}" mysql -u$user -p$password << EOF FLUSH TABLES WITH READ LOCK; system btrfs sub snap -r $basepath $snappath system btrfs sub sync $basepath UNLOCK TABLES; quit EOF echo "Databases unlocked ${datet}" [root@backuplogC7 ~]# ls -l /root/script/backup/backupsnap.sh -rwxr--r-- 1 root root 333 Aug 3 09:11 /root/script/backup/backupsnap.sh [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708070830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 1083.43 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708080830/ sending incremental file list ./ sent 3769 bytes received 19 bytes 841.78 bytes/sec total size is 718361496 speedup is 189641.37 (DRY RUN) [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/ sending incremental file list ./ ib_logfile1 ibdata1 sent 3779 bytes received 25 bytes 1086.86 bytes/sec total size is 718361496 speedup is 188843.72 (DRY RUN) [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708100830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708100830/ sending incremental file list ./ ib_logfile1 ibdata1 sent 3779 bytes received 25 bytes 1086.86 bytes/sec total size is 718361496 speedup is 188843.72 (DRY RUN) [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708110830/ root@192.168.45.166:/var/lib/mariadb/mysql_20170811830/ sending incremental file list created directory /var/lib/mariadb/mysql_20170811830 ./ aria_log.00000001 aria_log_control ib_logfile0 ib_logfile1 ibdata1 mysql-bin.000001 mysql-bin.index backups_db/ backups_db/db.opt backups_db/ints.frm backups_db/tpc_backup_detail.frm backups_db/tpc_backup_header.frm backups_db/tpc_backup_history.frm backups_db/tpc_backup_schedule_level_master.frm backups_db/tpc_backup_schedule_master.frm backups_db/tpc_backup_schedule_master_extra.frm backups_db/tpc_backup_schedule_old.frm backups_db/tpc_backup_server.frm backups_db/tpc_backup_step.frm backups_db/tpc_backup_user.frm backups_db/v_backup_header_status.frm backups_db/v_calendar.frm backups_db/v_dup_backup.frm mysql/ mysql/columns_priv.MYD mysql/columns_priv.MYI mysql/columns_priv.frm mysql/db.MYD mysql/db.MYI mysql/db.frm mysql/event.MYD mysql/event.MYI mysql/event.frm mysql/func.MYD mysql/func.MYI mysql/func.frm mysql/general_log.CSM mysql/general_log.CSV mysql/general_log.frm mysql/help_category.MYD mysql/help_category.MYI mysql/help_category.frm mysql/help_keyword.MYD mysql/help_keyword.MYI mysql/help_keyword.frm mysql/help_relation.MYD mysql/help_relation.MYI mysql/help_relation.frm mysql/help_topic.MYD mysql/help_topic.MYI mysql/help_topic.frm mysql/host.MYD mysql/host.MYI mysql/host.frm mysql/ndb_binlog_index.MYD mysql/ndb_binlog_index.MYI mysql/ndb_binlog_index.frm mysql/plugin.MYD mysql/plugin.MYI mysql/plugin.frm mysql/proc.MYD mysql/proc.MYI mysql/proc.frm mysql/procs_priv.MYD mysql/procs_priv.MYI mysql/procs_priv.frm mysql/proxies_priv.MYD mysql/proxies_priv.MYI mysql/proxies_priv.frm mysql/servers.MYD mysql/servers.MYI mysql/servers.frm mysql/slow_log.CSM mysql/slow_log.CSV mysql/slow_log.frm mysql/tables_priv.MYD mysql/tables_priv.MYI mysql/tables_priv.frm mysql/time_zone.MYD mysql/time_zone.MYI mysql/time_zone.frm mysql/time_zone_leap_second.MYD mysql/time_zone_leap_second.MYI mysql/time_zone_leap_second.frm mysql/time_zone_name.MYD mysql/time_zone_name.MYI mysql/time_zone_name.frm mysql/time_zone_transition.MYD mysql/time_zone_transition.MYI mysql/time_zone_transition.frm mysql/time_zone_transition_type.MYD mysql/time_zone_transition_type.MYI mysql/time_zone_transition_type.frm mysql/user.MYD mysql/user.MYI mysql/user.frm performance_schema/ performance_schema/cond_instances.frm performance_schema/db.opt performance_schema/events_waits_current.frm performance_schema/events_waits_history.frm performance_schema/events_waits_history_long.frm performance_schema/events_waits_summary_by_instance.frm performance_schema/events_waits_summary_by_thread_by_event_name.frm performance_schema/events_waits_summary_global_by_event_name.frm performance_schema/file_instances.frm performance_schema/file_summary_by_event_name.frm performance_schema/file_summary_by_instance.frm performance_schema/mutex_instances.frm performance_schema/performance_timers.frm performance_schema/rwlock_instances.frm performance_schema/setup_consumers.frm performance_schema/setup_instruments.frm performance_schema/setup_timers.frm performance_schema/threads.frm test/ test/db.opt sent 4124 bytes received 370 bytes 1797.60 bytes/sec total size is 718361496 speedup is 159849.02 (DRY RUN) [root@backuplogC7 ~]# [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708120830/ root@192.168.45.166:/var/lib/mariadb/mysql_20170812830/ sending incremental file list created directory /var/lib/mariadb/mysql_20170812830 ./ aria_log.00000001 aria_log_control ib_logfile0 ib_logfile1 ibdata1 mysql-bin.000001 mysql-bin.index backups_db/ backups_db/db.opt backups_db/ints.frm backups_db/tpc_backup_detail.frm backups_db/tpc_backup_header.frm backups_db/tpc_backup_history.frm backups_db/tpc_backup_schedule_level_master.frm backups_db/tpc_backup_schedule_master.frm backups_db/tpc_backup_schedule_master_extra.frm backups_db/tpc_backup_schedule_old.frm backups_db/tpc_backup_server.frm backups_db/tpc_backup_step.frm backups_db/tpc_backup_user.frm backups_db/v_backup_header_status.frm backups_db/v_calendar.frm backups_db/v_dup_backup.frm mysql/ mysql/columns_priv.MYD mysql/columns_priv.MYI mysql/columns_priv.frm mysql/db.MYD mysql/db.MYI mysql/db.frm mysql/event.MYD mysql/event.MYI mysql/event.frm mysql/func.MYD mysql/func.MYI mysql/func.frm mysql/general_log.CSM mysql/general_log.CSV mysql/general_log.frm mysql/help_category.MYD mysql/help_category.MYI mysql/help_category.frm mysql/help_keyword.MYD mysql/help_keyword.MYI mysql/help_keyword.frm mysql/help_relation.MYD mysql/help_relation.MYI mysql/help_relation.frm mysql/help_topic.MYD mysql/help_topic.MYI mysql/help_topic.frm mysql/host.MYD mysql/host.MYI mysql/host.frm mysql/ndb_binlog_index.MYD mysql/ndb_binlog_index.MYI mysql/ndb_binlog_index.frm mysql/plugin.MYD mysql/plugin.MYI mysql/plugin.frm mysql/proc.MYD mysql/proc.MYI mysql/proc.frm mysql/procs_priv.MYD mysql/procs_priv.MYI mysql/procs_priv.frm mysql/proxies_priv.MYD mysql/proxies_priv.MYI mysql/proxies_priv.frm mysql/servers.MYD mysql/servers.MYI mysql/servers.frm mysql/slow_log.CSM mysql/slow_log.CSV mysql/slow_log.frm mysql/tables_priv.MYD mysql/tables_priv.MYI mysql/tables_priv.frm mysql/time_zone.MYD mysql/time_zone.MYI mysql/time_zone.frm mysql/time_zone_leap_second.MYD mysql/time_zone_leap_second.MYI mysql/time_zone_leap_second.frm mysql/time_zone_name.MYD mysql/time_zone_name.MYI mysql/time_zone_name.frm mysql/time_zone_transition.MYD mysql/time_zone_transition.MYI mysql/time_zone_transition.frm mysql/time_zone_transition_type.MYD mysql/time_zone_transition_type.MYI mysql/time_zone_transition_type.frm mysql/user.MYD mysql/user.MYI mysql/user.frm performance_schema/ performance_schema/cond_instances.frm performance_schema/db.opt performance_schema/events_waits_current.frm performance_schema/events_waits_history.frm performance_schema/events_waits_history_long.frm performance_schema/events_waits_summary_by_instance.frm performance_schema/events_waits_summary_by_thread_by_event_name.frm performance_schema/events_waits_summary_global_by_event_name.frm performance_schema/file_instances.frm performance_schema/file_summary_by_event_name.frm performance_schema/file_summary_by_instance.frm performance_schema/mutex_instances.frm performance_schema/performance_timers.frm performance_schema/rwlock_instances.frm performance_schema/setup_consumers.frm performance_schema/setup_instruments.frm performance_schema/setup_timers.frm performance_schema/threads.frm test/ test/db.opt sent 4124 bytes received 370 bytes 1797.60 bytes/sec total size is 718361496 speedup is 159849.02 (DRY RUN) Best Regards, Siranee Jaraswachirakul > On Fri, Aug 11, 2017 at 8:38 PM, <siranee.ja@tpc.co.th> wrote: >> Hi Chris, >> >> I explain what I have done in the attached file. > > That is way too long. And please don't attach files directed at me, > that's not appropriate. Just message the list normally, so it can be > searched by others down the road. > > There is a problem with your rsync command, you really don't > understand what I was asking. The whole point of rsync is to compare > the origin and destination subvolumes to see if they are different > from each other. You ran the command on a single subvolume on a single > machine which is pointless. > > On machine A: > > $ rsync -avnc /mnt/snapshot6/ chris@192.168.1.116:/mnt/snapshot6/ > chris@192.168.1.116's password: > sending incremental file list > > sent 125 bytes received 12 bytes 39.14 bytes/sec > total size is 30,686 speedup is 223.99 (DRY RUN) > [chris@f26h ~]$ > > Because no files are listed, all are confirmed to be identical. If > files are listed, there's a difference. > > > >> Please suggest me what I should do next? > > Check this. I wonder if you're running into this obscure bug where > maybe your read only snapshot is not yet synced before it's being > sent. Hence it *is* changing on origin machine, but not because of > other snapshots being deleted. I have no idea if your scripting > handles errors like the bogus stale NFS handle error, maybe it's > possible it happens but you're not seeing it? Anyway, you might try > just adding a single sync right after taking the snapshot, change > nothing else and see if the problem reproduces itself. > > https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Initial_Bootstrapping > > I'm pretty sure this is fixed in newer kernels, I haven't run into it > in a long time myself. But I don't know when it was fixed. > > > > -- > Chris Murphy > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-12 5:08 ` siranee.ja @ 2017-08-12 21:34 ` Chris Murphy 2017-08-12 22:41 ` Janos Toth F. ` (2 more replies) 0 siblings, 3 replies; 31+ messages in thread From: Chris Murphy @ 2017-08-12 21:34 UTC (permalink / raw) To: siranee.ja; +Cc: Chris Murphy, Btrfs BTRFS, voravat On Fri, Aug 11, 2017 at 11:08 PM, <siranee.ja@tpc.co.th> wrote: > The backup script has the btrfs sync command since Aug 3 >From your script: > system btrfs sub snap -r $basepath $snappath > system btrfs sub sync $basepath >From the man page: sync <path> [subvolid...] Wait until given subvolume(s) are completely removed from the filesystem after deletion. This 'subvolume sync' command, per the man page, is only about subvolume deletion. I suggest replacing it with a regular sync command. I think the problem is that the script does things so fast that the snapshot is not always consistent on disk before btrfs send starts. It's just a guess though. If I'm right, this means the rsync mismaches mean the destination snapshots are bad. Here's what I would do: - delete all the bad/mismatching snapshots only on the destination computer. - he most recent good snapshot pair, which rsync shows origin and destination match, is mysql_201708080830 so you can keep that one on both sides. - manually do incremental send/receive, starting with mysql_201708090830/, to make the destination current again with the origin. - confirm with rsync that the snapshot pairs on origin and destination are the same - now resume using the modified script, which will do snapshot -> sync -> send. OPTIONAL, you could add to your script an rsync -avnc to double check that the incremental send receive is working. This is admittedly inefficient because it checks the *entire* contents of the snapshots on both sides, it's not just checking the incremental data. But if it doesn't take too long, it will help restore trust in send/receive, and confirm that a regular sync is needed in between snapshot and send. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-12 21:34 ` Chris Murphy @ 2017-08-12 22:41 ` Janos Toth F. 2017-08-12 23:07 ` Chris Murphy 2017-08-12 22:49 ` Hugo Mills 2017-08-13 2:20 ` siranee.ja 2 siblings, 1 reply; 31+ messages in thread From: Janos Toth F. @ 2017-08-12 22:41 UTC (permalink / raw) To: Chris Murphy; +Cc: siranee.ja, Btrfs BTRFS, voravat On Sat, Aug 12, 2017 at 11:34 PM, Chris Murphy <lists@colorremedies.com> wrote: > On Fri, Aug 11, 2017 at 11:08 PM, <siranee.ja@tpc.co.th> wrote: > > I think the problem is that the script does things so fast that the > snapshot is not always consistent on disk before btrfs send starts. > It's just a guess though. If I'm right, this means the rsync mismaches > mean the destination snapshots are bad. Hmm. I was wondering about this exact issue the other day when I fiddled with my own backup script. - Should I issue a sync between creating a snapshot and starting to rsync (or send/receive) the content of that fresh snapshot to an external backup storage? I dismissed the thought because I figured rsync should see the proper state regardless if some data is still waiting in the system write-back cache. Now I am confused again. Is it only for send/receive? (I don't use that feature yet but thought about switching to it from rsync.) ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-12 22:41 ` Janos Toth F. @ 2017-08-12 23:07 ` Chris Murphy 0 siblings, 0 replies; 31+ messages in thread From: Chris Murphy @ 2017-08-12 23:07 UTC (permalink / raw) To: Janos Toth F.; +Cc: Chris Murphy, siranee.ja, Btrfs BTRFS, voravat On Sat, Aug 12, 2017 at 4:41 PM, Janos Toth F. <toth.f.janos@gmail.com> wrote: > On Sat, Aug 12, 2017 at 11:34 PM, Chris Murphy <lists@colorremedies.com> wrote: >> On Fri, Aug 11, 2017 at 11:08 PM, <siranee.ja@tpc.co.th> wrote: >> >> I think the problem is that the script does things so fast that the >> snapshot is not always consistent on disk before btrfs send starts. >> It's just a guess though. If I'm right, this means the rsync mismaches >> mean the destination snapshots are bad. > > Hmm. I was wondering about this exact issue the other day when I > fiddled with my own backup script. > - Should I issue a sync between creating a snapshot and starting to > rsync (or send/receive) the content of that fresh snapshot to an > external backup storage? A sync won't hurt, but I think we need btrfs send to include sync if this is still required as stated by the Wiki. And if the problem stated in the Wiki has been fixed, then it'd be useful to know when, but a git log search doesn't reveal anything related to NFS and Btrfs. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-12 21:34 ` Chris Murphy 2017-08-12 22:41 ` Janos Toth F. @ 2017-08-12 22:49 ` Hugo Mills 2017-08-12 23:14 ` Chris Murphy 2017-08-13 2:20 ` siranee.ja 2 siblings, 1 reply; 31+ messages in thread From: Hugo Mills @ 2017-08-12 22:49 UTC (permalink / raw) To: Chris Murphy; +Cc: siranee.ja, Btrfs BTRFS, voravat [-- Attachment #1: Type: text/plain, Size: 1369 bytes --] On Sat, Aug 12, 2017 at 03:34:01PM -0600, Chris Murphy wrote: > On Fri, Aug 11, 2017 at 11:08 PM, <siranee.ja@tpc.co.th> wrote: > > > > The backup script has the btrfs sync command since Aug 3 > > > From your script: > > system btrfs sub snap -r $basepath $snappath > > system btrfs sub sync $basepath > > From the man page: sync <path> [subvolid...] > Wait until given subvolume(s) are completely removed from the > filesystem after deletion. > > > This 'subvolume sync' command, per the man page, is only about > subvolume deletion. I suggest replacing it with a regular sync > command. > > I think the problem is that the script does things so fast that the > snapshot is not always consistent on disk before btrfs send starts. > It's just a guess though. If I'm right, this means the rsync mismaches > mean the destination snapshots are bad. Here's what I would do: I don't see how that can happen. Snapshots are atomic -- they're either there or not there. It's not a matter even of copying the metadata part of the subvol. It's literally just adding a pointer to point at an existing FS tree. Hugo. -- Hugo Mills | If it's December 1941 in Casablanca, what time is it hugo@... carfax.org.uk | in New York? http://carfax.org.uk/ | PGP: E2AB1DE4 | Rick Blaine, Casablanca [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-12 22:49 ` Hugo Mills @ 2017-08-12 23:14 ` Chris Murphy 0 siblings, 0 replies; 31+ messages in thread From: Chris Murphy @ 2017-08-12 23:14 UTC (permalink / raw) To: Hugo Mills, Chris Murphy, siranee.ja, Btrfs BTRFS, voravat On Sat, Aug 12, 2017 at 4:49 PM, Hugo Mills <hugo@carfax.org.uk> wrote: > On Sat, Aug 12, 2017 at 03:34:01PM -0600, Chris Murphy wrote: >> On Fri, Aug 11, 2017 at 11:08 PM, <siranee.ja@tpc.co.th> wrote: >> >> >> > The backup script has the btrfs sync command since Aug 3 >> >> >> From your script: >> > system btrfs sub snap -r $basepath $snappath >> > system btrfs sub sync $basepath >> >> From the man page: sync <path> [subvolid...] >> Wait until given subvolume(s) are completely removed from the >> filesystem after deletion. >> >> >> This 'subvolume sync' command, per the man page, is only about >> subvolume deletion. I suggest replacing it with a regular sync >> command. >> >> I think the problem is that the script does things so fast that the >> snapshot is not always consistent on disk before btrfs send starts. >> It's just a guess though. If I'm right, this means the rsync mismaches >> mean the destination snapshots are bad. Here's what I would do: > > I don't see how that can happen. Snapshots are atomic -- they're > either there or not there. It's not a matter even of copying the > metadata part of the subvol. It's literally just adding a pointer to > point at an existing FS tree. Do snapshots come with sync and flush to disk? Or does it just set a checkpoint and the extents that are as yet uncommitted prior to the snapshot aren't for sure on disk yet until the next commit time or sync? Nevertheless the wiki explicitly says to do sync after taking a snapshot in the context of send/receive. https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Initial_Bootstrapping And people on the list have had this "stale NFS file handle" error . I have no idea if it's still a problem with current kernels, or if it applies to 4.4 kernel. Anyway, the OP does appear to be having a real problem. rync is very clearly showing a given snapshot with incremental send/receive differ by *a lot* on source and destination machines. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-12 21:34 ` Chris Murphy 2017-08-12 22:41 ` Janos Toth F. 2017-08-12 22:49 ` Hugo Mills @ 2017-08-13 2:20 ` siranee.ja 2017-08-13 2:59 ` Chris Murphy 2 siblings, 1 reply; 31+ messages in thread From: siranee.ja @ 2017-08-13 2:20 UTC (permalink / raw) To: Chris Murphy; +Cc: Chris Murphy, Btrfs BTRFS, voravat Hi Chris, I did as you suggest and the result was bad then I decided to start over with snapshot mysql_201708070830 and manually send incremental the result rsync always said "a log diff" but the dest can start mariadb until snapshot "mysql_201708100830" it couldn't start mariadb. The following are the result. - delete all the bad/mismatching snapshots only on the destination computer. tpcorp@virtualtrust3:~$ lxc exec joytest -- bash [root@joytest ~]# btrfs sub list /var/lib/mariadb ID 298 gen 177 top level 5 path mysql_201708070830 ID 301 gen 148 top level 5 path mysql_201708080830 ID 303 gen 156 top level 5 path mysql_201708090830 ID 305 gen 162 top level 5 path mysql_201708100830 ID 309 gen 175 top level 5 path mysql_201708110830 ID 310 gen 176 top level 5 path mysql ID 311 gen 180 top level 5 path mysql_201708120830 [root@joytest ~]# cd /var/lib/mariadb [root@joytest mariadb]# btrfs sub list . ID 298 gen 177 top level 5 path mysql_201708070830 ID 301 gen 148 top level 5 path mysql_201708080830 ID 303 gen 156 top level 5 path mysql_201708090830 ID 305 gen 162 top level 5 path mysql_201708100830 ID 309 gen 175 top level 5 path mysql_201708110830 ID 310 gen 176 top level 5 path mysql ID 311 gen 180 top level 5 path mysql_201708120830 [root@joytest mariadb]# btrfs sub del mysql_201708090830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' [root@joytest mariadb]# btrfs sub del mysql_201708100830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' [root@joytest mariadb]# btrfs sub del mysql_201708110830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' [root@joytest mariadb]# btrfs sub del mysql_201708120830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' [root@joytest mariadb]# btrfs sub sync . - The most recent good snapshot pair, which rsync shows origin and destination match, is mysql_201708080830 so you can keep that one on both sides. [root@joytest mariadb]# btrfs sub list . ID 298 gen 177 top level 5 path mysql_201708070830 ID 301 gen 148 top level 5 path mysql_201708080830 ID 310 gen 176 top level 5 path mysql - manually do incremental send/receive, starting with mysql_201708090830/, to make the destination current again with the origin. tpcorp@virtualtrust3:~$ lxc exec backuplogC7 -- bash [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb ID 257 gen 537 top level 5 path mysql ID 316 gen 498 top level 5 path mysql_201708060830 ID 317 gen 503 top level 5 path mysql_201708070830 ID 318 gen 507 top level 5 path mysql_201708080830 ID 319 gen 514 top level 5 path mysql_201708090830 ID 320 gen 524 top level 5 path mysql_201708100830 ID 321 gen 529 top level 5 path mysql_201708110830 ID 322 gen 533 top level 5 path mysql_201708120830 [root@backuplogC7 ~]# more /var/log/btrfs_send.log Start Send 201708040905 btrfs send -p /var/lib/mariadb/mysql_201708030830 /var/lib/mariadb/mysql_201708040830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708040830 Stop Send 201708040905 Start Send 201708050905 btrfs send -p /var/lib/mariadb/mysql_201708040830 /var/lib/mariadb/mysql_201708050830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708050830 Stop Send 201708050905 Start Send 201708060905 btrfs send -p /var/lib/mariadb/mysql_201708050830 /var/lib/mariadb/mysql_201708060830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708060830 Stop Send 201708060905 Start Send 201708070905 btrfs send -p /var/lib/mariadb/mysql_201708060830 /var/lib/mariadb/mysql_201708070830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708070830 Stop Send 201708070905 Start Send 201708080905 btrfs send -p /var/lib/mariadb/mysql_201708070830 /var/lib/mariadb/mysql_201708080830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708080830 Stop Send 201708080905 Start Send 201708090905 btrfs send -p /var/lib/mariadb/mysql_201708080830 /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708090830 Stop Send 201708090905 Start Send 201708100905 btrfs send -p /var/lib/mariadb/mysql_201708090830 /var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708100830 Stop Send 201708100905 Start Send 201708110905 btrfs send -p /var/lib/mariadb/mysql_201708100830 /var/lib/mariadb/mysql_201708110830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708110830 Stop Send 201708110905 Start Send 201708120905 btrfs send -p /var/lib/mariadb/mysql_201708110830 /var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708120830 Stop Send 201708120905 [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708090830 At snapshot mysql_201708090830 [root@backuplogC7 ~]# history|grep rsync 416 rsync 417 yum install rsync 424 rsync -anc mysql_201708110830 > /tmp/source_mysql_201708110830.txt 433 history|grep rsync 434 rsync -anc mysql_201708100830 > /tmp/source_mysql_201708100830.txt 435 rsync -anc /var/lib/mariadb/mysql_201708100830 > /tmp/source_mysql_201708100830.txt 442 history|grep rsync 443 rsync -anc /var/lib/mariadb/mysql_201708090830 > /tmp/source_mysql_201708090830.txt 444 rsync -anc /var/lib/mariadb/mysql_201708080830 > /tmp/source_mysql_201708080830.txt 445 rsync -anc /var/lib/mariadb/mysql_201708070830 > /tmp/source_mysql_201708070830.txt 486 history|grep rsync 495 history|grep rsync 496 rsync -anc /var/lib/mariadb/mysql_201708120830 > /tmp/source_mysql_201708120830.txt 498 rsync -anc /var/lib/mariadb/mysql_201708120830 499 rsync -anc /var/lib/mariadb/mysql_201708110830 504 history|grep rsync [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 root@192.168.45.166://var/lib/mariadb/mysql_201708090830 sending incremental file list mysql_201708090830/ mysql_201708090830/aria_log.00000001 mysql_201708090830/aria_log_control mysql_201708090830/ib_logfile0 mysql_201708090830/ib_logfile1 mysql_201708090830/ibdata1 mysql_201708090830/mysql-bin.000001 mysql_201708090830/mysql-bin.index mysql_201708090830/backups_db/ mysql_201708090830/backups_db/db.opt mysql_201708090830/backups_db/ints.frm mysql_201708090830/backups_db/tpc_backup_detail.frm mysql_201708090830/backups_db/tpc_backup_header.frm mysql_201708090830/backups_db/tpc_backup_history.frm mysql_201708090830/backups_db/tpc_backup_schedule_level_master.frm mysql_201708090830/backups_db/tpc_backup_schedule_master.frm mysql_201708090830/backups_db/tpc_backup_schedule_master_extra.frm mysql_201708090830/backups_db/tpc_backup_schedule_old.frm mysql_201708090830/backups_db/tpc_backup_server.frm mysql_201708090830/backups_db/tpc_backup_step.frm mysql_201708090830/backups_db/tpc_backup_user.frm mysql_201708090830/backups_db/v_backup_header_status.frm mysql_201708090830/backups_db/v_calendar.frm mysql_201708090830/backups_db/v_dup_backup.frm mysql_201708090830/mysql/ mysql_201708090830/mysql/columns_priv.MYD mysql_201708090830/mysql/columns_priv.MYI mysql_201708090830/mysql/columns_priv.frm mysql_201708090830/mysql/db.MYD mysql_201708090830/mysql/db.MYI mysql_201708090830/mysql/db.frm mysql_201708090830/mysql/event.MYD mysql_201708090830/mysql/event.MYI mysql_201708090830/mysql/event.frm mysql_201708090830/mysql/func.MYD mysql_201708090830/mysql/func.MYI mysql_201708090830/mysql/func.frm mysql_201708090830/mysql/general_log.CSM mysql_201708090830/mysql/general_log.CSV mysql_201708090830/mysql/general_log.frm mysql_201708090830/mysql/help_category.MYD mysql_201708090830/mysql/help_category.MYI mysql_201708090830/mysql/help_category.frm mysql_201708090830/mysql/help_keyword.MYD mysql_201708090830/mysql/help_keyword.MYI mysql_201708090830/mysql/help_keyword.frm mysql_201708090830/mysql/help_relation.MYD mysql_201708090830/mysql/help_relation.MYI mysql_201708090830/mysql/help_relation.frm mysql_201708090830/mysql/help_topic.MYD mysql_201708090830/mysql/help_topic.MYI mysql_201708090830/mysql/help_topic.frm mysql_201708090830/mysql/host.MYD mysql_201708090830/mysql/host.MYI mysql_201708090830/mysql/host.frm mysql_201708090830/mysql/ndb_binlog_index.MYD mysql_201708090830/mysql/ndb_binlog_index.MYI mysql_201708090830/mysql/ndb_binlog_index.frm mysql_201708090830/mysql/plugin.MYD mysql_201708090830/mysql/plugin.MYI mysql_201708090830/mysql/plugin.frm mysql_201708090830/mysql/proc.MYD mysql_201708090830/mysql/proc.MYI mysql_201708090830/mysql/proc.frm mysql_201708090830/mysql/procs_priv.MYD mysql_201708090830/mysql/procs_priv.MYI mysql_201708090830/mysql/procs_priv.frm mysql_201708090830/mysql/proxies_priv.MYD mysql_201708090830/mysql/proxies_priv.MYI mysql_201708090830/mysql/proxies_priv.frm mysql_201708090830/mysql/servers.MYD mysql_201708090830/mysql/servers.MYI mysql_201708090830/mysql/servers.frm mysql_201708090830/mysql/slow_log.CSM mysql_201708090830/mysql/slow_log.CSV mysql_201708090830/mysql/slow_log.frm mysql_201708090830/mysql/tables_priv.MYD mysql_201708090830/mysql/tables_priv.MYI mysql_201708090830/mysql/tables_priv.frm mysql_201708090830/mysql/time_zone.MYD mysql_201708090830/mysql/time_zone.MYI mysql_201708090830/mysql/time_zone.frm mysql_201708090830/mysql/time_zone_leap_second.MYD mysql_201708090830/mysql/time_zone_leap_second.MYI mysql_201708090830/mysql/time_zone_leap_second.frm mysql_201708090830/mysql/time_zone_name.MYD mysql_201708090830/mysql/time_zone_name.MYI mysql_201708090830/mysql/time_zone_name.frm mysql_201708090830/mysql/time_zone_transition.MYD mysql_201708090830/mysql/time_zone_transition.MYI mysql_201708090830/mysql/time_zone_transition.frm mysql_201708090830/mysql/time_zone_transition_type.MYD mysql_201708090830/mysql/time_zone_transition_type.MYI mysql_201708090830/mysql/time_zone_transition_type.frm mysql_201708090830/mysql/user.MYD mysql_201708090830/mysql/user.MYI mysql_201708090830/mysql/user.frm mysql_201708090830/performance_schema/ mysql_201708090830/performance_schema/cond_instances.frm mysql_201708090830/performance_schema/db.opt mysql_201708090830/performance_schema/events_waits_current.frm mysql_201708090830/performance_schema/events_waits_history.frm mysql_201708090830/performance_schema/events_waits_history_long.frm mysql_201708090830/performance_schema/events_waits_summary_by_instance.frm mysql_201708090830/performance_schema/events_waits_summary_by_thread_by_event_name.frm mysql_201708090830/performance_schema/events_waits_summary_global_by_event_name.frm mysql_201708090830/performance_schema/file_instances.frm mysql_201708090830/performance_schema/file_summary_by_event_name.frm mysql_201708090830/performance_schema/file_summary_by_instance.frm mysql_201708090830/performance_schema/mutex_instances.frm mysql_201708090830/performance_schema/performance_timers.frm mysql_201708090830/performance_schema/rwlock_instances.frm mysql_201708090830/performance_schema/setup_consumers.frm mysql_201708090830/performance_schema/setup_instruments.frm mysql_201708090830/performance_schema/setup_timers.frm mysql_201708090830/performance_schema/threads.frm mysql_201708090830/test/ mysql_201708090830/test/db.opt sent 4157 bytes received 371 bytes 1811.20 bytes/sec total size is 718361496 speedup is 158648.74 (DRY RUN) [root@backuplogC7 ~]# [root@joytest mariadb]# btrfs sub list . ID 298 gen 182 top level 5 path mysql_201708070830 ID 301 gen 148 top level 5 path mysql_201708080830 ID 310 gen 176 top level 5 path mysql ID 312 gen 185 top level 5 path mysql_201708090830 - now resume using the modified script, which will do snapshot -> sync -> send. [root@backuplogC7 ~]# more /etc/crontab SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # For details see man 4 crontabs # Example of job definition: # .---------------- minute (0 - 59) # | .------------- hour (0 - 23) # | | .---------- day of month (1 - 31) # | | | .------- month (1 - 12) OR jan,feb,mar,apr ... # | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat # | | | | | # * * * * * user-name command to be executed 30 08 * * * root /root/script/backup/backupsnap.sh root password /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log 05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166 /var/lib/mariadb >> /var/log/btrfs_send.log 0 19 * * * root /root/script/backup/doBackup_file_backuplog.sh 30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >> /var/log/btrfs_del.log # temp ##00 09 * * * root /root/script/backupbtrfs_all.sh /var/lib/mariadb /backupbtrfs mysql_201707210830 [root@backuplogC7 ~]# vi /root/script/backup/backupsnap.sh [root@backuplogC7 ~]# ls -l /root/script/backup/backupsnap.sh -rwxr--r-- 1 root root 314 Aug 13 08:12 /root/script/backup/backupsnap.sh [root@backuplogC7 ~]# cat /root/script/backup/backupsnap.sh #Backup # user=$1 password=$2 basepath=$3 datet=$(date +%Y%m%d%H%M) snappath=${basepath}_${datet} echo "Locking databases ${datet}" mysql -u$user -p$password << EOF FLUSH TABLES WITH READ LOCK; system btrfs sub snap -r $basepath $snappath system sync UNLOCK TABLES; quit EOF echo "Databases unlocked ${datet}" [root@backuplogC7 ~]# [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830 root@192.168.45.166://var/lib/mariadb/mysql_201708080830 sending incremental file list mysql_201708080830/ mysql_201708080830/aria_log.00000001 mysql_201708080830/aria_log_control mysql_201708080830/ib_logfile0 mysql_201708080830/ib_logfile1 mysql_201708080830/ibdata1 mysql_201708080830/mysql-bin.000001 mysql_201708080830/mysql-bin.index mysql_201708080830/backups_db/ mysql_201708080830/backups_db/db.opt mysql_201708080830/backups_db/ints.frm mysql_201708080830/backups_db/tpc_backup_detail.frm mysql_201708080830/backups_db/tpc_backup_header.frm mysql_201708080830/backups_db/tpc_backup_history.frm mysql_201708080830/backups_db/tpc_backup_schedule_level_master.frm mysql_201708080830/backups_db/tpc_backup_schedule_master.frm mysql_201708080830/backups_db/tpc_backup_schedule_master_extra.frm mysql_201708080830/backups_db/tpc_backup_schedule_old.frm mysql_201708080830/backups_db/tpc_backup_server.frm mysql_201708080830/backups_db/tpc_backup_step.frm mysql_201708080830/backups_db/tpc_backup_user.frm mysql_201708080830/backups_db/v_backup_header_status.frm mysql_201708080830/backups_db/v_calendar.frm mysql_201708080830/backups_db/v_dup_backup.frm mysql_201708080830/mysql/ mysql_201708080830/mysql/columns_priv.MYD mysql_201708080830/mysql/columns_priv.MYI mysql_201708080830/mysql/columns_priv.frm mysql_201708080830/mysql/db.MYD mysql_201708080830/mysql/db.MYI mysql_201708080830/mysql/db.frm mysql_201708080830/mysql/event.MYD mysql_201708080830/mysql/event.MYI mysql_201708080830/mysql/event.frm mysql_201708080830/mysql/func.MYD mysql_201708080830/mysql/func.MYI mysql_201708080830/mysql/func.frm mysql_201708080830/mysql/general_log.CSM mysql_201708080830/mysql/general_log.CSV mysql_201708080830/mysql/general_log.frm mysql_201708080830/mysql/help_category.MYD mysql_201708080830/mysql/help_category.MYI mysql_201708080830/mysql/help_category.frm mysql_201708080830/mysql/help_keyword.MYD mysql_201708080830/mysql/help_keyword.MYI mysql_201708080830/mysql/help_keyword.frm mysql_201708080830/mysql/help_relation.MYD mysql_201708080830/mysql/help_relation.MYI mysql_201708080830/mysql/help_relation.frm mysql_201708080830/mysql/help_topic.MYD mysql_201708080830/mysql/help_topic.MYI mysql_201708080830/mysql/help_topic.frm mysql_201708080830/mysql/host.MYD mysql_201708080830/mysql/host.MYI mysql_201708080830/mysql/host.frm mysql_201708080830/mysql/ndb_binlog_index.MYD mysql_201708080830/mysql/ndb_binlog_index.MYI mysql_201708080830/mysql/ndb_binlog_index.frm mysql_201708080830/mysql/plugin.MYD mysql_201708080830/mysql/plugin.MYI mysql_201708080830/mysql/plugin.frm mysql_201708080830/mysql/proc.MYD mysql_201708080830/mysql/proc.MYI mysql_201708080830/mysql/proc.frm mysql_201708080830/mysql/procs_priv.MYD mysql_201708080830/mysql/procs_priv.MYI mysql_201708080830/mysql/procs_priv.frm mysql_201708080830/mysql/proxies_priv.MYD mysql_201708080830/mysql/proxies_priv.MYI mysql_201708080830/mysql/proxies_priv.frm mysql_201708080830/mysql/servers.MYD mysql_201708080830/mysql/servers.MYI mysql_201708080830/mysql/servers.frm mysql_201708080830/mysql/slow_log.CSM mysql_201708080830/mysql/slow_log.CSV mysql_201708080830/mysql/slow_log.frm mysql_201708080830/mysql/tables_priv.MYD mysql_201708080830/mysql/tables_priv.MYI mysql_201708080830/mysql/tables_priv.frm mysql_201708080830/mysql/time_zone.MYD mysql_201708080830/mysql/time_zone.MYI mysql_201708080830/mysql/time_zone.frm mysql_201708080830/mysql/time_zone_leap_second.MYD mysql_201708080830/mysql/time_zone_leap_second.MYI mysql_201708080830/mysql/time_zone_leap_second.frm mysql_201708080830/mysql/time_zone_name.MYD mysql_201708080830/mysql/time_zone_name.MYI mysql_201708080830/mysql/time_zone_name.frm mysql_201708080830/mysql/time_zone_transition.MYD mysql_201708080830/mysql/time_zone_transition.MYI mysql_201708080830/mysql/time_zone_transition.frm mysql_201708080830/mysql/time_zone_transition_type.MYD mysql_201708080830/mysql/time_zone_transition_type.MYI mysql_201708080830/mysql/time_zone_transition_type.frm mysql_201708080830/mysql/user.MYD mysql_201708080830/mysql/user.MYI mysql_201708080830/mysql/user.frm mysql_201708080830/performance_schema/ mysql_201708080830/performance_schema/cond_instances.frm mysql_201708080830/performance_schema/db.opt mysql_201708080830/performance_schema/events_waits_current.frm mysql_201708080830/performance_schema/events_waits_history.frm mysql_201708080830/performance_schema/events_waits_history_long.frm mysql_201708080830/performance_schema/events_waits_summary_by_instance.frm mysql_201708080830/performance_schema/events_waits_summary_by_thread_by_event_name.frm mysql_201708080830/performance_schema/events_waits_summary_global_by_event_name.frm mysql_201708080830/performance_schema/file_instances.frm mysql_201708080830/performance_schema/file_summary_by_event_name.frm mysql_201708080830/performance_schema/file_summary_by_instance.frm mysql_201708080830/performance_schema/mutex_instances.frm mysql_201708080830/performance_schema/performance_timers.frm mysql_201708080830/performance_schema/rwlock_instances.frm mysql_201708080830/performance_schema/setup_consumers.frm mysql_201708080830/performance_schema/setup_instruments.frm mysql_201708080830/performance_schema/setup_timers.frm mysql_201708080830/performance_schema/threads.frm mysql_201708080830/test/ mysql_201708080830/test/db.opt sent 4153 bytes received 371 bytes 1809.60 bytes/sec total size is 718361496 speedup is 158789.01 (DRY RUN) [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830 root@192.168.45.166://var/lib/mariadb/mysql_201708070830 sending incremental file list mysql_201708070830/ mysql_201708070830/aria_log.00000001 mysql_201708070830/aria_log_control mysql_201708070830/ib_logfile0 mysql_201708070830/ib_logfile1 mysql_201708070830/ibdata1 mysql_201708070830/mysql-bin.000001 mysql_201708070830/mysql-bin.index mysql_201708070830/backups_db/ mysql_201708070830/backups_db/db.opt mysql_201708070830/backups_db/ints.frm mysql_201708070830/backups_db/tpc_backup_detail.frm mysql_201708070830/backups_db/tpc_backup_header.frm mysql_201708070830/backups_db/tpc_backup_history.frm mysql_201708070830/backups_db/tpc_backup_schedule_level_master.frm mysql_201708070830/backups_db/tpc_backup_schedule_master.frm mysql_201708070830/backups_db/tpc_backup_schedule_master_extra.frm mysql_201708070830/backups_db/tpc_backup_schedule_old.frm mysql_201708070830/backups_db/tpc_backup_server.frm mysql_201708070830/backups_db/tpc_backup_step.frm mysql_201708070830/backups_db/tpc_backup_user.frm mysql_201708070830/backups_db/v_backup_header_status.frm mysql_201708070830/backups_db/v_calendar.frm mysql_201708070830/backups_db/v_dup_backup.frm mysql_201708070830/mysql/ mysql_201708070830/mysql/columns_priv.MYD mysql_201708070830/mysql/columns_priv.MYI mysql_201708070830/mysql/columns_priv.frm mysql_201708070830/mysql/db.MYD mysql_201708070830/mysql/db.MYI mysql_201708070830/mysql/db.frm mysql_201708070830/mysql/event.MYD mysql_201708070830/mysql/event.MYI mysql_201708070830/mysql/event.frm mysql_201708070830/mysql/func.MYD mysql_201708070830/mysql/func.MYI mysql_201708070830/mysql/func.frm mysql_201708070830/mysql/general_log.CSM mysql_201708070830/mysql/general_log.CSV mysql_201708070830/mysql/general_log.frm mysql_201708070830/mysql/help_category.MYD mysql_201708070830/mysql/help_category.MYI mysql_201708070830/mysql/help_category.frm mysql_201708070830/mysql/help_keyword.MYD mysql_201708070830/mysql/help_keyword.MYI mysql_201708070830/mysql/help_keyword.frm mysql_201708070830/mysql/help_relation.MYD mysql_201708070830/mysql/help_relation.MYI mysql_201708070830/mysql/help_relation.frm mysql_201708070830/mysql/help_topic.MYD mysql_201708070830/mysql/help_topic.MYI mysql_201708070830/mysql/help_topic.frm mysql_201708070830/mysql/host.MYD mysql_201708070830/mysql/host.MYI mysql_201708070830/mysql/host.frm mysql_201708070830/mysql/ndb_binlog_index.MYD mysql_201708070830/mysql/ndb_binlog_index.MYI mysql_201708070830/mysql/ndb_binlog_index.frm mysql_201708070830/mysql/plugin.MYD mysql_201708070830/mysql/plugin.MYI mysql_201708070830/mysql/plugin.frm mysql_201708070830/mysql/proc.MYD mysql_201708070830/mysql/proc.MYI mysql_201708070830/mysql/proc.frm mysql_201708070830/mysql/procs_priv.MYD mysql_201708070830/mysql/procs_priv.MYI mysql_201708070830/mysql/procs_priv.frm mysql_201708070830/mysql/proxies_priv.MYD mysql_201708070830/mysql/proxies_priv.MYI mysql_201708070830/mysql/proxies_priv.frm mysql_201708070830/mysql/servers.MYD mysql_201708070830/mysql/servers.MYI mysql_201708070830/mysql/servers.frm mysql_201708070830/mysql/slow_log.CSM mysql_201708070830/mysql/slow_log.CSV mysql_201708070830/mysql/slow_log.frm mysql_201708070830/mysql/tables_priv.MYD mysql_201708070830/mysql/tables_priv.MYI mysql_201708070830/mysql/tables_priv.frm mysql_201708070830/mysql/time_zone.MYD mysql_201708070830/mysql/time_zone.MYI mysql_201708070830/mysql/time_zone.frm mysql_201708070830/mysql/time_zone_leap_second.MYD mysql_201708070830/mysql/time_zone_leap_second.MYI mysql_201708070830/mysql/time_zone_leap_second.frm mysql_201708070830/mysql/time_zone_name.MYD mysql_201708070830/mysql/time_zone_name.MYI mysql_201708070830/mysql/time_zone_name.frm mysql_201708070830/mysql/time_zone_transition.MYD mysql_201708070830/mysql/time_zone_transition.MYI mysql_201708070830/mysql/time_zone_transition.frm mysql_201708070830/mysql/time_zone_transition_type.MYD mysql_201708070830/mysql/time_zone_transition_type.MYI mysql_201708070830/mysql/time_zone_transition_type.frm mysql_201708070830/mysql/user.MYD mysql_201708070830/mysql/user.MYI mysql_201708070830/mysql/user.frm mysql_201708070830/performance_schema/ mysql_201708070830/performance_schema/cond_instances.frm mysql_201708070830/performance_schema/db.opt mysql_201708070830/performance_schema/events_waits_current.frm mysql_201708070830/performance_schema/events_waits_history.frm mysql_201708070830/performance_schema/events_waits_history_long.frm mysql_201708070830/performance_schema/events_waits_summary_by_instance.frm mysql_201708070830/performance_schema/events_waits_summary_by_thread_by_event_name.frm mysql_201708070830/performance_schema/events_waits_summary_global_by_event_name.frm mysql_201708070830/performance_schema/file_instances.frm mysql_201708070830/performance_schema/file_summary_by_event_name.frm mysql_201708070830/performance_schema/file_summary_by_instance.frm mysql_201708070830/performance_schema/mutex_instances.frm mysql_201708070830/performance_schema/performance_timers.frm mysql_201708070830/performance_schema/rwlock_instances.frm mysql_201708070830/performance_schema/setup_consumers.frm mysql_201708070830/performance_schema/setup_instruments.frm mysql_201708070830/performance_schema/setup_timers.frm mysql_201708070830/performance_schema/threads.frm mysql_201708070830/test/ mysql_201708070830/test/db.opt sent 4157 bytes received 371 bytes 1811.20 bytes/sec total size is 718361496 speedup is 158648.74 (DRY RUN) [root@backuplogC7 ~]# - I decide to start with new send "mysql_201708070830" [root@joytest mariadb]# btrfs sub del mysql* Delete subvolume (no-commit): '/var/lib/mariadb/mysql' Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708070830' Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708080830' Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' [root@joytest mariadb]# btrfs sub sync . ^C [root@joytest mariadb]# systemctl status mariadb ◠mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2017-08-11 14:00:20 ICT; 1 day 18h ago Process: 15671 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE) Process: 15670 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS) Process: 15642 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS) Main PID: 15670 (code=exited, status=0/SUCCESS) Aug 11 14:00:19 joytest systemd[1]: Starting MariaDB database server... Aug 11 14:00:19 joytest mysqld_safe[15670]: 170811 14:00:19 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. Aug 11 14:00:19 joytest mysqld_safe[15670]: 170811 14:00:19 mysqld_safe Starting mysqld daemon with databases from /var/lib/mariadb/mysql Aug 11 14:00:20 joytest systemd[1]: mariadb.service: control process exited, code=exited status=1 Aug 11 14:00:20 joytest systemd[1]: Failed to start MariaDB database server. Aug 11 14:00:20 joytest systemd[1]: Unit mariadb.service entered failed state. Aug 11 14:00:20 joytest systemd[1]: mariadb.service failed. [root@joytest mariadb]# btrfs sub sync . [root@joytest mariadb]# btrfs sub list . [root@backuplogC7 ~]# btrfs send /var/lib/mariadb/mysql_201708070830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708070830 At subvol mysql_201708070830 [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830 root@192.168.45.166://var/lib/mariadb/mysql_201708070830 sending incremental file list mysql_201708070830/ mysql_201708070830/aria_log.00000001 mysql_201708070830/aria_log_control mysql_201708070830/ib_logfile0 mysql_201708070830/ib_logfile1 mysql_201708070830/ibdata1 mysql_201708070830/mysql-bin.000001 mysql_201708070830/mysql-bin.index mysql_201708070830/backups_db/ mysql_201708070830/backups_db/db.opt mysql_201708070830/backups_db/ints.frm mysql_201708070830/backups_db/tpc_backup_detail.frm mysql_201708070830/backups_db/tpc_backup_header.frm mysql_201708070830/backups_db/tpc_backup_history.frm mysql_201708070830/backups_db/tpc_backup_schedule_level_master.frm mysql_201708070830/backups_db/tpc_backup_schedule_master.frm mysql_201708070830/backups_db/tpc_backup_schedule_master_extra.frm mysql_201708070830/backups_db/tpc_backup_schedule_old.frm mysql_201708070830/backups_db/tpc_backup_server.frm mysql_201708070830/backups_db/tpc_backup_step.frm mysql_201708070830/backups_db/tpc_backup_user.frm mysql_201708070830/backups_db/v_backup_header_status.frm mysql_201708070830/backups_db/v_calendar.frm mysql_201708070830/backups_db/v_dup_backup.frm mysql_201708070830/mysql/ mysql_201708070830/mysql/columns_priv.MYD mysql_201708070830/mysql/columns_priv.MYI mysql_201708070830/mysql/columns_priv.frm mysql_201708070830/mysql/db.MYD mysql_201708070830/mysql/db.MYI mysql_201708070830/mysql/db.frm mysql_201708070830/mysql/event.MYD mysql_201708070830/mysql/event.MYI mysql_201708070830/mysql/event.frm mysql_201708070830/mysql/func.MYD mysql_201708070830/mysql/func.MYI mysql_201708070830/mysql/func.frm mysql_201708070830/mysql/general_log.CSM mysql_201708070830/mysql/general_log.CSV mysql_201708070830/mysql/general_log.frm mysql_201708070830/mysql/help_category.MYD mysql_201708070830/mysql/help_category.MYI mysql_201708070830/mysql/help_category.frm mysql_201708070830/mysql/help_keyword.MYD mysql_201708070830/mysql/help_keyword.MYI mysql_201708070830/mysql/help_keyword.frm mysql_201708070830/mysql/help_relation.MYD mysql_201708070830/mysql/help_relation.MYI mysql_201708070830/mysql/help_relation.frm mysql_201708070830/mysql/help_topic.MYD mysql_201708070830/mysql/help_topic.MYI mysql_201708070830/mysql/help_topic.frm mysql_201708070830/mysql/host.MYD mysql_201708070830/mysql/host.MYI mysql_201708070830/mysql/host.frm mysql_201708070830/mysql/ndb_binlog_index.MYD mysql_201708070830/mysql/ndb_binlog_index.MYI mysql_201708070830/mysql/ndb_binlog_index.frm mysql_201708070830/mysql/plugin.MYD mysql_201708070830/mysql/plugin.MYI mysql_201708070830/mysql/plugin.frm mysql_201708070830/mysql/proc.MYD mysql_201708070830/mysql/proc.MYI mysql_201708070830/mysql/proc.frm mysql_201708070830/mysql/procs_priv.MYD mysql_201708070830/mysql/procs_priv.MYI mysql_201708070830/mysql/procs_priv.frm mysql_201708070830/mysql/proxies_priv.MYD mysql_201708070830/mysql/proxies_priv.MYI mysql_201708070830/mysql/proxies_priv.frm mysql_201708070830/mysql/servers.MYD mysql_201708070830/mysql/servers.MYI mysql_201708070830/mysql/servers.frm mysql_201708070830/mysql/slow_log.CSM mysql_201708070830/mysql/slow_log.CSV mysql_201708070830/mysql/slow_log.frm mysql_201708070830/mysql/tables_priv.MYD mysql_201708070830/mysql/tables_priv.MYI mysql_201708070830/mysql/tables_priv.frm mysql_201708070830/mysql/time_zone.MYD mysql_201708070830/mysql/time_zone.MYI mysql_201708070830/mysql/time_zone.frm mysql_201708070830/mysql/time_zone_leap_second.MYD mysql_201708070830/mysql/time_zone_leap_second.MYI mysql_201708070830/mysql/time_zone_leap_second.frm mysql_201708070830/mysql/time_zone_name.MYD mysql_201708070830/mysql/time_zone_name.MYI mysql_201708070830/mysql/time_zone_name.frm mysql_201708070830/mysql/time_zone_transition.MYD mysql_201708070830/mysql/time_zone_transition.MYI mysql_201708070830/mysql/time_zone_transition.frm mysql_201708070830/mysql/time_zone_transition_type.MYD mysql_201708070830/mysql/time_zone_transition_type.MYI mysql_201708070830/mysql/time_zone_transition_type.frm mysql_201708070830/mysql/user.MYD mysql_201708070830/mysql/user.MYI mysql_201708070830/mysql/user.frm mysql_201708070830/performance_schema/ mysql_201708070830/performance_schema/cond_instances.frm mysql_201708070830/performance_schema/db.opt mysql_201708070830/performance_schema/events_waits_current.frm mysql_201708070830/performance_schema/events_waits_history.frm mysql_201708070830/performance_schema/events_waits_history_long.frm mysql_201708070830/performance_schema/events_waits_summary_by_instance.frm mysql_201708070830/performance_schema/events_waits_summary_by_thread_by_event_name.frm mysql_201708070830/performance_schema/events_waits_summary_global_by_event_name.frm mysql_201708070830/performance_schema/file_instances.frm mysql_201708070830/performance_schema/file_summary_by_event_name.frm mysql_201708070830/performance_schema/file_summary_by_instance.frm mysql_201708070830/performance_schema/mutex_instances.frm mysql_201708070830/performance_schema/performance_timers.frm mysql_201708070830/performance_schema/rwlock_instances.frm mysql_201708070830/performance_schema/setup_consumers.frm mysql_201708070830/performance_schema/setup_instruments.frm mysql_201708070830/performance_schema/setup_timers.frm mysql_201708070830/performance_schema/threads.frm mysql_201708070830/test/ mysql_201708070830/test/db.opt sent 4157 bytes received 371 bytes 3018.67 bytes/sec total size is 718361496 speedup is 158648.74 (DRY RUN) - "A lot diff" but dest mariadb can start and the data is correct. [root@joytest mariadb]# btrfs sub list . ID 313 gen 190 top level 5 path mysql_201708070830 [root@joytest mariadb]# btrfs sub snap mysql_201708070830 mysql Create a snapshot of 'mysql_201708070830' in './mysql' [root@joytest mariadb]# systemctl start mariadb [root@joytest mariadb]# systemctl status mariadb ◠mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2017-08-13 08:30:17 ICT; 4min 36s ago Process: 17079 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=0/SUCCESS) Process: 17050 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS) Main PID: 17078 (mysqld_safe) CGroup: /system.slice/mariadb.service ├─17078 /bin/sh /usr/bin/mysqld_safe --basedir=/usr └─17630 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mariadb/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/m... Aug 13 08:30:12 joytest systemd[1]: Starting MariaDB database server... Aug 13 08:30:13 joytest mysqld_safe[17078]: 170813 08:30:13 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. Aug 13 08:30:13 joytest mysqld_safe[17078]: 170813 08:30:13 mysqld_safe Starting mysqld daemon with databases from /var/lib/mariadb/mysql Aug 13 08:30:17 joytest systemd[1]: Started MariaDB database server. [root@joytest mariadb]# mysql -uroot -ppassword Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 3 Server version: 5.5.52-MariaDB MariaDB Server Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> use backups_db; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A ^[[ADatabase changed MariaDB [backups_db]> select max(backup_start_date) from tpc_backup_header order by backup_start_date desc; +------------------------+ | max(backup_start_date) | +------------------------+ | 2017-08-07 06:30:01 | +------------------------+ 1 row in set (0.09 sec) MariaDB [backups_db]> Ctrl-C -- exit! Aborted - I continuously send the incremental and it work fine on dest (even rsync said "a lot diff") until the snapshot mysql_201708100830 mariadb can not start [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708090830 /var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708100830 At snapshot mysql_201708100830 [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708100830 root@192.168.45.166:/var/lib/mariadb/mysql_201708100830 sending incremental file list mysql_201708100830/ mysql_201708100830/aria_log.00000001 mysql_201708100830/aria_log_control mysql_201708100830/ib_logfile0 mysql_201708100830/ib_logfile1 mysql_201708100830/ibdata1 mysql_201708100830/mysql-bin.000001 mysql_201708100830/mysql-bin.index mysql_201708100830/backups_db/ mysql_201708100830/backups_db/db.opt mysql_201708100830/backups_db/ints.frm mysql_201708100830/backups_db/tpc_backup_detail.frm mysql_201708100830/backups_db/tpc_backup_header.frm mysql_201708100830/backups_db/tpc_backup_history.frm mysql_201708100830/backups_db/tpc_backup_schedule_level_master.frm mysql_201708100830/backups_db/tpc_backup_schedule_master.frm mysql_201708100830/backups_db/tpc_backup_schedule_master_extra.frm mysql_201708100830/backups_db/tpc_backup_schedule_old.frm mysql_201708100830/backups_db/tpc_backup_server.frm mysql_201708100830/backups_db/tpc_backup_step.frm mysql_201708100830/backups_db/tpc_backup_user.frm mysql_201708100830/backups_db/v_backup_header_status.frm mysql_201708100830/backups_db/v_calendar.frm mysql_201708100830/backups_db/v_dup_backup.frm mysql_201708100830/mysql/ mysql_201708100830/mysql/columns_priv.MYD mysql_201708100830/mysql/columns_priv.MYI mysql_201708100830/mysql/columns_priv.frm mysql_201708100830/mysql/db.MYD mysql_201708100830/mysql/db.MYI mysql_201708100830/mysql/db.frm mysql_201708100830/mysql/event.MYD mysql_201708100830/mysql/event.MYI mysql_201708100830/mysql/event.frm mysql_201708100830/mysql/func.MYD mysql_201708100830/mysql/func.MYI mysql_201708100830/mysql/func.frm mysql_201708100830/mysql/general_log.CSM mysql_201708100830/mysql/general_log.CSV mysql_201708100830/mysql/general_log.frm mysql_201708100830/mysql/help_category.MYD mysql_201708100830/mysql/help_category.MYI mysql_201708100830/mysql/help_category.frm mysql_201708100830/mysql/help_keyword.MYD mysql_201708100830/mysql/help_keyword.MYI mysql_201708100830/mysql/help_keyword.frm mysql_201708100830/mysql/help_relation.MYD mysql_201708100830/mysql/help_relation.MYI mysql_201708100830/mysql/help_relation.frm mysql_201708100830/mysql/help_topic.MYD mysql_201708100830/mysql/help_topic.MYI mysql_201708100830/mysql/help_topic.frm mysql_201708100830/mysql/host.MYD mysql_201708100830/mysql/host.MYI mysql_201708100830/mysql/host.frm mysql_201708100830/mysql/ndb_binlog_index.MYD mysql_201708100830/mysql/ndb_binlog_index.MYI mysql_201708100830/mysql/ndb_binlog_index.frm mysql_201708100830/mysql/plugin.MYD mysql_201708100830/mysql/plugin.MYI mysql_201708100830/mysql/plugin.frm mysql_201708100830/mysql/proc.MYD mysql_201708100830/mysql/proc.MYI mysql_201708100830/mysql/proc.frm mysql_201708100830/mysql/procs_priv.MYD mysql_201708100830/mysql/procs_priv.MYI mysql_201708100830/mysql/procs_priv.frm mysql_201708100830/mysql/proxies_priv.MYD mysql_201708100830/mysql/proxies_priv.MYI mysql_201708100830/mysql/proxies_priv.frm mysql_201708100830/mysql/servers.MYD mysql_201708100830/mysql/servers.MYI mysql_201708100830/mysql/servers.frm mysql_201708100830/mysql/slow_log.CSM mysql_201708100830/mysql/slow_log.CSV mysql_201708100830/mysql/slow_log.frm mysql_201708100830/mysql/tables_priv.MYD mysql_201708100830/mysql/tables_priv.MYI mysql_201708100830/mysql/tables_priv.frm mysql_201708100830/mysql/time_zone.MYD mysql_201708100830/mysql/time_zone.MYI mysql_201708100830/mysql/time_zone.frm mysql_201708100830/mysql/time_zone_leap_second.MYD mysql_201708100830/mysql/time_zone_leap_second.MYI mysql_201708100830/mysql/time_zone_leap_second.frm mysql_201708100830/mysql/time_zone_name.MYD mysql_201708100830/mysql/time_zone_name.MYI mysql_201708100830/mysql/time_zone_name.frm mysql_201708100830/mysql/time_zone_transition.MYD mysql_201708100830/mysql/time_zone_transition.MYI mysql_201708100830/mysql/time_zone_transition.frm mysql_201708100830/mysql/time_zone_transition_type.MYD mysql_201708100830/mysql/time_zone_transition_type.MYI mysql_201708100830/mysql/time_zone_transition_type.frm mysql_201708100830/mysql/user.MYD mysql_201708100830/mysql/user.MYI mysql_201708100830/mysql/user.frm mysql_201708100830/performance_schema/ mysql_201708100830/performance_schema/cond_instances.frm mysql_201708100830/performance_schema/db.opt mysql_201708100830/performance_schema/events_waits_current.frm mysql_201708100830/performance_schema/events_waits_history.frm mysql_201708100830/performance_schema/events_waits_history_long.frm mysql_201708100830/performance_schema/events_waits_summary_by_instance.frm mysql_201708100830/performance_schema/events_waits_summary_by_thread_by_event_name.frm mysql_201708100830/performance_schema/events_waits_summary_global_by_event_name.frm mysql_201708100830/performance_schema/file_instances.frm mysql_201708100830/performance_schema/file_summary_by_event_name.frm mysql_201708100830/performance_schema/file_summary_by_instance.frm mysql_201708100830/performance_schema/mutex_instances.frm mysql_201708100830/performance_schema/performance_timers.frm mysql_201708100830/performance_schema/rwlock_instances.frm mysql_201708100830/performance_schema/setup_consumers.frm mysql_201708100830/performance_schema/setup_instruments.frm mysql_201708100830/performance_schema/setup_timers.frm mysql_201708100830/performance_schema/threads.frm mysql_201708100830/test/ mysql_201708100830/test/db.opt sent 4157 bytes received 371 bytes 1811.20 bytes/sec total size is 718361496 speedup is 158648.74 (DRY RUN) [root@backuplogC7 ~]# [root@joytest mariadb]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 57: eth0@if58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 00:16:3e:e3:b3:6f brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.45.166/21 brd 192.168.47.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fee3:b36f/64 scope link valid_lft forever preferred_lft forever [root@joytest mariadb]# systemctl mariadb stop Unknown operation 'mariadb'. [root@joytest mariadb]# systemctl stop mariadb [root@joytest mariadb]# btrfs sub list . ID 313 gen 201 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 317 gen 204 top level 5 path mysql ID 318 gen 204 top level 5 path mysql_201708090830 [root@joytest mariadb]# btrfs sub del mysql Delete subvolume (no-commit): '/var/lib/mariadb/mysql' [root@joytest mariadb]# btrfs sub sync . Subvolume id 317 is gone [root@joytest mariadb]# btrfs sub snap mysql_201708090830 mysql Create a snapshot of 'mysql_201708090830' in './mysql' [root@joytest mariadb]# systemctl start mariadb [root@joytest mariadb]# mysql -uroot -ppassword Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 2 Server version: 5.5.52-MariaDB MariaDB Server Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> use backups_db; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [backups_db]> select max(backup_start_date) from tpc_backup_header order by backup_start_date desc; +------------------------+ | max(backup_start_date) | +------------------------+ | 2017-08-09 06:30:01 | +------------------------+ 1 row in set (0.09 sec) MariaDB [backups_db]> exit Bye [root@joytest mariadb]# systemctl stop mariadb [root@joytest mariadb]# btrfs sub list . ID 313 gen 207 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 318 gen 205 top level 5 path mysql_201708090830 ID 319 gen 206 top level 5 path mysql ID 320 gen 208 top level 5 path mysql_201708100830 [root@joytest mariadb]# btrfs sub del mysql Delete subvolume (no-commit): '/var/lib/mariadb/mysql' [root@joytest mariadb]# btrfs sub sync . [root@joytest mariadb]# btrfs sub snap mysql_201708100830 mysql Create a snapshot of 'mysql_201708100830' in './mysql' [root@joytest mariadb]# systemctl start mariadb Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details. [root@joytest mariadb]# - I still send the incremetal [root@joytest mariadb]# btrfs sub list . ID 313 gen 220 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 318 gen 205 top level 5 path mysql_201708090830 ID 320 gen 211 top level 5 path mysql_201708100830 ID 321 gen 212 top level 5 path mysql ID 322 gen 219 top level 5 path mysql_201708110830 ID 323 gen 219 top level 5 path mysql_201708120830 ID 324 gen 223 top level 5 path mysql_201708130830 [root@joytest mariadb]# btrfs sub del mysql Delete subvolume (no-commit): '/var/lib/mariadb/mysql' [root@joytest mariadb]# btrfs sub sync . Subvolume id 321 is gone [root@joytest mariadb]# btrfs sub snap mysql_201708130830 mysql Create a snapshot of 'mysql_201708130830' in './mysql' [root@joytest mariadb]# systemctl start mariadb Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details. [root@joytest mariadb]# - The latest log from the source send tested at 201708130818 for comfirm that the script run with correct syntax. Start Send 201708130818 btrfs send -p /var/lib/mariadb/mysql_201708110830 /var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb rsync -avnc /var/lib/mariadb/mysql_201708120830 root@192.168.45.166://var/lib/mariadb/mysql_201708120830 Stop Send 201708130818 - the real send log after change include rsync in send script Start Send 201708130905 btrfs send -p /var/lib/mariadb/mysql_201708120830 /var/lib/mariadb/mysql_201708130830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At snapshot mysql_201708130830 rsync -avnc /var/lib/mariadb/mysql_201708130830 root@192.168.45.166://var/lib/mariadb/mysql_201708130830 sending incremental file list mysql_201708130830/ mysql_201708130830/aria_log.00000001 mysql_201708130830/aria_log_control mysql_201708130830/ib_logfile0 mysql_201708130830/ib_logfile1 mysql_201708130830/ibdata1 mysql_201708130830/mysql-bin.000001 mysql_201708130830/mysql-bin.index mysql_201708130830/backups_db/ mysql_201708130830/backups_db/db.opt mysql_201708130830/backups_db/ints.frm mysql_201708130830/backups_db/tpc_backup_detail.frm mysql_201708130830/backups_db/tpc_backup_header.frm mysql_201708130830/backups_db/tpc_backup_history.frm mysql_201708130830/backups_db/tpc_backup_schedule_level_master.frm mysql_201708130830/backups_db/tpc_backup_schedule_master.frm mysql_201708130830/backups_db/tpc_backup_schedule_master_extra.frm mysql_201708130830/backups_db/tpc_backup_schedule_old.frm mysql_201708130830/backups_db/tpc_backup_server.frm mysql_201708130830/backups_db/tpc_backup_step.frm mysql_201708130830/backups_db/tpc_backup_user.frm mysql_201708130830/backups_db/v_backup_header_status.frm mysql_201708130830/backups_db/v_calendar.frm mysql_201708130830/backups_db/v_dup_backup.frm mysql_201708130830/mysql/ mysql_201708130830/mysql/columns_priv.MYD mysql_201708130830/mysql/columns_priv.MYI mysql_201708130830/mysql/columns_priv.frm mysql_201708130830/mysql/db.MYD mysql_201708130830/mysql/db.MYI mysql_201708130830/mysql/db.frm mysql_201708130830/mysql/event.MYD mysql_201708130830/mysql/event.MYI mysql_201708130830/mysql/event.frm mysql_201708130830/mysql/func.MYD mysql_201708130830/mysql/func.MYI mysql_201708130830/mysql/func.frm mysql_201708130830/mysql/general_log.CSM mysql_201708130830/mysql/general_log.CSV mysql_201708130830/mysql/general_log.frm mysql_201708130830/mysql/help_category.MYD mysql_201708130830/mysql/help_category.MYI mysql_201708130830/mysql/help_category.frm mysql_201708130830/mysql/help_keyword.MYD mysql_201708130830/mysql/help_keyword.MYI mysql_201708130830/mysql/help_keyword.frm mysql_201708130830/mysql/help_relation.MYD mysql_201708130830/mysql/help_relation.MYI mysql_201708130830/mysql/help_relation.frm mysql_201708130830/mysql/help_topic.MYD mysql_201708130830/mysql/help_topic.MYI mysql_201708130830/mysql/help_topic.frm mysql_201708130830/mysql/host.MYD mysql_201708130830/mysql/host.MYI mysql_201708130830/mysql/host.frm mysql_201708130830/mysql/ndb_binlog_index.MYD mysql_201708130830/mysql/ndb_binlog_index.MYI mysql_201708130830/mysql/ndb_binlog_index.frm mysql_201708130830/mysql/plugin.MYD mysql_201708130830/mysql/plugin.MYI mysql_201708130830/mysql/plugin.frm mysql_201708130830/mysql/proc.MYD mysql_201708130830/mysql/proc.MYI mysql_201708130830/mysql/proc.frm mysql_201708130830/mysql/procs_priv.MYD mysql_201708130830/mysql/procs_priv.MYI mysql_201708130830/mysql/procs_priv.frm mysql_201708130830/mysql/proxies_priv.MYD mysql_201708130830/mysql/proxies_priv.MYI mysql_201708130830/mysql/proxies_priv.frm mysql_201708130830/mysql/servers.MYD mysql_201708130830/mysql/servers.MYI mysql_201708130830/mysql/servers.frm mysql_201708130830/mysql/slow_log.CSM mysql_201708130830/mysql/slow_log.CSV mysql_201708130830/mysql/slow_log.frm mysql_201708130830/mysql/tables_priv.MYD mysql_201708130830/mysql/tables_priv.MYI mysql_201708130830/mysql/tables_priv.frm mysql_201708130830/mysql/time_zone.MYD mysql_201708130830/mysql/time_zone.MYI mysql_201708130830/mysql/time_zone.frm mysql_201708130830/mysql/time_zone_leap_second.MYD mysql_201708130830/mysql/time_zone_leap_second.MYI mysql_201708130830/mysql/time_zone_leap_second.frm mysql_201708130830/mysql/time_zone_name.MYD mysql_201708130830/mysql/time_zone_name.MYI mysql_201708130830/mysql/time_zone_name.frm mysql_201708130830/mysql/time_zone_transition.MYD mysql_201708130830/mysql/time_zone_transition.MYI mysql_201708130830/mysql/time_zone_transition.frm mysql_201708130830/mysql/time_zone_transition_type.MYD mysql_201708130830/mysql/time_zone_transition_type.MYI mysql_201708130830/mysql/time_zone_transition_type.frm mysql_201708130830/mysql/user.MYD mysql_201708130830/mysql/user.MYI mysql_201708130830/mysql/user.frm mysql_201708130830/performance_schema/ mysql_201708130830/performance_schema/cond_instances.frm mysql_201708130830/performance_schema/db.opt mysql_201708130830/performance_schema/events_waits_current.frm mysql_201708130830/performance_schema/events_waits_history.frm mysql_201708130830/performance_schema/events_waits_history_long.frm mysql_201708130830/performance_schema/events_waits_summary_by_instance.frm mysql_201708130830/performance_schema/events_waits_summary_by_thread_by_event_name.frm mysql_201708130830/performance_schema/events_waits_summary_global_by_event_name.frm mysql_201708130830/performance_schema/file_instances.frm mysql_201708130830/performance_schema/file_summary_by_event_name.frm mysql_201708130830/performance_schema/file_summary_by_instance.frm mysql_201708130830/performance_schema/mutex_instances.frm mysql_201708130830/performance_schema/performance_timers.frm mysql_201708130830/performance_schema/rwlock_instances.frm mysql_201708130830/performance_schema/setup_consumers.frm mysql_201708130830/performance_schema/setup_instruments.frm mysql_201708130830/performance_schema/setup_timers.frm mysql_201708130830/performance_schema/threads.frm mysql_201708130830/test/ mysql_201708130830/test/db.opt sent 4157 bytes received 371 bytes 823.27 bytes/sec total size is 718361496 speedup is 158648.74 (DRY RUN) Stop Send 201708130905 [root@backuplogC7 ~]# ^C [root@backuplogC7 ~]# Am I lost? Best Regards, Siranee Jaraswachirakul. > On Fri, Aug 11, 2017 at 11:08 PM, <siranee.ja@tpc.co.th> wrote: > > >> The backup script has the btrfs sync command since Aug 3 > > > From your script: >> system btrfs sub snap -r $basepath $snappath >> system btrfs sub sync $basepath > > From the man page: sync <path> [subvolid...] > Wait until given subvolume(s) are completely removed from the > filesystem after deletion. > > > This 'subvolume sync' command, per the man page, is only about > subvolume deletion. I suggest replacing it with a regular sync > command. > > I think the problem is that the script does things so fast that the > snapshot is not always consistent on disk before btrfs send starts. > It's just a guess though. If I'm right, this means the rsync mismaches > mean the destination snapshots are bad. Here's what I would do: > > > - delete all the bad/mismatching snapshots only on the destination computer. > > - he most recent good snapshot pair, which rsync shows origin and > destination match, is mysql_201708080830 so you can keep that one on > both sides. > > - manually do incremental send/receive, starting with > mysql_201708090830/, to make the destination current again with the > origin. > > - confirm with rsync that the snapshot pairs on origin and destination > are the same > > - now resume using the modified script, which will do snapshot -> sync -> send. > > OPTIONAL, you could add to your script an rsync -avnc to double check > that the incremental send receive is working. This is admittedly > inefficient because it checks the *entire* contents of the snapshots > on both sides, it's not just checking the incremental data. But if it > doesn't take too long, it will help restore trust in send/receive, and > confirm that a regular sync is needed in between snapshot and send. > > > > > -- > Chris Murphy > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 2:20 ` siranee.ja @ 2017-08-13 2:59 ` Chris Murphy 2017-08-13 3:40 ` siranee.ja 0 siblings, 1 reply; 31+ messages in thread From: Chris Murphy @ 2017-08-13 2:59 UTC (permalink / raw) To: siranee.ja; +Cc: Chris Murphy, Btrfs BTRFS, voravat On Sat, Aug 12, 2017 at 8:20 PM, <siranee.ja@tpc.co.th> wrote: > [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 > root@192.168.45.166://var/lib/mariadb/mysql_201708090830 You need trailing / for the first directory with -a option. rsync -a dir dir is not the same command as rsync -a dir/ dir It's confusing but your command is trying to create mysql_201708090830 directory on the source, in the mysql_201708090830 on the destination. That is why everything mismatches. To make it mean "contents of" you need trailing slash on at least the origin. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 2:59 ` Chris Murphy @ 2017-08-13 3:40 ` siranee.ja 2017-08-13 4:34 ` Chris Murphy 2017-08-13 6:20 ` A L 0 siblings, 2 replies; 31+ messages in thread From: siranee.ja @ 2017-08-13 3:40 UTC (permalink / raw) To: Chris Murphy; +Cc: Chris Murphy, Btrfs BTRFS, voravat Hi Chris, I started as your suggestion again. The diff occured since snapshot mysql_201708090830 manually send. What should I do next? - delete all the bad/mismatching snapshots only on the destination computer. [root@joytest ~]# date Sun Aug 13 10:27:23 ICT 2017 [root@joytest ~]# cd /var/lib/mariadb [root@joytest mariadb]# btrfs sub list . ID 313 gen 220 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 318 gen 205 top level 5 path mysql_201708090830 ID 320 gen 211 top level 5 path mysql_201708100830 ID 322 gen 219 top level 5 path mysql_201708110830 ID 323 gen 219 top level 5 path mysql_201708120830 ID 324 gen 224 top level 5 path mysql_201708130830 ID 325 gen 225 top level 5 path mysql [root@joytest mariadb]# btrfs sub del mysql_201708130830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' [root@joytest mariadb]# btrfs sub del mysql_201708120830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' [root@joytest mariadb]# btrfs sub del mysql_201708110830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' [root@joytest mariadb]# btrfs sub del mysql_201708100830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' [root@joytest mariadb]# btrfs sub del mysql_201708090830 Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' [root@joytest mariadb]# btrfs sub sync . [root@joytest mariadb]# systemctl status mariadb â— mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min ago Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE) Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS) Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS) Main PID: 19870 (code=exited, status=0/SUCCESS) Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'. Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting mysqld daemon with databases from /var/lib/mariadb/mysql Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, code=exited status=1 Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state. Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed. [root@joytest mariadb]# btrfs sub list . ID 313 gen 220 top level 5 path mysql_201708070830 ID 316 gen 199 top level 5 path mysql_201708080830 ID 325 gen 225 top level 5 path mysql [root@joytest mariadb]# - The most recent good snapshot pair, which rsync shows origin and destination match, is mysql_201708080830 so you can keep that one on both sides. [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb ID 257 gen 538 top level 5 path mysql ID 316 gen 498 top level 5 path mysql_201708060830 ID 317 gen 503 top level 5 path mysql_201708070830 ID 318 gen 507 top level 5 path mysql_201708080830 ID 319 gen 514 top level 5 path mysql_201708090830 ID 320 gen 524 top level 5 path mysql_201708100830 ID 321 gen 529 top level 5 path mysql_201708110830 ID 322 gen 533 top level 5 path mysql_201708120830 ID 323 gen 538 top level 5 path mysql_201708130830 [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ root@192.168.45.166://var/lib/mariadb/mysql_201708070830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 842.67 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ root@192.168.45.166://var/lib/mariadb/mysql_201708080830/ sending incremental file list ./ sent 3769 bytes received 19 bytes 841.78 bytes/sec total size is 718361496 speedup is 189641.37 (DRY RUN) [root@backuplogC7 ~]# date Sun Aug 13 10:34:05 ICT 2017 [root@backuplogC7 ~]# - manually do incremental send/receive, starting with mysql_201708090830/, to make the destination current again with the origin. [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708090830 At snapshot mysql_201708090830 [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ sending incremental file list ./ ib_logfile1 ibdata1 sent 3779 bytes received 25 bytes 507.20 bytes/sec total size is 718361496 speedup is 188843.72 (DRY RUN) [root@backuplogC7 ~]# Best Regards, Siranee Jaraswachirakul. > On Sat, Aug 12, 2017 at 8:20 PM, <siranee.ja@tpc.co.th> wrote: > >> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 >> root@192.168.45.166://var/lib/mariadb/mysql_201708090830 > > > You need trailing / for the first directory with -a option. > > rsync -a dir dir > > is not the same command as > > rsync -a dir/ dir > > It's confusing but your command is trying to create mysql_201708090830 > directory on the source, in the mysql_201708090830 on the destination. > That is why everything mismatches. To make it mean "contents of" you > need trailing slash on at least the origin. > > > > -- > Chris Murphy > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 3:40 ` siranee.ja @ 2017-08-13 4:34 ` Chris Murphy 2017-08-13 10:49 ` siranee.ja 2017-08-13 6:20 ` A L 1 sibling, 1 reply; 31+ messages in thread From: Chris Murphy @ 2017-08-13 4:34 UTC (permalink / raw) To: siranee.ja; +Cc: Chris Murphy, Btrfs BTRFS, voravat On Sat, Aug 12, 2017 at 9:40 PM, <siranee.ja@tpc.co.th> wrote: > [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ > root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ > sending incremental file list > ./ > ib_logfile1 > ibdata1 > > sent 3779 bytes received 25 bytes 507.20 bytes/sec > total size is 718361496 speedup is 188843.72 (DRY RUN) OK so I don't think this can be a sync related problem. That snapshot has been committed to disk days ago. There's definitely something wrong with the incremental send/receive, but it's unclear whether this is a kernel bug (send side) or btrfs-progs (receive side), or if there's any chance of file system corruption/confusion happening with either of the two subvolumes on the origin or the subvolume (parent) on the destination. So that means you're really in the weeds on what to do next. Try deleting mysql_201708090830/ snapshot on the destination. And resend but this time do a full send of that snapshot, don't use -p. I wonder if a full send, rather than incremental makes a difference. Follow it up with the rsync command to compare origin and destination. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 4:34 ` Chris Murphy @ 2017-08-13 10:49 ` siranee.ja 2017-08-13 19:31 ` Chris Murphy 0 siblings, 1 reply; 31+ messages in thread From: siranee.ja @ 2017-08-13 10:49 UTC (permalink / raw) To: Chris Murphy; +Cc: Chris Murphy, Btrfs BTRFS, voravat Hi Chris, Try deleting mysql_201708090830/ snapshot on the destination. And resend but this time do a full send of that snapshot, don't use -p. I wonder if a full send, rather than incremental makes a difference. Follow it up with the rsync command to compare origin and destination. Yes, It's different. [root@backuplogC7 ~]# btrfs send /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708090830 At subvol mysql_201708090830 [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 842.67 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) Best Regards, Siranee Jaraswachirakul > On Sat, Aug 12, 2017 at 9:40 PM, <siranee.ja@tpc.co.th> wrote: > >> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ >> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ >> sending incremental file list >> ./ >> ib_logfile1 >> ibdata1 >> >> sent 3779 bytes received 25 bytes 507.20 bytes/sec >> total size is 718361496 speedup is 188843.72 (DRY RUN) > > > OK so I don't think this can be a sync related problem. That snapshot > has been committed to disk days ago. There's definitely something > wrong with the incremental send/receive, but it's unclear whether this > is a kernel bug (send side) or btrfs-progs (receive side), or if > there's any chance of file system corruption/confusion happening with > either of the two subvolumes on the origin or the subvolume (parent) > on the destination. > > So that means you're really in the weeds on what to do next. > > Try deleting mysql_201708090830/ snapshot on the destination. And > resend but this time do a full send of that snapshot, don't use -p. I > wonder if a full send, rather than incremental makes a difference. > Follow it up with the rsync command to compare origin and destination. > > > > -- > Chris Murphy > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 10:49 ` siranee.ja @ 2017-08-13 19:31 ` Chris Murphy 0 siblings, 0 replies; 31+ messages in thread From: Chris Murphy @ 2017-08-13 19:31 UTC (permalink / raw) To: siranee.ja; +Cc: Chris Murphy, Btrfs BTRFS, voravat On Sun, Aug 13, 2017 at 4:49 AM, <siranee.ja@tpc.co.th> wrote: > [root@backuplogC7 ~]# btrfs send /var/lib/mariadb/mysql_201708090830 | ssh > 192.168.45.166 btrfs receive /var/lib/mariadb > At subvol /var/lib/mariadb/mysql_201708090830 > At subvol mysql_201708090830 > [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ > root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/ > sending incremental file list > ./ > > sent 3773 bytes received 19 bytes 842.67 bytes/sec > total size is 718361496 speedup is 189441.32 (DRY RUN) OK so the full send is sane. That suggests the file systems on both sides are OK, and that the problem is related strictly to incremental send/receive. It suggests that in a critical way either an origin parent or its copy on the destination are not identical. Why will be hard to figure out, and I'm not even sure it's worth it. Right now you basically have a work around: a known good full send, you can start doing incremental send/receive again using this full send/receive snapshot as the parent (with -p). Where confusion can happen is if you have to restore a snapshot, I think you must commit to a whole new canonical tree with the reverse send/receive being a full one, NOT incremental. I'm asserting this, I do not know if it is true, I don't know if the tools should prevent reverse incremental send/receive, but the very idea of reversing an incremental send receive to me just seems logically problematic. Any time you reverse the send/receive direction it must be full. I think. I did once get into trouble with this myself, having to do a restore from backup, to a new file system. I then took a rw snapshot of it, and made that the canonical live modify subvolume. I then ro snapshotted it, and tried to do an incremental send *back* to backup and it failed with an error I don't remember but I think it was later send/receive versions than 4.4 where there's some extract sanity checking to prevent this. And I was able to work around it with -c (clone) option. Anyway, I got sufficiently confused myself after all of that, that I pretty much gave up on such a strategy, blew away all the ro snapshots on both sides, and started from scratch with a new full send, and then resumed the incrementals unidirectionally. And anytime they are reversed I assume it must be a full send. Anyway, I think it's perilous to reverse directions while also trying to do incremental send/receive without a lot of testing to thoroughly understand what's going on. And I basically got to fuck this, not worth the complexity. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 3:40 ` siranee.ja 2017-08-13 4:34 ` Chris Murphy @ 2017-08-13 6:20 ` A L 2017-08-13 10:52 ` siranee.ja 1 sibling, 1 reply; 31+ messages in thread From: A L @ 2017-08-13 6:20 UTC (permalink / raw) To: Btrfs BTRFS; +Cc: siranee.ja Have you checked that there is no Received UUID on the source subvolume? # btrfs sub show volume/mysql/ volume/mysql Name: mysql UUID: 8a94524e-a956-c14b-bb8d-d453627f27d5 Parent UUID: - Received UUID: - Creation time: 2017-04-17 11:46:20 +0200 Subvolume ID: 1469 Generation: 122934 Gen at creation: 78671 Parent ID: 5 Top level ID: 5 Flags: - Snapshot(s): There is no Received UUID here. If it has, then btrfs send-receive will have problems, since all snapshots of the source subvolume will have the same Received UUID and it can't tell the differences between the snapshots. On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote: > Hi Chris, > > I started as your suggestion again. The diff occured since snapshot > mysql_201708090830 manually send. What should I do next? > > - delete all the bad/mismatching snapshots only on the destination computer. > [root@joytest ~]# date > Sun Aug 13 10:27:23 ICT 2017 > [root@joytest ~]# cd /var/lib/mariadb > [root@joytest mariadb]# btrfs sub list . > ID 313 gen 220 top level 5 path mysql_201708070830 > ID 316 gen 199 top level 5 path mysql_201708080830 > ID 318 gen 205 top level 5 path mysql_201708090830 > ID 320 gen 211 top level 5 path mysql_201708100830 > ID 322 gen 219 top level 5 path mysql_201708110830 > ID 323 gen 219 top level 5 path mysql_201708120830 > ID 324 gen 224 top level 5 path mysql_201708130830 > ID 325 gen 225 top level 5 path mysql > [root@joytest mariadb]# btrfs sub del mysql_201708130830 > Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' > [root@joytest mariadb]# btrfs sub del mysql_201708120830 > Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' > [root@joytest mariadb]# btrfs sub del mysql_201708110830 > Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' > [root@joytest mariadb]# btrfs sub del mysql_201708100830 > Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' > [root@joytest mariadb]# btrfs sub del mysql_201708090830 > Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' > [root@joytest mariadb]# btrfs sub sync . > [root@joytest mariadb]# systemctl status mariadb > â— mariadb.service - MariaDB database server > Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: > disabled) > Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min ago > Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID > (code=exited, status=1/FAILURE) > Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, > status=0/SUCCESS) > Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, > status=0/SUCCESS) > Main PID: 19870 (code=exited, status=0/SUCCESS) > > Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... > Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to > '/var/log/mariadb/mariadb.log'. > Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting > mysqld daemon with databases from /var/lib/mariadb/mysql > Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, > code=exited status=1 > Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. > Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state. > Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed. > [root@joytest mariadb]# btrfs sub list . > ID 313 gen 220 top level 5 path mysql_201708070830 > ID 316 gen 199 top level 5 path mysql_201708080830 > ID 325 gen 225 top level 5 path mysql > [root@joytest mariadb]# > > - The most recent good snapshot pair, which rsync shows origin and > destination match, is mysql_201708080830 so you can keep that one on > both sides. > > [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb > ID 257 gen 538 top level 5 path mysql > ID 316 gen 498 top level 5 path mysql_201708060830 > ID 317 gen 503 top level 5 path mysql_201708070830 > ID 318 gen 507 top level 5 path mysql_201708080830 > ID 319 gen 514 top level 5 path mysql_201708090830 > ID 320 gen 524 top level 5 path mysql_201708100830 > ID 321 gen 529 top level 5 path mysql_201708110830 > ID 322 gen 533 top level 5 path mysql_201708120830 > ID 323 gen 538 top level 5 path mysql_201708130830 > [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ > root@192.168.45.166://var/lib/mariadb/mysql_201708070830/ > sending incremental file list > ./ > > sent 3773 bytes received 19 bytes 842.67 bytes/sec > total size is 718361496 speedup is 189441.32 (DRY RUN) > [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ > root@192.168.45.166://var/lib/mariadb/mysql_201708080830/ > sending incremental file list > ./ > > sent 3769 bytes received 19 bytes 841.78 bytes/sec > total size is 718361496 speedup is 189641.37 (DRY RUN) > [root@backuplogC7 ~]# date > Sun Aug 13 10:34:05 ICT 2017 > [root@backuplogC7 ~]# > > - manually do incremental send/receive, starting with > mysql_201708090830/, to make the destination current again with the > origin. > > [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 > /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive > /var/lib/mariadb > At subvol /var/lib/mariadb/mysql_201708090830 > At snapshot mysql_201708090830 > [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ > root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ > sending incremental file list > ./ > ib_logfile1 > ibdata1 > > sent 3779 bytes received 25 bytes 507.20 bytes/sec > total size is 718361496 speedup is 188843.72 (DRY RUN) > [root@backuplogC7 ~]# > > Best Regards, > > Siranee Jaraswachirakul. > >> On Sat, Aug 12, 2017 at 8:20 PM, <siranee.ja@tpc.co.th> wrote: >> >>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 >>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830 >> >> You need trailing / for the first directory with -a option. >> >> rsync -a dir dir >> >> is not the same command as >> >> rsync -a dir/ dir >> >> It's confusing but your command is trying to create mysql_201708090830 >> directory on the source, in the mysql_201708090830 on the destination. >> That is why everything mismatches. To make it mean "contents of" you >> need trailing slash on at least the origin. >> >> >> >> -- >> Chris Murphy >> > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 6:20 ` A L @ 2017-08-13 10:52 ` siranee.ja 2017-08-13 12:51 ` A L 2017-08-13 19:50 ` Chris Murphy 0 siblings, 2 replies; 31+ messages in thread From: siranee.ja @ 2017-08-13 10:52 UTC (permalink / raw) To: A L; +Cc: Btrfs BTRFS Hi "A L", [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql /var/lib/mariadb/mysql Name: mysql UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 Parent UUID: - Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 Creation time: 2017-06-21 13:27:41 +0700 Subvolume ID: 257 Generation: 539 Gen at creation: 9 Parent ID: 5 Top level ID: 5 Flags: - Snapshot(s): mysql_201708060830 mysql_201708070830 mysql_201708080830 mysql_201708090830 mysql_201708100830 mysql_201708110830 mysql_201708120830 mysql_201708130830 yes I think it has Received UUID because I restored the source from snapshot mysql_201708040830 for prove that the local snapshot was work. How to clear the Received UUID ? What to do next? Best Regards, Siranee Jaraswachirakul. > Have you checked that there is no Received UUID on the source subvolume? > > # btrfs sub show volume/mysql/ > volume/mysql > Name: mysql > UUID: 8a94524e-a956-c14b-bb8d-d453627f27d5 > Parent UUID: - > Received UUID: - > Creation time: 2017-04-17 11:46:20 +0200 > Subvolume ID: 1469 > Generation: 122934 > Gen at creation: 78671 > Parent ID: 5 > Top level ID: 5 > Flags: - > Snapshot(s): > > There is no Received UUID here. If it has, then btrfs send-receive will > have problems, since all snapshots of the source subvolume will have the > same Received UUID and it can't tell the differences between the snapshots. > > On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote: >> Hi Chris, >> >> I started as your suggestion again. The diff occured since snapshot >> mysql_201708090830 manually send. What should I do next? >> >> - delete all the bad/mismatching snapshots only on the destination computer. >> [root@joytest ~]# date >> Sun Aug 13 10:27:23 ICT 2017 >> [root@joytest ~]# cd /var/lib/mariadb >> [root@joytest mariadb]# btrfs sub list . >> ID 313 gen 220 top level 5 path mysql_201708070830 >> ID 316 gen 199 top level 5 path mysql_201708080830 >> ID 318 gen 205 top level 5 path mysql_201708090830 >> ID 320 gen 211 top level 5 path mysql_201708100830 >> ID 322 gen 219 top level 5 path mysql_201708110830 >> ID 323 gen 219 top level 5 path mysql_201708120830 >> ID 324 gen 224 top level 5 path mysql_201708130830 >> ID 325 gen 225 top level 5 path mysql >> [root@joytest mariadb]# btrfs sub del mysql_201708130830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' >> [root@joytest mariadb]# btrfs sub del mysql_201708120830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' >> [root@joytest mariadb]# btrfs sub del mysql_201708110830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' >> [root@joytest mariadb]# btrfs sub del mysql_201708100830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' >> [root@joytest mariadb]# btrfs sub del mysql_201708090830 >> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' >> [root@joytest mariadb]# btrfs sub sync . >> [root@joytest mariadb]# systemctl status mariadb >> â— mariadb.service - MariaDB database server >> Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor >> preset: >> disabled) >> Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min >> ago >> Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID >> (code=exited, status=1/FAILURE) >> Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, >> status=0/SUCCESS) >> Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n >> (code=exited, >> status=0/SUCCESS) >> Main PID: 19870 (code=exited, status=0/SUCCESS) >> >> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... >> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to >> '/var/log/mariadb/mariadb.log'. >> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting >> mysqld daemon with databases from /var/lib/mariadb/mysql >> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, >> code=exited status=1 >> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. >> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state. >> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed. >> [root@joytest mariadb]# btrfs sub list . >> ID 313 gen 220 top level 5 path mysql_201708070830 >> ID 316 gen 199 top level 5 path mysql_201708080830 >> ID 325 gen 225 top level 5 path mysql >> [root@joytest mariadb]# >> >> - The most recent good snapshot pair, which rsync shows origin and >> destination match, is mysql_201708080830 so you can keep that one on >> both sides. >> >> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb >> ID 257 gen 538 top level 5 path mysql >> ID 316 gen 498 top level 5 path mysql_201708060830 >> ID 317 gen 503 top level 5 path mysql_201708070830 >> ID 318 gen 507 top level 5 path mysql_201708080830 >> ID 319 gen 514 top level 5 path mysql_201708090830 >> ID 320 gen 524 top level 5 path mysql_201708100830 >> ID 321 gen 529 top level 5 path mysql_201708110830 >> ID 322 gen 533 top level 5 path mysql_201708120830 >> ID 323 gen 538 top level 5 path mysql_201708130830 >> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ >> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/ >> sending incremental file list >> ./ >> >> sent 3773 bytes received 19 bytes 842.67 bytes/sec >> total size is 718361496 speedup is 189441.32 (DRY RUN) >> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ >> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/ >> sending incremental file list >> ./ >> >> sent 3769 bytes received 19 bytes 841.78 bytes/sec >> total size is 718361496 speedup is 189641.37 (DRY RUN) >> [root@backuplogC7 ~]# date >> Sun Aug 13 10:34:05 ICT 2017 >> [root@backuplogC7 ~]# >> >> - manually do incremental send/receive, starting with >> mysql_201708090830/, to make the destination current again with the >> origin. >> >> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 >> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive >> /var/lib/mariadb >> At subvol /var/lib/mariadb/mysql_201708090830 >> At snapshot mysql_201708090830 >> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ >> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ >> sending incremental file list >> ./ >> ib_logfile1 >> ibdata1 >> >> sent 3779 bytes received 25 bytes 507.20 bytes/sec >> total size is 718361496 speedup is 188843.72 (DRY RUN) >> [root@backuplogC7 ~]# >> >> Best Regards, >> >> Siranee Jaraswachirakul. >> >>> On Sat, Aug 12, 2017 at 8:20 PM, <siranee.ja@tpc.co.th> wrote: >>> >>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 >>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830 >>> >>> You need trailing / for the first directory with -a option. >>> >>> rsync -a dir dir >>> >>> is not the same command as >>> >>> rsync -a dir/ dir >>> >>> It's confusing but your command is trying to create mysql_201708090830 >>> directory on the source, in the mysql_201708090830 on the destination. >>> That is why everything mismatches. To make it mean "contents of" you >>> need trailing slash on at least the origin. >>> >>> >>> >>> -- >>> Chris Murphy >>> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 10:52 ` siranee.ja @ 2017-08-13 12:51 ` A L 2017-08-13 14:00 ` siranee.ja 2017-08-13 19:50 ` Chris Murphy 1 sibling, 1 reply; 31+ messages in thread From: A L @ 2017-08-13 12:51 UTC (permalink / raw) To: Btrfs BTRFS; +Cc: siranee.ja On 8/13/2017 12:52 PM, siranee.ja@tpc.co.th wrote: > Hi "A L", > > [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql > /var/lib/mariadb/mysql > Name: mysql > UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 > Parent UUID: - > Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 > Creation time: 2017-06-21 13:27:41 +0700 > Subvolume ID: 257 > Generation: 539 > Gen at creation: 9 > Parent ID: 5 > Top level ID: 5 > Flags: - > Snapshot(s): > mysql_201708060830 > mysql_201708070830 > mysql_201708080830 > mysql_201708090830 > mysql_201708100830 > mysql_201708110830 > mysql_201708120830 > mysql_201708130830 > > yes I think it has Received UUID because I restored the source from snapshot > mysql_201708040830 for prove that the local snapshot was work. > > How to clear the Received UUID ? > What to do next? You need to make a read-write snapshot of /var/lib/mariadb/mysql and then remove the old subvolume and all its snapshots. Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md # cd /mnt/btr_pool # mv mysubvolume mysubvolume.broken # btrfs subvolume snapshot mysubvolume.broken mysubvolume You can do the same with each of your snapshots too, and send them as full snapshots (without -p). ~A > Best Regards, > Siranee Jaraswachirakul. > >> Have you checked that there is no Received UUID on the source subvolume? >> >> # btrfs sub show volume/mysql/ >> volume/mysql >> Name: mysql >> UUID: 8a94524e-a956-c14b-bb8d-d453627f27d5 >> Parent UUID: - >> Received UUID: - >> Creation time: 2017-04-17 11:46:20 +0200 >> Subvolume ID: 1469 >> Generation: 122934 >> Gen at creation: 78671 >> Parent ID: 5 >> Top level ID: 5 >> Flags: - >> Snapshot(s): >> >> There is no Received UUID here. If it has, then btrfs send-receive will >> have problems, since all snapshots of the source subvolume will have the >> same Received UUID and it can't tell the differences between the snapshots. >> >> On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote: >>> Hi Chris, >>> >>> I started as your suggestion again. The diff occured since snapshot >>> mysql_201708090830 manually send. What should I do next? >>> >>> - delete all the bad/mismatching snapshots only on the destination computer. >>> [root@joytest ~]# date >>> Sun Aug 13 10:27:23 ICT 2017 >>> [root@joytest ~]# cd /var/lib/mariadb >>> [root@joytest mariadb]# btrfs sub list . >>> ID 313 gen 220 top level 5 path mysql_201708070830 >>> ID 316 gen 199 top level 5 path mysql_201708080830 >>> ID 318 gen 205 top level 5 path mysql_201708090830 >>> ID 320 gen 211 top level 5 path mysql_201708100830 >>> ID 322 gen 219 top level 5 path mysql_201708110830 >>> ID 323 gen 219 top level 5 path mysql_201708120830 >>> ID 324 gen 224 top level 5 path mysql_201708130830 >>> ID 325 gen 225 top level 5 path mysql >>> [root@joytest mariadb]# btrfs sub del mysql_201708130830 >>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' >>> [root@joytest mariadb]# btrfs sub del mysql_201708120830 >>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' >>> [root@joytest mariadb]# btrfs sub del mysql_201708110830 >>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' >>> [root@joytest mariadb]# btrfs sub del mysql_201708100830 >>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' >>> [root@joytest mariadb]# btrfs sub del mysql_201708090830 >>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' >>> [root@joytest mariadb]# btrfs sub sync . >>> [root@joytest mariadb]# systemctl status mariadb >>> â— mariadb.service - MariaDB database server >>> Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor >>> preset: >>> disabled) >>> Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min >>> ago >>> Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID >>> (code=exited, status=1/FAILURE) >>> Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, >>> status=0/SUCCESS) >>> Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n >>> (code=exited, >>> status=0/SUCCESS) >>> Main PID: 19870 (code=exited, status=0/SUCCESS) >>> >>> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... >>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to >>> '/var/log/mariadb/mariadb.log'. >>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting >>> mysqld daemon with databases from /var/lib/mariadb/mysql >>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, >>> code=exited status=1 >>> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. >>> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state. >>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed. >>> [root@joytest mariadb]# btrfs sub list . >>> ID 313 gen 220 top level 5 path mysql_201708070830 >>> ID 316 gen 199 top level 5 path mysql_201708080830 >>> ID 325 gen 225 top level 5 path mysql >>> [root@joytest mariadb]# >>> >>> - The most recent good snapshot pair, which rsync shows origin and >>> destination match, is mysql_201708080830 so you can keep that one on >>> both sides. >>> >>> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb >>> ID 257 gen 538 top level 5 path mysql >>> ID 316 gen 498 top level 5 path mysql_201708060830 >>> ID 317 gen 503 top level 5 path mysql_201708070830 >>> ID 318 gen 507 top level 5 path mysql_201708080830 >>> ID 319 gen 514 top level 5 path mysql_201708090830 >>> ID 320 gen 524 top level 5 path mysql_201708100830 >>> ID 321 gen 529 top level 5 path mysql_201708110830 >>> ID 322 gen 533 top level 5 path mysql_201708120830 >>> ID 323 gen 538 top level 5 path mysql_201708130830 >>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ >>> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/ >>> sending incremental file list >>> ./ >>> >>> sent 3773 bytes received 19 bytes 842.67 bytes/sec >>> total size is 718361496 speedup is 189441.32 (DRY RUN) >>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ >>> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/ >>> sending incremental file list >>> ./ >>> >>> sent 3769 bytes received 19 bytes 841.78 bytes/sec >>> total size is 718361496 speedup is 189641.37 (DRY RUN) >>> [root@backuplogC7 ~]# date >>> Sun Aug 13 10:34:05 ICT 2017 >>> [root@backuplogC7 ~]# >>> >>> - manually do incremental send/receive, starting with >>> mysql_201708090830/, to make the destination current again with the >>> origin. >>> >>> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 >>> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive >>> /var/lib/mariadb >>> At subvol /var/lib/mariadb/mysql_201708090830 >>> At snapshot mysql_201708090830 >>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ >>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ >>> sending incremental file list >>> ./ >>> ib_logfile1 >>> ibdata1 >>> >>> sent 3779 bytes received 25 bytes 507.20 bytes/sec >>> total size is 718361496 speedup is 188843.72 (DRY RUN) >>> [root@backuplogC7 ~]# >>> >>> Best Regards, >>> >>> Siranee Jaraswachirakul. >>> >>>> On Sat, Aug 12, 2017 at 8:20 PM, <siranee.ja@tpc.co.th> wrote: >>>> >>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 >>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830 >>>> You need trailing / for the first directory with -a option. >>>> >>>> rsync -a dir dir >>>> >>>> is not the same command as >>>> >>>> rsync -a dir/ dir >>>> >>>> It's confusing but your command is trying to create mysql_201708090830 >>>> directory on the source, in the mysql_201708090830 on the destination. >>>> That is why everything mismatches. To make it mean "contents of" you >>>> need trailing slash on at least the origin. >>>> >>>> >>>> >>>> -- >>>> Chris Murphy >>>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 12:51 ` A L @ 2017-08-13 14:00 ` siranee.ja 2017-08-13 21:31 ` A L 0 siblings, 1 reply; 31+ messages in thread From: siranee.ja @ 2017-08-13 14:00 UTC (permalink / raw) To: A L; +Cc: Btrfs BTRFS Hi "A L", As your suggestion. Does it mean I have to make all my current subvolumes to clear "Receive UUID" and send them with full snapshots? >> mysql >> mysql_201708060830 >> mysql_201708070830 >> mysql_201708080830 >> mysql_201708090830 >> mysql_201708100830 >> mysql_201708110830 >> mysql_201708120830 >> mysql_201708130830 The next day that I take snapshot with the script will be mysql_201708140830 which will be sent with incremental from mysql_201708130830 to mysql_201708140830 Is this correct? I am not understand with the url that you gave me which said "I'm getting an error: Aborted: "Received UUID" is set You probably restored a backup with send-receive, and made it read/write using btrfs property set. This is bad, as all snapshots and backups will inherit this identical "Received UUID", which results in all these subvolumes will be treated as "containing same data". To fix this, create a "proper" snapshot: - This is as your suggestion for the subvolume "mysql" # cd /mnt/btr_pool # mv mysubvolume mysubvolume.broken # btrfs subvolume snapshot mysubvolume.broken mysubvolume Now, mysubvolume should have an empty "Received UUID". Note that in order to have a clean environment, you also need to fix all subvolumes (snapshots as well as backups) that you created with the broken subvolume. Check if there are more broken subvolumes: # btrfs subvolume show mysubvolume.broken # btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above> # btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above> - This guide seem that I have to clear <"Received UUID" > only the subvolume "mysql" and the others ("mysql_201708070830" should using btrfs subvolume snapshot -r instead of btrfs subvolume snapshot. Is this correct? Now clean all subvolume listed (same as above, but using btrfs subvolume snapshot -r now). Then delete all the broken subvolumes: # btrfs subvolume delete *.broken Finally, you should have a clean environment, and btrbk will not complain any more. Best Regards, Siranee Jaraswachirakul. > > > On 8/13/2017 12:52 PM, siranee.ja@tpc.co.th wrote: >> Hi "A L", >> >> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql >> /var/lib/mariadb/mysql >> Name: mysql >> UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 >> Parent UUID: - >> Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 >> Creation time: 2017-06-21 13:27:41 +0700 >> Subvolume ID: 257 >> Generation: 539 >> Gen at creation: 9 >> Parent ID: 5 >> Top level ID: 5 >> Flags: - >> Snapshot(s): >> mysql_201708060830 >> mysql_201708070830 >> mysql_201708080830 >> mysql_201708090830 >> mysql_201708100830 >> mysql_201708110830 >> mysql_201708120830 >> mysql_201708130830 >> >> yes I think it has Received UUID because I restored the source from snapshot >> mysql_201708040830 for prove that the local snapshot was work. >> >> How to clear the Received UUID ? >> What to do next? > You need to make a read-write snapshot of /var/lib/mariadb/mysql and > then remove the old subvolume and all its snapshots. > > Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md > > # cd /mnt/btr_pool > # mv mysubvolume mysubvolume.broken > # btrfs subvolume snapshot mysubvolume.broken mysubvolume > > You can do the same with each of your snapshots too, and send them as > full snapshots (without -p). > > ~A > >> Best Regards, >> Siranee Jaraswachirakul. >> >>> Have you checked that there is no Received UUID on the source subvolume? >>> >>> # btrfs sub show volume/mysql/ >>> volume/mysql >>> Name: mysql >>> UUID: 8a94524e-a956-c14b-bb8d-d453627f27d5 >>> Parent UUID: - >>> Received UUID: - >>> Creation time: 2017-04-17 11:46:20 +0200 >>> Subvolume ID: 1469 >>> Generation: 122934 >>> Gen at creation: 78671 >>> Parent ID: 5 >>> Top level ID: 5 >>> Flags: - >>> Snapshot(s): >>> >>> There is no Received UUID here. If it has, then btrfs send-receive will >>> have problems, since all snapshots of the source subvolume will have the >>> same Received UUID and it can't tell the differences between the snapshots. >>> >>> On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote: >>>> Hi Chris, >>>> >>>> I started as your suggestion again. The diff occured since snapshot >>>> mysql_201708090830 manually send. What should I do next? >>>> >>>> - delete all the bad/mismatching snapshots only on the destination computer. >>>> [root@joytest ~]# date >>>> Sun Aug 13 10:27:23 ICT 2017 >>>> [root@joytest ~]# cd /var/lib/mariadb >>>> [root@joytest mariadb]# btrfs sub list . >>>> ID 313 gen 220 top level 5 path mysql_201708070830 >>>> ID 316 gen 199 top level 5 path mysql_201708080830 >>>> ID 318 gen 205 top level 5 path mysql_201708090830 >>>> ID 320 gen 211 top level 5 path mysql_201708100830 >>>> ID 322 gen 219 top level 5 path mysql_201708110830 >>>> ID 323 gen 219 top level 5 path mysql_201708120830 >>>> ID 324 gen 224 top level 5 path mysql_201708130830 >>>> ID 325 gen 225 top level 5 path mysql >>>> [root@joytest mariadb]# btrfs sub del mysql_201708130830 >>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' >>>> [root@joytest mariadb]# btrfs sub del mysql_201708120830 >>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' >>>> [root@joytest mariadb]# btrfs sub del mysql_201708110830 >>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' >>>> [root@joytest mariadb]# btrfs sub del mysql_201708100830 >>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' >>>> [root@joytest mariadb]# btrfs sub del mysql_201708090830 >>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' >>>> [root@joytest mariadb]# btrfs sub sync . >>>> [root@joytest mariadb]# systemctl status mariadb >>>> â— mariadb.service - MariaDB database server >>>> Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor >>>> preset: >>>> disabled) >>>> Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h >>>> 24min >>>> ago >>>> Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID >>>> (code=exited, status=1/FAILURE) >>>> Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, >>>> status=0/SUCCESS) >>>> Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n >>>> (code=exited, >>>> status=0/SUCCESS) >>>> Main PID: 19870 (code=exited, status=0/SUCCESS) >>>> >>>> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... >>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging >>>> to >>>> '/var/log/mariadb/mariadb.log'. >>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting >>>> mysqld daemon with databases from /var/lib/mariadb/mysql >>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, >>>> code=exited status=1 >>>> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. >>>> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state. >>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed. >>>> [root@joytest mariadb]# btrfs sub list . >>>> ID 313 gen 220 top level 5 path mysql_201708070830 >>>> ID 316 gen 199 top level 5 path mysql_201708080830 >>>> ID 325 gen 225 top level 5 path mysql >>>> [root@joytest mariadb]# >>>> >>>> - The most recent good snapshot pair, which rsync shows origin and >>>> destination match, is mysql_201708080830 so you can keep that one on >>>> both sides. >>>> >>>> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb >>>> ID 257 gen 538 top level 5 path mysql >>>> ID 316 gen 498 top level 5 path mysql_201708060830 >>>> ID 317 gen 503 top level 5 path mysql_201708070830 >>>> ID 318 gen 507 top level 5 path mysql_201708080830 >>>> ID 319 gen 514 top level 5 path mysql_201708090830 >>>> ID 320 gen 524 top level 5 path mysql_201708100830 >>>> ID 321 gen 529 top level 5 path mysql_201708110830 >>>> ID 322 gen 533 top level 5 path mysql_201708120830 >>>> ID 323 gen 538 top level 5 path mysql_201708130830 >>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ >>>> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/ >>>> sending incremental file list >>>> ./ >>>> >>>> sent 3773 bytes received 19 bytes 842.67 bytes/sec >>>> total size is 718361496 speedup is 189441.32 (DRY RUN) >>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ >>>> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/ >>>> sending incremental file list >>>> ./ >>>> >>>> sent 3769 bytes received 19 bytes 841.78 bytes/sec >>>> total size is 718361496 speedup is 189641.37 (DRY RUN) >>>> [root@backuplogC7 ~]# date >>>> Sun Aug 13 10:34:05 ICT 2017 >>>> [root@backuplogC7 ~]# >>>> >>>> - manually do incremental send/receive, starting with >>>> mysql_201708090830/, to make the destination current again with the >>>> origin. >>>> >>>> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 >>>> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive >>>> /var/lib/mariadb >>>> At subvol /var/lib/mariadb/mysql_201708090830 >>>> At snapshot mysql_201708090830 >>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ >>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ >>>> sending incremental file list >>>> ./ >>>> ib_logfile1 >>>> ibdata1 >>>> >>>> sent 3779 bytes received 25 bytes 507.20 bytes/sec >>>> total size is 718361496 speedup is 188843.72 (DRY RUN) >>>> [root@backuplogC7 ~]# >>>> >>>> Best Regards, >>>> >>>> Siranee Jaraswachirakul. >>>> >>>>> On Sat, Aug 12, 2017 at 8:20 PM, <siranee.ja@tpc.co.th> wrote: >>>>> >>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 >>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830 >>>>> You need trailing / for the first directory with -a option. >>>>> >>>>> rsync -a dir dir >>>>> >>>>> is not the same command as >>>>> >>>>> rsync -a dir/ dir >>>>> >>>>> It's confusing but your command is trying to create mysql_201708090830 >>>>> directory on the source, in the mysql_201708090830 on the destination. >>>>> That is why everything mismatches. To make it mean "contents of" you >>>>> need trailing slash on at least the origin. >>>>> >>>>> >>>>> >>>>> -- >>>>> Chris Murphy >>>>> >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> > > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 14:00 ` siranee.ja @ 2017-08-13 21:31 ` A L 2017-08-14 1:57 ` siranee.ja 0 siblings, 1 reply; 31+ messages in thread From: A L @ 2017-08-13 21:31 UTC (permalink / raw) To: siranee.ja; +Cc: Btrfs BTRFS ---- From: siranee.ja@tpc.co.th -- Sent: 2017-08-13 - 16:00 ---- > Hi "A L", > > As your suggestion. Does it mean I have to make all my current subvolumes to clear > "Receive UUID" and send them with full snapshots? You need to clear all subvolumes with 'Received UUID' in order to use send-receive safely. once you have done that you can start over with new incremental sends (after the initial full send). I suggest to remove any existing snapshots of those volumes to avoid confusion. >>> mysql >>> mysql_201708060830 >>> mysql_201708070830 >>> mysql_201708080830 >>> mysql_201708090830 >>> mysql_201708100830 >>> mysql_201708110830 >>> mysql_201708120830 >>> mysql_201708130830 > > The next day that I take snapshot with the script will be > mysql_201708140830 > > which will be sent with incremental from mysql_201708130830 to mysql_201708140830 > > Is this correct? yes, but your current snapshots can't be used because of the 'Received UUID'. > > I am not understand with the url that you gave me which said btrbk is a backup tool for btrfs. It detects the problem of the 'Received UUID'. The FAQ explains how to remove the 'Received UUID' from the source volumes and snapshots. It is safer to remove all existing snapshots then trying to fix them and risking the integrity of the data. > > "I'm getting an error: Aborted: "Received UUID" is set > > You probably restored a backup with send-receive, and made it read/write using btrfs > property set. This is bad, as all snapshots and backups will inherit this identical > "Received UUID", which results in all these subvolumes will be treated as > "containing same data". > > To fix this, create a "proper" snapshot: > > - This is as your suggestion for the subvolume "mysql" > > # cd /mnt/btr_pool > # mv mysubvolume mysubvolume.broken > # btrfs subvolume snapshot mysubvolume.broken mysubvolume > > Now, mysubvolume should have an empty "Received UUID". Note that in order to have a > clean environment, you also need to fix all subvolumes (snapshots as well as > backups) that you created with the broken subvolume. > > Check if there are more broken subvolumes: > > # btrfs subvolume show mysubvolume.broken > # btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above> > # btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above> > > - This guide seem that I have to clear <"Received UUID" > only the subvolume "mysql" > and the others ("mysql_201708070830" should using btrfs subvolume snapshot -r > instead of btrfs subvolume snapshot. Is this correct? > > Now clean all subvolume listed (same as above, but using btrfs subvolume snapshot -r > now). Then delete all the broken subvolumes: > > # btrfs subvolume delete *.broken > > Finally, you should have a clean environment, and btrbk will not complain any more. > > > Best Regards, > Siranee Jaraswachirakul. > >> >> >> On 8/13/2017 12:52 PM, siranee.ja@tpc.co.th wrote: >>> Hi "A L", >>> >>> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql >>> /var/lib/mariadb/mysql >>> Name: mysql >>> UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 >>> Parent UUID: - >>> Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 >>> Creation time: 2017-06-21 13:27:41 +0700 >>> Subvolume ID: 257 >>> Generation: 539 >>> Gen at creation: 9 >>> Parent ID: 5 >>> Top level ID: 5 >>> Flags: - >>> Snapshot(s): >>> mysql_201708060830 >>> mysql_201708070830 >>> mysql_201708080830 >>> mysql_201708090830 >>> mysql_201708100830 >>> mysql_201708110830 >>> mysql_201708120830 >>> mysql_201708130830 >>> >>> yes I think it has Received UUID because I restored the source from snapshot >>> mysql_201708040830 for prove that the local snapshot was work. >>> >>> How to clear the Received UUID ? >>> What to do next? >> You need to make a read-write snapshot of /var/lib/mariadb/mysql and >> then remove the old subvolume and all its snapshots. >> >> Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md >> >> # cd /mnt/btr_pool >> # mv mysubvolume mysubvolume.broken >> # btrfs subvolume snapshot mysubvolume.broken mysubvolume >> >> You can do the same with each of your snapshots too, and send them as >> full snapshots (without -p). >> >> ~A >> >>> Best Regards, >>> Siranee Jaraswachirakul. >>> >>>> Have you checked that there is no Received UUID on the source subvolume? >>>> >>>> # btrfs sub show volume/mysql/ >>>> volume/mysql >>>> Name: mysql >>>> UUID: 8a94524e-a956-c14b-bb8d-d453627f27d5 >>>> Parent UUID: - >>>> Received UUID: - >>>> Creation time: 2017-04-17 11:46:20 +0200 >>>> Subvolume ID: 1469 >>>> Generation: 122934 >>>> Gen at creation: 78671 >>>> Parent ID: 5 >>>> Top level ID: 5 >>>> Flags: - >>>> Snapshot(s): >>>> >>>> There is no Received UUID here. If it has, then btrfs send-receive will >>>> have problems, since all snapshots of the source subvolume will have the >>>> same Received UUID and it can't tell the differences between the snapshots. >>>> >>>> On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote: >>>>> Hi Chris, >>>>> >>>>> I started as your suggestion again. The diff occured since snapshot >>>>> mysql_201708090830 manually send. What should I do next? >>>>> >>>>> - delete all the bad/mismatching snapshots only on the destination computer. >>>>> [root@joytest ~]# date >>>>> Sun Aug 13 10:27:23 ICT 2017 >>>>> [root@joytest ~]# cd /var/lib/mariadb >>>>> [root@joytest mariadb]# btrfs sub list . >>>>> ID 313 gen 220 top level 5 path mysql_201708070830 >>>>> ID 316 gen 199 top level 5 path mysql_201708080830 >>>>> ID 318 gen 205 top level 5 path mysql_201708090830 >>>>> ID 320 gen 211 top level 5 path mysql_201708100830 >>>>> ID 322 gen 219 top level 5 path mysql_201708110830 >>>>> ID 323 gen 219 top level 5 path mysql_201708120830 >>>>> ID 324 gen 224 top level 5 path mysql_201708130830 >>>>> ID 325 gen 225 top level 5 path mysql >>>>> [root@joytest mariadb]# btrfs sub del mysql_201708130830 >>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' >>>>> [root@joytest mariadb]# btrfs sub del mysql_201708120830 >>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' >>>>> [root@joytest mariadb]# btrfs sub del mysql_201708110830 >>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' >>>>> [root@joytest mariadb]# btrfs sub del mysql_201708100830 >>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' >>>>> [root@joytest mariadb]# btrfs sub del mysql_201708090830 >>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' >>>>> [root@joytest mariadb]# btrfs sub sync . >>>>> [root@joytest mariadb]# systemctl status mariadb >>>>> â— mariadb.service - MariaDB database server >>>>> Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor >>>>> preset: >>>>> disabled) >>>>> Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h >>>>> 24min >>>>> ago >>>>> Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID >>>>> (code=exited, status=1/FAILURE) >>>>> Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, >>>>> status=0/SUCCESS) >>>>> Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n >>>>> (code=exited, >>>>> status=0/SUCCESS) >>>>> Main PID: 19870 (code=exited, status=0/SUCCESS) >>>>> >>>>> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... >>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging >>>>> to >>>>> '/var/log/mariadb/mariadb.log'. >>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting >>>>> mysqld daemon with databases from /var/lib/mariadb/mysql >>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, >>>>> code=exited status=1 >>>>> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. >>>>> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state. >>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed. >>>>> [root@joytest mariadb]# btrfs sub list . >>>>> ID 313 gen 220 top level 5 path mysql_201708070830 >>>>> ID 316 gen 199 top level 5 path mysql_201708080830 >>>>> ID 325 gen 225 top level 5 path mysql >>>>> [root@joytest mariadb]# >>>>> >>>>> - The most recent good snapshot pair, which rsync shows origin and >>>>> destination match, is mysql_201708080830 so you can keep that one on >>>>> both sides. >>>>> >>>>> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb >>>>> ID 257 gen 538 top level 5 path mysql >>>>> ID 316 gen 498 top level 5 path mysql_201708060830 >>>>> ID 317 gen 503 top level 5 path mysql_201708070830 >>>>> ID 318 gen 507 top level 5 path mysql_201708080830 >>>>> ID 319 gen 514 top level 5 path mysql_201708090830 >>>>> ID 320 gen 524 top level 5 path mysql_201708100830 >>>>> ID 321 gen 529 top level 5 path mysql_201708110830 >>>>> ID 322 gen 533 top level 5 path mysql_201708120830 >>>>> ID 323 gen 538 top level 5 path mysql_201708130830 >>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ >>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/ >>>>> sending incremental file list >>>>> ./ >>>>> >>>>> sent 3773 bytes received 19 bytes 842.67 bytes/sec >>>>> total size is 718361496 speedup is 189441.32 (DRY RUN) >>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ >>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/ >>>>> sending incremental file list >>>>> ./ >>>>> >>>>> sent 3769 bytes received 19 bytes 841.78 bytes/sec >>>>> total size is 718361496 speedup is 189641.37 (DRY RUN) >>>>> [root@backuplogC7 ~]# date >>>>> Sun Aug 13 10:34:05 ICT 2017 >>>>> [root@backuplogC7 ~]# >>>>> >>>>> - manually do incremental send/receive, starting with >>>>> mysql_201708090830/, to make the destination current again with the >>>>> origin. >>>>> >>>>> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 >>>>> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive >>>>> /var/lib/mariadb >>>>> At subvol /var/lib/mariadb/mysql_201708090830 >>>>> At snapshot mysql_201708090830 >>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ >>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ >>>>> sending incremental file list >>>>> ./ >>>>> ib_logfile1 >>>>> ibdata1 >>>>> >>>>> sent 3779 bytes received 25 bytes 507.20 bytes/sec >>>>> total size is 718361496 speedup is 188843.72 (DRY RUN) >>>>> [root@backuplogC7 ~]# >>>>> >>>>> Best Regards, >>>>> >>>>> Siranee Jaraswachirakul. >>>>> >>>>>> On Sat, Aug 12, 2017 at 8:20 PM, <siranee.ja@tpc.co.th> wrote: >>>>>> >>>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 >>>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830 >>>>>> You need trailing / for the first directory with -a option. >>>>>> >>>>>> rsync -a dir dir >>>>>> >>>>>> is not the same command as >>>>>> >>>>>> rsync -a dir/ dir >>>>>> >>>>>> It's confusing but your command is trying to create mysql_201708090830 >>>>>> directory on the source, in the mysql_201708090830 on the destination. >>>>>> That is why everything mismatches. To make it mean "contents of" you >>>>>> need trailing slash on at least the origin. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Chris Murphy >>>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> >> >> > > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 21:31 ` A L @ 2017-08-14 1:57 ` siranee.ja 0 siblings, 0 replies; 31+ messages in thread From: siranee.ja @ 2017-08-14 1:57 UTC (permalink / raw) To: A L; +Cc: Btrfs BTRFS Hi "A L", Thank you very much for your suggestion. I've got it. It work properly right now. what I have done are following. mv all current mysql and snapshot mysql to mbroken and mbroken_yymmddhhmm btrfs subvolume snapshot mbroken to mysql (to make it rw subvolume without Recieve UUID) btrfs subvolume snapshot mbroken_yymmddhhmm to rw_yymmddhhmm (to make it rw subvolume without Recieve UUID) btrfs subvolume snap -r rw_yymmddhhmm to mysql_yymmddhhmm ( to make ro snapshot) after this just send full for the first ro snapshot (mysql_201708070830) and send incremental until the end (mysql_201708130830). [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708070830 rw_201708070830 Create a snapshot of 'mbroken_201708070830' in './rw_201708070830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 568 top level 5 path mbroken_201708080830 ID 319 gen 569 top level 5 path mbroken_201708090830 ID 320 gen 570 top level 5 path mbroken_201708100830 ID 321 gen 571 top level 5 path mbroken_201708110830 ID 322 gen 572 top level 5 path mbroken_201708120830 ID 323 gen 573 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708080830 rw_201708080830 Create a snapshot of 'mbroken_201708080830' in './rw_201708080830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708090830 rw_201708090830 Create a snapshot of 'mbroken_201708090830' in './rw_201708090830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708100830 rw_201708100830 Create a snapshot of 'mbroken_201708100830' in './rw_201708100830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708110830 rw_201708110830 Create a snapshot of 'mbroken_201708110830' in './rw_201708110830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708120830 rw_201708120830 Create a snapshot of 'mbroken_201708120830' in './rw_201708120830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708130830 rw_201708130830 Create a snapshot of 'mbroken_201708130830' in './rw_201708130830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 577 top level 5 path mbroken_201708080830 ID 319 gen 578 top level 5 path mbroken_201708090830 ID 320 gen 579 top level 5 path mbroken_201708100830 ID 321 gen 580 top level 5 path mbroken_201708110830 ID 322 gen 581 top level 5 path mbroken_201708120830 ID 323 gen 582 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 ID 349 gen 577 top level 5 path rw_201708080830 ID 350 gen 578 top level 5 path rw_201708090830 ID 351 gen 579 top level 5 path rw_201708100830 ID 352 gen 580 top level 5 path rw_201708110830 ID 353 gen 581 top level 5 path rw_201708120830 ID 354 gen 582 top level 5 path rw_201708130830 [root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep "3ad0334a-4063-654c-add6-b1cbcdeaa639" ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708070830 ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708080830 ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708090830 ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708100830 ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708110830 ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708120830 ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708130830 [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708070830 mysql_201708070830 Create a readonly snapshot of 'rw_201708070830' in './mysql_201708070830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708080830 mysql_201708080830 Create a readonly snapshot of 'rw_201708080830' in './mysql_201708080830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708090830 mysql_201708090830 Create a readonly snapshot of 'rw_201708090830' in './mysql_201708090830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708100830 mysql_201708100830 Create a readonly snapshot of 'rw_201708100830' in './mysql_201708100830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708110830 mysql_201708110830 Create a readonly snapshot of 'rw_201708110830' in './mysql_201708110830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708120830 mysql_201708120830 Create a readonly snapshot of 'rw_201708120830' in './mysql_201708120830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708130830 mysql_201708130830 Create a readonly snapshot of 'rw_201708130830' in './mysql_201708130830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 577 top level 5 path mbroken_201708080830 ID 319 gen 578 top level 5 path mbroken_201708090830 ID 320 gen 579 top level 5 path mbroken_201708100830 ID 321 gen 580 top level 5 path mbroken_201708110830 ID 322 gen 581 top level 5 path mbroken_201708120830 ID 323 gen 582 top level 5 path mbroken_201708130830 ID 324 gen 584 top level 5 path mysql ID 348 gen 583 top level 5 path rw_201708070830 ID 349 gen 584 top level 5 path rw_201708080830 ID 350 gen 585 top level 5 path rw_201708090830 ID 351 gen 586 top level 5 path rw_201708100830 ID 352 gen 587 top level 5 path rw_201708110830 ID 353 gen 588 top level 5 path rw_201708120830 ID 354 gen 589 top level 5 path rw_201708130830 ID 355 gen 583 top level 5 path mysql_201708070830 ID 356 gen 584 top level 5 path mysql_201708080830 ID 357 gen 585 top level 5 path mysql_201708090830 ID 358 gen 586 top level 5 path mysql_201708100830 ID 359 gen 587 top level 5 path mysql_201708110830 ID 360 gen 588 top level 5 path mysql_201708120830 ID 361 gen 589 top level 5 path mysql_201708130830 [root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep "3ad0334a-4063-654c-add6-b1cbcdeaa639" ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708070830 ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708080830 ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708090830 ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708100830 ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708110830 ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708120830 ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708130830 [root@backuplogC7 mariadb]# btrfs send /var/lib/mariadb/mysql_201708070830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708070830 At subvol mysql_201708070830 [root@backuplogC7 mariadb]# btrfs sub show mysql_201708070830 /var/lib/mariadb/mysql_201708070830 Name: mysql_201708070830 UUID: 70ee3c31-126d-574a-814c-e3b4c81b414e Parent UUID: 1d5bb8eb-b0df-2549-8b62-552cfa517609 Received UUID: - Creation time: 2017-08-14 07:00:08 +0700 Subvolume ID: 355 Generation: 583 Gen at creation: 583 Parent ID: 5 Top level ID: 5 Flags: readonly Snapshot(s): [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708070830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 1083.43 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708070830 /var/lib/mariadb/mysql_201708080830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708080830 At snapshot mysql_201708080830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708080830/ sending incremental file list ./ sent 3769 bytes received 19 bytes 688.73 bytes/sec total size is 718361496 speedup is 189641.37 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708080830 /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708090830 At snapshot mysql_201708090830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 583.38 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708090830 /var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708100830 At snapshot mysql_201708100830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708100830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708100830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708100830 /var/lib/mariadb/mysql_201708110830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708110830 At snapshot mysql_201708110830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708110830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708110830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708110830 /var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708120830 At snapshot mysql_201708120830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708120830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708120830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708120830 /var/lib/mariadb/mysql_201708130830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708130830 At snapshot mysql_201708130830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708130830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708130830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# > > > ---- From: siranee.ja@tpc.co.th -- Sent: 2017-08-13 - 16:00 ---- > >> Hi "A L", >> >> As your suggestion. Does it mean I have to make all my current subvolumes to clear >> "Receive UUID" and send them with full snapshots? > > You need to clear all subvolumes with 'Received UUID' in order to use send-receive > safely. once you have done that you can start over with new incremental sends (after > the initial full send). > > I suggest to remove any existing snapshots of those volumes to avoid confusion. > >>>> mysql >>>> mysql_201708060830 >>>> mysql_201708070830 >>>> mysql_201708080830 >>>> mysql_201708090830 >>>> mysql_201708100830 >>>> mysql_201708110830 >>>> mysql_201708120830 >>>> mysql_201708130830 >> >> The next day that I take snapshot with the script will be >> mysql_201708140830 >> >> which will be sent with incremental from mysql_201708130830 to mysql_201708140830 >> >> Is this correct? > > yes, but your current snapshots can't be used because of the 'Received UUID'. >> >> I am not understand with the url that you gave me which said > > btrbk is a backup tool for btrfs. It detects the problem of the 'Received UUID'. The > FAQ explains how to remove the 'Received UUID' from the source volumes and > snapshots. > > It is safer to remove all existing snapshots then trying to fix them and risking > the integrity of the data. > > >> >> "I'm getting an error: Aborted: "Received UUID" is set >> >> You probably restored a backup with send-receive, and made it read/write using >> btrfs >> property set. This is bad, as all snapshots and backups will inherit this >> identical >> "Received UUID", which results in all these subvolumes will be treated as >> "containing same data". >> >> To fix this, create a "proper" snapshot: >> >> - This is as your suggestion for the subvolume "mysql" >> >> # cd /mnt/btr_pool >> # mv mysubvolume mysubvolume.broken >> # btrfs subvolume snapshot mysubvolume.broken mysubvolume >> >> Now, mysubvolume should have an empty "Received UUID". Note that in order to have >> a >> clean environment, you also need to fix all subvolumes (snapshots as well as >> backups) that you created with the broken subvolume. >> >> Check if there are more broken subvolumes: >> >> # btrfs subvolume show mysubvolume.broken >> # btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above> >> # btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above> >> >> - This guide seem that I have to clear <"Received UUID" > only the subvolume >> "mysql" >> and the others ("mysql_201708070830" should using btrfs subvolume snapshot -r >> instead of btrfs subvolume snapshot. Is this correct? >> >> Now clean all subvolume listed (same as above, but using btrfs subvolume snapshot >> -r >> now). Then delete all the broken subvolumes: >> >> # btrfs subvolume delete *.broken >> >> Finally, you should have a clean environment, and btrbk will not complain any >> more. >> >> >> Best Regards, >> Siranee Jaraswachirakul. >> >>> >>> >>> On 8/13/2017 12:52 PM, siranee.ja@tpc.co.th wrote: >>>> Hi "A L", >>>> >>>> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql >>>> /var/lib/mariadb/mysql >>>> Name: mysql >>>> UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 >>>> Parent UUID: - >>>> Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 >>>> Creation time: 2017-06-21 13:27:41 +0700 >>>> Subvolume ID: 257 >>>> Generation: 539 >>>> Gen at creation: 9 >>>> Parent ID: 5 >>>> Top level ID: 5 >>>> Flags: - >>>> Snapshot(s): >>>> mysql_201708060830 >>>> mysql_201708070830 >>>> mysql_201708080830 >>>> mysql_201708090830 >>>> mysql_201708100830 >>>> mysql_201708110830 >>>> mysql_201708120830 >>>> mysql_201708130830 >>>> >>>> yes I think it has Received UUID because I restored the source from snapshot >>>> mysql_201708040830 for prove that the local snapshot was work. >>>> >>>> How to clear the Received UUID ? >>>> What to do next? >>> You need to make a read-write snapshot of? /var/lib/mariadb/mysql and >>> then remove the old subvolume and all its snapshots. >>> >>> Example from https://github.com/digint/btrbk/blob/master/doc/FAQ.md >>> >>> # cd /mnt/btr_pool >>> # mv mysubvolume mysubvolume.broken >>> # btrfs subvolume snapshot mysubvolume.broken mysubvolume >>> >>> You can do the same with each of your snapshots too, and send them as >>> full snapshots (without -p). >>> >>> ~A >>> >>>> Best Regards, >>>> Siranee Jaraswachirakul. >>>> >>>>> Have you checked that there is no Received UUID on the source subvolume? >>>>> >>>>> # btrfs sub show volume/mysql/ >>>>> volume/mysql >>>>> ??????? Name:?????????????????? mysql >>>>> ??????? UUID:?????????????????? 8a94524e-a956-c14b-bb8d-d453627f27d5 >>>>> ??????? Parent UUID:??????????? - >>>>> ??????? Received UUID:????????? - >>>>> ??????? Creation time:????????? 2017-04-17 11:46:20 +0200 >>>>> ??????? Subvolume ID:?????????? 1469 >>>>> ??????? Generation:???????????? 122934 >>>>> ??????? Gen at creation:??????? 78671 >>>>> ??????? Parent ID:????????????? 5 >>>>> ??????? Top level ID:?????????? 5 >>>>> ??????? Flags:????????????????? - >>>>> ??????? Snapshot(s): >>>>> >>>>> There is no Received UUID here. If it has, then btrfs send-receive will >>>>> have problems, since all snapshots of the source subvolume will have the >>>>> same Received UUID and it can't tell the differences between the snapshots. >>>>> >>>>> On 8/13/2017 5:40 AM, siranee.ja@tpc.co.th wrote: >>>>>> Hi Chris, >>>>>> >>>>>> I started as your suggestion again. The diff occured since snapshot >>>>>> mysql_201708090830 manually send. What should I do next? >>>>>> >>>>>> - delete all the bad/mismatching snapshots only on the destination computer. >>>>>> [root@joytest ~]# date >>>>>> Sun Aug 13 10:27:23 ICT 2017 >>>>>> [root@joytest ~]# cd /var/lib/mariadb >>>>>> [root@joytest mariadb]# btrfs sub list . >>>>>> ID 313 gen 220 top level 5 path mysql_201708070830 >>>>>> ID 316 gen 199 top level 5 path mysql_201708080830 >>>>>> ID 318 gen 205 top level 5 path mysql_201708090830 >>>>>> ID 320 gen 211 top level 5 path mysql_201708100830 >>>>>> ID 322 gen 219 top level 5 path mysql_201708110830 >>>>>> ID 323 gen 219 top level 5 path mysql_201708120830 >>>>>> ID 324 gen 224 top level 5 path mysql_201708130830 >>>>>> ID 325 gen 225 top level 5 path mysql >>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708130830 >>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830' >>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708120830 >>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830' >>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708110830 >>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830' >>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708100830 >>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830' >>>>>> [root@joytest mariadb]# btrfs sub del mysql_201708090830 >>>>>> Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830' >>>>>> [root@joytest mariadb]# btrfs sub sync . >>>>>> [root@joytest mariadb]# systemctl status mariadb >>>>>> â? mariadb.service - MariaDB database server >>>>>> Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor >>>>>> preset: >>>>>> disabled) >>>>>> Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h >>>>>> 24min >>>>>> ago >>>>>> Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID >>>>>> (code=exited, status=1/FAILURE) >>>>>> Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, >>>>>> status=0/SUCCESS) >>>>>> Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n >>>>>> (code=exited, >>>>>> status=0/SUCCESS) >>>>>> Main PID: 19870 (code=exited, status=0/SUCCESS) >>>>>> >>>>>> Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server... >>>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe >>>>>> Logging >>>>>> to >>>>>> '/var/log/mariadb/mariadb.log'. >>>>>> Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe >>>>>> Starting >>>>>> mysqld daemon with databases from /var/lib/mariadb/mysql >>>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited, >>>>>> code=exited status=1 >>>>>> Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server. >>>>>> Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state. >>>>>> Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed. >>>>>> [root@joytest mariadb]# btrfs sub list . >>>>>> ID 313 gen 220 top level 5 path mysql_201708070830 >>>>>> ID 316 gen 199 top level 5 path mysql_201708080830 >>>>>> ID 325 gen 225 top level 5 path mysql >>>>>> [root@joytest mariadb]# >>>>>> >>>>>> - The most recent good snapshot pair, which rsync shows origin and >>>>>> destination match, is mysql_201708080830 so you can keep that one on >>>>>> both sides. >>>>>> >>>>>> [root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb >>>>>> ID 257 gen 538 top level 5 path mysql >>>>>> ID 316 gen 498 top level 5 path mysql_201708060830 >>>>>> ID 317 gen 503 top level 5 path mysql_201708070830 >>>>>> ID 318 gen 507 top level 5 path mysql_201708080830 >>>>>> ID 319 gen 514 top level 5 path mysql_201708090830 >>>>>> ID 320 gen 524 top level 5 path mysql_201708100830 >>>>>> ID 321 gen 529 top level 5 path mysql_201708110830 >>>>>> ID 322 gen 533 top level 5 path mysql_201708120830 >>>>>> ID 323 gen 538 top level 5 path mysql_201708130830 >>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ >>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708070830/ >>>>>> sending incremental file list >>>>>> ./ >>>>>> >>>>>> sent 3773 bytes received 19 bytes 842.67 bytes/sec >>>>>> total size is 718361496 speedup is 189441.32 (DRY RUN) >>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ >>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708080830/ >>>>>> sending incremental file list >>>>>> ./ >>>>>> >>>>>> sent 3769 bytes received 19 bytes 841.78 bytes/sec >>>>>> total size is 718361496 speedup is 189641.37 (DRY RUN) >>>>>> [root@backuplogC7 ~]# date >>>>>> Sun Aug 13 10:34:05 ICT 2017 >>>>>> [root@backuplogC7 ~]# >>>>>> >>>>>> - manually do incremental send/receive, starting with >>>>>> mysql_201708090830/, to make the destination current again with the >>>>>> origin. >>>>>> >>>>>> [root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830 >>>>>> /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive >>>>>> /var/lib/mariadb >>>>>> At subvol /var/lib/mariadb/mysql_201708090830 >>>>>> At snapshot mysql_201708090830 >>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ >>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830/ >>>>>> sending incremental file list >>>>>> ./ >>>>>> ib_logfile1 >>>>>> ibdata1 >>>>>> >>>>>> sent 3779 bytes received 25 bytes 507.20 bytes/sec >>>>>> total size is 718361496 speedup is 188843.72 (DRY RUN) >>>>>> [root@backuplogC7 ~]# >>>>>> >>>>>> Best Regards, >>>>>> >>>>>> Siranee Jaraswachirakul. >>>>>> >>>>>>> On Sat, Aug 12, 2017 at 8:20 PM, <siranee.ja@tpc.co.th> wrote: >>>>>>> >>>>>>>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830 >>>>>>>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830 >>>>>>> You need trailing / for the first directory with -a option. >>>>>>> >>>>>>> rsync -a dir dir >>>>>>> >>>>>>> is not the same command as >>>>>>> >>>>>>> rsync -a dir/ dir >>>>>>> >>>>>>> It's confusing but your command is trying to create mysql_201708090830 >>>>>>> directory on the source, in the mysql_201708090830 on the destination. >>>>>>> That is why everything mismatches. To make it mean "contents of" you >>>>>>> need trailing slash on at least the origin. >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Chris Murphy >>>>>>> >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> >>>> >>> >>> >> >> > > > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 10:52 ` siranee.ja 2017-08-13 12:51 ` A L @ 2017-08-13 19:50 ` Chris Murphy 2017-08-14 2:04 ` siranee.ja 1 sibling, 1 reply; 31+ messages in thread From: Chris Murphy @ 2017-08-13 19:50 UTC (permalink / raw) To: siranee.ja; +Cc: A L, Btrfs BTRFS On Sun, Aug 13, 2017 at 4:52 AM, <siranee.ja@tpc.co.th> wrote: > Hi "A L", > > [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql > /var/lib/mariadb/mysql > Name: mysql > UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 > Parent UUID: - > Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 > Creation time: 2017-06-21 13:27:41 +0700 > Subvolume ID: 257 > Generation: 539 > Gen at creation: 9 > Parent ID: 5 > Top level ID: 5 > Flags: - > Snapshot(s): > mysql_201708060830 > mysql_201708070830 > mysql_201708080830 > mysql_201708090830 > mysql_201708100830 > mysql_201708110830 > mysql_201708120830 > mysql_201708130830 > > yes I think it has Received UUID because I restored the source from snapshot > mysql_201708040830 for prove that the local snapshot was work. > > How to clear the Received UUID ? > What to do next? I'm using btrfs-progs 4.12, and I just did a btrfs send/receive from file system A to B; and then on B I took a rw snapshot of the ro snapshot. The ro snapshot has Received UUID, but the rw snapshot made from it does not have Received UUID. So now I'm confused how you have a rw snapshot with Received UUID set. Did you ever use 'btrfs property set' to change a ro snapshot to rw? That's the only way I can think it's possible. The only other thing is maybe the behavior has changed since btrfs-progs 4.4. It is possible this behavior has changed since btrfs-progs 4.4. The changelogs shows there are improvements in send/receive in particular 4.8.3 and 4.8.4, but also others. I have no idea which are related. But anyway, this is one of the reasons why the expert users on this almost always say use something newer because it's just too hard to remember, even with changelogs, what's fixed and where. My suggestion is to investigate moving to kernel 4.9 series; and it should be safe to move to btrfs-progs 4.12 now. Any new features in 4.12 that can't be supported with an older kernel should fail gracefully. No matter what, you must have separate backups. You can't depend exclusively on one backup, no matter the method. It's fine to use send/receive for backup1. But backup2 should be a conventional backup, e.g. XFS with rsync. Basically this is a game of risk and the more independent backups you have, the more failure cases (especially user error) you can recover from, at the expense of some complexity. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-13 19:50 ` Chris Murphy @ 2017-08-14 2:04 ` siranee.ja 0 siblings, 0 replies; 31+ messages in thread From: siranee.ja @ 2017-08-14 2:04 UTC (permalink / raw) To: Chris Murphy; +Cc: A L, Btrfs BTRFS Hi Chris, Thank you very much for your suggestion. I didn't remember which steps that I made a mistake and made the mysql had Received UUID. I have done the following and it work as it should be right now. [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708070830 rw_201708070830 Create a snapshot of 'mbroken_201708070830' in './rw_201708070830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 568 top level 5 path mbroken_201708080830 ID 319 gen 569 top level 5 path mbroken_201708090830 ID 320 gen 570 top level 5 path mbroken_201708100830 ID 321 gen 571 top level 5 path mbroken_201708110830 ID 322 gen 572 top level 5 path mbroken_201708120830 ID 323 gen 573 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708080830 rw_201708080830 Create a snapshot of 'mbroken_201708080830' in './rw_201708080830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708090830 rw_201708090830 Create a snapshot of 'mbroken_201708090830' in './rw_201708090830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708100830 rw_201708100830 Create a snapshot of 'mbroken_201708100830' in './rw_201708100830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708110830 rw_201708110830 Create a snapshot of 'mbroken_201708110830' in './rw_201708110830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708120830 rw_201708120830 Create a snapshot of 'mbroken_201708120830' in './rw_201708120830' [root@backuplogC7 mariadb]# btrfs subvolume snapshot mbroken_201708130830 rw_201708130830 Create a snapshot of 'mbroken_201708130830' in './rw_201708130830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 577 top level 5 path mbroken_201708080830 ID 319 gen 578 top level 5 path mbroken_201708090830 ID 320 gen 579 top level 5 path mbroken_201708100830 ID 321 gen 580 top level 5 path mbroken_201708110830 ID 322 gen 581 top level 5 path mbroken_201708120830 ID 323 gen 582 top level 5 path mbroken_201708130830 ID 324 gen 543 top level 5 path mysql ID 348 gen 576 top level 5 path rw_201708070830 ID 349 gen 577 top level 5 path rw_201708080830 ID 350 gen 578 top level 5 path rw_201708090830 ID 351 gen 579 top level 5 path rw_201708100830 ID 352 gen 580 top level 5 path rw_201708110830 ID 353 gen 581 top level 5 path rw_201708120830 ID 354 gen 582 top level 5 path rw_201708130830 [root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep "3ad0334a-4063-654c-add6-b1cbcdeaa639" ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708070830 ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708080830 ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708090830 ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708100830 ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708110830 ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708120830 ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708130830 [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708070830 mysql_201708070830 Create a readonly snapshot of 'rw_201708070830' in './mysql_201708070830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708080830 mysql_201708080830 Create a readonly snapshot of 'rw_201708080830' in './mysql_201708080830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708090830 mysql_201708090830 Create a readonly snapshot of 'rw_201708090830' in './mysql_201708090830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708100830 mysql_201708100830 Create a readonly snapshot of 'rw_201708100830' in './mysql_201708100830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708110830 mysql_201708110830 Create a readonly snapshot of 'rw_201708110830' in './mysql_201708110830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708120830 mysql_201708120830 Create a readonly snapshot of 'rw_201708120830' in './mysql_201708120830' [root@backuplogC7 mariadb]# btrfs sub snap -r rw_201708130830 mysql_201708130830 Create a readonly snapshot of 'rw_201708130830' in './mysql_201708130830' [root@backuplogC7 mariadb]# btrfs sub list . ID 257 gen 542 top level 5 path mbroken ID 317 gen 576 top level 5 path mbroken_201708070830 ID 318 gen 577 top level 5 path mbroken_201708080830 ID 319 gen 578 top level 5 path mbroken_201708090830 ID 320 gen 579 top level 5 path mbroken_201708100830 ID 321 gen 580 top level 5 path mbroken_201708110830 ID 322 gen 581 top level 5 path mbroken_201708120830 ID 323 gen 582 top level 5 path mbroken_201708130830 ID 324 gen 584 top level 5 path mysql ID 348 gen 583 top level 5 path rw_201708070830 ID 349 gen 584 top level 5 path rw_201708080830 ID 350 gen 585 top level 5 path rw_201708090830 ID 351 gen 586 top level 5 path rw_201708100830 ID 352 gen 587 top level 5 path rw_201708110830 ID 353 gen 588 top level 5 path rw_201708120830 ID 354 gen 589 top level 5 path rw_201708130830 ID 355 gen 583 top level 5 path mysql_201708070830 ID 356 gen 584 top level 5 path mysql_201708080830 ID 357 gen 585 top level 5 path mysql_201708090830 ID 358 gen 586 top level 5 path mysql_201708100830 ID 359 gen 587 top level 5 path mysql_201708110830 ID 360 gen 588 top level 5 path mysql_201708120830 ID 361 gen 589 top level 5 path mysql_201708130830 [root@backuplogC7 mariadb]# btrfs subvolume list -a -R . | grep "3ad0334a-4063-654c-add6-b1cbcdeaa639" ID 257 gen 542 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken ID 317 gen 576 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708070830 ID 318 gen 577 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708080830 ID 319 gen 578 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708090830 ID 320 gen 579 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708100830 ID 321 gen 580 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708110830 ID 322 gen 581 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708120830 ID 323 gen 582 top level 5 received_uuid 3ad0334a-4063-654c-add6-b1cbcdeaa639 path mbroken_201708130830 [root@backuplogC7 mariadb]# btrfs send /var/lib/mariadb/mysql_201708070830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708070830 At subvol mysql_201708070830 [root@backuplogC7 mariadb]# btrfs sub show mysql_201708070830 /var/lib/mariadb/mysql_201708070830 Name: mysql_201708070830 UUID: 70ee3c31-126d-574a-814c-e3b4c81b414e Parent UUID: 1d5bb8eb-b0df-2549-8b62-552cfa517609 Received UUID: - Creation time: 2017-08-14 07:00:08 +0700 Subvolume ID: 355 Generation: 583 Gen at creation: 583 Parent ID: 5 Top level ID: 5 Flags: readonly Snapshot(s): [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708070830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708070830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 1083.43 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708070830 /var/lib/mariadb/mysql_201708080830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708080830 At snapshot mysql_201708080830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708080830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708080830/ sending incremental file list ./ sent 3769 bytes received 19 bytes 688.73 bytes/sec total size is 718361496 speedup is 189641.37 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708080830 /var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708090830 At snapshot mysql_201708090830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708090830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708090830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 583.38 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708090830 /var/lib/mariadb/mysql_201708100830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708100830 At snapshot mysql_201708100830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708100830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708100830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708100830 /var/lib/mariadb/mysql_201708110830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708110830 At snapshot mysql_201708110830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708110830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708110830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708110830 /var/lib/mariadb/mysql_201708120830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708120830 At snapshot mysql_201708120830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708120830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708120830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# btrfs send -p /var/lib/mariadb/mysql_201708120830 /var/lib/mariadb/mysql_201708130830 | ssh 192.168.45.166 btrfs receive /var/lib/mariadb At subvol /var/lib/mariadb/mysql_201708130830 At snapshot mysql_201708130830 [root@backuplogC7 mariadb]# rsync -avnc /var/lib/mariadb/mysql_201708130830/ root@192.168.45.166:/var/lib/mariadb/mysql_201708130830/ sending incremental file list ./ sent 3773 bytes received 19 bytes 689.45 bytes/sec total size is 718361496 speedup is 189441.32 (DRY RUN) [root@backuplogC7 mariadb]# Best Regards, Siranee Jaraswachirakul. > On Sun, Aug 13, 2017 at 4:52 AM, <siranee.ja@tpc.co.th> wrote: >> Hi "A L", >> >> [root@backuplogC7 ~]# btrfs sub show /var/lib/mariadb/mysql >> /var/lib/mariadb/mysql >> Name: mysql >> UUID: 92f319c5-e132-3249-9b13-d39ee77a2b44 >> Parent UUID: - >> Received UUID: 3ad0334a-4063-654c-add6-b1cbcdeaa639 >> Creation time: 2017-06-21 13:27:41 +0700 >> Subvolume ID: 257 >> Generation: 539 >> Gen at creation: 9 >> Parent ID: 5 >> Top level ID: 5 >> Flags: - >> Snapshot(s): >> mysql_201708060830 >> mysql_201708070830 >> mysql_201708080830 >> mysql_201708090830 >> mysql_201708100830 >> mysql_201708110830 >> mysql_201708120830 >> mysql_201708130830 >> >> yes I think it has Received UUID because I restored the source from snapshot >> mysql_201708040830 for prove that the local snapshot was work. >> >> How to clear the Received UUID ? >> What to do next? > > > I'm using btrfs-progs 4.12, and I just did a btrfs send/receive from > file system A to B; and then on B I took a rw snapshot of the ro > snapshot. The ro snapshot has Received UUID, but the rw snapshot made > from it does not have Received UUID. So now I'm confused how you have > a rw snapshot with Received UUID set. > > > Did you ever use 'btrfs property set' to change a ro snapshot to rw? > That's the only way I can think it's possible. The only other thing is > maybe the behavior has changed since btrfs-progs 4.4. > > It is possible this behavior has changed since btrfs-progs 4.4. The > changelogs shows there are improvements in send/receive in particular > 4.8.3 and 4.8.4, but also others. I have no idea which are related. > But anyway, this is one of the reasons why the expert users on this > almost always say use something newer because it's just too hard to > remember, even with changelogs, what's fixed and where. > > My suggestion is to investigate moving to kernel 4.9 series; and it > should be safe to move to btrfs-progs 4.12 now. Any new features in > 4.12 that can't be supported with an older kernel should fail > gracefully. > > No matter what, you must have separate backups. You can't depend > exclusively on one backup, no matter the method. It's fine to use > send/receive for backup1. But backup2 should be a conventional backup, > e.g. XFS with rsync. Basically this is a game of risk and the more > independent backups you have, the more failure cases (especially user > error) you can recover from, at the expense of some complexity. > > > > -- > Chris Murphy > ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: btrfs issue with mariadb incremental backup 2017-08-11 4:40 ` siranee.ja 2017-08-11 6:00 ` siranee.ja @ 2017-08-11 15:06 ` Chris Murphy 1 sibling, 0 replies; 31+ messages in thread From: Chris Murphy @ 2017-08-11 15:06 UTC (permalink / raw) To: siranee.ja; +Cc: Chris Murphy, Btrfs BTRFS, voravat On Thu, Aug 10, 2017 at 10:40 PM, <siranee.ja@tpc.co.th> wrote: > Hi Chris, > The kernel version that I test is "4.4.0-89-generic" as I tested on ubuntu lxd If I > want to change the kernel version I have to upgrade the host box. I can't parse what kernel that really is compared to upstream, which is at 4.4.80. > As you suggest the rsync to compare the subvolumes. I found the point. > the subvolumes are different only after I start to del old subvolumes on machine A > the steps are You're saying a read only snapshot's contents are changing after deleting other (child) snapshots? If you have a subvolume "subvolume" and you make read only snapshots of it over time "snapshot1" "snapshot2" "snapshot3" "snapshot4" "snapshot5" "snapshot6" on both machine A and machine B. And rsync -anc machineA/snapshot6 machineB/snapshot6 they are identical. And then you delete "snapshot1" "snapshot2" "snapshot3" "snapshot4" on machine A. And rsync -anc machineA/snapshot6 machineB/snapshot6 they are no longer identical? I've never heard of this before. > > 30 08 * * * root /root/script/backup/backupsnap.sh root password > /var/lib/mariadb/mysql >> /var/log/btrfs_snap.log > 05 09 * * * root /root/script/backupbtrfs_inc.sh /var/lib/mariadb 192.168.45.166 > /var/lib/mariadb >> /var/log/btrfs_send.log > 30 19 * * * root /root/script/delete_btrfs_sub_snap_volume.sh /var/lib/mariadb 7 >> > /var/log/btrfs_del.log I have no idea what this means or why it's relevant. > > > The following script maintain snapshot to currently only 7 snapshots. > [root@backuplogC7 ~]# cat /root/script/delete_btrfs_sub_snap_volume.sh > basepath=$1 > keepcount=$2 > havecount=`btrfs sub list -s ${basepath}|cut -d' ' -f14|wc -l` > delcount=$[$keepcount-$havecount]; > datet=$(date +%Y%m%d%H%M) > echo "Start Delete ${datet}" > if [ $delcount -lt 0 ]; then > # list only snapshot subvolume > for i in `btrfs sub list -s ${basepath}|cut -d' ' -f14|head ${delcount}` > do > echo "btrfs sub delete ${basepath}/$i" > btrfs sub delete ${basepath}/$i > btrfs sub sync ${basepath} Seems sane. > Does it mean my delete script is not the properly way of the btrfs purge old > snapshot on source? I don't think so. But I still don't understand the problem, and what exactly has changed and when, and it might be my confusion. But if you have an ro snapshot that's correct at one moment, but then changes when some other (child) snapshots are deleted, that's pretty remarkable so I have to assume I'm confused. -- Chris Murphy ^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2017-08-14 2:04 UTC | newest] Thread overview: 31+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-08-09 4:32 btrfs issue with mariadb incremental backup siranee.ja 2017-08-09 5:46 ` Chris Murphy 2017-08-09 6:36 ` siranee.ja 2017-08-09 17:59 ` Chris Murphy 2017-08-10 2:03 ` siranee.ja 2017-08-11 4:40 ` siranee.ja 2017-08-11 6:00 ` siranee.ja 2017-08-11 15:35 ` Chris Murphy 2017-08-12 2:38 ` siranee.ja 2017-08-12 4:31 ` Chris Murphy 2017-08-12 5:08 ` siranee.ja 2017-08-12 21:34 ` Chris Murphy 2017-08-12 22:41 ` Janos Toth F. 2017-08-12 23:07 ` Chris Murphy 2017-08-12 22:49 ` Hugo Mills 2017-08-12 23:14 ` Chris Murphy 2017-08-13 2:20 ` siranee.ja 2017-08-13 2:59 ` Chris Murphy 2017-08-13 3:40 ` siranee.ja 2017-08-13 4:34 ` Chris Murphy 2017-08-13 10:49 ` siranee.ja 2017-08-13 19:31 ` Chris Murphy 2017-08-13 6:20 ` A L 2017-08-13 10:52 ` siranee.ja 2017-08-13 12:51 ` A L 2017-08-13 14:00 ` siranee.ja 2017-08-13 21:31 ` A L 2017-08-14 1:57 ` siranee.ja 2017-08-13 19:50 ` Chris Murphy 2017-08-14 2:04 ` siranee.ja 2017-08-11 15:06 ` Chris Murphy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).