Re: btrfs issue with mariadb incremental backup

[siranee.ja@tpc.co.th]

Hi Chris,

I started as your suggestion again. The diff occured since snapshot
mysql_201708090830 manually send. What should I do next?

- delete all the bad/mismatching snapshots only on the destination computer.
[root@joytest ~]# date
Sun Aug 13 10:27:23 ICT 2017
[root@joytest ~]# cd /var/lib/mariadb
[root@joytest mariadb]# btrfs sub list .
ID 313 gen 220 top level 5 path mysql_201708070830
ID 316 gen 199 top level 5 path mysql_201708080830
ID 318 gen 205 top level 5 path mysql_201708090830
ID 320 gen 211 top level 5 path mysql_201708100830
ID 322 gen 219 top level 5 path mysql_201708110830
ID 323 gen 219 top level 5 path mysql_201708120830
ID 324 gen 224 top level 5 path mysql_201708130830
ID 325 gen 225 top level 5 path mysql
[root@joytest mariadb]# btrfs sub del mysql_201708130830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708130830'
[root@joytest mariadb]# btrfs sub del mysql_201708120830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708120830'
[root@joytest mariadb]# btrfs sub del mysql_201708110830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708110830'
[root@joytest mariadb]# btrfs sub del mysql_201708100830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708100830'
[root@joytest mariadb]# btrfs sub del mysql_201708090830
Delete subvolume (no-commit): '/var/lib/mariadb/mysql_201708090830'
[root@joytest mariadb]# btrfs sub sync .
[root@joytest mariadb]# systemctl status mariadb
â— mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset:
disabled)
   Active: failed (Result: exit-code) since Sun 2017-08-13 09:07:00 ICT; 1h 24min ago
  Process: 19871 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID
(code=exited, status=1/FAILURE)
  Process: 19870 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited,
status=0/SUCCESS)
  Process: 19842 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited,
status=0/SUCCESS)
 Main PID: 19870 (code=exited, status=0/SUCCESS)

Aug 13 09:06:58 joytest systemd[1]: Starting MariaDB database server...
Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Logging to
'/var/log/mariadb/mariadb.log'.
Aug 13 09:06:58 joytest mysqld_safe[19870]: 170813 09:06:58 mysqld_safe Starting
mysqld daemon with databases from /var/lib/mariadb/mysql
Aug 13 09:07:00 joytest systemd[1]: mariadb.service: control process exited,
code=exited status=1
Aug 13 09:07:00 joytest systemd[1]: Failed to start MariaDB database server.
Aug 13 09:07:00 joytest systemd[1]: Unit mariadb.service entered failed state.
Aug 13 09:07:00 joytest systemd[1]: mariadb.service failed.
[root@joytest mariadb]# btrfs sub list .
ID 313 gen 220 top level 5 path mysql_201708070830
ID 316 gen 199 top level 5 path mysql_201708080830
ID 325 gen 225 top level 5 path mysql
[root@joytest mariadb]#

- The most recent good snapshot pair, which rsync shows origin and
destination match, is mysql_201708080830 so you can keep that one on
both sides.

[root@backuplogC7 ~]# btrfs sub list /var/lib/mariadb
ID 257 gen 538 top level 5 path mysql
ID 316 gen 498 top level 5 path mysql_201708060830
ID 317 gen 503 top level 5 path mysql_201708070830
ID 318 gen 507 top level 5 path mysql_201708080830
ID 319 gen 514 top level 5 path mysql_201708090830
ID 320 gen 524 top level 5 path mysql_201708100830
ID 321 gen 529 top level 5 path mysql_201708110830
ID 322 gen 533 top level 5 path mysql_201708120830
ID 323 gen 538 top level 5 path mysql_201708130830
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708070830/
root@192.168.45.166://var/lib/mariadb/mysql_201708070830/
sending incremental file list
./

sent 3773 bytes  received 19 bytes  842.67 bytes/sec
total size is 718361496  speedup is 189441.32 (DRY RUN)
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708080830/
root@192.168.45.166://var/lib/mariadb/mysql_201708080830/
sending incremental file list
./

sent 3769 bytes  received 19 bytes  841.78 bytes/sec
total size is 718361496  speedup is 189641.37 (DRY RUN)
[root@backuplogC7 ~]# date
Sun Aug 13 10:34:05 ICT 2017
[root@backuplogC7 ~]#

- manually do incremental send/receive, starting with
mysql_201708090830/, to make the destination current again with the
origin.

[root@backuplogC7 ~]# btrfs send -p /var/lib/mariadb/mysql_201708080830
/var/lib/mariadb/mysql_201708090830 | ssh 192.168.45.166 btrfs receive
/var/lib/mariadb
At subvol /var/lib/mariadb/mysql_201708090830
At snapshot mysql_201708090830
[root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830/
root@192.168.45.166://var/lib/mariadb/mysql_201708090830/
sending incremental file list
./
ib_logfile1
ibdata1

sent 3779 bytes  received 25 bytes  507.20 bytes/sec
total size is 718361496  speedup is 188843.72 (DRY RUN)
[root@backuplogC7 ~]#

Best Regards,

Siranee Jaraswachirakul.

> On Sat, Aug 12, 2017 at 8:20 PM,  <siranee.ja@tpc.co.th> wrote:
>
>> [root@backuplogC7 ~]# rsync -avnc /var/lib/mariadb/mysql_201708090830
>> root@192.168.45.166://var/lib/mariadb/mysql_201708090830
>
>
> You need trailing / for the first directory with -a option.
>
> rsync -a dir dir
>
> is not the same command as
>
> rsync -a dir/ dir
>
> It's confusing but your command is trying to create mysql_201708090830
> directory on the source, in the mysql_201708090830 on the destination.
> That is why everything mismatches. To make it mean "contents of" you
> need trailing slash on at least the origin.
>
>
>
> --
> Chris Murphy
>