public inbox for linux-xfs@vger.kernel.org
 help / color / mirror / Atom feed
* xfsrestore report SUCCESS but not restore all files in kernel 3.17
@ 2014-10-29  4:06 Tommy Wu
  2014-10-29  4:09 ` Eric Sandeen
  2014-10-29 21:25 ` Dave Chinner
  0 siblings, 2 replies; 30+ messages in thread
From: Tommy Wu @ 2014-10-29  4:06 UTC (permalink / raw)
  To: xfs


[-- Attachment #1.1: Type: text/plain, Size: 4144 bytes --]

kernel 3.17.0 (same command in kernel 3.16.x work fine)
xfsdump 3.1.4
xfsprogs 3.2.1

test case:
1. create a new filesystem in LVM

lvcreate -L 5G -n test vg
mkfs.xfs -f /dev/vg/test
mount /dev/vg/test /mnt/x

2. extract linux-3.17.tar.xz to the new filesystem

cd /mnt/x
tar Jxvf /usr/src/linux-3.17.tar.xz

3. backup the filesystem

xfsdump -l 0 -o -p 300 -J -F -M test -L test - /mnt/x | /bin/gzip -qv >
/vol/test.xfsdump.gz

4. test the dump file, it show a lots of NOTE, and it said restore status
is SUCCESS.

cat /vol/test.xfsdump.gz | gzip -dqv | xfsrestore -p 300 -J -t - | grep
"^/sbin/xfsrestore:"
/sbin/xfsrestore: using file dump (drive_simple) strategy
/sbin/xfsrestore: version 3.1.4 (dump format 3.0)
/sbin/xfsrestore: searching media for dump
/sbin/xfsrestore: examining media file 0
/sbin/xfsrestore: dump description:
/sbin/xfsrestore: hostname: mail.teatime.com.tw
/sbin/xfsrestore: mount point: /mnt/x
/sbin/xfsrestore: volume: /dev/mapper/vg-test
/sbin/xfsrestore: session time: Wed Oct 29 11:45:54 2014
/sbin/xfsrestore: level: 0
/sbin/xfsrestore: session label: "test"
/sbin/xfsrestore: media label: "test"
/sbin/xfsrestore: file system id: d757277d-5167-4cdb-975d-c64b2f94a47e
/sbin/xfsrestore: session id: 6f114712-6fb6-4f35-a2f3-30ca1a0af5bb
/sbin/xfsrestore: media id: 2eb2ff54-1838-47b2-a8d5-5f520deaf5f5
/sbin/xfsrestore: searching media for directory dump
/sbin/xfsrestore: reading directories
/sbin/xfsrestore: 2035 directories and 33045 entries processed
/sbin/xfsrestore: directory post-processing
/sbin/xfsrestore: reading non-directory files
/sbin/xfsrestore: NOTE: ino 144 salvaging file, placing in
orphanage/143.0/Makefile
/sbin/xfsrestore: NOTE: ino 145 salvaging file, placing in
orphanage/143.0/bayer.png.b64
/sbin/xfsrestore: NOTE: ino 146 salvaging file, placing in
orphanage/143.0/constraints.png.b64
/sbin/xfsrestore: NOTE: ino 147 salvaging file, placing in
orphanage/143.0/crop.gif.b64
/sbin/xfsrestore: NOTE: ino 148 salvaging file, placing in
orphanage/143.0/dvbstb.png.b64
/sbin/xfsrestore: NOTE: ino 149 salvaging file, placing in
orphanage/143.0/fieldseq_bt.gif.b64
/sbin/xfsrestore: NOTE: ino 150 salvaging file, placing in
orphanage/143.0/fieldseq_tb.gif.b64
.... skip it....
/sbin/xfsrestore: NOTE: ino 50363235 gen 1622742662 not referenced: placing
in orphanage
/sbin/xfsrestore: table of contents display complete: 3 seconds elapsed
/sbin/xfsrestore: Restore Status: SUCCESS

5. restore it, show a lots of NOTE, and resotre status is SUCCESS, but...
compare them, it's different.

cat test.xfsdump.gz | gzip -dqv | xfsrestore -p 300 -J - /vol/x/
/sbin/xfsrestore: using file dump (drive_simple) strategy
/sbin/xfsrestore: version 3.1.4 (dump format 3.0)
/sbin/xfsrestore: searching media for dump
/sbin/xfsrestore: examining media file 0
/sbin/xfsrestore: dump description:
/sbin/xfsrestore: hostname: mail.teatime.com.tw
/sbin/xfsrestore: mount point: /mnt/x
/sbin/xfsrestore: volume: /dev/mapper/vg-test
/sbin/xfsrestore: session time: Wed Oct 29 11:45:54 2014
/sbin/xfsrestore: level: 0
/sbin/xfsrestore: session label: "test"
/sbin/xfsrestore: media label: "test"
/sbin/xfsrestore: file system id: d757277d-5167-4cdb-975d-c64b2f94a47e
/sbin/xfsrestore: session id: 6f114712-6fb6-4f35-a2f3-30ca1a0af5bb
/sbin/xfsrestore: media id: 2eb2ff54-1838-47b2-a8d5-5f520deaf5f5
/sbin/xfsrestore: searching media for directory dump
/sbin/xfsrestore: reading directories
/sbin/xfsrestore: 2035 directories and 33045 entries processed
/sbin/xfsrestore: directory post-processing
/sbin/xfsrestore: restoring non-directory files
/sbin/xfsrestore: NOTE: ino 144 salvaging file, placing in
orphanage/143.0/Makefile
/sbin/xfsrestore: NOTE: ino 145 salvaging file, placing in
orphanage/143.0/bayer.png.b64
/sbin/xfsrestore: NOTE: ino 146 salvaging file, placing in
orphanage/143.0/constraints.png.b64
.... skip it....
/sbin/xfsrestore: NOTE: ino 50363235 gen 1622742662 not referenced: placing
in orphanage
/sbin/xfsrestore: WARNING: unable to rmdir /vol/x//orphanage: Directory not
empty
/sbin/xfsrestore: restore complete: 3 seconds elapsed
/sbin/xfsrestore: Restore Status: SUCCESS


-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 4628 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  4:06 xfsrestore report SUCCESS but not restore all files in kernel 3.17 Tommy Wu
@ 2014-10-29  4:09 ` Eric Sandeen
  2014-10-29  4:27   ` Tommy Wu
  2014-10-29 21:25 ` Dave Chinner
  1 sibling, 1 reply; 30+ messages in thread
From: Eric Sandeen @ 2014-10-29  4:09 UTC (permalink / raw)
  To: Tommy Wu, xfs

On 10/28/14 11:06 PM, Tommy Wu wrote:
> 
> kernel 3.17.0 (same command in kernel 3.16.x work fine)
> xfsdump 3.1.4
> xfsprogs 3.2.1
> 
> test case:
> 1. create a new filesystem in LVM
> 
> lvcreate -L 5G -n test vg
> mkfs.xfs -f /dev/vg/test
> mount /dev/vg/test /mnt/x
> 
> 2. extract linux-3.17.tar.xz to the new filesystem
> 
> cd /mnt/x
> tar Jxvf /usr/src/linux-3.17.tar.xz
> 
> 3. backup the filesystem
> 
> xfsdump -l 0 -o -p 300 -J -F -M test -L test - /mnt/x | /bin/gzip -qv > /vol/test.xfsdump.gz
> 
> 4. test the dump file, it show a lots of NOTE, and it said restore status is SUCCESS.

Is this not the same bug we already discussed in your prior email,
"xfsdump not work in 3.17" ?  Are you testing with or without those bugfixes?

-Eric

> cat /vol/test.xfsdump.gz | gzip -dqv | xfsrestore -p 300 -J -t - | grep "^/sbin/xfsrestore:"
> /sbin/xfsrestore: using file dump (drive_simple) strategy
> /sbin/xfsrestore: version 3.1.4 (dump format 3.0)
> /sbin/xfsrestore: searching media for dump
> /sbin/xfsrestore: examining media file 0
> /sbin/xfsrestore: dump description:
> /sbin/xfsrestore: hostname: mail.teatime.com.tw <http://mail.teatime.com.tw>
> /sbin/xfsrestore: mount point: /mnt/x
> /sbin/xfsrestore: volume: /dev/mapper/vg-test
> /sbin/xfsrestore: session time: Wed Oct 29 11:45:54 2014
> /sbin/xfsrestore: level: 0
> /sbin/xfsrestore: session label: "test"
> /sbin/xfsrestore: media label: "test"
> /sbin/xfsrestore: file system id: d757277d-5167-4cdb-975d-c64b2f94a47e
> /sbin/xfsrestore: session id: 6f114712-6fb6-4f35-a2f3-30ca1a0af5bb
> /sbin/xfsrestore: media id: 2eb2ff54-1838-47b2-a8d5-5f520deaf5f5
> /sbin/xfsrestore: searching media for directory dump
> /sbin/xfsrestore: reading directories
> /sbin/xfsrestore: 2035 directories and 33045 entries processed
> /sbin/xfsrestore: directory post-processing
> /sbin/xfsrestore: reading non-directory files
> /sbin/xfsrestore: NOTE: ino 144 salvaging file, placing in orphanage/143.0/Makefile
> /sbin/xfsrestore: NOTE: ino 145 salvaging file, placing in orphanage/143.0/bayer.png.b64
> /sbin/xfsrestore: NOTE: ino 146 salvaging file, placing in orphanage/143.0/constraints.png.b64
> /sbin/xfsrestore: NOTE: ino 147 salvaging file, placing in orphanage/143.0/crop.gif.b64
> /sbin/xfsrestore: NOTE: ino 148 salvaging file, placing in orphanage/143.0/dvbstb.png.b64
> /sbin/xfsrestore: NOTE: ino 149 salvaging file, placing in orphanage/143.0/fieldseq_bt.gif.b64
> /sbin/xfsrestore: NOTE: ino 150 salvaging file, placing in orphanage/143.0/fieldseq_tb.gif.b64
> .... skip it....
> /sbin/xfsrestore: NOTE: ino 50363235 gen 1622742662 not referenced: placing in orphanage
> /sbin/xfsrestore: table of contents display complete: 3 seconds elapsed
> /sbin/xfsrestore: Restore Status: SUCCESS
> 
> 5. restore it, show a lots of NOTE, and resotre status is SUCCESS, but... compare them, it's different.
> 
> cat test.xfsdump.gz | gzip -dqv | xfsrestore -p 300 -J - /vol/x/
> /sbin/xfsrestore: using file dump (drive_simple) strategy
> /sbin/xfsrestore: version 3.1.4 (dump format 3.0)
> /sbin/xfsrestore: searching media for dump
> /sbin/xfsrestore: examining media file 0
> /sbin/xfsrestore: dump description:
> /sbin/xfsrestore: hostname: mail.teatime.com.tw <http://mail.teatime.com.tw>
> /sbin/xfsrestore: mount point: /mnt/x
> /sbin/xfsrestore: volume: /dev/mapper/vg-test
> /sbin/xfsrestore: session time: Wed Oct 29 11:45:54 2014
> /sbin/xfsrestore: level: 0
> /sbin/xfsrestore: session label: "test"
> /sbin/xfsrestore: media label: "test"
> /sbin/xfsrestore: file system id: d757277d-5167-4cdb-975d-c64b2f94a47e
> /sbin/xfsrestore: session id: 6f114712-6fb6-4f35-a2f3-30ca1a0af5bb
> /sbin/xfsrestore: media id: 2eb2ff54-1838-47b2-a8d5-5f520deaf5f5
> /sbin/xfsrestore: searching media for directory dump
> /sbin/xfsrestore: reading directories
> /sbin/xfsrestore: 2035 directories and 33045 entries processed
> /sbin/xfsrestore: directory post-processing
> /sbin/xfsrestore: restoring non-directory files
> /sbin/xfsrestore: NOTE: ino 144 salvaging file, placing in orphanage/143.0/Makefile
> /sbin/xfsrestore: NOTE: ino 145 salvaging file, placing in orphanage/143.0/bayer.png.b64
> /sbin/xfsrestore: NOTE: ino 146 salvaging file, placing in orphanage/143.0/constraints.png.b64
> .... skip it....
> /sbin/xfsrestore: NOTE: ino 50363235 gen 1622742662 not referenced: placing in orphanage
> /sbin/xfsrestore: WARNING: unable to rmdir /vol/x//orphanage: Directory not empty
> /sbin/xfsrestore: restore complete: 3 seconds elapsed
> /sbin/xfsrestore: Restore Status: SUCCESS
> 
> 
> -- 
> 
> Tommy Wu
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  4:09 ` Eric Sandeen
@ 2014-10-29  4:27   ` Tommy Wu
  2014-10-29  4:28     ` Eric Sandeen
  0 siblings, 1 reply; 30+ messages in thread
From: Tommy Wu @ 2014-10-29  4:27 UTC (permalink / raw)
  To: Eric Sandeen; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 516 bytes --]

In "xfsdump not work in 3.17", after apply the 2 patches, xfsdump work
(won't hang).
But xfsrestore can't restore the dump correctly (like the case in this
email).

Is other patch I miss?  I check the thread in Archive again, I don't see
any patch about the xfsrestore issue.

2014-10-29 12:09 GMT+08:00 Eric Sandeen <sandeen@sandeen.net>:

>
> Is this not the same bug we already discussed in your prior email,
> "xfsdump not work in 3.17" ?  Are you testing with or without those
> bugfixes?
>
>
>


-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 1008 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  4:27   ` Tommy Wu
@ 2014-10-29  4:28     ` Eric Sandeen
  2014-10-29  4:30       ` Tommy Wu
  0 siblings, 1 reply; 30+ messages in thread
From: Eric Sandeen @ 2014-10-29  4:28 UTC (permalink / raw)
  To: Tommy Wu; +Cc: xfs

On 10/28/14 11:27 PM, Tommy Wu wrote:
> In "xfsdump not work in 3.17", after apply the 2 patches, xfsdump work (won't hang).
> But xfsrestore can't restore the dump correctly (like the case in this email).
> 
> Is other patch I miss?  I check the thread in Archive again, I don't see any patch about the xfsrestore issue.

Was the dump you are restoring created with those 2 patches in place?  Or was
it created with the stock/unpatched 3.17 kernel?

-Eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  4:28     ` Eric Sandeen
@ 2014-10-29  4:30       ` Tommy Wu
  2014-10-29  4:40         ` Dave Chinner
  0 siblings, 1 reply; 30+ messages in thread
From: Tommy Wu @ 2014-10-29  4:30 UTC (permalink / raw)
  To: Eric Sandeen; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 610 bytes --]

both dump and restore with those 2 patches.

2014-10-29 12:28 GMT+08:00 Eric Sandeen <sandeen@sandeen.net>:

> On 10/28/14 11:27 PM, Tommy Wu wrote:
> > In "xfsdump not work in 3.17", after apply the 2 patches, xfsdump work
> (won't hang).
> > But xfsrestore can't restore the dump correctly (like the case in this
> email).
> >
> > Is other patch I miss?  I check the thread in Archive again, I don't see
> any patch about the xfsrestore issue.
>
> Was the dump you are restoring created with those 2 patches in place?  Or
> was
> it created with the stock/unpatched 3.17 kernel?
>
> -Eric
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 1104 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  4:30       ` Tommy Wu
@ 2014-10-29  4:40         ` Dave Chinner
  2014-10-29  6:46           ` Tommy Wu
  0 siblings, 1 reply; 30+ messages in thread
From: Dave Chinner @ 2014-10-29  4:40 UTC (permalink / raw)
  To: Tommy Wu; +Cc: Eric Sandeen, xfs

On Wed, Oct 29, 2014 at 12:30:33PM +0800, Tommy Wu wrote:
> both dump and restore with those 2 patches.

Can you try the xfs-fixes-for-3.18-rc3 branch I pushed out this
morning to the upstream git repo here:

git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  4:40         ` Dave Chinner
@ 2014-10-29  6:46           ` Tommy Wu
  2014-10-29  7:51             ` Dave Chinner
  0 siblings, 1 reply; 30+ messages in thread
From: Tommy Wu @ 2014-10-29  6:46 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Eric Sandeen, xfs


[-- Attachment #1.1: Type: text/plain, Size: 666 bytes --]

it will hang after this message (the kernel in above git repo):

/sbin/xfsdump: dumping non-directory files

console show the below message every 2 minutes:
[  480.387308] INFO: task xfsdump:2717 blocked for more than 120 seconds.



2014-10-29 12:40 GMT+08:00 Dave Chinner <david@fromorbit.com>:

> On Wed, Oct 29, 2014 at 12:30:33PM +0800, Tommy Wu wrote:
> > both dump and restore with those 2 patches.
>
> Can you try the xfs-fixes-for-3.18-rc3 branch I pushed out this
> morning to the upstream git repo here:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 1351 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  6:46           ` Tommy Wu
@ 2014-10-29  7:51             ` Dave Chinner
  2014-10-29  9:03               ` Tommy Wu
  2014-10-29  9:21               ` Tommy Wu
  0 siblings, 2 replies; 30+ messages in thread
From: Dave Chinner @ 2014-10-29  7:51 UTC (permalink / raw)
  To: Tommy Wu; +Cc: Eric Sandeen, xfs

On Wed, Oct 29, 2014 at 02:46:33PM +0800, Tommy Wu wrote:
> it will hang after this message (the kernel in above git repo):
> 
> /sbin/xfsdump: dumping non-directory files
> 
> console show the below message every 2 minutes:
> [  480.387308] INFO: task xfsdump:2717 blocked for more than 120 seconds.

I think there might be something else wrong with your system.

What's the stack trace associated with that? If there's none,
what's the output of 'echo w > /proc/sysrq-trigger' in dmesg?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  7:51             ` Dave Chinner
@ 2014-10-29  9:03               ` Tommy Wu
  2014-10-29  9:21               ` Tommy Wu
  1 sibling, 0 replies; 30+ messages in thread
From: Tommy Wu @ 2014-10-29  9:03 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Eric Sandeen, xfs


[-- Attachment #1.1: Type: text/plain, Size: 5579 bytes --]

Oct 29 17:01:58 mail kernel: [  360.367165] INFO: task xfsdump:2549 blocked
for more than 120 seconds.
Oct 29 17:01:58 mail kernel: [  360.367182]       Tainted: G           O
3.18.0-rc2-1029 #1
Oct 29 17:01:58 mail kernel: [  360.367195] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 29 17:01:58 mail kernel: [  360.367211] xfsdump         D
0000000000000000     0  2549   2265 0x00000000
Oct 29 17:01:58 mail kernel: [  360.367229]  ffff8808110b8190
0000000000000082 ffff8807eb0e4800 ffff8807fa527fd8
Oct 29 17:01:58 mail kernel: [  360.367248]  ffff8808110b8190
000000000000a000 0000000000000296 000000007a9966c4
Oct 29 17:01:58 mail kernel: [  360.367266]  ffff8807fa557800
000000007a9966c4 ffff8807eb0e4800 7fffffffffffffff
Oct 29 17:01:58 mail kernel: [  360.367285] Call Trace:
Oct 29 17:01:58 mail kernel: [  360.367296]  [<ffffffff814106ad>] ?
schedule_timeout+0x15d/0x1b0
Oct 29 17:01:58 mail kernel: [  360.367310]  [<ffffffff8140f835>] ?
__down+0x65/0xb0
Oct 29 17:01:58 mail kernel: [  360.367322]  [<ffffffff8107e09d>] ?
down+0x3d/0x60
Oct 29 17:01:58 mail kernel: [  360.367359]  [<ffffffffa03637bd>] ?
xfs_buf_lock+0x1d/0x40 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367378]  [<ffffffffa0363975>] ?
_xfs_buf_find+0x195/0x270 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367398]  [<ffffffffa0363a71>] ?
xfs_buf_get_map+0x21/0x120 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367418]  [<ffffffffa03640ce>] ?
xfs_buf_read_map+0x1e/0xc0 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367438]  [<ffffffffa038c9db>] ?
xfs_trans_read_buf_map+0x10b/0x300 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367459]  [<ffffffffa0354440>] ?
xfs_read_agi+0x80/0xb0 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367478]  [<ffffffffa0354481>] ?
xfs_ialloc_read_agi+0x11/0x70 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367499]  [<ffffffffa0376532>] ?
xfs_bulkstat+0x192/0x530 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367517]  [<ffffffffa03761e0>] ?
xfs_bulkstat_one_int+0x2e0/0x2e0 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367538]  [<ffffffffa036bda2>] ?
xfs_ioc_bulkstat+0xe2/0x1c0 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367558]  [<ffffffffa036e518>] ?
xfs_file_ioctl+0x7f8/0xab0 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367577]  [<ffffffffa037221c>] ?
xfs_iunlock+0x3c/0x80 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367596]  [<ffffffffa0360ed5>] ?
xfs_free_eofblocks+0xd5/0x2a0 [xfs]
Oct 29 17:01:58 mail kernel: [  360.367611]  [<ffffffff810f6834>] ?
list_lru_add+0x64/0x100
Oct 29 17:01:58 mail kernel: [  360.367625]  [<ffffffff81152800>] ?
do_vfs_ioctl+0x2e0/0x4c0
Oct 29 17:01:58 mail kernel: [  360.367638]  [<ffffffff8105e3d4>] ?
task_work_run+0xb4/0xf0
Oct 29 17:01:58 mail kernel: [  360.367651]  [<ffffffff81152a59>] ?
SyS_ioctl+0x79/0x90
Oct 29 17:01:58 mail kernel: [  360.367662]  [<ffffffff814119e0>] ?
int_signal+0x12/0x17
Oct 29 17:01:58 mail kernel: [  360.367674]  [<ffffffff81411796>] ?
system_call_fastpath+0x16/0x1b

[  418.302083] SysRq : Show Blocked State
[  418.302096]   task                        PC stack   pid father
[  418.302127] xfsdump         D 0000000000000000     0  2549   2265
0x00000000
[  418.302144]  ffff8808110b8190 0000000000000082 ffff8807eb0e4800
ffff8807fa527fd8
[  418.302163]  ffff8808110b8190 000000000000a000 0000000000000296
000000007a9966c4
[  418.302181]  ffff8807fa557800 000000007a9966c4 ffff8807eb0e4800
7fffffffffffffff
[  418.302200] Call Trace:
[  418.302210]  [<ffffffff814106ad>] ? schedule_timeout+0x15d/0x1b0
[  418.302224]  [<ffffffff8140f835>] ? __down+0x65/0xb0
[  418.302237]  [<ffffffff8107e09d>] ? down+0x3d/0x60
[  418.302715]  [<ffffffffa03637bd>] ? xfs_buf_lock+0x1d/0x40 [xfs]
[  418.303160]  [<ffffffffa0363975>] ? _xfs_buf_find+0x195/0x270 [xfs]
[  418.303585]  [<ffffffffa0363a71>] ? xfs_buf_get_map+0x21/0x120 [xfs]
[  418.304007]  [<ffffffffa03640ce>] ? xfs_buf_read_map+0x1e/0xc0 [xfs]
[  418.304421]  [<ffffffffa038c9db>] ? xfs_trans_read_buf_map+0x10b/0x300
[xfs]
[  418.304837]  [<ffffffffa0354440>] ? xfs_read_agi+0x80/0xb0 [xfs]
[  418.305250]  [<ffffffffa0354481>] ? xfs_ialloc_read_agi+0x11/0x70 [xfs]
[  418.305664]  [<ffffffffa0376532>] ? xfs_bulkstat+0x192/0x530 [xfs]
[  418.306078]  [<ffffffffa03761e0>] ? xfs_bulkstat_one_int+0x2e0/0x2e0
[xfs]
[  418.306492]  [<ffffffffa036bda2>] ? xfs_ioc_bulkstat+0xe2/0x1c0 [xfs]
[  418.306939]  [<ffffffffa036e518>] ? xfs_file_ioctl+0x7f8/0xab0 [xfs]
[  418.307350]  [<ffffffffa037221c>] ? xfs_iunlock+0x3c/0x80 [xfs]
[  418.307754]  [<ffffffffa0360ed5>] ? xfs_free_eofblocks+0xd5/0x2a0 [xfs]
[  418.308163]  [<ffffffff810f6834>] ? list_lru_add+0x64/0x100
[  418.308567]  [<ffffffff81152800>] ? do_vfs_ioctl+0x2e0/0x4c0
[  418.308961]  [<ffffffff8105e3d4>] ? task_work_run+0xb4/0xf0
[  418.309352]  [<ffffffff81152a59>] ? SyS_ioctl+0x79/0x90
[  418.309734]  [<ffffffff814119e0>] ? int_signal+0x12/0x17
[  418.310113]  [<ffffffff81411796>] ? system_call_fastpath+0x16/0x1b



2014-10-29 15:51 GMT+08:00 Dave Chinner <david@fromorbit.com>:

> On Wed, Oct 29, 2014 at 02:46:33PM +0800, Tommy Wu wrote:
> > it will hang after this message (the kernel in above git repo):
> >
> > /sbin/xfsdump: dumping non-directory files
> >
> > console show the below message every 2 minutes:
> > [  480.387308] INFO: task xfsdump:2717 blocked for more than 120 seconds.
>
> I think there might be something else wrong with your system.
>
> What's the stack trace associated with that? If there's none,
> what's the output of 'echo w > /proc/sysrq-trigger' in dmesg?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 6794 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  7:51             ` Dave Chinner
  2014-10-29  9:03               ` Tommy Wu
@ 2014-10-29  9:21               ` Tommy Wu
  2014-10-29 10:33                 ` Dave Chinner
  1 sibling, 1 reply; 30+ messages in thread
From: Tommy Wu @ 2014-10-29  9:21 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Eric Sandeen, xfs


[-- Attachment #1.1: Type: text/plain, Size: 1003 bytes --]

Same testing in same machine (just different kernel).

3.18-rc2:
xfsdump will hang (the test.xfsdump.gz size stop around 57M)

3.17.1 (with 2 patches):
xfsdump generate the dump file, but xfsrestore can't restore all files from
the dump file.
I think the dump is corrupted, can't restore it also in kernel 3.16.5.

3.16.5:
everything work fine.


2014-10-29 15:51 GMT+08:00 Dave Chinner <david@fromorbit.com>:

> On Wed, Oct 29, 2014 at 02:46:33PM +0800, Tommy Wu wrote:
> > it will hang after this message (the kernel in above git repo):
> >
> > /sbin/xfsdump: dumping non-directory files
> >
> > console show the below message every 2 minutes:
> > [  480.387308] INFO: task xfsdump:2717 blocked for more than 120 seconds.
>
> I think there might be something else wrong with your system.
>
> What's the stack trace associated with that? If there's none,
> what's the output of 'echo w > /proc/sysrq-trigger' in dmesg?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 1670 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  9:21               ` Tommy Wu
@ 2014-10-29 10:33                 ` Dave Chinner
  2014-10-29 10:43                   ` Dave Chinner
  0 siblings, 1 reply; 30+ messages in thread
From: Dave Chinner @ 2014-10-29 10:33 UTC (permalink / raw)
  To: Tommy Wu; +Cc: Eric Sandeen, xfs

On Wed, Oct 29, 2014 at 05:21:52PM +0800, Tommy Wu wrote:
> Same testing in same machine (just different kernel).
> 
> 3.18-rc2:
> xfsdump will hang (the test.xfsdump.gz size stop around 57M)

Ugh, colliding fixes to the bulkstat code. pull again from
the xfs-fixes-for-3.18-rc3 branch and the hang should be fixed.

/me wonders why the new dump xfstest didn't pick this up

Cheers,

Dave.

-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29 10:33                 ` Dave Chinner
@ 2014-10-29 10:43                   ` Dave Chinner
  2014-10-29 12:32                     ` Tommy Wu
  0 siblings, 1 reply; 30+ messages in thread
From: Dave Chinner @ 2014-10-29 10:43 UTC (permalink / raw)
  To: Tommy Wu; +Cc: Eric Sandeen, xfs

On Wed, Oct 29, 2014 at 09:33:55PM +1100, Dave Chinner wrote:
> On Wed, Oct 29, 2014 at 05:21:52PM +0800, Tommy Wu wrote:
> > Same testing in same machine (just different kernel).
> > 
> > 3.18-rc2:
> > xfsdump will hang (the test.xfsdump.gz size stop around 57M)
> 
> Ugh, colliding fixes to the bulkstat code. pull again from
> the xfs-fixes-for-3.18-rc3 branch and the hang should be fixed.

Ugh, I forgot the stable tag on the new commit, so I'm going to have
to rebase that branch before it's ready to send to Linus. I'll do
it in the morning when I'm more awake.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29 10:43                   ` Dave Chinner
@ 2014-10-29 12:32                     ` Tommy Wu
  2014-10-29 19:50                       ` Dave Chinner
  0 siblings, 1 reply; 30+ messages in thread
From: Tommy Wu @ 2014-10-29 12:32 UTC (permalink / raw)
  To: Dave Chinner; +Cc: Eric Sandeen, xfs


[-- Attachment #1.1: Type: text/plain, Size: 999 bytes --]

Does it just fix the hang issue or fix the dump corrupted issue?

I just clone the git repo and checkout xfs-fixes-for-3.18-rc3, build it
again.
Now, xfsdump won't hang, but still can't restore the dump file correctly.
The result just same as kernel 3.17.1.


2014-10-29 18:43 GMT+08:00 Dave Chinner <david@fromorbit.com>:

> On Wed, Oct 29, 2014 at 09:33:55PM +1100, Dave Chinner wrote:
> > On Wed, Oct 29, 2014 at 05:21:52PM +0800, Tommy Wu wrote:
> > > Same testing in same machine (just different kernel).
> > >
> > > 3.18-rc2:
> > > xfsdump will hang (the test.xfsdump.gz size stop around 57M)
> >
> > Ugh, colliding fixes to the bulkstat code. pull again from
> > the xfs-fixes-for-3.18-rc3 branch and the hang should be fixed.
>
> Ugh, I forgot the stable tag on the new commit, so I'm going to have
> to rebase that branch before it's ready to send to Linus. I'll do
> it in the morning when I'm more awake.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 1636 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29 12:32                     ` Tommy Wu
@ 2014-10-29 19:50                       ` Dave Chinner
  2014-10-29 20:52                         ` Dave Chinner
  0 siblings, 1 reply; 30+ messages in thread
From: Dave Chinner @ 2014-10-29 19:50 UTC (permalink / raw)
  To: Tommy Wu; +Cc: Eric Sandeen, xfs

On Wed, Oct 29, 2014 at 08:32:52PM +0800, Tommy Wu wrote:
> Does it just fix the hang issue or fix the dump corrupted issue?

The hang.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29 19:50                       ` Dave Chinner
@ 2014-10-29 20:52                         ` Dave Chinner
  0 siblings, 0 replies; 30+ messages in thread
From: Dave Chinner @ 2014-10-29 20:52 UTC (permalink / raw)
  To: Tommy Wu; +Cc: Eric Sandeen, xfs

On Thu, Oct 30, 2014 at 06:50:06AM +1100, Dave Chinner wrote:
> On Wed, Oct 29, 2014 at 08:32:52PM +0800, Tommy Wu wrote:
> > Does it just fix the hang issue or fix the dump corrupted issue?
> 
> The hang.

FWIW, can you run xfs_repair -n on your filesystem and post the
output? Can you also run a full dump and restore and post the
*entire* output of both?

Also, all the other information noted here would be useful:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-29  4:06 xfsrestore report SUCCESS but not restore all files in kernel 3.17 Tommy Wu
  2014-10-29  4:09 ` Eric Sandeen
@ 2014-10-29 21:25 ` Dave Chinner
       [not found]   ` <CAGdb-8dbGRu_GWB4qoeNP8Kso2hzkLcuFS9Pwtnm2xOkP2T0rQ@mail.gmail.com>
  1 sibling, 1 reply; 30+ messages in thread
From: Dave Chinner @ 2014-10-29 21:25 UTC (permalink / raw)
  To: Tommy Wu; +Cc: xfs

On Wed, Oct 29, 2014 at 12:06:10PM +0800, Tommy Wu wrote:
> /sbin/xfsrestore: session id: 6f114712-6fb6-4f35-a2f3-30ca1a0af5bb
> /sbin/xfsrestore: media id: 2eb2ff54-1838-47b2-a8d5-5f520deaf5f5
> /sbin/xfsrestore: searching media for directory dump
> /sbin/xfsrestore: reading directories
> /sbin/xfsrestore: 2035 directories and 33045 entries processed
> /sbin/xfsrestore: directory post-processing
> /sbin/xfsrestore: reading non-directory files
> /sbin/xfsrestore: NOTE: ino 144 salvaging file, placing in
> orphanage/143.0/Makefile
> /sbin/xfsrestore: NOTE: ino 145 salvaging file, placing in
> orphanage/143.0/bayer.png.b64
.....
> .... skip it....
> /sbin/xfsrestore: NOTE: ino 50363235 gen 1622742662 not referenced: placing
> in orphanage

FWIW, this can be expected behaviour if you run xfsdump on a live
filesystem. From the DIAGNOSTICS section of the xfsdump man page:

	The message ``xfsdump: WARNING: unable to open directory:
	ino N: Invalid argument'' can occur with filesystems which
	are actively being modified while xfsdump is running.
	This can happen to either directory or regular file inodes -
	affected files will not end up in the dump, files below
	affected directories will be placed in the orphanage
	directory by xfsrestore.

This is exactly what you are seeing - files being placed in the
orphanage on restore because they appear to have no parent directory
inode referencing them.

IOWs, you're likely seeing the effect of the filesystem changing
while you are running xfsdump, not a bug in xfsdump or xfs_restore.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
       [not found]   ` <CAGdb-8dbGRu_GWB4qoeNP8Kso2hzkLcuFS9Pwtnm2xOkP2T0rQ@mail.gmail.com>
@ 2014-10-30  5:03     ` Dave Chinner
  2014-10-30  8:39       ` Tommy Wu
  0 siblings, 1 reply; 30+ messages in thread
From: Dave Chinner @ 2014-10-30  5:03 UTC (permalink / raw)
  To: Tommy Wu; +Cc: xfs

On Thu, Oct 30, 2014 at 11:07:21AM +0800, Tommy Wu wrote:
> I just create a x86-64 VM (using VirtualBox 4.3.18, under windows 8).
> Install Debian jessie on it. Create a test partition in LVM for testing.
> (so I think it's not hardware issue here)
> 
> test procedure:
> 1. mkfs.xfs the 1st partition in LVM
> 2. mount  the 1st partition
> 3. extract linux-3.17.tar.xz to the partition
> 4. xfsdump the 1st partition
> 5. mkfs.xfs the 2nd partition in LVM
> 6. mount the 2nd partition
> 7. xfsrestore the dump file to 2nd partition
> 8. compare the file/directory in 2 partitions.

Partition sizes are roughly a 5GB source and 10GB destination.

So there's this many files in the fs:

> xfsrestore: 3054 directories and 50556 entries processed

OK, so on 3.18-rc2:

> root@debian:/mnt/xfsdump# mount /dev/debian/xfsrestore /mnt/xfsrestore
> root@debian:/mnt/xfsdump# cd /mnt/xfsrestore/
> root@debian:/mnt/xfsrestore# xfsdump -l 0 -o -p 300 -J -F -M test -L test -
> /mnt/xfsdump | gzip -qv > /mnt/xfsrestore/xfsdump.gz
....
> xfsdump: media file size 575648352 bytes
> xfsdump: dump size (non-dir files) : 560258760 bytes

These are a handful of bytes different.

> xfsdump: dump complete: 21 seconds elapsed
> xfsdump: Dump Status: SUCCESS
>  78.5%
> root@debian:/mnt/xfsrestore# ls -la
> total 120680
> drwxr-xr-x 2 root root        23 Oct 30 10:48 .
> drwxr-xr-x 4 root root      4096 Oct 30 10:37 ..
> -rw-r--r-- 1 root root 123569739 Oct 30 10:48 xfsdump.gz
> root@debian:/mnt/xfsrestore# cat /mnt/xfsrestore/xfsdump.gz | gzip -dqv |
> xfsrestore -p 300 -J -t - | grep xfsrestore

and the test shows:

> xfsrestore: reading directories
> xfsrestore: 2035 directories and 33045 entries processed

A thousand less directories and 20000 less files, which doesn't make
much sense for a dump that has a few bytes difference in size.

So, I've done almost exactly the same test locally (I used the
3.18-rc2 tarball) and on both the upstream 3.18-rc2 kernel and my
current xfs-fixes-for-3.18-rc3 branch I can't reproduce your
problem:

$  find /mnt/test -type d |wc -l
3080
$  find /mnt/test  |wc -l
51068
$ zcat dump.gz | sudo xfsrestore -p 300 -J - . |grep xfsrestore
.....
xfsrestore: reading directories
xfsrestore: 3080 directories and 51068 entries processed
xfsrestore: directory post-processing
xfsrestore: restoring non-directory files
xfsrestore: restore complete: 7 seconds elapsed
xfsrestore: Restore Status: SUCCESS
$ find /mnt/scratch -type d |wc -l
3080
$ find /mnt/scratch |wc -l
51069
$ ls /mnt/scratch
dump.gz  linux-3.18-rc2
$

Which shows that it's working just fine.  I've tried cold cache
dumps. I've tried crc enabled filesystems, I've tried a bunch of
random permutations, but I cannot get dump/restore to do what you
are seeing it do. Whatever problem you are seeing is specific to
your system.

Hmmm - can you run 'ldd xfsdump' and 'ldd xfsrestore' and paste the
output for me?

Can you also please run a full test again on a kernel built from the
current xfs-fixes-for-3.18-rc3 branch, with full dump/restore debug
enabled and send me the output? i.e.

$ xfsdump -v debug ...

and 

$ xfsrestore -v debug,tree=nitty ...

They ar going to produce a lot of output, so please capture it to
files rather than trying to cut/paste output to an email.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-30  5:03     ` Dave Chinner
@ 2014-10-30  8:39       ` Tommy Wu
  2014-10-30 10:39         ` Tommy Wu
  2014-10-30 23:04         ` Dave Chinner
  0 siblings, 2 replies; 30+ messages in thread
From: Tommy Wu @ 2014-10-30  8:39 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 5217 bytes --]

I'm confusing now...

in the test VM and one of my machine, rebuild xfsdump/xfsprogs will make
the test work.

In one of my machine, still got error.

fw1:/vol/backup/x# ldd /sbin/xfsdump
        linux-vdso.so.1 (0x00007fff3f7fe000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1
(0x00007f8c2bf9d000)
        libhandle.so.1 => /lib/libhandle.so.1 (0x00007f8c2bd99000)
        libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1
(0x00007f8c2bb93000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f8c2b976000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8c2b5cd000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8c2c1b5000)
fw1:/vol/backup/x# ldd /sbin/xfsrestore
        linux-vdso.so.1 (0x00007fffcee8d000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1
(0x00007ff53d507000)
        libhandle.so.1 => /lib/libhandle.so.1 (0x00007ff53d303000)
        libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1
(0x00007ff53d0fd000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007ff53cee0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff53cb37000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff53d71f000)

the xfsdump command I use:
xfsdump -v debug -l 0 -o -p 300 -J -F -M test -L test -f test.xfsdump
/mnt/backup > xfsdump.log
the log:
https://mega.co.nz/#!xIhTjD6Y!Gwp_ilW5UyRiqn2DSxPTHM0TH5qBghMlKXAmHoWk2iQ

the xfsrestore command I use:
xfsrestore -v debug,tree=nitty -p 300 -J -f test.xfsdump /vol/backup/x/ >
xfsrestore.log
the log:
https://mega.co.nz/#!oZIFWQST!VUY90EZ8XvZDYy9GwF5mqtQVtWNW90Sar5MZhNzGbYI



2014-10-30 13:03 GMT+08:00 Dave Chinner <david@fromorbit.com>:

> On Thu, Oct 30, 2014 at 11:07:21AM +0800, Tommy Wu wrote:
> > I just create a x86-64 VM (using VirtualBox 4.3.18, under windows 8).
> > Install Debian jessie on it. Create a test partition in LVM for testing.
> > (so I think it's not hardware issue here)
> >
> > test procedure:
> > 1. mkfs.xfs the 1st partition in LVM
> > 2. mount  the 1st partition
> > 3. extract linux-3.17.tar.xz to the partition
> > 4. xfsdump the 1st partition
> > 5. mkfs.xfs the 2nd partition in LVM
> > 6. mount the 2nd partition
> > 7. xfsrestore the dump file to 2nd partition
> > 8. compare the file/directory in 2 partitions.
>
> Partition sizes are roughly a 5GB source and 10GB destination.
>
> So there's this many files in the fs:
>
> > xfsrestore: 3054 directories and 50556 entries processed
>
> OK, so on 3.18-rc2:
>
> > root@debian:/mnt/xfsdump# mount /dev/debian/xfsrestore /mnt/xfsrestore
> > root@debian:/mnt/xfsdump# cd /mnt/xfsrestore/
> > root@debian:/mnt/xfsrestore# xfsdump -l 0 -o -p 300 -J -F -M test -L
> test -
> > /mnt/xfsdump | gzip -qv > /mnt/xfsrestore/xfsdump.gz
> ....
> > xfsdump: media file size 575648352 bytes
> > xfsdump: dump size (non-dir files) : 560258760 bytes
>
> These are a handful of bytes different.
>
> > xfsdump: dump complete: 21 seconds elapsed
> > xfsdump: Dump Status: SUCCESS
> >  78.5%
> > root@debian:/mnt/xfsrestore# ls -la
> > total 120680
> > drwxr-xr-x 2 root root        23 Oct 30 10:48 .
> > drwxr-xr-x 4 root root      4096 Oct 30 10:37 ..
> > -rw-r--r-- 1 root root 123569739 Oct 30 10:48 xfsdump.gz
> > root@debian:/mnt/xfsrestore# cat /mnt/xfsrestore/xfsdump.gz | gzip -dqv
> |
> > xfsrestore -p 300 -J -t - | grep xfsrestore
>
> and the test shows:
>
> > xfsrestore: reading directories
> > xfsrestore: 2035 directories and 33045 entries processed
>
> A thousand less directories and 20000 less files, which doesn't make
> much sense for a dump that has a few bytes difference in size.
>
> So, I've done almost exactly the same test locally (I used the
> 3.18-rc2 tarball) and on both the upstream 3.18-rc2 kernel and my
> current xfs-fixes-for-3.18-rc3 branch I can't reproduce your
> problem:
>
> $  find /mnt/test -type d |wc -l
> 3080
> $  find /mnt/test  |wc -l
> 51068
> $ zcat dump.gz | sudo xfsrestore -p 300 -J - . |grep xfsrestore
> .....
> xfsrestore: reading directories
> xfsrestore: 3080 directories and 51068 entries processed
> xfsrestore: directory post-processing
> xfsrestore: restoring non-directory files
> xfsrestore: restore complete: 7 seconds elapsed
> xfsrestore: Restore Status: SUCCESS
> $ find /mnt/scratch -type d |wc -l
> 3080
> $ find /mnt/scratch |wc -l
> 51069
> $ ls /mnt/scratch
> dump.gz  linux-3.18-rc2
> $
>
> Which shows that it's working just fine.  I've tried cold cache
> dumps. I've tried crc enabled filesystems, I've tried a bunch of
> random permutations, but I cannot get dump/restore to do what you
> are seeing it do. Whatever problem you are seeing is specific to
> your system.
>
> Hmmm - can you run 'ldd xfsdump' and 'ldd xfsrestore' and paste the
> output for me?
>
> Can you also please run a full test again on a kernel built from the
> current xfs-fixes-for-3.18-rc3 branch, with full dump/restore debug
> enabled and send me the output? i.e.
>
> $ xfsdump -v debug ...
>
> and
>
> $ xfsrestore -v debug,tree=nitty ...
>
> They ar going to produce a lot of output, so please capture it to
> files rather than trying to cut/paste output to an email.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 6851 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-30  8:39       ` Tommy Wu
@ 2014-10-30 10:39         ` Tommy Wu
  2014-10-30 23:04         ` Dave Chinner
  1 sibling, 0 replies; 30+ messages in thread
From: Tommy Wu @ 2014-10-30 10:39 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 5591 bytes --]

It' look like no help after rebuild xfsdump/xfsprogs, it still got the same
error for some case.

2014-10-30 16:39 GMT+08:00 Tommy Wu <wu.tommy@gmail.com>:

> I'm confusing now...
>
> in the test VM and one of my machine, rebuild xfsdump/xfsprogs will make
> the test work.
>
> In one of my machine, still got error.
>
> fw1:/vol/backup/x# ldd /sbin/xfsdump
>         linux-vdso.so.1 (0x00007fff3f7fe000)
>         libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1
> (0x00007f8c2bf9d000)
>         libhandle.so.1 => /lib/libhandle.so.1 (0x00007f8c2bd99000)
>         libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1
> (0x00007f8c2bb93000)
>         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007f8c2b976000)
>         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8c2b5cd000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007f8c2c1b5000)
> fw1:/vol/backup/x# ldd /sbin/xfsrestore
>         linux-vdso.so.1 (0x00007fffcee8d000)
>         libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1
> (0x00007ff53d507000)
>         libhandle.so.1 => /lib/libhandle.so.1 (0x00007ff53d303000)
>         libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1
> (0x00007ff53d0fd000)
>         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x00007ff53cee0000)
>         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff53cb37000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007ff53d71f000)
>
> the xfsdump command I use:
> xfsdump -v debug -l 0 -o -p 300 -J -F -M test -L test -f test.xfsdump
> /mnt/backup > xfsdump.log
> the log:
> https://mega.co.nz/#!xIhTjD6Y!Gwp_ilW5UyRiqn2DSxPTHM0TH5qBghMlKXAmHoWk2iQ
>
> the xfsrestore command I use:
> xfsrestore -v debug,tree=nitty -p 300 -J -f test.xfsdump /vol/backup/x/ >
> xfsrestore.log
> the log:
> https://mega.co.nz/#!oZIFWQST!VUY90EZ8XvZDYy9GwF5mqtQVtWNW90Sar5MZhNzGbYI
>
>
>
> 2014-10-30 13:03 GMT+08:00 Dave Chinner <david@fromorbit.com>:
>
>> On Thu, Oct 30, 2014 at 11:07:21AM +0800, Tommy Wu wrote:
>> > I just create a x86-64 VM (using VirtualBox 4.3.18, under windows 8).
>> > Install Debian jessie on it. Create a test partition in LVM for testing.
>> > (so I think it's not hardware issue here)
>> >
>> > test procedure:
>> > 1. mkfs.xfs the 1st partition in LVM
>> > 2. mount  the 1st partition
>> > 3. extract linux-3.17.tar.xz to the partition
>> > 4. xfsdump the 1st partition
>> > 5. mkfs.xfs the 2nd partition in LVM
>> > 6. mount the 2nd partition
>> > 7. xfsrestore the dump file to 2nd partition
>> > 8. compare the file/directory in 2 partitions.
>>
>> Partition sizes are roughly a 5GB source and 10GB destination.
>>
>> So there's this many files in the fs:
>>
>> > xfsrestore: 3054 directories and 50556 entries processed
>>
>> OK, so on 3.18-rc2:
>>
>> > root@debian:/mnt/xfsdump# mount /dev/debian/xfsrestore /mnt/xfsrestore
>> > root@debian:/mnt/xfsdump# cd /mnt/xfsrestore/
>> > root@debian:/mnt/xfsrestore# xfsdump -l 0 -o -p 300 -J -F -M test -L
>> test -
>> > /mnt/xfsdump | gzip -qv > /mnt/xfsrestore/xfsdump.gz
>> ....
>> > xfsdump: media file size 575648352 bytes
>> > xfsdump: dump size (non-dir files) : 560258760 bytes
>>
>> These are a handful of bytes different.
>>
>> > xfsdump: dump complete: 21 seconds elapsed
>> > xfsdump: Dump Status: SUCCESS
>> >  78.5%
>> > root@debian:/mnt/xfsrestore# ls -la
>> > total 120680
>> > drwxr-xr-x 2 root root        23 Oct 30 10:48 .
>> > drwxr-xr-x 4 root root      4096 Oct 30 10:37 ..
>> > -rw-r--r-- 1 root root 123569739 Oct 30 10:48 xfsdump.gz
>> > root@debian:/mnt/xfsrestore# cat /mnt/xfsrestore/xfsdump.gz | gzip
>> -dqv |
>> > xfsrestore -p 300 -J -t - | grep xfsrestore
>>
>> and the test shows:
>>
>> > xfsrestore: reading directories
>> > xfsrestore: 2035 directories and 33045 entries processed
>>
>> A thousand less directories and 20000 less files, which doesn't make
>> much sense for a dump that has a few bytes difference in size.
>>
>> So, I've done almost exactly the same test locally (I used the
>> 3.18-rc2 tarball) and on both the upstream 3.18-rc2 kernel and my
>> current xfs-fixes-for-3.18-rc3 branch I can't reproduce your
>> problem:
>>
>> $  find /mnt/test -type d |wc -l
>> 3080
>> $  find /mnt/test  |wc -l
>> 51068
>> $ zcat dump.gz | sudo xfsrestore -p 300 -J - . |grep xfsrestore
>> .....
>> xfsrestore: reading directories
>> xfsrestore: 3080 directories and 51068 entries processed
>> xfsrestore: directory post-processing
>> xfsrestore: restoring non-directory files
>> xfsrestore: restore complete: 7 seconds elapsed
>> xfsrestore: Restore Status: SUCCESS
>> $ find /mnt/scratch -type d |wc -l
>> 3080
>> $ find /mnt/scratch |wc -l
>> 51069
>> $ ls /mnt/scratch
>> dump.gz  linux-3.18-rc2
>> $
>>
>> Which shows that it's working just fine.  I've tried cold cache
>> dumps. I've tried crc enabled filesystems, I've tried a bunch of
>> random permutations, but I cannot get dump/restore to do what you
>> are seeing it do. Whatever problem you are seeing is specific to
>> your system.
>>
>> Hmmm - can you run 'ldd xfsdump' and 'ldd xfsrestore' and paste the
>> output for me?
>>
>> Can you also please run a full test again on a kernel built from the
>> current xfs-fixes-for-3.18-rc3 branch, with full dump/restore debug
>> enabled and send me the output? i.e.
>>
>> $ xfsdump -v debug ...
>>
>> and
>>
>> $ xfsrestore -v debug,tree=nitty ...
>>
>> They ar going to produce a lot of output, so please capture it to
>> files rather than trying to cut/paste output to an email.
>>
>> Cheers,
>>
>> Dave.
>> --
>> Dave Chinner
>> david@fromorbit.com
>>
>
>
>
> --
>
> Tommy Wu
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 7410 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-30  8:39       ` Tommy Wu
  2014-10-30 10:39         ` Tommy Wu
@ 2014-10-30 23:04         ` Dave Chinner
  2014-10-31  0:09           ` Dave Chinner
  1 sibling, 1 reply; 30+ messages in thread
From: Dave Chinner @ 2014-10-30 23:04 UTC (permalink / raw)
  To: Tommy Wu; +Cc: xfs

On Thu, Oct 30, 2014 at 04:39:01PM +0800, Tommy Wu wrote:
> I'm confusing now...
> 
> in the test VM and one of my machine, rebuild xfsdump/xfsprogs will make
> the test work.

I know why, too.

> the xfsdump command I use:
> xfsdump -v debug -l 0 -o -p 300 -J -F -M test -L test -f test.xfsdump
> /mnt/backup > xfsdump.log
> the log:
> https://mega.co.nz/#!xIhTjD6Y!Gwp_ilW5UyRiqn2DSxPTHM0TH5qBghMlKXAmHoWk2iQ

This indicates the bulkstat on the dump side was terminated
prematurely.

> the xfsrestore command I use:
> xfsrestore -v debug,tree=nitty -p 300 -J -f test.xfsdump /vol/backup/x/ >
> xfsrestore.log
> the log:
> https://mega.co.nz/#!oZIFWQST!VUY90EZ8XvZDYy9GwF5mqtQVtWNW90Sar5MZhNzGbYI

And the errors in the restore side are because there are missing
directory inodes in the dump, hence orphaning occurs.

It might take me a little while to come up with a fix - I've noticed
a few things that I need to check out in more detail in tracking
this down....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-30 23:04         ` Dave Chinner
@ 2014-10-31  0:09           ` Dave Chinner
  2014-10-31  1:51             ` Tommy Wu
  0 siblings, 1 reply; 30+ messages in thread
From: Dave Chinner @ 2014-10-31  0:09 UTC (permalink / raw)
  To: Tommy Wu; +Cc: xfs

On Fri, Oct 31, 2014 at 10:04:57AM +1100, Dave Chinner wrote:
> On Thu, Oct 30, 2014 at 04:39:01PM +0800, Tommy Wu wrote:
> > I'm confusing now...
> > 
> > in the test VM and one of my machine, rebuild xfsdump/xfsprogs will make
> > the test work.
> 
> I know why, too.
> 
> > the xfsdump command I use:
> > xfsdump -v debug -l 0 -o -p 300 -J -F -M test -L test -f test.xfsdump
> > /mnt/backup > xfsdump.log
> > the log:
> > https://mega.co.nz/#!xIhTjD6Y!Gwp_ilW5UyRiqn2DSxPTHM0TH5qBghMlKXAmHoWk2iQ
> 
> This indicates the bulkstat on the dump side was terminated
> prematurely.
> 
> > the xfsrestore command I use:
> > xfsrestore -v debug,tree=nitty -p 300 -J -f test.xfsdump /vol/backup/x/ >
> > xfsrestore.log
> > the log:
> > https://mega.co.nz/#!oZIFWQST!VUY90EZ8XvZDYy9GwF5mqtQVtWNW90Sar5MZhNzGbYI
> 
> And the errors in the restore side are because there are missing
> directory inodes in the dump, hence orphaning occurs.
> 
> It might take me a little while to come up with a fix - I've noticed
> a few things that I need to check out in more detail in tracking
> this down....

Can you try this patch?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

xfs: handle munged bulkstat cookies correctly

From: Dave Chinner <dchinner@redhat.com>

xfsdump munges the last inode cookie that bulkstat passes to
userspace so subsequent calls can restart at the last inode that
bulkstat parsed. xfsdump thinks that it needs to decrement the
cookie so that it catches the last inode, and this affects the
tree search initialisation code.

By decrementing the lastino cookie, the restart inode number can
point to non-existent inode number. In 3.16, bulkstat would simply
increment the cookie until it either found the next valid inode or
the end of the AG, at which point it would just continue on as
though nothing had happened. This is a relatively difficult
situation to contrive so is extremely hard to test for, but is
likely to occur fairly frequently in the real world.

The current code, however, treats a cookie that points off into the
weeds as invalid and returns a error. This has the effect of causing
xfsdump to think it has reached the end of the bulkstat scan, and so
if it's cookie munging happens just wrong it will cause incomplete
dumps to occur. These dumps still have all the file data in them,
just he directory structure is broken and so restore will end up
with lots and lots of files in the orphanage (i.e. lost+found).

IOWs, we broke xfsdump in a nasty, subtle way that users don't
notice until they restore a dump.

To fix this, restore the old behaviour of incrementing the inode
number derived from the cookie when we fail to find a matching inode
record.

cc: stable@vger.kernel.org
Reported-by: Tommy Wu <wu.tommy@gmail.com>
Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 fs/xfs/xfs_itable.c | 34 ++++++++++++++++++++++++++++++----
 1 file changed, 30 insertions(+), 4 deletions(-)

diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
index 7765ff7..dcc0c99 100644
--- a/fs/xfs/xfs_itable.c
+++ b/fs/xfs/xfs_itable.c
@@ -208,6 +208,15 @@ xfs_bulkstat_ichunk_ra(
  * if we found the chunk.  If the inode was not the last in the chunk and there
  * are some left allocated, update the data for the pointed-to record as well as
  * return the count of grabbed inodes.
+ *
+ * If this is the first lookup in a bulkstat call, we have to be careful about
+ * the agino that is passed - it may not point to a valid inode record. xfsdump,
+ * in particular, screws with the lastino cookie between bulkstat calls and so
+ * can point it off into the weeds. Hence we can't return an error here if
+ * the agino doesn't point to the correct inode chunk. The previous code simply
+ * incremented agino and tried again. This will work for xfsdump as the next
+ * lookup will either find the correct chunk or the end of the AG, so let's just
+ * keep doing that.
  */
 STATIC int
 xfs_bulkstat_grab_ichunk(
@@ -226,7 +235,7 @@ xfs_bulkstat_grab_ichunk(
 		return error;
 	if (!stat) {
 		*icount = 0;
-		return error;
+		return 0;
 	}
 
 	/* Get the record, should always work */
@@ -236,8 +245,11 @@ xfs_bulkstat_grab_ichunk(
 	XFS_WANT_CORRUPTED_RETURN(stat == 1);
 
 	/* Check if the record contains the inode in request */
-	if (irec->ir_startino + XFS_INODES_PER_CHUNK <= agino)
-		return -EINVAL;
+	if (irec->ir_startino + XFS_INODES_PER_CHUNK <= agino) {
+		/* oops, someone is feeding us garbage! */
+		*icount = -1;
+		return 0;
+	}
 
 	idx = agino - irec->ir_startino + 1;
 	if (idx < XFS_INODES_PER_CHUNK &&
@@ -428,6 +440,17 @@ xfs_bulkstat(
 			error = xfs_bulkstat_grab_ichunk(cur, agino, &icount, &r);
 			if (error)
 				goto del_cursor;
+			if (icount < 0) {
+				/*
+				 * A pox be cast over xfsdump!
+				 *
+				 * No record containing agino was found.  Bump
+				 * it, try again until we find a chunk or fall
+				 * off the end of the AG.
+				 */
+				agino++;
+				goto del_cursor;
+			}
 			if (icount) {
 				irbp->ir_startino = r.ir_startino;
 				irbp->ir_freecount = r.ir_freecount;
@@ -435,14 +458,17 @@ xfs_bulkstat(
 				irbp++;
 				agino = r.ir_startino + XFS_INODES_PER_CHUNK;
 			}
+
 			/* Increment to the next record */
 			error = xfs_btree_increment(cur, 0, &tmp);
 		} else {
 			/* Start of ag.  Lookup the first inode chunk */
 			error = xfs_inobt_lookup(cur, 0, XFS_LOOKUP_GE, &tmp);
 		}
-		if (error)
+		if (error || tmp == 0) {
+			end_of_ag = 1;
 			goto del_cursor;
+		}
 
 		/*
 		 * Loop through inode btree records in this ag,

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply related	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-31  0:09           ` Dave Chinner
@ 2014-10-31  1:51             ` Tommy Wu
  2014-10-31  3:49               ` Dave Chinner
  0 siblings, 1 reply; 30+ messages in thread
From: Tommy Wu @ 2014-10-31  1:51 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 7008 bytes --]

After apply this patch, xfsdump will hang again.

the log for xfsdump:
https://mega.co.nz/#!tUgz3QrC!q7Ix53sl9lYCg0iKRHjgOqF0tZJAVKHBNz0OHWof-ds

But 'echo w > /proc/sysrq-trigger' no any blocked task in dmesg.


2014-10-31 8:09 GMT+08:00 Dave Chinner <david@fromorbit.com>:

> On Fri, Oct 31, 2014 at 10:04:57AM +1100, Dave Chinner wrote:
> > On Thu, Oct 30, 2014 at 04:39:01PM +0800, Tommy Wu wrote:
> > > I'm confusing now...
> > >
> > > in the test VM and one of my machine, rebuild xfsdump/xfsprogs will
> make
> > > the test work.
> >
> > I know why, too.
> >
> > > the xfsdump command I use:
> > > xfsdump -v debug -l 0 -o -p 300 -J -F -M test -L test -f test.xfsdump
> > > /mnt/backup > xfsdump.log
> > > the log:
> > >
> https://mega.co.nz/#!xIhTjD6Y!Gwp_ilW5UyRiqn2DSxPTHM0TH5qBghMlKXAmHoWk2iQ
> >
> > This indicates the bulkstat on the dump side was terminated
> > prematurely.
> >
> > > the xfsrestore command I use:
> > > xfsrestore -v debug,tree=nitty -p 300 -J -f test.xfsdump
> /vol/backup/x/ >
> > > xfsrestore.log
> > > the log:
> > >
> https://mega.co.nz/#!oZIFWQST!VUY90EZ8XvZDYy9GwF5mqtQVtWNW90Sar5MZhNzGbYI
> >
> > And the errors in the restore side are because there are missing
> > directory inodes in the dump, hence orphaning occurs.
> >
> > It might take me a little while to come up with a fix - I've noticed
> > a few things that I need to check out in more detail in tracking
> > this down....
>
> Can you try this patch?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>
> xfs: handle munged bulkstat cookies correctly
>
> From: Dave Chinner <dchinner@redhat.com>
>
> xfsdump munges the last inode cookie that bulkstat passes to
> userspace so subsequent calls can restart at the last inode that
> bulkstat parsed. xfsdump thinks that it needs to decrement the
> cookie so that it catches the last inode, and this affects the
> tree search initialisation code.
>
> By decrementing the lastino cookie, the restart inode number can
> point to non-existent inode number. In 3.16, bulkstat would simply
> increment the cookie until it either found the next valid inode or
> the end of the AG, at which point it would just continue on as
> though nothing had happened. This is a relatively difficult
> situation to contrive so is extremely hard to test for, but is
> likely to occur fairly frequently in the real world.
>
> The current code, however, treats a cookie that points off into the
> weeds as invalid and returns a error. This has the effect of causing
> xfsdump to think it has reached the end of the bulkstat scan, and so
> if it's cookie munging happens just wrong it will cause incomplete
> dumps to occur. These dumps still have all the file data in them,
> just he directory structure is broken and so restore will end up
> with lots and lots of files in the orphanage (i.e. lost+found).
>
> IOWs, we broke xfsdump in a nasty, subtle way that users don't
> notice until they restore a dump.
>
> To fix this, restore the old behaviour of incrementing the inode
> number derived from the cookie when we fail to find a matching inode
> record.
>
> cc: stable@vger.kernel.org
> Reported-by: Tommy Wu <wu.tommy@gmail.com>
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
>  fs/xfs/xfs_itable.c | 34 ++++++++++++++++++++++++++++++----
>  1 file changed, 30 insertions(+), 4 deletions(-)
>
> diff --git a/fs/xfs/xfs_itable.c b/fs/xfs/xfs_itable.c
> index 7765ff7..dcc0c99 100644
> --- a/fs/xfs/xfs_itable.c
> +++ b/fs/xfs/xfs_itable.c
> @@ -208,6 +208,15 @@ xfs_bulkstat_ichunk_ra(
>   * if we found the chunk.  If the inode was not the last in the chunk and
> there
>   * are some left allocated, update the data for the pointed-to record as
> well as
>   * return the count of grabbed inodes.
> + *
> + * If this is the first lookup in a bulkstat call, we have to be careful
> about
> + * the agino that is passed - it may not point to a valid inode record.
> xfsdump,
> + * in particular, screws with the lastino cookie between bulkstat calls
> and so
> + * can point it off into the weeds. Hence we can't return an error here if
> + * the agino doesn't point to the correct inode chunk. The previous code
> simply
> + * incremented agino and tried again. This will work for xfsdump as the
> next
> + * lookup will either find the correct chunk or the end of the AG, so
> let's just
> + * keep doing that.
>   */
>  STATIC int
>  xfs_bulkstat_grab_ichunk(
> @@ -226,7 +235,7 @@ xfs_bulkstat_grab_ichunk(
>                 return error;
>         if (!stat) {
>                 *icount = 0;
> -               return error;
> +               return 0;
>         }
>
>         /* Get the record, should always work */
> @@ -236,8 +245,11 @@ xfs_bulkstat_grab_ichunk(
>         XFS_WANT_CORRUPTED_RETURN(stat == 1);
>
>         /* Check if the record contains the inode in request */
> -       if (irec->ir_startino + XFS_INODES_PER_CHUNK <= agino)
> -               return -EINVAL;
> +       if (irec->ir_startino + XFS_INODES_PER_CHUNK <= agino) {
> +               /* oops, someone is feeding us garbage! */
> +               *icount = -1;
> +               return 0;
> +       }
>
>         idx = agino - irec->ir_startino + 1;
>         if (idx < XFS_INODES_PER_CHUNK &&
> @@ -428,6 +440,17 @@ xfs_bulkstat(
>                         error = xfs_bulkstat_grab_ichunk(cur, agino,
> &icount, &r);
>                         if (error)
>                                 goto del_cursor;
> +                       if (icount < 0) {
> +                               /*
> +                                * A pox be cast over xfsdump!
> +                                *
> +                                * No record containing agino was found.
> Bump
> +                                * it, try again until we find a chunk or
> fall
> +                                * off the end of the AG.
> +                                */
> +                               agino++;
> +                               goto del_cursor;
> +                       }
>                         if (icount) {
>                                 irbp->ir_startino = r.ir_startino;
>                                 irbp->ir_freecount = r.ir_freecount;
> @@ -435,14 +458,17 @@ xfs_bulkstat(
>                                 irbp++;
>                                 agino = r.ir_startino +
> XFS_INODES_PER_CHUNK;
>                         }
> +
>                         /* Increment to the next record */
>                         error = xfs_btree_increment(cur, 0, &tmp);
>                 } else {
>                         /* Start of ag.  Lookup the first inode chunk */
>                         error = xfs_inobt_lookup(cur, 0, XFS_LOOKUP_GE,
> &tmp);
>                 }
> -               if (error)
> +               if (error || tmp == 0) {
> +                       end_of_ag = 1;
>                         goto del_cursor;
> +               }
>
>                 /*
>                  * Loop through inode btree records in this ag,
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 9222 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-31  1:51             ` Tommy Wu
@ 2014-10-31  3:49               ` Dave Chinner
  2014-10-31  4:25                 ` Dave Chinner
  2014-10-31  4:40                 ` Tommy Wu
  0 siblings, 2 replies; 30+ messages in thread
From: Dave Chinner @ 2014-10-31  3:49 UTC (permalink / raw)
  To: Tommy Wu; +Cc: xfs

On Fri, Oct 31, 2014 at 09:51:27AM +0800, Tommy Wu wrote:
> After apply this patch, xfsdump will hang again.
> 
> the log for xfsdump:
> https://mega.co.nz/#!tUgz3QrC!q7Ix53sl9lYCg0iKRHjgOqF0tZJAVKHBNz0OHWof-ds
> 
> But 'echo w > /proc/sysrq-trigger' no any blocked task in dmesg.

echo l > /proc/sysrq-trigger
echo t > /proc/sysrq-trigger

-Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-31  3:49               ` Dave Chinner
@ 2014-10-31  4:25                 ` Dave Chinner
  2014-10-31  4:42                   ` Tommy Wu
  2014-10-31  5:40                   ` Tommy Wu
  2014-10-31  4:40                 ` Tommy Wu
  1 sibling, 2 replies; 30+ messages in thread
From: Dave Chinner @ 2014-10-31  4:25 UTC (permalink / raw)
  To: Tommy Wu; +Cc: xfs

On Fri, Oct 31, 2014 at 02:49:25PM +1100, Dave Chinner wrote:
> On Fri, Oct 31, 2014 at 09:51:27AM +0800, Tommy Wu wrote:
> > After apply this patch, xfsdump will hang again.
> > 
> > the log for xfsdump:
> > https://mega.co.nz/#!tUgz3QrC!q7Ix53sl9lYCg0iKRHjgOqF0tZJAVKHBNz0OHWof-ds
> > 
> > But 'echo w > /proc/sysrq-trigger' no any blocked task in dmesg.
> 
> echo l > /proc/sysrq-trigger
> echo t > /proc/sysrq-trigger

Also, I just cannot reproduce a hung dump or a broken restore,
so can you take an unobfuscated metadump of the source filesystem
(i.e. the one you are running dump on) so I can try on an exact
image of the filesystem you are having troubles on?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-31  3:49               ` Dave Chinner
  2014-10-31  4:25                 ` Dave Chinner
@ 2014-10-31  4:40                 ` Tommy Wu
  1 sibling, 0 replies; 30+ messages in thread
From: Tommy Wu @ 2014-10-31  4:40 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 585 bytes --]

the dmesg log:
https://mega.co.nz/#!VIgnQYbK!cC0l8WKOtgJkmi0pCbpJf9mm357nzi0fpV0extUmbC0

2014-10-31 11:49 GMT+08:00 Dave Chinner <david@fromorbit.com>:

> On Fri, Oct 31, 2014 at 09:51:27AM +0800, Tommy Wu wrote:
> > After apply this patch, xfsdump will hang again.
> >
> > the log for xfsdump:
> >
> https://mega.co.nz/#!tUgz3QrC!q7Ix53sl9lYCg0iKRHjgOqF0tZJAVKHBNz0OHWof-ds
> >
> > But 'echo w > /proc/sysrq-trigger' no any blocked task in dmesg.
>
> echo l > /proc/sysrq-trigger
> echo t > /proc/sysrq-trigger
>
> -Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 1365 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-31  4:25                 ` Dave Chinner
@ 2014-10-31  4:42                   ` Tommy Wu
  2014-10-31  4:55                     ` Eric Sandeen
  2014-10-31  5:40                   ` Tommy Wu
  1 sibling, 1 reply; 30+ messages in thread
From: Tommy Wu @ 2014-10-31  4:42 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 888 bytes --]

dump whole filesystem?
I'll try dd later.

2014-10-31 12:25 GMT+08:00 Dave Chinner <david@fromorbit.com>:

> On Fri, Oct 31, 2014 at 02:49:25PM +1100, Dave Chinner wrote:
> > On Fri, Oct 31, 2014 at 09:51:27AM +0800, Tommy Wu wrote:
> > > After apply this patch, xfsdump will hang again.
> > >
> > > the log for xfsdump:
> > >
> https://mega.co.nz/#!tUgz3QrC!q7Ix53sl9lYCg0iKRHjgOqF0tZJAVKHBNz0OHWof-ds
> > >
> > > But 'echo w > /proc/sysrq-trigger' no any blocked task in dmesg.
> >
> > echo l > /proc/sysrq-trigger
> > echo t > /proc/sysrq-trigger
>
> Also, I just cannot reproduce a hung dump or a broken restore,
> so can you take an unobfuscated metadump of the source filesystem
> (i.e. the one you are running dump on) so I can try on an exact
> image of the filesystem you are having troubles on?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 1622 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-31  4:42                   ` Tommy Wu
@ 2014-10-31  4:55                     ` Eric Sandeen
  0 siblings, 0 replies; 30+ messages in thread
From: Eric Sandeen @ 2014-10-31  4:55 UTC (permalink / raw)
  To: Tommy Wu, Dave Chinner; +Cc: xfs

On 10/30/14 11:42 PM, Tommy Wu wrote:
> dump whole filesystem?
> I'll try dd later.

no, use xfs_metadump -o, compress & send it (or a link to it) to Dave
on a private channel.

-Eric

> 2014-10-31 12:25 GMT+08:00 Dave Chinner <david@fromorbit.com <mailto:david@fromorbit.com>>:
> 
>     On Fri, Oct 31, 2014 at 02:49:25PM +1100, Dave Chinner wrote:
>     > On Fri, Oct 31, 2014 at 09:51:27AM +0800, Tommy Wu wrote:
>     > > After apply this patch, xfsdump will hang again.
>     > >
>     > > the log for xfsdump:
>     > > https://mega.co.nz/#!tUgz3QrC!q7Ix53sl9lYCg0iKRHjgOqF0tZJAVKHBNz0OHWof-ds
>     > >
>     > > But 'echo w > /proc/sysrq-trigger' no any blocked task in dmesg.
>     >
>     > echo l > /proc/sysrq-trigger
>     > echo t > /proc/sysrq-trigger
> 
>     Also, I just cannot reproduce a hung dump or a broken restore,
>     so can you take an unobfuscated metadump of the source filesystem
>     (i.e. the one you are running dump on) so I can try on an exact
>     image of the filesystem you are having troubles on?
> 
>     Cheers,
> 
>     Dave.
>     --
>     Dave Chinner
>     david@fromorbit.com <mailto:david@fromorbit.com>
> 
> 
> 
> 
> -- 
> 
> Tommy Wu
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-31  4:25                 ` Dave Chinner
  2014-10-31  4:42                   ` Tommy Wu
@ 2014-10-31  5:40                   ` Tommy Wu
  2014-11-07  1:05                     ` Tommy Wu
  1 sibling, 1 reply; 30+ messages in thread
From: Tommy Wu @ 2014-10-31  5:40 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 987 bytes --]

metadump (it's just a test partition with linux 3.17 source code):
https://mega.co.nz/#!FQImhABD!8D6A0ZtXmGPhpm2gtrMBej6xPS5pKF-WM4MZxfGPbD0

2014-10-31 12:25 GMT+08:00 Dave Chinner <david@fromorbit.com>:

> On Fri, Oct 31, 2014 at 02:49:25PM +1100, Dave Chinner wrote:
> > On Fri, Oct 31, 2014 at 09:51:27AM +0800, Tommy Wu wrote:
> > > After apply this patch, xfsdump will hang again.
> > >
> > > the log for xfsdump:
> > >
> https://mega.co.nz/#!tUgz3QrC!q7Ix53sl9lYCg0iKRHjgOqF0tZJAVKHBNz0OHWof-ds
> > >
> > > But 'echo w > /proc/sysrq-trigger' no any blocked task in dmesg.
> >
> > echo l > /proc/sysrq-trigger
> > echo t > /proc/sysrq-trigger
>
> Also, I just cannot reproduce a hung dump or a broken restore,
> so can you take an unobfuscated metadump of the source filesystem
> (i.e. the one you are running dump on) so I can try on an exact
> image of the filesystem you are having troubles on?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 1787 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-10-31  5:40                   ` Tommy Wu
@ 2014-11-07  1:05                     ` Tommy Wu
  2014-11-07  4:37                       ` Dave Chinner
  0 siblings, 1 reply; 30+ messages in thread
From: Tommy Wu @ 2014-11-07  1:05 UTC (permalink / raw)
  To: Dave Chinner; +Cc: xfs


[-- Attachment #1.1: Type: text/plain, Size: 1163 bytes --]

just update to xfs-fixes-for-3.18-rc3.
it work fine now.

2014-10-31 13:40 GMT+08:00 Tommy Wu <wu.tommy@gmail.com>:

> metadump (it's just a test partition with linux 3.17 source code):
> https://mega.co.nz/#!FQImhABD!8D6A0ZtXmGPhpm2gtrMBej6xPS5pKF-WM4MZxfGPbD0
>
> 2014-10-31 12:25 GMT+08:00 Dave Chinner <david@fromorbit.com>:
>
>> On Fri, Oct 31, 2014 at 02:49:25PM +1100, Dave Chinner wrote:
>> > On Fri, Oct 31, 2014 at 09:51:27AM +0800, Tommy Wu wrote:
>> > > After apply this patch, xfsdump will hang again.
>> > >
>> > > the log for xfsdump:
>> > >
>> https://mega.co.nz/#!tUgz3QrC!q7Ix53sl9lYCg0iKRHjgOqF0tZJAVKHBNz0OHWof-ds
>> > >
>> > > But 'echo w > /proc/sysrq-trigger' no any blocked task in dmesg.
>> >
>> > echo l > /proc/sysrq-trigger
>> > echo t > /proc/sysrq-trigger
>>
>> Also, I just cannot reproduce a hung dump or a broken restore,
>> so can you take an unobfuscated metadump of the source filesystem
>> (i.e. the one you are running dump on) so I can try on an exact
>> image of the filesystem you are having troubles on?
>>
>> Cheers,
>>
>> Dave.
>> --
>> Dave Chinner
>> david@fromorbit.com
>>
>
>
>
> --
>
> Tommy Wu
>



-- 

Tommy Wu

[-- Attachment #1.2: Type: text/html, Size: 2365 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
* Re: xfsrestore report SUCCESS but not restore all files in kernel 3.17
  2014-11-07  1:05                     ` Tommy Wu
@ 2014-11-07  4:37                       ` Dave Chinner
  0 siblings, 0 replies; 30+ messages in thread
From: Dave Chinner @ 2014-11-07  4:37 UTC (permalink / raw)
  To: Tommy Wu; +Cc: xfs

On Fri, Nov 07, 2014 at 09:05:55AM +0800, Tommy Wu wrote:
> just update to xfs-fixes-for-3.18-rc3.
> it work fine now.

Thanks for reporting the problem, providing all the information
necessary to fix it and for testing the fixes!

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

^ permalink raw reply	[flat|nested] 30+ messages in thread
end of thread, other threads:[~2014-11-07  4:37 UTC | newest]

Thread overview: 30+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-10-29  4:06 xfsrestore report SUCCESS but not restore all files in kernel 3.17 Tommy Wu
2014-10-29  4:09 ` Eric Sandeen
2014-10-29  4:27   ` Tommy Wu
2014-10-29  4:28     ` Eric Sandeen
2014-10-29  4:30       ` Tommy Wu
2014-10-29  4:40         ` Dave Chinner
2014-10-29  6:46           ` Tommy Wu
2014-10-29  7:51             ` Dave Chinner
2014-10-29  9:03               ` Tommy Wu
2014-10-29  9:21               ` Tommy Wu
2014-10-29 10:33                 ` Dave Chinner
2014-10-29 10:43                   ` Dave Chinner
2014-10-29 12:32                     ` Tommy Wu
2014-10-29 19:50                       ` Dave Chinner
2014-10-29 20:52                         ` Dave Chinner
2014-10-29 21:25 ` Dave Chinner
     [not found]   ` <CAGdb-8dbGRu_GWB4qoeNP8Kso2hzkLcuFS9Pwtnm2xOkP2T0rQ@mail.gmail.com>
2014-10-30  5:03     ` Dave Chinner
2014-10-30  8:39       ` Tommy Wu
2014-10-30 10:39         ` Tommy Wu
2014-10-30 23:04         ` Dave Chinner
2014-10-31  0:09           ` Dave Chinner
2014-10-31  1:51             ` Tommy Wu
2014-10-31  3:49               ` Dave Chinner
2014-10-31  4:25                 ` Dave Chinner
2014-10-31  4:42                   ` Tommy Wu
2014-10-31  4:55                     ` Eric Sandeen
2014-10-31  5:40                   ` Tommy Wu
2014-11-07  1:05                     ` Tommy Wu
2014-11-07  4:37                       ` Dave Chinner
2014-10-31  4:40                 ` Tommy Wu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox