Problems Implementing Snappy Patches

Problems Implementing Snappy Patches

[Mitch Harder]

I've been trying to test the snappy compression patches, but I'm
getting corruptions when trying to use snappy as built on my system.

I'm checking out the Linux 3.2.6 kernel, merging that with the latest
'for-linus' branch on Chris Mason's kernel.org repo, and then
integrating the snappy and lz4 patches from David Sterba's git
repository (dev/compression-squad branch).

I've tried a simple merge of the dev/compression-squad branch (which
merged without complaint by git), and I also tried a second build of
the kernel integrating the dev/compression-squad branch patches with
format-patch (git format-patch -k -m -U5 --stdout <range> | git am -3
-k).

I've also tried sourcing my snappy patches from Chris Mason's snappy
branch on kernel.org with the same result.

I get the same results either way.

My system is a Core 2 Duo x86_64 Sabayon based system (Gentoo is our
parent distro).  My target btrfs/snappy partition is a 16 GB partition
I use for testing on a 500GB Western Digital Hard Drive.

When I copy directory containing a kernel sources git repository to a
freshly formated partition mounted with snappy, I get a corrupted
copy.  If I mount with lzo or lz4 compression, I don't see any
corruptions from the copy.

I'm not showing any errors in dmesg.

My procedure to replicate is as follows:

mkfs.btrfs -m single /dev/sdb6
mount -o compress-force=snappy,autodefrag /dev/sdb6 /mnt/benchmark/
cp -a /var/tmp/portage/subdir1 /mnt/benchmark/
diff -Naur /var/tmp/portage/subdir1 /mnt/benchmark/subdir1 | less

I've spot checked the corrupted files with filefrag, and the one's
I've checked are *NOT* in-lined.

The following is a brief, representative excerpt of the diff (the full
diff is very long):

diff -Naur /var/tmp/portage/subdir1/linux-btrfs-backport/drivers/net/bnx2x/bnx2x_reg.h
/mnt/benchmark/subdir1/linux-btrfs-backport/drivers/net/bnx2x/bnx2x_reg.h
--- /var/tmp/portage/subdir1/linux-btrfs-backport/drivers/net/bnx2x/bnx2x_reg.h
2011-01-12 16:31:39.000000000 -0600
+++ /mnt/benchmark/subdir1/linux-btrfs-backport/drivers/net/bnx2x/bnx2x_reg.h
  2011-01-12 16:31:39.000000000 -0600
@@ -4051,76 +4051,73 @@
 #define UCM_REG_TM_INIT_CRD                                     0xe021c
 /* [RW 28] The CM header for Timers expiration command. */
 #define UCM_REG_TM_UCM_HDR                                      0xe009c
-/* [RW 1] Timers - CM Interface enable. If 0 - the valid input is
+/* [RW  Timimers - CM Interface enable. If 0 - the valid input is
    disregarded; acknowledge output is deasserted; all other signals are
    treated as usual; if 1 - normal activity. */
-#define UCM_REG_TM_UCM_IFEN                                     0xe001c
-/* [RW 3] The weight of the Timers input in the WRR mechanism. 0 stands for
-   weight 8 (the most prioritised); 1 stands for weight 1(least
-   prioritised); 2 stands for weight 2; tc. */
+#define UCM_REG_TM_UCM_IFEN                                     0xe00
+/* * [RW 3] The weight of the mers input i in the WRR mechanism. 0 stands for
+   ightht 8 (the most prioritised); 1 stands foreighght 1(least
+   ioritised); ; 2 stands for weight 2; tc. */
 #define UCM_REG_TM_WEIGHT                                       0xe00d4
-/* [RW 1] Input tsem Interface enable. If 0 - the valid input is
+/* [RW  TiInput tsem Interface enable. If 0 - the valid input is
    disregarded; acknowledge output is deasserted; all other signals are
    treated as usual; if 1 - normal activity. */
-#define UCM_REG_TSEM_IFEN                                       0xe0024
-/* [RC 1] Set when the message length mismatch (relative to last indication)
-   at the tsem interface is detected. */
+#define UCM_REG_TSEM_IFEN                                       0xe00
+/* * [RC 1] Set when the message length mismatch (relative to last indication)
+    thehe tsem terfrface is detected. */
 #define UCM_REG_TSEM_LENGTH_MIS                                 0xe015c


Here's a list of the top commits in my git repository:

54ca0b5 btrfs: lz4: tune speed/compression ratio
7ead5fb btrfs: lz4: add wrapper functions and enable it
3a611a7 btrfs: lz4: add wrapper for context size estimation
ae2de6b btrfs: add LZ4 compression method
56e0aa6 btrfs: prepare incompat flags for more compression methods
540f832 Btrfs: fix decompressing of snappy-compressed inline extents
8f8b19f8 Btrfs: fix incompat flags setting
369cc38 SNAPPY: Add dual GPL/BSD license to snappy module
b479967 Add snappy interface to crypto API
e68c69a BTRFS: Add snappy support v2
428ac51 Add the snappy-c compressor to lib v2
4d0defa Merge branch 'for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs into
linux-3.2.6-btrfs-lz4-v6.3
c2db2e2 Linux 3.2.6

Re: Problems Implementing Snappy Patches

[Duncan]

Mitch Harder posted on Wed, 15 Feb 2012 16:46:00 -0600 as excerpted:

> I've been trying to test the snappy compression patches, but I'm getting
> corruptions when trying to use snappy as built on my system.
> 
> I'm checking out the Linux 3.2.6 kernel, merging that with the latest
> 'for-linus' branch on Chris Mason's kernel.org repo, and then
> integrating the snappy and lz4 patches from David Sterba's git
> repository (dev/compression-squad branch).

Given that we're past the 3.3 kernel merge window, wouldn't Chris's "for-
linus" branch be based on the patches that went into 3.3, now?  IOW, 
you're basing on 3.2.x but AFAIK you should be basing on 3.3, now, so 
you're missing the 3.3 patches which CM's for-linus tree branch should be 
assuming, at this point.

Unless you know otherwise of course.  I run the mainline tree here, and 
don't actually run anything btrfs yet, as once I started investigating I 
realized that it's missing features (like multi-mirror not just dual-
mirror) that I need, and isn't yet as stable as I had hoped, either.  So 
I don't know that much about the specifics of CM's tree, but in the 
general case, a for-linus branch would be based on 3.3 now, since it's 
well past the merge window.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Problems Implementing Snappy Patches

[Mitch Harder]

On Wed, Feb 15, 2012 at 5:03 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Mitch Harder posted on Wed, 15 Feb 2012 16:46:00 -0600 as excerpted:
>
>> I've been trying to test the snappy compression patches, but I'm get=
ting
>> corruptions when trying to use snappy as built on my system.
>>
>> I'm checking out the Linux 3.2.6 kernel, merging that with the lates=
t
>> 'for-linus' branch on Chris Mason's kernel.org repo, and then
>> integrating the snappy and lz4 patches from David Sterba's git
>> repository (dev/compression-squad branch).
>
> Given that we're past the 3.3 kernel merge window, wouldn't Chris's "=
for-
> linus" branch be based on the patches that went into 3.3, now? =A0IOW=
,
> you're basing on 3.2.x but AFAIK you should be basing on 3.3, now, so
> you're missing the 3.3 patches which CM's for-linus tree branch shoul=
d be
> assuming, at this point.
>

Chris's 'for-linus' branch is currently based on 3.2 (but not 3.2.6).
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems Implementing Snappy Patches

[Li Zefan]

Mitch Harder wrote:
> I've been trying to test the snappy compression patches, but I'm
> getting corruptions when trying to use snappy as built on my system.
> 
> I'm checking out the Linux 3.2.6 kernel, merging that with the latest
> 'for-linus' branch on Chris Mason's kernel.org repo, and then
> integrating the snappy and lz4 patches from David Sterba's git
> repository (dev/compression-squad branch).
> 
> I've tried a simple merge of the dev/compression-squad branch (which
> merged without complaint by git), and I also tried a second build of
> the kernel integrating the dev/compression-squad branch patches with
> format-patch (git format-patch -k -m -U5 --stdout <range> | git am -3
> -k).
> 
> I've also tried sourcing my snappy patches from Chris Mason's snappy
> branch on kernel.org with the same result.
> 

A month ago I saved emails as patches and applied them on linux-btrfs'
for-linus branch, and I got the same problem. Then I used the snappy
branch (plus my fix), and it was fine. Don't know why.

> I get the same results either way.
> 
> My system is a Core 2 Duo x86_64 Sabayon based system (Gentoo is our
> parent distro).  My target btrfs/snappy partition is a 16 GB partition
> I use for testing on a 500GB Western Digital Hard Drive.
> 
> When I copy directory containing a kernel sources git repository to a
> freshly formated partition mounted with snappy, I get a corrupted
> copy.  If I mount with lzo or lz4 compression, I don't see any
> corruptions from the copy.
> 
> I'm not showing any errors in dmesg.
> 

You're not seeing any errors, right?

Re: Problems Implementing Snappy Patches

[Mitch Harder]

On Wed, Feb 15, 2012 at 8:06 PM, Li Zefan <lizf@cn.fujitsu.com> wrote:
> Mitch Harder wrote:
>> When I copy directory containing a kernel sources git repository to =
a
>> freshly formated partition mounted with snappy, I get a corrupted
>> copy. =A0If I mount with lzo or lz4 compression, I don't see any
>> corruptions from the copy.
>>
>> I'm not showing any errors in dmesg.
>>
>
> You're not seeing any errors, right?

That is correct.

I'm not seeing any errors thrown out directly from the copy command,
nor am I seeing anything else in the dmesg log.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Problems Implementing Snappy Patches

[David Sterba]

On Thu, Feb 16, 2012 at 10:06:13AM +0800, Li Zefan wrote:
> Mitch Harder wrote:
> > I've been trying to test the snappy compression patches, but I'm
> > getting corruptions when trying to use snappy as built on my system.
> > 
> > I've also tried sourcing my snappy patches from Chris Mason's snappy
> > branch on kernel.org with the same result.
> > 
> 
> A month ago I saved emails as patches and applied them on linux-btrfs'
> for-linus branch, and I got the same problem. Then I used the snappy
> branch (plus my fix), and it was fine. Don't know why.

The decompression corruptions Mitch reported look same as what you sent.
And I don't think they are caused by misapplied patches, I've checked
both 'mbox via mutt' -> git or pull from Chris' branch, + your patches to
fix the inlined extents, no other changes.

I did this test:
* mkfs
* mount with compress-force=snappy
* copy linux-3.2
* drop caches  (or umount/mount)
* check md5sums from original

Result:
md5sum: WARNING: 32986 computed checksums did NOT match

there are 37617 files in linux-3.2/

syslog says just:
[  728.744179] Btrfs loaded
[  728.746332] device fsid 194bbb84-1f87-4a37-8091-957d557633f1 devid 1 transid 4 /dev/sda9
[  728.747693] btrfs: enabling auto defrag
[  728.747698] btrfs: enabling inode map caching
[  728.747702] btrfs: use snappy compression
[  728.747704] btrfs: disk space caching is enabled

> > My system is a Core 2 Duo x86_64 Sabayon based system (Gentoo is our
> > parent distro).  My target btrfs/snappy partition is a 16 GB partition
> > I use for testing on a 500GB Western Digital Hard Drive.

I use the same testbox and disk as always, no related hw problems in
syslog.


david

Re: Problems Implementing Snappy Patches

[Mitch Harder]

Just to add another data point, I built Chris Mason's 'snappy' branch
from git.kernel.org as-is (a 3.2.0 kernel plus the snappy patches)
with no additional patches.

This 3.2.0 snappy kernel also exhibits the problem copying the kernel
sources to a snappy-compressed partition.

Re: Problems Implementing Snappy Patches

[David Sterba]

On Wed, Feb 15, 2012 at 04:46:00PM -0600, Mitch Harder wrote:
> I've been trying to test the snappy compression patches, but I'm
> getting corruptions when trying to use snappy as built on my system.

I went through the original C++ implementation (svn release r52) and
compared it to the C implementation. I did not see any problems with it,
it's direct transformation from C++ -> C, nothing like off-by-one
somewhere or typos in condition expressions. No clue where the
corruption comes from.


david