[PATCH v2 0/3] iotests improvements for mirror-sparse and others

[PATCH v2 0/3] iotests improvements for mirror-sparse and others

[Eric Blake]

v1 was here:
https://lists.gnu.org/archive/html/qemu-devel/2025-05/msg05449.html

Since then
 - add more patches
 - don't try sync; that wasn't enough for ZFS

I don't have ZFS locally, so patch 3 WILL need to be tested there (but
I did test that the skip mechanism works on ext4 when I intentionally
tweak the common.rc code to fail failure); and this only addresses the
subset of tests in Fiona's email where I could obviously see some sort
of probing of image sizes, and not every test mentioned.

Eric Blake (3):
  iotests: Use disk_usage in more places
  iotests: Improve mirror-sparse on ext4
  iotests: Filter out ZFS in several tests

 tests/qemu-iotests/common.rc                | 30 +++++++++++++++++++++
 tests/qemu-iotests/106                      |  1 +
 tests/qemu-iotests/125                      |  2 +-
 tests/qemu-iotests/175                      |  1 +
 tests/qemu-iotests/221                      |  1 +
 tests/qemu-iotests/253                      |  1 +
 tests/qemu-iotests/308                      |  5 ++--
 tests/qemu-iotests/tests/mirror-sparse      |  5 +++-
 tests/qemu-iotests/tests/write-zeroes-unmap |  1 +
 9 files changed, 43 insertions(+), 4 deletions(-)

-- 
2.49.0

[PATCH v2 1/3] iotests: Use disk_usage in more places

[Eric Blake]

Commit be9bac07 added a utility disk_usage function, but there are
a couple of other tests that could also use it.

Signed-off-by: Eric Blake <eblake@redhat.com>
---
 tests/qemu-iotests/125 | 2 +-
 tests/qemu-iotests/308 | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/125 b/tests/qemu-iotests/125
index 46279d6b382..708e7c5ba21 100755
--- a/tests/qemu-iotests/125
+++ b/tests/qemu-iotests/125
@@ -35,7 +35,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15

 get_image_size_on_host()
 {
-    echo $(($(stat -c '%b * %B' "$TEST_IMG_FILE")))
+    disk_usage "$TEST_IMG_FILE"
 }

 # get standard environment and filters
diff --git a/tests/qemu-iotests/308 b/tests/qemu-iotests/308
index ea81dc496a0..437a9014da5 100755
--- a/tests/qemu-iotests/308
+++ b/tests/qemu-iotests/308
@@ -290,7 +290,7 @@ echo '--- Try growing non-growable export ---'

 # Get the current size so we can write beyond the EOF
 orig_len=$(get_proto_len "$EXT_MP" "$TEST_IMG")
-orig_disk_usage=$(stat -c '%b' "$TEST_IMG")
+orig_disk_usage=$(disk_usage "$TEST_IMG")

 # Should fail (exports are non-growable by default)
 # (Note that qemu-io can never write beyond the EOF, so we have to use
@@ -312,7 +312,7 @@ else
     echo 'OK: Post-truncate image size is as expected'
 fi

-new_disk_usage=$(stat -c '%b' "$TEST_IMG")
+new_disk_usage=$(disk_usage "$TEST_IMG")
 if [ "$new_disk_usage" -gt "$orig_disk_usage" ]; then
     echo 'OK: Disk usage grew with fallocate'
 else
-- 
2.49.0

[PATCH v2 2/3] iotests: Improve mirror-sparse on ext4

[Eric Blake]

Fiona reported that an ext4 filesystem on top of LVM can sometimes
report over-allocation to du (based on the hueristics the filesystem
is making while observing the contents being mirrored); even though
the contents and actual size matched, about 50% of the time the size
reported by disk_usage was too large by 4k, failing the test.  In
auditing other iotests, this is a common problem we've had to deal
with.

Reported-by: Fiona Ebner <f.ebner@proxmox.com>
Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>

---
v2: Drop sync for ZFS
---
 tests/qemu-iotests/tests/mirror-sparse | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/tests/mirror-sparse b/tests/qemu-iotests/tests/mirror-sparse
index 8c52a4e2448..338d6cfbb35 100755
--- a/tests/qemu-iotests/tests/mirror-sparse
+++ b/tests/qemu-iotests/tests/mirror-sparse
@@ -96,10 +96,12 @@ _send_qemu_cmd $h1 '{"execute": "blockdev-del", "arguments":
                 {"node-name": "dst"}}' 'return' \
                   | _filter_block_job_offset | _filter_block_job_len
 $QEMU_IMG compare -U -f $IMGFMT -F raw $TEST_IMG.base $TEST_IMG
+# Some filesystems can fudge allocations for various reasons; rather
+# than expecting precise 2M and 20M images, it is better to allow for slop.
 result=$(disk_usage $TEST_IMG)
 if test $result -lt $((3*1024*1024)); then
     actual=sparse
-elif test $result = $((20*1024*1024)); then
+elif test $result -gt $((19*1024*1024)); then
     actual=full
 else
     actual=unknown
-- 
2.49.0

[PATCH v2 3/3] iotests: Filter out ZFS in several tests

[Eric Blake]

Fiona reported that ZFS makes sparse file testing awkward, since:
- it has asynchronous allocation (not even 'fsync $file' makes du see
  the desired size; it takes the slower 'fsync -f $file' which is not
  appropriate for the tests)
- for tests of fully allocated files, ZFS with compression enabled
  still reports smaller disk usage

Add a new _require_sparse_file that quickly probes whether an attempt
to create a sparse 5M file shows as less than 1M usage, while the same
file with -o preallocation=full shows as more than 4M usage without
sync, which should filter out ZFS behavior.  Then use it in various
affected tests.

This does not add the new filter on all tests that Fiona is seeing ZFS
failures on, but only those where I could quickly spot that there is
at least one place where the test depends on the output of 'du -b' or
'stat -c %b'.

Reported-by: Fiona Ebner <f.ebner@proxmox.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
---
 tests/qemu-iotests/common.rc                | 30 +++++++++++++++++++++
 tests/qemu-iotests/106                      |  1 +
 tests/qemu-iotests/175                      |  1 +
 tests/qemu-iotests/221                      |  1 +
 tests/qemu-iotests/253                      |  1 +
 tests/qemu-iotests/308                      |  1 +
 tests/qemu-iotests/tests/mirror-sparse      |  1 +
 tests/qemu-iotests/tests/write-zeroes-unmap |  1 +
 8 files changed, 37 insertions(+)

diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests/common.rc
index 237f746af88..e977cb4eb61 100644
--- a/tests/qemu-iotests/common.rc
+++ b/tests/qemu-iotests/common.rc
@@ -996,6 +996,36 @@ _require_large_file()
     rm "$FILENAME"
 }

+# Check whether disk_usage can be reliably used.
+_require_disk_usage()
+{
+    local unusable=false
+    # ZFS triggers known failures on this front; it does not immediately
+    # allocate files, and then aggressively compresses writes even when full
+    # allocation was requested.
+    if [ -z "$TEST_IMG_FILE" ]; then
+        FILENAME="$TEST_IMG"
+    else
+        FILENAME="$TEST_IMG_FILE"
+    fi
+    if [ -e "FILENAME" ]; then
+        echo "unwilling to overwrite existing file"
+        exit 1
+    fi
+    $QEMU_IMG create -f raw "$FILENAME" 5M > /dev/null
+    if [ $(disk_usage "$FILENAME") -gt $((1024*1024)) ]; then
+        unusable=true
+    fi
+    $QEMU_IMG create -f raw -o preallocation=full "$FILENAME" 5M > /dev/null
+    if [ $(disk_usage "$FILENAME") -lt $((4*1024*1024)) ]; then
+        unusable=true
+    fi
+    rm -f "$FILENAME"
+    if $unusable; then
+        _notrun "file system on $TEST_DIR does not handle sparse files nicely"
+    fi
+}
+
 # Check that a set of devices is available in the QEMU binary
 #
 _require_devices()
diff --git a/tests/qemu-iotests/106 b/tests/qemu-iotests/106
index ae0fc466910..55548439aad 100755
--- a/tests/qemu-iotests/106
+++ b/tests/qemu-iotests/106
@@ -40,6 +40,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt raw
 _supported_proto file fuse
 _supported_os Linux
+_require_disk_usage

 # in kB
 CREATION_SIZE=128
diff --git a/tests/qemu-iotests/175 b/tests/qemu-iotests/175
index f74f053b719..bbbf550a5af 100755
--- a/tests/qemu-iotests/175
+++ b/tests/qemu-iotests/175
@@ -77,6 +77,7 @@ _supported_os Linux

 _default_cache_mode none
 _supported_cache_modes none directsync
+_require_disk_usage

 size=$((1 * 1024 * 1024))

diff --git a/tests/qemu-iotests/221 b/tests/qemu-iotests/221
index c463fd4b113..eba00b80adb 100755
--- a/tests/qemu-iotests/221
+++ b/tests/qemu-iotests/221
@@ -41,6 +41,7 @@ _supported_os Linux

 _default_cache_mode writeback
 _supported_cache_modes writeback writethrough unsafe
+_require_disk_usage

 echo
 echo "=== Check mapping of unaligned raw image ==="
diff --git a/tests/qemu-iotests/253 b/tests/qemu-iotests/253
index 35039d20a89..6da85e6a113 100755
--- a/tests/qemu-iotests/253
+++ b/tests/qemu-iotests/253
@@ -41,6 +41,7 @@ _supported_os Linux

 _default_cache_mode none
 _supported_cache_modes none directsync
+_require_disk_usage

 echo
 echo "=== Check mapping of unaligned raw image ==="
diff --git a/tests/qemu-iotests/308 b/tests/qemu-iotests/308
index 437a9014da5..6eced3aefb9 100755
--- a/tests/qemu-iotests/308
+++ b/tests/qemu-iotests/308
@@ -51,6 +51,7 @@ _unsupported_fmt vpc

 _supported_proto file # We create the FUSE export manually
 _supported_os Linux # We need /dev/urandom
+_require_disk_usage

 # $1: Export ID
 # $2: Options (beyond the node-name and ID)
diff --git a/tests/qemu-iotests/tests/mirror-sparse b/tests/qemu-iotests/tests/mirror-sparse
index 338d6cfbb35..39e3196c811 100755
--- a/tests/qemu-iotests/tests/mirror-sparse
+++ b/tests/qemu-iotests/tests/mirror-sparse
@@ -40,6 +40,7 @@ cd ..
 _supported_fmt qcow2 raw  # Format of the source. dst is always raw file
 _supported_proto file
 _supported_os Linux
+_require_disk_usage

 echo
 echo "=== Initial image setup ==="
diff --git a/tests/qemu-iotests/tests/write-zeroes-unmap b/tests/qemu-iotests/tests/write-zeroes-unmap
index 7cfeeaf8391..f90fb8e8d27 100755
--- a/tests/qemu-iotests/tests/write-zeroes-unmap
+++ b/tests/qemu-iotests/tests/write-zeroes-unmap
@@ -32,6 +32,7 @@ cd ..
 _supported_fmt raw
 _supported_proto file
 _supported_os Linux
+_require_disk_usage

 create_test_image() {
     _make_test_img -f $IMGFMT 1m
-- 
2.49.0

Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4

[Eric Blake]

On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
> Fiona reported that an ext4 filesystem on top of LVM can sometimes
> report over-allocation to du (based on the hueristics the filesystem

and despite my other edits, I still missed fixing my typo on heuristics

> is making while observing the contents being mirrored); even though
> the contents and actual size matched, about 50% of the time the size
> reported by disk_usage was too large by 4k, failing the test.  In
> auditing other iotests, this is a common problem we've had to deal
> with.
> 
> Reported-by: Fiona Ebner <f.ebner@proxmox.com>
> Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> Signed-off-by: Eric Blake <eblake@redhat.com>
> Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
> Tested-by: Fiona Ebner <f.ebner@proxmox.com>
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: [PATCH v2 1/3] iotests: Use disk_usage in more places

[Fiona Ebner]

Am 23.05.25 um 18:27 schrieb Eric Blake:
> Commit be9bac07 added a utility disk_usage function, but there are
> a couple of other tests that could also use it.
> 
> Signed-off-by: Eric Blake <eblake@redhat.com>

Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>

Re: [PATCH v2 3/3] iotests: Filter out ZFS in several tests

[Fiona Ebner]

Am 23.05.25 um 18:27 schrieb Eric Blake:
> Fiona reported that ZFS makes sparse file testing awkward, since:
> - it has asynchronous allocation (not even 'fsync $file' makes du see
>   the desired size; it takes the slower 'fsync -f $file' which is not
>   appropriate for the tests)
> - for tests of fully allocated files, ZFS with compression enabled
>   still reports smaller disk usage
> 
> Add a new _require_sparse_file that quickly probes whether an attempt

It's called _require_disk_usage in the implementation.

> to create a sparse 5M file shows as less than 1M usage, while the same
> file with -o preallocation=full shows as more than 4M usage without
> sync, which should filter out ZFS behavior.  Then use it in various
> affected tests.
> 
> This does not add the new filter on all tests that Fiona is seeing ZFS
> failures on, but only those where I could quickly spot that there is
> at least one place where the test depends on the output of 'du -b' or
> 'stat -c %b'.
> 
> Reported-by: Fiona Ebner <f.ebner@proxmox.com>
> Signed-off-by: Eric Blake <eblake@redhat.com>

Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>

Re: [PATCH v2 3/3] iotests: Filter out ZFS in several tests

[Eric Blake]

On Mon, May 26, 2025 at 11:42:24AM +0200, Fiona Ebner wrote:
> Am 23.05.25 um 18:27 schrieb Eric Blake:
> > Fiona reported that ZFS makes sparse file testing awkward, since:
> > - it has asynchronous allocation (not even 'fsync $file' makes du see
> >   the desired size; it takes the slower 'fsync -f $file' which is not
> >   appropriate for the tests)
> > - for tests of fully allocated files, ZFS with compression enabled
> >   still reports smaller disk usage
> > 
> > Add a new _require_sparse_file that quickly probes whether an attempt
> 
> It's called _require_disk_usage in the implementation.

I'll fix it (obviously, my efforts to rebase to a better name missed
this one.)

> 
> > to create a sparse 5M file shows as less than 1M usage, while the same
> > file with -o preallocation=full shows as more than 4M usage without
> > sync, which should filter out ZFS behavior.  Then use it in various
> > affected tests.
> > 
> > This does not add the new filter on all tests that Fiona is seeing ZFS
> > failures on, but only those where I could quickly spot that there is
> > at least one place where the test depends on the output of 'du -b' or
> > 'stat -c %b'.
> > 
> > Reported-by: Fiona Ebner <f.ebner@proxmox.com>
> > Signed-off-by: Eric Blake <eblake@redhat.com>
> 
> Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
> Tested-by: Fiona Ebner <f.ebner@proxmox.com>

Series is now queued in my NBD tree.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4

[Eric Blake]

On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
> Fiona reported that an ext4 filesystem on top of LVM can sometimes
> report over-allocation to du (based on the hueristics the filesystem
> is making while observing the contents being mirrored); even though
> the contents and actual size matched, about 50% of the time the size
> reported by disk_usage was too large by 4k, failing the test.  In
> auditing other iotests, this is a common problem we've had to deal
> with.
> 
> Reported-by: Fiona Ebner <f.ebner@proxmox.com>
> Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> Signed-off-by: Eric Blake <eblake@redhat.com>
> Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
> Tested-by: Fiona Ebner <f.ebner@proxmox.com>
> 
> ---
> v2: Drop sync for ZFS
> ---
>  tests/qemu-iotests/tests/mirror-sparse | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)

Squashing this in since Markus is reporting failures even with this
patch applied, and I'm lacking enough context to see what else might
be interfering:

diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
index 39e3196c811..a30e52de41d 100755
--- i/tests/qemu-iotests/tests/mirror-sparse
+++ w/tests/qemu-iotests/tests/mirror-sparse
@@ -105,7 +105,7 @@ if test $result -lt $((3*1024*1024)); then
 elif test $result -gt $((19*1024*1024)); then
     actual=full
 else
-    actual=unknown
+    actual="unexpected size ($result)"
 fi
 echo "Destination is $actual; expected $expected"
 }


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4

[Eric Blake]

On Wed, May 28, 2025 at 10:45:35AM -0500, Eric Blake wrote:
> On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
> > Fiona reported that an ext4 filesystem on top of LVM can sometimes
> > report over-allocation to du (based on the hueristics the filesystem
> > is making while observing the contents being mirrored); even though
> > the contents and actual size matched, about 50% of the time the size
> > reported by disk_usage was too large by 4k, failing the test.  In
> > auditing other iotests, this is a common problem we've had to deal
> > with.
> > 
> > Reported-by: Fiona Ebner <f.ebner@proxmox.com>
> > Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> > Signed-off-by: Eric Blake <eblake@redhat.com>
> > Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
> > Tested-by: Fiona Ebner <f.ebner@proxmox.com>
> > 
> > ---
> > v2: Drop sync for ZFS
> > ---
> >  tests/qemu-iotests/tests/mirror-sparse | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> Squashing this in since Markus is reporting failures even with this
> patch applied, and I'm lacking enough context to see what else might
> be interfering:
> 
> diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
> index 39e3196c811..a30e52de41d 100755
> --- i/tests/qemu-iotests/tests/mirror-sparse
> +++ w/tests/qemu-iotests/tests/mirror-sparse
> @@ -105,7 +105,7 @@ if test $result -lt $((3*1024*1024)); then
>  elif test $result -gt $((19*1024*1024)); then
>      actual=full
>  else
> -    actual=unknown
> +    actual="unexpected size ($result)"

I reproduced Markus' situation on XFS (since XFS is easier than ZFS to
install on Fedora) - it turns out that XFS rounds disk usage in terms
of allocation groups which default to a size of 1M; so I need to treat
exactly 3M as sparse, not unknown.  I'm also squashing in:

diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
index a30e52de41d..cfcaa600ab4 100755
--- i/tests/qemu-iotests/tests/mirror-sparse
+++ w/tests/qemu-iotests/tests/mirror-sparse
@@ -100,7 +100,7 @@ $QEMU_IMG compare -U -f $IMGFMT -F raw $TEST_IMG.base $TEST_IMG
 # Some filesystems can fudge allocations for various reasons; rather
 # than expecting precise 2M and 20M images, it is better to allow for slop.
 result=$(disk_usage $TEST_IMG)
-if test $result -lt $((3*1024*1024)); then
+if test $result -lt $((4*1024*1024)); then
     actual=sparse
 elif test $result -gt $((19*1024*1024)); then
     actual=full


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On Wed, May 28, 2025 at 10:45:35AM -0500, Eric Blake wrote:
>> On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
>> > Fiona reported that an ext4 filesystem on top of LVM can sometimes
>> > report over-allocation to du (based on the hueristics the filesystem
>> > is making while observing the contents being mirrored); even though
>> > the contents and actual size matched, about 50% of the time the size
>> > reported by disk_usage was too large by 4k, failing the test.  In
>> > auditing other iotests, this is a common problem we've had to deal
>> > with.
>> > 
>> > Reported-by: Fiona Ebner <f.ebner@proxmox.com>
>> > Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
>> > Signed-off-by: Eric Blake <eblake@redhat.com>
>> > Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
>> > Tested-by: Fiona Ebner <f.ebner@proxmox.com>
>> > 
>> > ---
>> > v2: Drop sync for ZFS
>> > ---
>> >  tests/qemu-iotests/tests/mirror-sparse | 4 +++-
>> >  1 file changed, 3 insertions(+), 1 deletion(-)
>> 
>> Squashing this in since Markus is reporting failures even with this
>> patch applied, and I'm lacking enough context to see what else might
>> be interfering:
>> 
>> diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
>> index 39e3196c811..a30e52de41d 100755
>> --- i/tests/qemu-iotests/tests/mirror-sparse
>> +++ w/tests/qemu-iotests/tests/mirror-sparse
>> @@ -105,7 +105,7 @@ if test $result -lt $((3*1024*1024)); then
>>  elif test $result -gt $((19*1024*1024)); then
>>      actual=full
>>  else
>> -    actual=unknown
>> +    actual="unexpected size ($result)"
>
> I reproduced Markus' situation on XFS (since XFS is easier than ZFS to
> install on Fedora) - it turns out that XFS rounds disk usage in terms
> of allocation groups which default to a size of 1M; so I need to treat
> exactly 3M as sparse, not unknown.  I'm also squashing in:
>
> diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
> index a30e52de41d..cfcaa600ab4 100755
> --- i/tests/qemu-iotests/tests/mirror-sparse
> +++ w/tests/qemu-iotests/tests/mirror-sparse
> @@ -100,7 +100,7 @@ $QEMU_IMG compare -U -f $IMGFMT -F raw $TEST_IMG.base $TEST_IMG
>  # Some filesystems can fudge allocations for various reasons; rather
>  # than expecting precise 2M and 20M images, it is better to allow for slop.
>  result=$(disk_usage $TEST_IMG)
> -if test $result -lt $((3*1024*1024)); then
> +if test $result -lt $((4*1024*1024)); then
>      actual=sparse
>  elif test $result -gt $((19*1024*1024)); then
>      actual=full

This one-liner appears to fix it for me.  Thanks!