3.17.0+ files disappearing after playing old dos game on nfsroot laptop

3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Hans de Bruin]

After playing an old dos game i am missing files on my nfsroot installed 
laptop. Which one, wel /bin/ls is at least one of them. After a reboot 
there all back again.  But not al is well. Some icons on my kde panel 
where gone. I have seen this twice in the last day's

-- 
Hans

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Hans de Bruin]

On 10/15/2014 10:00 PM, Hans de Bruin wrote:
> After playing an old dos game i am missing files on my nfsroot installed
> laptop. Which one, wel /bin/ls is at least one of them. After a reboot
> there all back again.  But not al is well. Some icons on my kde panel
> where gone. I have seen this twice in the last day's
>

The problem appears immediately after starting dosemu so this bisectable

-- 
Hans

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Hans de Bruin]

On 10/19/2014 07:32 PM, Hans de Bruin wrote:
> On 10/15/2014 10:00 PM, Hans de Bruin wrote:
>> After playing an old dos game i am missing files on my nfsroot installed
>> laptop. Which one, wel /bin/ls is at least one of them. After a reboot
>> there all back again.  But not al is well. Some icons on my kde panel
>> where gone. I have seen this twice in the last day's
>>
>
> The problem appears immediately after starting dosemu so this bisectable
>


My bisect ended here:

commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe
Author: Eric W. Biederman <ebiederman@twitter.com>
Date:   Tue Oct 1 18:33:48 2013 -0700


     vfs: Lazily remove mounts on unlinked files and directories.


I haven reverted it yet.



Eric,

Immediately after starting dosemu all files under mount point /usr are 
gone. When I play a old dos game for a while (binaries are under /home) 
even /bin/ls disappears. Al mounts are nfs, even /.

Could you look in to this and cc this to the relevant kernel mailinglist?

-- 
Hans

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Hans de Bruin]

On 10/19/2014 07:32 PM, Hans de Bruin wrote:
> On 10/15/2014 10:00 PM, Hans de Bruin wrote:
>> After playing an old dos game i am missing files on my nfsroot installed
>> laptop. Which one, wel /bin/ls is at least one of them. After a reboot
>> there all back again.  But not al is well. Some icons on my kde panel
>> where gone. I have seen this twice in the last day's
>>
>
> The problem appears immediately after starting dosemu so this bisectable
>


My bisect ended here:

commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe
Author: Eric W. Biederman <ebiederman@twitter.com>
Date:   Tue Oct 1 18:33:48 2013 -0700


      vfs: Lazily remove mounts on unlinked files and directories.


I haven reverted it yet.



Eric,

Immediately after starting dosemu all files under mount point /usr are
gone. When I play a old dos game for a while (binaries are under /home)
even /bin/ls disappears. Al mounts are nfs, even /.

Could you look in to this and cc this to the relevant kernel mailinglist?

-- 
Hans

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Eric W. Biederman]

Hans de Bruin <jmdebruin@xmsnet.nl> writes:

> On 10/19/2014 07:32 PM, Hans de Bruin wrote:
>> On 10/15/2014 10:00 PM, Hans de Bruin wrote:
>>> After playing an old dos game i am missing files on my nfsroot installed
>>> laptop. Which one, wel /bin/ls is at least one of them. After a reboot
>>> there all back again.  But not al is well. Some icons on my kde panel
>>> where gone. I have seen this twice in the last day's
>>>
>>
>> The problem appears immediately after starting dosemu so this bisectable
>>
>
>
> My bisect ended here:
>
> commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe
> Author: Eric W. Biederman <ebiederman@twitter.com>
> Date:   Tue Oct 1 18:33:48 2013 -0700
>
>
>      vfs: Lazily remove mounts on unlinked files and directories.
>
>
> I haven reverted it yet.
>
>
>
> Eric,
>
> Immediately after starting dosemu all files under mount point /usr are
> gone. When I play a old dos game for a while (binaries are under /home)
> even /bin/ls disappears. Al mounts are nfs, even /.
>
> Could you look in to this and cc this to the relevant kernel
> mailinglist?

At this point I don't know enough to reproduce this.
What does /proc/mounts look like before you start dosemu?

My expectation is that you should only see this if the mount points are
removed on the nfs server (which does not sound like it is the case).

Although a transient malfunction of the nfs server or misplaced call to
check_submounts_and_drop could cause mounts to disappear as well.  
During testing autofs was observed to have an inappropriate call
to d_invalidate and it is unlikely but possible something like
that is going on with nfs as well.

Are your nfs mounts read-only or read-write?

What is your nfs-server and what is it exporting?
Which distro are you running?
Which version of dosemu are you running?
How is dosemu configured to access files on your filesystem?

Eric

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Hans de Bruin]

On 10/24/2014 08:18 PM, Eric W. Biederman wrote:
> Hans de Bruin <jmdebruin@xmsnet.nl> writes:
>
>> On 10/19/2014 07:32 PM, Hans de Bruin wrote:
>>> On 10/15/2014 10:00 PM, Hans de Bruin wrote:
>>>> After playing an old dos game i am missing files on my nfsroot installed
>>>> laptop. Which one, wel /bin/ls is at least one of them. After a reboot
>>>> there all back again.  But not al is well. Some icons on my kde panel
>>>> where gone. I have seen this twice in the last day's
>>>>
>>>
>>> The problem appears immediately after starting dosemu so this bisectable
>>>
>>
>>
>> My bisect ended here:
>>
>> commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe
>> Author: Eric W. Biederman <ebiederman@twitter.com>
>> Date:   Tue Oct 1 18:33:48 2013 -0700
>>
>>
>>       vfs: Lazily remove mounts on unlinked files and directories.
>>
>>
>> I haven reverted it yet.
>>
>>
>>
>> Eric,
>>
>> Immediately after starting dosemu all files under mount point /usr are
>> gone. When I play a old dos game for a while (binaries are under /home)
>> even /bin/ls disappears. Al mounts are nfs, even /.
>>
>> Could you look in to this and cc this to the relevant kernel
>> mailinglist?
>
> At this point I don't know enough to reproduce this.
> What does /proc/mounts look like before you start dosemu?

bash-4.2$ cat /proc/mounts
rootfs / rootfs rw 0 0
10.10.0.1:/nfs/root/psion_14.1 / nfs 
rw,relatime,vers=3,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountproto=tcp,local_lock=all,addr=10.10.0.1 
0 0
devtmpfs /dev devtmpfs 
rw,relatime,size=1031016k,nr_inodes=220978,mode=755 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
tmpfs /run tmpfs rw,relatime,mode=755 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620 0 0
cgroup /sys/fs/cgroup cgroup rw,relatime,cpu 0 0
/dev/shm /tmp tmpfs rw,relatime,size=524288k 0 0
/dev/shm /dev/shm tmpfs rw,relatime,size=524288k 0 0
nfs:/nfs/usr/slackware-14.1/usr /usr nfs 
ro,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 
0 0
nfs:/nfs/home /home nfs 
rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 
0 0
nfs:/nfs/mp3 /mp3 nfs 
rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 
0 0
nfs:/nfs/src /usr/src nfs 
rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 
0 0
nfs:/nfs/video /video nfs 
rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1 
0 0

I now know why I do not see any file in /usr after running dosemu. The 
whole /usr mount disappears in /proc/mounts. When I remount it I have a 
usable laptop again. Running dosemu a second time does not remove the 
mount again. In the mean time I have seen /usr disappear after running 
other programs like xterm and firefox. But until now never after 
remouting it.

>
> My expectation is that you should only see this if the mount points are
> removed on the nfs server (which does not sound like it is the case).

This is a at home environment with a nfs server in the meter cupboard. I 
have not changed the exports.


> Although a transient malfunction of the nfs server or misplaced call to
> check_submounts_and_drop could cause mounts to disappear as well.
> During testing autofs was observed to have an inappropriate call

I am not using autofs

> to d_invalidate and it is unlikely but possible something like
> that is going on with nfs as well.
>
> Are your nfs mounts read-only or read-write?

/usr is mounted ro and exported ro

>
> What is your nfs-server and what is it exporting?
> Which distro are you running?

The nfs-server is slackware64 14.0 with a 3.4 kernel
part from exports:
/nfs/usr       -ro,async,no_subtree_check 10.10.0.0/16

The laptop is slackware(32 bit) 14.1 with yesterday's linus kernel

> Which version of dosemu are you running?

1.4.0.8

> How is dosemu configured to access files on your filesystem?

I do not understand what you mean by this. It uses ~/.dosemu/drive_c

-- 
Hans

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Eric W. Biederman]

Hans de Bruin <jmdebruin@xmsnet.nl> writes:

> On 10/24/2014 08:18 PM, Eric W. Biederman wrote:

[...]

>> At this point I don't know enough to reproduce this.
>> What does /proc/mounts look like before you start dosemu?
>
> bash-4.2$ cat /proc/mounts
> rootfs / rootfs rw 0 0
> 10.10.0.1:/nfs/root/psion_14.1 / nfs
> rw,relatime,vers=3,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountproto=tcp,local_lock=all,addr=10.10.0.1
> 0 0
> devtmpfs /dev devtmpfs rw,relatime,size=1031016k,nr_inodes=220978,mode=755 0 0
> proc /proc proc rw,relatime 0 0
> sysfs /sys sysfs rw,relatime 0 0
> tmpfs /run tmpfs rw,relatime,mode=755 0 0
> devpts /dev/pts devpts rw,relatime,gid=5,mode=620 0 0
> cgroup /sys/fs/cgroup cgroup rw,relatime,cpu 0 0
> /dev/shm /tmp tmpfs rw,relatime,size=524288k 0 0
> /dev/shm /dev/shm tmpfs rw,relatime,size=524288k 0 0
> nfs:/nfs/usr/slackware-14.1/usr /usr nfs
> ro,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
> 0 0
> nfs:/nfs/home /home nfs
> rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
> 0 0
> nfs:/nfs/mp3 /mp3 nfs
> rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
> 0 0
> nfs:/nfs/src /usr/src nfs
> rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
> 0 0
> nfs:/nfs/video /video nfs
> rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
> 0 0
>
> I now know why I do not see any file in /usr after running dosemu. The whole
> /usr mount disappears in /proc/mounts. When I remount it I have a usable laptop
> again. Running dosemu a second time does not remove the mount again. In the mean
> time I have seen /usr disappear after running other programs like xterm and
> firefox. But until now never after remouting it.

That is interesting.

It sounds like occassionally your /home mount disappears as well.

Both of those would be consistent with my patch working doing what it
was designed to do.

My guess is that your nfs mount on / is being weird, and causing valid
directory dentries (AKA /usr and /home) to disappear temporariliy, and
it looks like my change is magnifying that weirdness by causing the
mounts on those directory entries to disappear.

What I don't understand is exactly why nfs is being weird that way yet.

I expect what needs to happen is to confirm that nfs directory entry
revalidation is buggy, and at least for the short term re-add the nfs
logic that will avoid dropping a dentry if it is a mount point, or
path to a mount point, to avoid the nfs bugs.

>> My expectation is that you should only see this if the mount points are
>> removed on the nfs server (which does not sound like it is the case).
>
> This is a at home environment with a nfs server in the meter cupboard. I have
> not changed the exports.

On the nfs server doing "rmdir .../nfs/posion_14.1/home" or
"rmdir .../nfs/posion_14.1/usr" should cause the behavior you are
seeing.

As that you aren't doing that something is weird is going on.

>> Although a transient malfunction of the nfs server or misplaced call to
>> check_submounts_and_drop could cause mounts to disappear as well.
>> During testing autofs was observed to have an inappropriate call
>
> I am not using autofs

I mentioned autofs because I think something similar is probably going
on with nfs, as was observed with autofs.

>> to d_invalidate and it is unlikely but possible something like
>> that is going on with nfs as well.
>>
>> Are your nfs mounts read-only or read-write?

> /usr is mounted ro and exported ro

/ is mounted read-write but that seems seems immaterial at the moment.

>> What is your nfs-server and what is it exporting?
>> Which distro are you running?
>
> The nfs-server is slackware64 14.0 with a 3.4 kernel
> part from exports:
> /nfs/usr       -ro,async,no_subtree_check 10.10.0.0/16
>
> The laptop is slackware(32 bit) 14.1 with yesterday's linus kernel
>
>> Which version of dosemu are you running?
>
> 1.4.0.8

Thanks that information helps.

Eric

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Hans de Bruin]

On 10/25/2014 04:55 PM, Eric W. Biederman wrote:
> Hans de Bruin <jmdebruin@xmsnet.nl> writes:
>
>> On 10/24/2014 08:18 PM, Eric W. Biederman wrote:
>
> [...]
>
>>> At this point I don't know enough to reproduce this.
>>> What does /proc/mounts look like before you start dosemu?
>>
>> bash-4.2$ cat /proc/mounts
>> rootfs / rootfs rw 0 0
>> 10.10.0.1:/nfs/root/psion_14.1 / nfs
>> rw,relatime,vers=3,rsize=4096,wsize=4096,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountproto=tcp,local_lock=all,addr=10.10.0.1
>> 0 0
>> devtmpfs /dev devtmpfs rw,relatime,size=1031016k,nr_inodes=220978,mode=755 0 0
>> proc /proc proc rw,relatime 0 0
>> sysfs /sys sysfs rw,relatime 0 0
>> tmpfs /run tmpfs rw,relatime,mode=755 0 0
>> devpts /dev/pts devpts rw,relatime,gid=5,mode=620 0 0
>> cgroup /sys/fs/cgroup cgroup rw,relatime,cpu 0 0
>> /dev/shm /tmp tmpfs rw,relatime,size=524288k 0 0
>> /dev/shm /dev/shm tmpfs rw,relatime,size=524288k 0 0
>> nfs:/nfs/usr/slackware-14.1/usr /usr nfs
>> ro,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
>> 0 0
>> nfs:/nfs/home /home nfs
>> rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
>> 0 0
>> nfs:/nfs/mp3 /mp3 nfs
>> rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
>> 0 0
>> nfs:/nfs/src /usr/src nfs
>> rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
>> 0 0
>> nfs:/nfs/video /video nfs
>> rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.0.1,mountvers=3,mountport=38337,mountproto=udp,local_lock=none,addr=10.10.0.1
>> 0 0
>>
>> I now know why I do not see any file in /usr after running dosemu. The whole
>> /usr mount disappears in /proc/mounts. When I remount it I have a usable laptop
>> again. Running dosemu a second time does not remove the mount again. In the mean
>> time I have seen /usr disappear after running other programs like xterm and
>> firefox. But until now never after remouting it.
>
> That is interesting.

boot
run dosemu -> /usr unmounted
mount /usr
run dosemu -> /usr still mounted
sync; echo 3 > /proc/sys/vm/drop_caches
run dosemu -> /usr unmounted

>
> It sounds like occassionally your /home mount disappears as well.
>

I have not seen that happening

...

>
> I expect what needs to happen is to confirm that nfs directory entry
> revalidation is buggy, and at least for the short term re-add the nfs
> logic that will avoid dropping a dentry if it is a mount point, or
> path to a mount point, to avoid the nfs bugs.
>

ok, I will go in waiting mode.

-- 
Hans

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Eric W. Biederman]

Hans de Bruin <jmdebruin@xmsnet.nl> writes:

>> I expect what needs to happen is to confirm that nfs directory entry
>> revalidation is buggy, and at least for the short term re-add the nfs
>> logic that will avoid dropping a dentry if it is a mount point, or
>> path to a mount point, to avoid the nfs bugs.
>>
>
> ok, I will go in waiting mode.

I dropped the ball on this but it looks like someone else hit the
problem and the following two commits fixed this issue:

Can you confirm that things are working again?

commit fa9233699cc1dc236f4cf42245d13e40966938c5
Author: Trond Myklebust <trond.myklebust@primarydata.com>
Date:   Mon Feb 23 18:51:32 2015 -0500

    NFS: Don't require a filehandle to refresh the inode in nfs_prime_dcache()
    
    If the server does not return a valid set of attributes that we can
    use to either create a file or refresh the inode, then there is no
    value in calling nfs_prime_dcache().
    
    However if we're just refreshing the inode using the attributes that
    the server returned, then it shouldn't matter whether or not we have
    a filehandle, as long as we check the fsid+fileid combination.
    
    Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>

commit 6c441c254eea2354d686be7f5544bcd79fb6a61f
Author: Trond Myklebust <trond.myklebust@primarydata.com>
Date:   Sun Feb 22 16:35:36 2015 -0500

    NFS: Don't invalidate a submounted dentry in nfs_prime_dcache()
    
    If we're traversing a directory which contains a submounted filesystem,
    or one that has a referral, the NFS server that is processing the READDIR
    request will often return information for the underlying (mounted-on)
    directory. It may, or may not, also return filehandle information.
    
    If this happens, and the lookup in nfs_prime_dcache() returns the
    dentry for the submounted directory, the filehandle comparison will
    fail, and we call d_invalidate(). Post-commit 8ed936b5671bf
    ("vfs: Lazily remove mounts on unlinked files and directories."), this
    means the entire subtree is unmounted.
    
    The following minimal patch addresses this problem by punting on
    the invalidation if there is a submount.
    
    Kudos to Neil Brown <neilb@suse.de> for having tracked down this
    issue (see link).
    
    Reported-by: Nix <nix@esperi.org.uk>
    Link: http://lkml.kernel.org/r/87iofju9ht.fsf@spindle.srvr.nix
    Cc: stable@vger.kernel.org # 3.18+
    Signed-off-by: Trond Myklebust <trond.myklebust@primarydata.com>

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Hans de Bruin]

On 05/01/2015 06:35 AM, Eric W. Biederman wrote:
> Hans de Bruin <jmdebruin@xmsnet.nl> writes:
>
>>> I expect what needs to happen is to confirm that nfs directory entry
>>> revalidation is buggy, and at least for the short term re-add the nfs
>>> logic that will avoid dropping a dentry if it is a mount point, or
>>> path to a mount point, to avoid the nfs bugs.
>>>
>>
>> ok, I will go in waiting mode.
>
> I dropped the ball on this but it looks like someone else hit the
> problem and the following two commits fixed this issue:
>
> Can you confirm that things are working again?

I noticed the issue was gone in one of the previous releases. I was 
wondering whether you changed something or not. Sorry for not notifying you.

-- 
Hans

Re: 3.17.0+ files disappearing after playing old dos game on nfsroot laptop

[Eric W. Biederman]

Hans de Bruin <jmdebruin@xmsnet.nl> writes:

> On 05/01/2015 06:35 AM, Eric W. Biederman wrote:
>> Hans de Bruin <jmdebruin@xmsnet.nl> writes:
>>
>>>> I expect what needs to happen is to confirm that nfs directory entry
>>>> revalidation is buggy, and at least for the short term re-add the nfs
>>>> logic that will avoid dropping a dentry if it is a mount point, or
>>>> path to a mount point, to avoid the nfs bugs.
>>>>
>>>
>>> ok, I will go in waiting mode.
>>
>> I dropped the ball on this but it looks like someone else hit the
>> problem and the following two commits fixed this issue:
>>
>> Can you confirm that things are working again?
>
> I noticed the issue was gone in one of the previous releases. I was
> wondering whether you changed something or not. Sorry for not
> notifying you.

No problem, and thank you for confirming the regression is fixed.
One more thing I can now mark off my list.

Eric