AutoFS5

AutoFS5

[Jim Summers]

Hello All,

Just hit a hiccup with autofs5.  Everything was going along fine and then had 
a user call and say they could not get their home on a machine.  I did some 
quick checking and some mounts were working and others not.  below is a bit of 
the debug log and some other info.  the user, gerardo is the one who called 
me.  entries in the debug log for him are the same as the tmac user.  in that 
it get his ldap info tries to mount but then says it is already mounted and 
then fails????

client machine is FC6 hand compiled autofs version:
autofs-5.0.1-20
autofs-debuginfo-5.0.1-20
with a patch from Ian.

Yum still isn't pulling the patched one in for me.

==============================

[root@saturn ~]# ls /home
amarack  granville


[root@saturn ~]# who
amarack  pts/0        2007-08-13 11:27 (pluto.it.ou.edu)
root     pts/5        2007-08-13 14:49 (redear.it.ou.edu)
gerardo  pts/2        2007-08-14 13:26 (dave.it.ou.edu)
root     pts/3        2007-08-14 13:44 (redear.it.ou.edu)

[root@saturn ~]# su - tmac
su: warning: cannot change directory to /home/tmac: No such file or directory

sun_mount: parse(sun): mounting root /home, mountpoint tmac, what 
topdawg:/raid/ds9/tmac, fstype nfs, options 
rw,actimeo=60,rsize=32768,wsize=32768,nocto
Aug 14 14:02:10 saturn automount[2743]: mount_mount: mount(nfs): root=/home 
name=tmac what=topdawg:/raid/ds9/tmac, fstype=nfs, 
options=rw,actimeo=60,rsize=32768,wsize=32768,nocto
Aug 14 14:02:10 saturn automount[2743]: mount_mount: mount(nfs): nfs 
options="rw,actimeo=60,rsize=32768,wsize=32768,nocto", nosymlink=0, ro=0
Aug 14 14:02:10 saturn automount[2743]: mount_mount: mount(nfs): calling 
mkdir_path /home/tmac
Aug 14 14:02:10 saturn automount[2743]: mount_mount: mount(nfs): calling mount 
-t nfs -s -o rw,actimeo=60,rsize=32768,wsize=32768,nocto 
topdawg:/raid/ds9/tmac /home/tmac
Aug 14 14:02:10 saturn automount[2743]: >> mount.nfs: /home/tmac is already 
mounted or busy
Aug 14 14:02:10 saturn automount[2743]: mount(nfs): nfs: mount failure 
topdawg:/raid/ds9/tmac on /home/tmac
Aug 14 14:02:10 saturn automount[2743]: send_fail: token = 284
Aug 14 14:02:10 saturn automount[2743]: failed to mount /home/tmac
Aug 14 14:02:10 saturn automount[2743]: handle_packet: type = 3
Aug 14 14:02:10 saturn automount[2743]: handle_packet_missing_indirect: token 
285, name tmac, request pid 21166
Aug 14 14:02:10 saturn automount[2743]: attempting to mount entry /home/tmac
Aug 14 14:02:10 saturn automount[2743]: lookup_mount: lookup(ldap): looking up 
tmac
Aug 14 14:02:10 saturn automount[2743]: send_fail: token = 285
Aug 14 14:02:10 saturn automount[2743]: failed to mount /home/tmac
Aug 14 14:02:10 saturn automount[2743]: handle_packet: type = 3
Aug 14 14:02:10 saturn automount[2743]: handle_packet_missing_indirect: token 
286, name tmac, request pid 21166
Aug 14 14:02:10 saturn automount[2743]: attempting to mount entry /home/tmac
Aug 14 14:02:10 saturn automount[2743]: lookup_mount: lookup(ldap): looking up 
tmac
Aug 14 14:02:10 saturn automount[2743]: send_fail: token = 286
Aug 14 14:02:10 saturn automount[2743]: failed to mount /home/tmac
Aug 14 14:02:10 saturn automount[2743]: handle_packet: type = 3
Aug 14 14:02:10 saturn automount[2743]: handle_packet_missing_indirect: token 
287, name tmac, request pid 21166
Aug 14 14:02:10 saturn automount[2743]: attempting to mount entry /home/tmac
Aug 14 14:02:10 saturn automount[2743]: lookup_mount: lookup(ldap): looking up 
tmac
Aug 14 14:02:10 saturn automount[2743]: send_fail: token = 287
Aug 14 14:02:10 saturn automount[2743]: failed to mount /home/tmac
Aug 14 14:02:10 saturn automount[2743]: handle_packet: type = 3
===========

ideas /  suggestions??

tia

-- 
Jim Summers
School of Computer Science-University of Oklahoma
-------------------------------------------------

Re: AutoFS5

[Jim Summers]

Jim Summers wrote:
> Hello All,
> 
> Just hit a hiccup with autofs5.  Everything was going along fine and then had 
> a user call and say they could not get their home on a machine.  I did some 
> quick checking and some mounts were working and others not.  below is a bit of 
> the debug log and some other info.  the user, gerardo is the one who called 
> me.  entries in the debug log for him are the same as the tmac user.  in that 
> it get his ldap info tries to mount but then says it is already mounted and 
> then fails????
> 
> client machine is FC6 hand compiled autofs version:
> autofs-5.0.1-20
> autofs-debuginfo-5.0.1-20

the rpms were actually built on a FC7 machine and then installed here.

thanks

> with a patch from Ian.
> 
> Yum still isn't pulling the patched one in for me.
> 
> ==============================
> 
> [root@saturn ~]# ls /home
> amarack  granville
> 
> 
> [root@saturn ~]# who
> amarack  pts/0        2007-08-13 11:27 (pluto.it.ou.edu)
> root     pts/5        2007-08-13 14:49 (redear.it.ou.edu)
> gerardo  pts/2        2007-08-14 13:26 (dave.it.ou.edu)
> root     pts/3        2007-08-14 13:44 (redear.it.ou.edu)
> 
> [root@saturn ~]# su - tmac
> su: warning: cannot change directory to /home/tmac: No such file or directory
> 
> sun_mount: parse(sun): mounting root /home, mountpoint tmac, what 
> topdawg:/raid/ds9/tmac, fstype nfs, options 
> rw,actimeo=60,rsize=32768,wsize=32768,nocto
> Aug 14 14:02:10 saturn automount[2743]: mount_mount: mount(nfs): root=/home 
> name=tmac what=topdawg:/raid/ds9/tmac, fstype=nfs, 
> options=rw,actimeo=60,rsize=32768,wsize=32768,nocto
> Aug 14 14:02:10 saturn automount[2743]: mount_mount: mount(nfs): nfs 
> options="rw,actimeo=60,rsize=32768,wsize=32768,nocto", nosymlink=0, ro=0
> Aug 14 14:02:10 saturn automount[2743]: mount_mount: mount(nfs): calling 
> mkdir_path /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: mount_mount: mount(nfs): calling mount 
> -t nfs -s -o rw,actimeo=60,rsize=32768,wsize=32768,nocto 
> topdawg:/raid/ds9/tmac /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: >> mount.nfs: /home/tmac is already 
> mounted or busy
> Aug 14 14:02:10 saturn automount[2743]: mount(nfs): nfs: mount failure 
> topdawg:/raid/ds9/tmac on /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: send_fail: token = 284
> Aug 14 14:02:10 saturn automount[2743]: failed to mount /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: handle_packet: type = 3
> Aug 14 14:02:10 saturn automount[2743]: handle_packet_missing_indirect: token 
> 285, name tmac, request pid 21166
> Aug 14 14:02:10 saturn automount[2743]: attempting to mount entry /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: lookup_mount: lookup(ldap): looking up 
> tmac
> Aug 14 14:02:10 saturn automount[2743]: send_fail: token = 285
> Aug 14 14:02:10 saturn automount[2743]: failed to mount /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: handle_packet: type = 3
> Aug 14 14:02:10 saturn automount[2743]: handle_packet_missing_indirect: token 
> 286, name tmac, request pid 21166
> Aug 14 14:02:10 saturn automount[2743]: attempting to mount entry /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: lookup_mount: lookup(ldap): looking up 
> tmac
> Aug 14 14:02:10 saturn automount[2743]: send_fail: token = 286
> Aug 14 14:02:10 saturn automount[2743]: failed to mount /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: handle_packet: type = 3
> Aug 14 14:02:10 saturn automount[2743]: handle_packet_missing_indirect: token 
> 287, name tmac, request pid 21166
> Aug 14 14:02:10 saturn automount[2743]: attempting to mount entry /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: lookup_mount: lookup(ldap): looking up 
> tmac
> Aug 14 14:02:10 saturn automount[2743]: send_fail: token = 287
> Aug 14 14:02:10 saturn automount[2743]: failed to mount /home/tmac
> Aug 14 14:02:10 saturn automount[2743]: handle_packet: type = 3
> ===========
> 
> ideas /  suggestions??
> 
> tia
> 

-- 
Jim Summers
School of Computer Science-University of Oklahoma
-------------------------------------------------

Re: AutoFS5

[Jeff Moyer]

Jim Summers <jsummers@bachman.cs.ou.edu> writes:

> Hello All,
>
> Just hit a hiccup with autofs5.  Everything was going along fine and then had 
> a user call and say they could not get their home on a machine.  I did some 
> quick checking and some mounts were working and others not.  below is a bit of 
> the debug log and some other info.  the user, gerardo is the one who called 
> me.  entries in the debug log for him are the same as the tmac user.  in that 
> it get his ldap info tries to mount but then says it is already mounted and 
> then fails????
>
> client machine is FC6 hand compiled autofs version:
> autofs-5.0.1-20
> autofs-debuginfo-5.0.1-20
> with a patch from Ian.

What is the kernel version on the client?

-Jeff

Re: AutoFS5

[Jeff Moyer]

Jeff Moyer <jmoyer@redhat.com> writes:

> Jim Summers <jsummers@bachman.cs.ou.edu> writes:
>
>> Hello All,
>>
>> Just hit a hiccup with autofs5.  Everything was going along fine and then had 
>> a user call and say they could not get their home on a machine.  I did some 
>> quick checking and some mounts were working and others not.  below is a bit of 
>> the debug log and some other info.  the user, gerardo is the one who called 
>> me.  entries in the debug log for him are the same as the tmac user.  in that 
>> it get his ldap info tries to mount but then says it is already mounted and 
>> then fails????
>>
>> client machine is FC6 hand compiled autofs version:
>> autofs-5.0.1-20
>> autofs-debuginfo-5.0.1-20
>> with a patch from Ian.
>
> What is the kernel version on the client?

I'm guessing that you have 2.6.22.1-13.fc6 or later installed.  These
kernels include the "nosharecache" patch for NFS.  This patch can
cause some mounts to fail with -EBUSY.  You can revert the
nosharecache patch to get the old behaviour back.

Could you let us know what nfs mount options are used for your maps,
though, as this should only bite you if you try to mount from the same
server file system with different sets of mount options.  In other
words, your first mount from the server should succeed, successive
mount attempts of the same file system with differring options will
fail.

-Jeff

Re: AutoFS5

[Jim Summers]

Jeff Moyer wrote:
> Jeff Moyer <jmoyer@redhat.com> writes:
> 
>> Jim Summers <jsummers@bachman.cs.ou.edu> writes:
>>
>>> Hello All,
>>>
>>> Just hit a hiccup with autofs5.  Everything was going along fine and then had 
>>> a user call and say they could not get their home on a machine.  I did some 
>>> quick checking and some mounts were working and others not.  below is a bit of 
>>> the debug log and some other info.  the user, gerardo is the one who called 
>>> me.  entries in the debug log for him are the same as the tmac user.  in that 
>>> it get his ldap info tries to mount but then says it is already mounted and 
>>> then fails????
>>>
>>> client machine is FC6 hand compiled autofs version:
>>> autofs-5.0.1-20
>>> autofs-debuginfo-5.0.1-20
>>> with a patch from Ian.
>> What is the kernel version on the client?

2.6.22.1-32.fc6

> 
> I'm guessing that you have 2.6.22.1-13.fc6 or later installed.  These
> kernels include the "nosharecache" patch for NFS.  This patch can
> cause some mounts to fail with -EBUSY.  You can revert the
> nosharecache patch to get the old behaviour back.

is this accomplished with boot option similar to 'noacpi' or something like that?

> 
> Could you let us know what nfs mount options are used for your maps,
> though, as this should only bite you if you try to mount from the same
> server file system with different sets of mount options.  In other
> words, your first mount from the server should succeed, successive
> mount attempts of the same file system with differring options will
> fail.

options:

amarak:
-rw,rsize=4096,wsize=4096
here is tmac:
  -rw,actimeo=60,rsize=32768,wsize=32768,nocto
here is gerardo:
-rw,rsize=8192,wsize=8192
here is granville:
-rw,rsize=4096,wsize=4096
jsummers:
-rw,actimeo=60,rsize=32768,wsize=32768,nocto

some results:
[root@saturn ~]# su - jsummers
[jsummers@saturn ~]$ exit
logout
[root@saturn ~]# su - tmac
su: warning: cannot change directory to /home/tmac: No such file or directory
-bash-3.1$ exit
logout
[root@saturn ~]# su - granville
saturn@~> logout
[root@saturn ~]# su - amarack
saturn@~> logout
[root@saturn ~]# su - gerardo
su: warning: cannot change directory to /home/gerardo: No such file or directory
-bash-3.1$ logout

weird, huh?

thanks again

> 
> -Jeff
> 
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs

Re: AutoFS5

[Jeff Moyer]

Jim Summers <jsummers@bachman.cs.ou.edu> writes:

> Jeff Moyer wrote:
>> Jeff Moyer <jmoyer@redhat.com> writes:
>>
>>> Jim Summers <jsummers@bachman.cs.ou.edu> writes:
>>>
>>>> Hello All,
>>>>
>>>> Just hit a hiccup with autofs5.  Everything was going along fine
>>>> and then had a user call and say they could not get their home on
>>>> a machine.  I did some quick checking and some mounts were working
>>>> and others not.  below is a bit of the debug log and some other
>>>> info.  the user, gerardo is the one who called me.  entries in the
>>>> debug log for him are the same as the tmac user.  in that it get
>>>> his ldap info tries to mount but then says it is already mounted
>>>> and then fails????
>>>>
>>>> client machine is FC6 hand compiled autofs version:
>>>> autofs-5.0.1-20
>>>> autofs-debuginfo-5.0.1-20
>>>> with a patch from Ian.
>>> What is the kernel version on the client?
>
> 2.6.22.1-32.fc6
>
>>
>> I'm guessing that you have 2.6.22.1-13.fc6 or later installed.  These
>> kernels include the "nosharecache" patch for NFS.  This patch can
>> cause some mounts to fail with -EBUSY.  You can revert the
>> nosharecache patch to get the old behaviour back.
>
> is this accomplished with boot option similar to 'noacpi' or something like that?

Oh, yeah, if you update nfs-utils, you should get access to the
nosharecache mount option.  You can probably specify this globally in
your /etc/sysconfig/autofs file.  OPTIONS="-nosharecache", I think.

>> Could you let us know what nfs mount options are used for your maps,
>> though, as this should only bite you if you try to mount from the same
>> server file system with different sets of mount options.  In other
>> words, your first mount from the server should succeed, successive
>> mount attempts of the same file system with differring options will
>> fail.
>
> options:
>
> amarak:
> -rw,rsize=4096,wsize=4096
> here is tmac:
>  -rw,actimeo=60,rsize=32768,wsize=32768,nocto
> here is gerardo:
> -rw,rsize=8192,wsize=8192
> here is granville:
> -rw,rsize=4096,wsize=4096
> jsummers:
> -rw,actimeo=60,rsize=32768,wsize=32768,nocto

So, these days superblocks are shared when mounting the same remote
file system.  As such, when you mount the first export it defines the
mount options that will be used for subsequent mounts that come from
the same file system.

This is a change in behaviour from what we've had in the past, and
there is still ongoing discussion as to how we should resolve the
issue.

> some results:
> [root@saturn ~]# su - jsummers
> [jsummers@saturn ~]$ exit
> logout
> [root@saturn ~]# su - tmac
> su: warning: cannot change directory to /home/tmac: No such file or directory
> -bash-3.1$ exit
> logout

Interesting.  I thought that this one would have worked.  If you can
provide a debug log, we can probably get to the bottom of this.  It
would also be interesting to know what the contents of /proc/mounts
was before each of these commands was run.

For information on collecting a debug log, you can consult my people
page:
  http://people.redhat.com/jmoyer/

> [root@saturn ~]# su - granville
> saturn@~> logout
> [root@saturn ~]# su - amarack
> saturn@~> logout
> [root@saturn ~]# su - gerardo
> su: warning: cannot change directory to /home/gerardo: No such file or directory
> -bash-3.1$ logout
>
> weird, huh?

Definitely.  Try the nosharecache mount option.  If that resolves the
problem, I'm not quite as interested in looking at the debug log.

Thanks!

Jeff

Re: AutoFS5

[Ian Kent]

On Tue, 2007-08-14 at 15:39 -0500, Jim Summers wrote:
> Jeff Moyer wrote:
> > Jeff Moyer <jmoyer@redhat.com> writes:
> > 
> >> Jim Summers <jsummers@bachman.cs.ou.edu> writes:
> >>
> >>> Hello All,
> >>>
> >>> Just hit a hiccup with autofs5.  Everything was going along fine and then had 
> >>> a user call and say they could not get their home on a machine.  I did some 
> >>> quick checking and some mounts were working and others not.  below is a bit of 
> >>> the debug log and some other info.  the user, gerardo is the one who called 
> >>> me.  entries in the debug log for him are the same as the tmac user.  in that 
> >>> it get his ldap info tries to mount but then says it is already mounted and 
> >>> then fails????
> >>>
> >>> client machine is FC6 hand compiled autofs version:
> >>> autofs-5.0.1-20
> >>> autofs-debuginfo-5.0.1-20
> >>> with a patch from Ian.
> >> What is the kernel version on the client?
> 
> 2.6.22.1-32.fc6
> 
> > 
> > I'm guessing that you have 2.6.22.1-13.fc6 or later installed.  These
> > kernels include the "nosharecache" patch for NFS.  This patch can
> > cause some mounts to fail with -EBUSY.  You can revert the
> > nosharecache patch to get the old behaviour back.
> 
> is this accomplished with boot option similar to 'noacpi' or something like that?

It's a mount option.
Check your nfs-utils. I believe that revision 14 and above will
understand the "nosharecache" option.

Add this as a global option by changing:

#OPTIONS=""

to

OPTIONS="-O nosharecache"

in /etc/sysconfig/autofs, near the bottom and let us know if this helps.

Ian

Re: AutoFS5

[Ian Kent]

On Wed, 2007-08-15 at 13:18 +0800, Ian Kent wrote:
> > > 
> > > I'm guessing that you have 2.6.22.1-13.fc6 or later installed.  These
> > > kernels include the "nosharecache" patch for NFS.  This patch can
> > > cause some mounts to fail with -EBUSY.  You can revert the
> > > nosharecache patch to get the old behaviour back.
> > 
> > is this accomplished with boot option similar to 'noacpi' or something like that?
> 
> It's a mount option.
> Check your nfs-utils. I believe that revision 14 and above will
> understand the "nosharecache" option.

That's nfs-utils-1.0.10-14 for FC6.

If you happen to be using F7 or Rawhide then it's nfs-utils-1.1.0-3, but
it doesn't look like the F7 package has got to the mirrors yet. The
patch for the option was inadvertently missed from F7 and Rawhide when
the patch went into the kernel.

Ian

Re: AutoFS5

[Jim Summers]

Ian Kent wrote:
> On Tue, 2007-08-14 at 15:39 -0500, Jim Summers wrote:
>> Jeff Moyer wrote:
>>> Jeff Moyer <jmoyer@redhat.com> writes:
>>>
>>>> Jim Summers <jsummers@bachman.cs.ou.edu> writes:
>>>>
>>>>> Hello All,
>>>>>
>>>>> Just hit a hiccup with autofs5.  Everything was going along fine and then had 
>>>>> a user call and say they could not get their home on a machine.  I did some 
>>>>> quick checking and some mounts were working and others not.  below is a bit of 
>>>>> the debug log and some other info.  the user, gerardo is the one who called 
>>>>> me.  entries in the debug log for him are the same as the tmac user.  in that 
>>>>> it get his ldap info tries to mount but then says it is already mounted and 
>>>>> then fails????
>>>>>
>>>>> client machine is FC6 hand compiled autofs version:
>>>>> autofs-5.0.1-20
>>>>> autofs-debuginfo-5.0.1-20
>>>>> with a patch from Ian.
>>>> What is the kernel version on the client?
>> 2.6.22.1-32.fc6
>>
>>> I'm guessing that you have 2.6.22.1-13.fc6 or later installed.  These
>>> kernels include the "nosharecache" patch for NFS.  This patch can
>>> cause some mounts to fail with -EBUSY.  You can revert the
>>> nosharecache patch to get the old behaviour back.
>> is this accomplished with boot option similar to 'noacpi' or something like that?
> 
> It's a mount option.
> Check your nfs-utils. I believe that revision 14 and above will
> understand the "nosharecache" option.
> 
> Add this as a global option by changing:
> 
> #OPTIONS=""
> 
> to
> 
> OPTIONS="-O nosharecache"
> 
> in /etc/sysconfig/autofs, near the bottom and let us know if this helps.
> 

seems to have resolved the problem.  i added the nosharecache option and 
restarted the daemon.  then was able to su to the affected and unaffected 
accounts and get the mounts.

the really cool thing was, i was able to do, "/etc/init.d/autofs restart" 
while a couple of users were logged in and it actually restarted.  I was not 
able to do that with autofs4, it would pretty much hang.

i will monitor it for a few days to verify all is well.

many thanks





> Ian
> 

-- 
Jim Summers
School of Computer Science-University of Oklahoma
-------------------------------------------------

Re: AutoFS5

[Ian Kent]

On Wed, 2007-08-15 at 08:48 -0500, Jim Summers wrote:
> Ian Kent wrote:
> > On Tue, 2007-08-14 at 15:39 -0500, Jim Summers wrote:
> >> Jeff Moyer wrote:
> >>> Jeff Moyer <jmoyer@redhat.com> writes:
> >>>
> >>>> Jim Summers <jsummers@bachman.cs.ou.edu> writes:
> >>>>
> >>>>> Hello All,
> >>>>>
> >>>>> Just hit a hiccup with autofs5.  Everything was going along fine and then had 
> >>>>> a user call and say they could not get their home on a machine.  I did some 
> >>>>> quick checking and some mounts were working and others not.  below is a bit of 
> >>>>> the debug log and some other info.  the user, gerardo is the one who called 
> >>>>> me.  entries in the debug log for him are the same as the tmac user.  in that 
> >>>>> it get his ldap info tries to mount but then says it is already mounted and 
> >>>>> then fails????
> >>>>>
> >>>>> client machine is FC6 hand compiled autofs version:
> >>>>> autofs-5.0.1-20
> >>>>> autofs-debuginfo-5.0.1-20
> >>>>> with a patch from Ian.
> >>>> What is the kernel version on the client?
> >> 2.6.22.1-32.fc6
> >>
> >>> I'm guessing that you have 2.6.22.1-13.fc6 or later installed.  These
> >>> kernels include the "nosharecache" patch for NFS.  This patch can
> >>> cause some mounts to fail with -EBUSY.  You can revert the
> >>> nosharecache patch to get the old behaviour back.
> >> is this accomplished with boot option similar to 'noacpi' or something like that?
> > 
> > It's a mount option.
> > Check your nfs-utils. I believe that revision 14 and above will
> > understand the "nosharecache" option.
> > 
> > Add this as a global option by changing:
> > 
> > #OPTIONS=""
> > 
> > to
> > 
> > OPTIONS="-O nosharecache"
> > 
> > in /etc/sysconfig/autofs, near the bottom and let us know if this helps.
> > 
> 
> seems to have resolved the problem.  i added the nosharecache option and 
> restarted the daemon.  then was able to su to the affected and unaffected 
> accounts and get the mounts.
> 
> the really cool thing was, i was able to do, "/etc/init.d/autofs restart" 
> while a couple of users were logged in and it actually restarted.  I was not 
> able to do that with autofs4, it would pretty much hang.

Yes, people are starting to notice that.

Ian