Inexplicable disk activity trying to load modules on devfs

Inexplicable disk activity trying to load modules on devfs

[Daniel Jacobowitz]

I just booted into 2.4.19-pre10-ac2 for the first time, and noticed
something very odd: my disk activity light was flashing at about
half-second intervals, very regularly, and I could hear the disk
moving.  I was only able to track it down to which disk controller, via
/proc/interrupts (are there any tools for monitoring VFS activity? 
They'd be really useful).  Eventually I hunted down the program causing
it: xmms.

The reason turned out to be that I hadn't remembered to build my sound
driver for this kernel version.  Every half-second xmms tried to open
/dev/mixer (and failed, ENOENT).  Every time it did that there was
actual disk activity.  Easily reproducible without xmms.  Reproducible
on any non-existant device in devfs, but not for nonexisting files on
other filesystems.  Is something bypassing the normal disk cache
mechanisms here?  That doesn't seem right at all.

-- 
Daniel Jacobowitz                           Debian GNU/Linux Developer
MontaVista Software                         Carnegie Mellon University

Re: Inexplicable disk activity trying to load modules on devfs

[Andrew Morton]

Daniel Jacobowitz wrote:
> 
> I just booted into 2.4.19-pre10-ac2 for the first time, and noticed
> something very odd: my disk activity light was flashing at about
> half-second intervals, very regularly, and I could hear the disk
> moving.  I was only able to track it down to which disk controller, via
> /proc/interrupts (are there any tools for monitoring VFS activity?
> They'd be really useful).  Eventually I hunted down the program causing
> it: xmms.
> 
> The reason turned out to be that I hadn't remembered to build my sound
> driver for this kernel version.  Every half-second xmms tried to open
> /dev/mixer (and failed, ENOENT).  Every time it did that there was
> actual disk activity.  Easily reproducible without xmms.  Reproducible
> on any non-existant device in devfs, but not for nonexisting files on
> other filesystems.  Is something bypassing the normal disk cache
> mechanisms here?  That doesn't seem right at all.
> 

syslog activity from a printk, perhaps?

-

Re: Inexplicable disk activity trying to load modules on devfs

[Daniel Jacobowitz]

On Sat, Jun 15, 2002 at 03:44:30PM -0700, Andrew Morton wrote:
> Daniel Jacobowitz wrote:
> > 
> > I just booted into 2.4.19-pre10-ac2 for the first time, and noticed
> > something very odd: my disk activity light was flashing at about
> > half-second intervals, very regularly, and I could hear the disk
> > moving.  I was only able to track it down to which disk controller, via
> > /proc/interrupts (are there any tools for monitoring VFS activity?
> > They'd be really useful).  Eventually I hunted down the program causing
> > it: xmms.
> > 
> > The reason turned out to be that I hadn't remembered to build my sound
> > driver for this kernel version.  Every half-second xmms tried to open
> > /dev/mixer (and failed, ENOENT).  Every time it did that there was
> > actual disk activity.  Easily reproducible without xmms.  Reproducible
> > on any non-existant device in devfs, but not for nonexisting files on
> > other filesystems.  Is something bypassing the normal disk cache
> > mechanisms here?  That doesn't seem right at all.
> > 
> 
> syslog activity from a printk, perhaps?

Nope.  No log activity whatsoever.

-- 
Daniel Jacobowitz                           Debian GNU/Linux Developer
MontaVista Software                         Carnegie Mellon University

Re: Inexplicable disk activity trying to load modules on devfs

[Rob Landley]

On Saturday 15 June 2002 07:04 pm, Daniel Jacobowitz wrote:
> On Sat, Jun 15, 2002 at 03:44:30PM -0700, Andrew Morton wrote:
> > Daniel Jacobowitz wrote:
> > > I just booted into 2.4.19-pre10-ac2 for the first time, and noticed
> > > something very odd: my disk activity light was flashing at about
> > > half-second intervals, very regularly, and I could hear the disk
> > > moving.  I was only able to track it down to which disk controller, via
> > > /proc/interrupts (are there any tools for monitoring VFS activity?
> > > They'd be really useful).  Eventually I hunted down the program causing
> > > it: xmms.
> > >
> > > The reason turned out to be that I hadn't remembered to build my sound
> > > driver for this kernel version.  Every half-second xmms tried to open
> > > /dev/mixer (and failed, ENOENT).  Every time it did that there was
> > > actual disk activity.  Easily reproducible without xmms.  Reproducible
> > > on any non-existant device in devfs, but not for nonexisting files on
> > > other filesystems.  Is something bypassing the normal disk cache
> > > mechanisms here?  That doesn't seem right at all.
> >
> > syslog activity from a printk, perhaps?
>
> Nope.  No log activity whatsoever.

Updated atime on the /dev/blah node?

Random guess...

Rob

Re: Inexplicable disk activity trying to load modules on devfs

[bert hubert]

On Sat, Jun 15, 2002 at 05:22:44PM -0500, Daniel Jacobowitz wrote:
> I just booted into 2.4.19-pre10-ac2 for the first time, and noticed
> something very odd: my disk activity light was flashing at about
> half-second intervals, very regularly, and I could hear the disk

I have a similar problem with the popular 'dig' utility. Running 'dig 2>
/dev/null > /dev/null' suffices to cause disk activity, even when run many
times in succession.

As far as stracing can tell (dig is multithreaded), there is no reason for
this.

Regards,

bert

-- 
http://www.PowerDNS.com          Versatile DNS Software & Services
http://www.tk                              the dot in .tk
http://lartc.org           Linux Advanced Routing & Traffic Control HOWTO

Re: Inexplicable disk activity trying to load modules on devfs

[Borsenkow Andrej]

>> 
>> I just booted into 2.4.19-pre10-ac2 for the first time, and noticed 
>> something very odd: my disk activity light was flashing at about 
>> half-second intervals, very regularly, and I could hear the disk 
>> moving. I was only able to track it down to which disk controller, via 
>> /proc/interrupts (are there any tools for monitoring VFS activity? 
>> They'd be really useful). Eventually I hunted down the program causing 
>> it: xmms. 
>> 
>> The reason turned out to be that I hadn't remembered to build my sound 
>> driver for this kernel version. Every half-second xmms tried to open 
>> /dev/mixer (and failed, ENOENT). Every time it did that there was 
>> actual disk activity. Easily reproducible without xmms. Reproducible 
>> on any non-existant device in devfs, but not for nonexisting files on 
>> other filesystems. Is something bypassing the normal disk cache 
>> mechanisms here? That doesn't seem right at all. 
>> 
>
>
>syslog activity from a printk, perhaps? 

No. It is most probably devfsd trying to load sound modules.

This is exactly the reason Mandrake does not enable devfs in kernel-secure.
You can badly hit your system by doing in a loop ls /dev/foo for some device
foo that is configured for module autoloading.

It is very fascist decision; the slightly more forgiving way is to disable
devfsd module autoloading (or disable devfsd entirely, just run it once
after all drivers are loaded to execute actions) but then you lose support
for hot plugging and some people do use kernel-secure on desktops. 

-andrej

Re: Inexplicable disk activity trying to load modules on devfs

[Daniel Jacobowitz]

On Mon, Jun 17, 2002 at 10:59:26AM +0400, Borsenkow Andrej wrote:
> >> 
> >> I just booted into 2.4.19-pre10-ac2 for the first time, and noticed 
> >> something very odd: my disk activity light was flashing at about 
> >> half-second intervals, very regularly, and I could hear the disk 
> >> moving. I was only able to track it down to which disk controller, via 
> >> /proc/interrupts (are there any tools for monitoring VFS activity? 
> >> They'd be really useful). Eventually I hunted down the program causing 
> >> it: xmms. 
> >> 
> >> The reason turned out to be that I hadn't remembered to build my sound 
> >> driver for this kernel version. Every half-second xmms tried to open 
> >> /dev/mixer (and failed, ENOENT). Every time it did that there was 
> >> actual disk activity. Easily reproducible without xmms. Reproducible 
> >> on any non-existant device in devfs, but not for nonexisting files on 
> >> other filesystems. Is something bypassing the normal disk cache 
> >> mechanisms here? That doesn't seem right at all. 
> >> 
> >
> >
> >syslog activity from a printk, perhaps? 
> 
> No. It is most probably devfsd trying to load sound modules.
> 
> This is exactly the reason Mandrake does not enable devfs in kernel-secure.
> You can badly hit your system by doing in a loop ls /dev/foo for some device
> foo that is configured for module autoloading.
> 
> It is very fascist decision; the slightly more forgiving way is to disable
> devfsd module autoloading (or disable devfsd entirely, just run it once
> after all drivers are loaded to execute actions) but then you lose support
> for hot plugging and some people do use kernel-secure on desktops. 

For the curious, the reason is that modprobe writes even failed attempts
to a log in /var/log/ksymoops, and calls fdatasync() on that file
afterwards.  There is no way to disable this without removing that
directory, as a design decision.  I don't personally see the point in
logging attempts which fail because there is no driver...


-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: Inexplicable disk activity trying to load modules on devfs

[Richard Gooch]

Daniel Jacobowitz writes:
> On Mon, Jun 17, 2002 at 10:59:26AM +0400, Borsenkow Andrej wrote:
> > >> 
> > >> I just booted into 2.4.19-pre10-ac2 for the first time, and noticed 
> > >> something very odd: my disk activity light was flashing at about 
> > >> half-second intervals, very regularly, and I could hear the disk 
> > >> moving. I was only able to track it down to which disk controller, via 
> > >> /proc/interrupts (are there any tools for monitoring VFS activity? 
> > >> They'd be really useful). Eventually I hunted down the program causing 
> > >> it: xmms. 
> > >> 
> > >> The reason turned out to be that I hadn't remembered to build my sound 
> > >> driver for this kernel version. Every half-second xmms tried to open 
> > >> /dev/mixer (and failed, ENOENT). Every time it did that there was 
> > >> actual disk activity. Easily reproducible without xmms. Reproducible 
> > >> on any non-existant device in devfs, but not for nonexisting files on 
> > >> other filesystems. Is something bypassing the normal disk cache 
> > >> mechanisms here? That doesn't seem right at all. 
> > >> 
> > >
> > >
> > >syslog activity from a printk, perhaps? 
> > 
> > No. It is most probably devfsd trying to load sound modules.
> > 
> > This is exactly the reason Mandrake does not enable devfs in kernel-secure.
> > You can badly hit your system by doing in a loop ls /dev/foo for some device
> > foo that is configured for module autoloading.
> > 
> > It is very fascist decision; the slightly more forgiving way is to disable
> > devfsd module autoloading (or disable devfsd entirely, just run it once
> > after all drivers are loaded to execute actions) but then you lose support
> > for hot plugging and some people do use kernel-secure on desktops. 
> 
> For the curious, the reason is that modprobe writes even failed
> attempts to a log in /var/log/ksymoops, and calls fdatasync() on
> that file afterwards.  There is no way to disable this without
> removing that directory, as a design decision.  I don't personally
> see the point in logging attempts which fail because there is no
> driver...

Sounds like the behaviour of modprobe needs to be fixed.

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: Inexplicable disk activity trying to load modules on devfs

[Keith Owens]

On Tue, 25 Jun 2002 23:38:48 -0400, 
Richard Gooch <rgooch@ras.ucalgary.ca> wrote:
>Daniel Jacobowitz writes:
>> For the curious, the reason is that modprobe writes even failed
>> attempts to a log in /var/log/ksymoops, and calls fdatasync() on
>> that file afterwards.  There is no way to disable this without
>> removing that directory, as a design decision.  I don't personally
>> see the point in logging attempts which fail because there is no
>> driver...
>
>Sounds like the behaviour of modprobe needs to be fixed.

People wanted to know what was invoking modprobe and with what
parameters, especially for failed attempts.  The call to fdatasync() is
to "ensure" that the log data hits the disk _before_ the module is
loaded, otherwise debugging data is lost if the module init routine
oopses.

Re: Inexplicable disk activity trying to load modules on devfs

[Richard Gooch]

Keith Owens writes:
> On Tue, 25 Jun 2002 23:38:48 -0400, 
> Richard Gooch <rgooch@ras.ucalgary.ca> wrote:
> >Daniel Jacobowitz writes:
> >> For the curious, the reason is that modprobe writes even failed
> >> attempts to a log in /var/log/ksymoops, and calls fdatasync() on
> >> that file afterwards.  There is no way to disable this without
> >> removing that directory, as a design decision.  I don't personally
> >> see the point in logging attempts which fail because there is no
> >> driver...
> >
> >Sounds like the behaviour of modprobe needs to be fixed.
> 
> People wanted to know what was invoking modprobe and with what
> parameters, especially for failed attempts.  The call to fdatasync()
> is to "ensure" that the log data hits the disk _before_ the module
> is loaded, otherwise debugging data is lost if the module init
> routine oopses.

Then there needs to be a way of enabling/disabling this. Maybe a
run-time config option?

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: Inexplicable disk activity trying to load modules on devfs

[Richard Gooch]

Borsenkow Andrej writes:
> >> 
> >> I just booted into 2.4.19-pre10-ac2 for the first time, and noticed 
> >> something very odd: my disk activity light was flashing at about 
> >> half-second intervals, very regularly, and I could hear the disk 
> >> moving. I was only able to track it down to which disk controller, via 
> >> /proc/interrupts (are there any tools for monitoring VFS activity? 
> >> They'd be really useful). Eventually I hunted down the program causing 
> >> it: xmms. 
> >> 
> >> The reason turned out to be that I hadn't remembered to build my sound 
> >> driver for this kernel version. Every half-second xmms tried to open 
> >> /dev/mixer (and failed, ENOENT). Every time it did that there was 
> >> actual disk activity. Easily reproducible without xmms. Reproducible 
> >> on any non-existant device in devfs, but not for nonexisting files on 
> >> other filesystems. Is something bypassing the normal disk cache 
> >> mechanisms here? That doesn't seem right at all. 
> >> 
> >
> >
> >syslog activity from a printk, perhaps? 
> 
> No. It is most probably devfsd trying to load sound modules.
> 
> This is exactly the reason Mandrake does not enable devfs in
> kernel-secure.  You can badly hit your system by doing in a loop ls
> /dev/foo for some device foo that is configured for module
> autoloading.
> 
> It is very fascist decision; the slightly more forgiving way is to
> disable devfsd module autoloading (or disable devfsd entirely, just
> run it once after all drivers are loaded to execute actions) but
> then you lose support for hot plugging and some people do use
> kernel-secure on desktops.

Or you can use the IGNORE action to selectively disable loading of
some modules (whether or not they exist).

Another option is to write a shared object extension to devfsd which
has rate-limiting. All the mechanisms you need are there or can be
built on top of the existing infrastructure.

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca