Work

Work

[Nick Krause]

Is there any work for a kernel newbie that you guys known of?
Cheers Nick

Work

[Kristofer Hallin]

Take a look at drivers/staging/. You will find something there to work on.
On Jul 24, 2014 6:38 PM, "Nick Krause" <xerofoify@gmail.com> wrote:

> Is there any work for a kernel newbie that you guys known of?
> Cheers Nick
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140724/3ec4fab5/attachment.html 

Work

[Lucas Tanure]

Could start cleaning a driver ?
http://kernelnewbies.org/OPWfirstpatch#head-4abafc61af197fb6e3d6cda623b00bdd90a52c26

The job here is about how to send patchs and not doing a awesome code.


--
Lucas Tanure
+55 (19) 988176559


On Thu, Jul 24, 2014 at 1:38 PM, Nick Krause <xerofoify@gmail.com> wrote:

> Is there any work for a kernel newbie that you guys known of?
> Cheers Nick
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140724/84b47c43/attachment.html 

Work

[Nick Krause]

On Thu, Jul 24, 2014 at 12:44 PM, Lucas Tanure <tanure@linux.com> wrote:
> Could start cleaning a driver ?
> http://kernelnewbies.org/OPWfirstpatch#head-4abafc61af197fb6e3d6cda623b00bdd90a52c26
>
> The job here is about how to send patchs and not doing a awesome code.
>
>
> --
> Lucas Tanure
> +55 (19) 988176559
>
>
> On Thu, Jul 24, 2014 at 1:38 PM, Nick Krause <xerofoify@gmail.com> wrote:
>>
>> Is there any work for a kernel newbie that you guys known of?
>> Cheers Nick
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>

I sent in some patches to Greg on the linux-next tree for staging.
Cheers Nick

Work

[Yi Li]

? 2014/7/25 0:44, Lucas Tanure ??:
> Could start cleaning a driver ?
> http://kernelnewbies.org/OPWfirstpatch#head-4abafc61af197fb6e3d6cda623b00bdd90a52c26
>
> The job here is about how to send patchs and not doing a awesome code.
>
>
Hi Lucas,
Because you have mentioned OPW, which is short for "Outstreach Program 
for Women".
but I am a *boy*, could I do some works on this program ?

regards,
Yi
> --
> Lucas Tanure
> +55 (19) 988176559
>
>
> On Thu, Jul 24, 2014 at 1:38 PM, Nick Krause <xerofoify@gmail.com 
> <mailto:xerofoify@gmail.com>> wrote:
>
>     Is there any work for a kernel newbie that you guys known of?
>     Cheers Nick
>
>     _______________________________________________
>     Kernelnewbies mailing list
>     Kernelnewbies at kernelnewbies.org
>     <mailto:Kernelnewbies@kernelnewbies.org>
>     http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140727/a897647e/attachment.html 

Work

[Lucas Tanure]

Hi Yi,

  No. You can't.

But this page http://kernelnewbies.org/OPWfirstpatch  is a awesome
start point for anyone, including myself.
Do a clean up driver as a start point. After that try to find some
spot in linux to help/maintain, but be sure that understand a lot of
that point before sending anything to kernel main list.

Thanks!


--
Lucas Tanure
+55 (19) 988176559


On Sat, Jul 26, 2014 at 6:13 PM, Yi Li <lovelylich@gmail.com> wrote:
>
> ? 2014/7/25 0:44, Lucas Tanure ??:
>
> Could start cleaning a driver ?
> http://kernelnewbies.org/OPWfirstpatch#head-4abafc61af197fb6e3d6cda623b00bdd90a52c26
>
> The job here is about how to send patchs and not doing a awesome code.
>
>
> Hi Lucas,
> Because you have mentioned OPW, which is short for "Outstreach Program for
> Women".
> but I am a *boy*, could I do some works on this program ?
>
> regards,
> Yi
>
> --
> Lucas Tanure
> +55 (19) 988176559
>
>
> On Thu, Jul 24, 2014 at 1:38 PM, Nick Krause <xerofoify@gmail.com> wrote:
>>
>> Is there any work for a kernel newbie that you guys known of?
>> Cheers Nick
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>

Work

[Andev]

On Thu, Jul 24, 2014 at 12:38 PM, Nick Krause <xerofoify@gmail.com> wrote:
> Is there any work for a kernel newbie that you guys known of?
> Cheers Nick
>

Your first task will be reading and _understanding_ the following:

Greg Kroah-Hartman, "How to piss off a kernel subsystem maintainer".
  <http://www.kroah.com/log/linux/maintainer.html>
  <http://www.kroah.com/log/linux/maintainer-02.html>
  <http://www.kroah.com/log/linux/maintainer-03.html>
  <http://www.kroah.com/log/linux/maintainer-04.html>
  <http://www.kroah.com/log/linux/maintainer-05.html>

from Documentation/SubmittingPatches.

-- 
Andev

Work

[Nick Krause]

On Thu, Jul 24, 2014 at 12:51 PM, Andev <debiandev@gmail.com> wrote:
> On Thu, Jul 24, 2014 at 12:38 PM, Nick Krause <xerofoify@gmail.com> wrote:
>> Is there any work for a kernel newbie that you guys known of?
>> Cheers Nick
>>
>
> Your first task will be reading and _understanding_ the following:
>
> Greg Kroah-Hartman, "How to piss off a kernel subsystem maintainer".
>   <http://www.kroah.com/log/linux/maintainer.html>
>   <http://www.kroah.com/log/linux/maintainer-02.html>
>   <http://www.kroah.com/log/linux/maintainer-03.html>
>   <http://www.kroah.com/log/linux/maintainer-04.html>
>   <http://www.kroah.com/log/linux/maintainer-05.html>
>
> from Documentation/SubmittingPatches.
>
> --
> Andev


Yes, I didn't build test and that is a huge mistake.
Cheers Nick

Work

[Nick Krause]

On Thu, Jul 24, 2014 at 1:10 PM, Nick Krause <xerofoify@gmail.com> wrote:
> On Thu, Jul 24, 2014 at 12:51 PM, Andev <debiandev@gmail.com> wrote:
>> On Thu, Jul 24, 2014 at 12:38 PM, Nick Krause <xerofoify@gmail.com> wrote:
>>> Is there any work for a kernel newbie that you guys known of?
>>> Cheers Nick
>>>
>>
>> Your first task will be reading and _understanding_ the following:
>>
>> Greg Kroah-Hartman, "How to piss off a kernel subsystem maintainer".
>>   <http://www.kroah.com/log/linux/maintainer.html>
>>   <http://www.kroah.com/log/linux/maintainer-02.html>
>>   <http://www.kroah.com/log/linux/maintainer-03.html>
>>   <http://www.kroah.com/log/linux/maintainer-04.html>
>>   <http://www.kroah.com/log/linux/maintainer-05.html>
>>
>> from Documentation/SubmittingPatches.
>>
>> --
>> Andev
>
>
> Yes, I didn't build test and that is a huge mistake.
> Cheers Nick

Andev,
I having been doing build tests and checkpatch in staging for the last month.
It doesn't seem like it's worth my time as so my other people are doing it. I
want an interesting project one that is challenging and rewarding :).
Nick

Work

[ravi ranjan Mishra]

How to make environment like (build and test) for working on kernel patch,
what are the resources should be available for that.
thanks.


On Fri, Jul 25, 2014 at 7:53 AM, Nick Krause <xerofoify@gmail.com> wrote:

> On Thu, Jul 24, 2014 at 1:10 PM, Nick Krause <xerofoify@gmail.com> wrote:
> > On Thu, Jul 24, 2014 at 12:51 PM, Andev <debiandev@gmail.com> wrote:
> >> On Thu, Jul 24, 2014 at 12:38 PM, Nick Krause <xerofoify@gmail.com>
> wrote:
> >>> Is there any work for a kernel newbie that you guys known of?
> >>> Cheers Nick
> >>>
> >>
> >> Your first task will be reading and _understanding_ the following:
> >>
> >> Greg Kroah-Hartman, "How to piss off a kernel subsystem maintainer".
> >>   <http://www.kroah.com/log/linux/maintainer.html>
> >>   <http://www.kroah.com/log/linux/maintainer-02.html>
> >>   <http://www.kroah.com/log/linux/maintainer-03.html>
> >>   <http://www.kroah.com/log/linux/maintainer-04.html>
> >>   <http://www.kroah.com/log/linux/maintainer-05.html>
> >>
> >> from Documentation/SubmittingPatches.
> >>
> >> --
> >> Andev
> >
> >
> > Yes, I didn't build test and that is a huge mistake.
> > Cheers Nick
>
> Andev,
> I having been doing build tests and checkpatch in staging for the last
> month.
> It doesn't seem like it's worth my time as so my other people are doing
> it. I
> want an interesting project one that is challenging and rewarding :).
> Nick
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140725/5e50455a/attachment.html 

Work

[Lucas Tanure]

Hi,

Watch : http://www.youtube.com/watch?v=LLBrBBImJt4

Read :
http://kernelnewbies.org/OPWfirstpatch
http://kernelnewbies.org/KernelBuild
http://lwn.net/Articles/571980/

Goal : Clone, build and run linux-next.

After that you can look for how to clean up a staging driver.

Thanks





--
Lucas Tanure
+55 (19) 988176559


On Fri, Jul 25, 2014 at 2:33 AM, ravi ranjan Mishra <
raviranjanmishra24@gmail.com> wrote:

> How to make environment like (build and test) for working on kernel patch,
> what are the resources should be available for that.
> thanks.
>
>
> On Fri, Jul 25, 2014 at 7:53 AM, Nick Krause <xerofoify@gmail.com> wrote:
>
>> On Thu, Jul 24, 2014 at 1:10 PM, Nick Krause <xerofoify@gmail.com> wrote:
>> > On Thu, Jul 24, 2014 at 12:51 PM, Andev <debiandev@gmail.com> wrote:
>> >> On Thu, Jul 24, 2014 at 12:38 PM, Nick Krause <xerofoify@gmail.com>
>> wrote:
>> >>> Is there any work for a kernel newbie that you guys known of?
>> >>> Cheers Nick
>> >>>
>> >>
>> >> Your first task will be reading and _understanding_ the following:
>> >>
>> >> Greg Kroah-Hartman, "How to piss off a kernel subsystem maintainer".
>> >>   <http://www.kroah.com/log/linux/maintainer.html>
>> >>   <http://www.kroah.com/log/linux/maintainer-02.html>
>> >>   <http://www.kroah.com/log/linux/maintainer-03.html>
>> >>   <http://www.kroah.com/log/linux/maintainer-04.html>
>> >>   <http://www.kroah.com/log/linux/maintainer-05.html>
>> >>
>> >> from Documentation/SubmittingPatches.
>> >>
>> >> --
>> >> Andev
>> >
>> >
>> > Yes, I didn't build test and that is a huge mistake.
>> > Cheers Nick
>>
>> Andev,
>> I having been doing build tests and checkpatch in staging for the last
>> month.
>> It doesn't seem like it's worth my time as so my other people are doing
>> it. I
>> want an interesting project one that is challenging and rewarding :).
>> Nick
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>>
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140725/f8ef5d76/attachment-0001.html 

Work

[Robert P. J. Day]

On Fri, 25 Jul 2014, Lucas Tanure wrote:

> Hi,?
> Watch :?http://www.youtube.com/watch?v=LLBrBBImJt4
>
> Read :?
> http://kernelnewbies.org/OPWfirstpatch
> http://kernelnewbies.org/KernelBuild?
> http://lwn.net/Articles/571980/
>
> Goal : Clone, build and run linux-next.?
>
> After that you can look for how to clean up a?staging driver.

  if i may (since i'm sure i antagonized my share of kernel gurus once
upon a time as i was getting my feet wet), start with cleaning up the
docs. read up on a subsystem, peruse the code, check the comments, see
if there's accompanying explanation in the Documentation/ directory,
and either correct obvious mistakes or improve what's there. the
obvious advantage to this is that you can't break stuff by working on
documentation. and everyone wants better documentation.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Work

[Nick Krause]

On Fri, Jul 25, 2014 at 8:17 AM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
> On Fri, 25 Jul 2014, Lucas Tanure wrote:
>
>> Hi,
>> Watch : http://www.youtube.com/watch?v=LLBrBBImJt4
>>
>> Read :
>> http://kernelnewbies.org/OPWfirstpatch
>> http://kernelnewbies.org/KernelBuild
>> http://lwn.net/Articles/571980/
>>
>> Goal : Clone, build and run linux-next.
>>
>> After that you can look for how to clean up a staging driver.
>
>   if i may (since i'm sure i antagonized my share of kernel gurus once
> upon a time as i was getting my feet wet), start with cleaning up the
> docs. read up on a subsystem, peruse the code, check the comments, see
> if there's accompanying explanation in the Documentation/ directory,
> and either correct obvious mistakes or improve what's there. the
> obvious advantage to this is that you can't break stuff by working on
> documentation. and everyone wants better documentation.
>
> rday
>
> --
I have already watched this and I have no idea on how to write the docs as my
knowledge of kernel subsystems is rather limited.
Cheers Nick

Work

[Robert P. J. Day]

On Fri, 25 Jul 2014, Nick Krause wrote:

> I have already watched this and I have no idea on how to write the
> docs as my knowledge of kernel subsystems is rather limited.

  um ... i think we've identified your problem, then.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Work

[Valdis.Kletnieks at vt.edu]

On Fri, 25 Jul 2014 11:03:36 +0530, ravi ranjan Mishra said:

> How to make environment like (build and test) for working on kernel patch,
> what are the resources should be available for that.
> thanks.

Depends on what exactly you're trying to do.  If it involves hardware
support, you'll need the hardware and a computer it will attach to.

Other than that, you'll need gcc, probably git, and if your target
system is a Raspberry Pi or similar *really* slow box, you'll want a faster
box to do cross-compiles.

But I've done essentially all my kernel hacking on bog-standard Dell Latitude
laptops...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140725/1f4d2937/attachment.bin 

Work

[Valdis.Kletnieks at vt.edu]

On Thu, 24 Jul 2014 22:23:00 -0400, Nick Krause said:

> I having been doing build tests and checkpatch in staging for the last month.
> It doesn't seem like it's worth my time as so my other people are doing it. I
> want an interesting project one that is challenging and rewarding :).

OK. I'm gonna be blunt.  That's about as dumb as people who post "What should
my next blog post be about?", and for exactly the same reasons.

If you aren't *internally* motivated by a specific goal/interest, we're
not going to be able to help.  Personally, I'm motivated to do testing and
debugging because I have a quarter acre of Linux boxes across the hall that
need stable kernels, and more users over on campus.  And every single bug
I catch in linux-next or Fedora Rawhide and file a *good* bug report on
is one less bug that one of my users can trip over and file a poor bug
report about. ;)

Other people are motivated by the fact their employer is paying them
to write a Foobar 2890 driver.  Some people find filesystems intellectually
interesting, and others worry far too much about security.  And so on.

But if nothing like that is jumping out at you, maybe you should go look
around and see if there's something in userspace that *does* jump out at you.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140725/47ef1790/attachment.bin 

Work

[Nick Krause]

On Fri, Jul 25, 2014 at 1:42 PM,  <Valdis.Kletnieks@vt.edu> wrote:
> On Thu, 24 Jul 2014 22:23:00 -0400, Nick Krause said:
>
>> I having been doing build tests and checkpatch in staging for the last month.
>> It doesn't seem like it's worth my time as so my other people are doing it. I
>> want an interesting project one that is challenging and rewarding :).
>
> OK. I'm gonna be blunt.  That's about as dumb as people who post "What should
> my next blog post be about?", and for exactly the same reasons.
>
> If you aren't *internally* motivated by a specific goal/interest, we're
> not going to be able to help.  Personally, I'm motivated to do testing and
> debugging because I have a quarter acre of Linux boxes across the hall that
> need stable kernels, and more users over on campus.  And every single bug
> I catch in linux-next or Fedora Rawhide and file a *good* bug report on
> is one less bug that one of my users can trip over and file a poor bug
> report about. ;)
>
> Other people are motivated by the fact their employer is paying them
> to write a Foobar 2890 driver.  Some people find filesystems intellectually
> interesting, and others worry far too much about security.  And so on.
>
> But if nothing like that is jumping out at you, maybe you should go look
> around and see if there's something in userspace that *does* jump out at you.

I am interested in file systems and will be working on brtfs and ext4.
Cheers Nick

Work

[Arlie Stephens]

On Jul 25 2014, Nick Krause wrote:
> > But if nothing like that is jumping out at you, maybe you should go look
> > around and see if there's something in userspace that *does* jump out at you.
> 
> I am interested in file systems and will be working on brtfs and ext4.
> Cheers Nick

If you want an annoying problem, explain and/or fix directory
performance on ext4. I've got a server where an ls of a directory took
5 seconds, according to "time", even though it only has 295 entries at
present.

It's a rather large directory, 4 times the size of a normal directory
in the opinion of ls -ld, and filefrag reports it has 7 extents. 

I'm also getting anecdotal reports of rename() taking unreasonable
amounts of time, probably with at least one of the directories
involved being in a similar state.  (They would have been created and
populated by the same software.) 

Probably the directory had a large number of files and/or
subdirectories some time in the past, and this is "expected
behaviour".  Expected or not, it seems to me that it's about time that 
linux handles directories with something closer to the efficiency with 
which it handles files ;-) 

-- 
Arlie

(Arlie Stephens					arlie at worldash.org)

Work

[Nick Krause]

On Fri, Jul 25, 2014 at 6:23 PM, Arlie Stephens <arlie@worldash.org> wrote:
> On Jul 25 2014, Nick Krause wrote:
>> > But if nothing like that is jumping out at you, maybe you should go look
>> > around and see if there's something in userspace that *does* jump out at you.
>>
>> I am interested in file systems and will be working on brtfs and ext4.
>> Cheers Nick
>
> If you want an annoying problem, explain and/or fix directory
> performance on ext4. I've got a server where an ls of a directory took
> 5 seconds, according to "time", even though it only has 295 entries at
> present.
>
> It's a rather large directory, 4 times the size of a normal directory
> in the opinion of ls -ld, and filefrag reports it has 7 extents.
>
> I'm also getting anecdotal reports of rename() taking unreasonable
> amounts of time, probably with at least one of the directories
> involved being in a similar state.  (They would have been created and
> populated by the same software.)
>
> Probably the directory had a large number of files and/or
> subdirectories some time in the past, and this is "expected
> behaviour".  Expected or not, it seems to me that it's about time that
> linux handles directories with something closer to the efficiency with
> which it handles files ;-)
>
> --
> Arlie
>
> (Arlie Stephens                                 arlie at worldash.org)


Sure Arlie,
I don't mind helping you out but you have to test this for me as this problem is
not happening on my hardware at least :).
Cheers Nick

Work

[Valdis.Kletnieks at vt.edu]

On Fri, 25 Jul 2014 15:23:42 -0700, Arlie Stephens said:

> If you want an annoying problem, explain and/or fix directory
> performance on ext4. I've got a server where an ls of a directory took
> 5 seconds, according to "time", even though it only has 295 entries at
> present.

I don't suppose you could get a trace of where that ls is spending its
time with the kernel's trace facilities, or even just getting a stack trace
of where that ls is in the kernel?

I'll go out on a limb and ask if a *second* ls of the same directory runs
quickly because it's now cache-hot.  If so, I'd start looking at whether
there's large amounts of *other* disk activity going on, and the reads of the
directory are getting hung in the I/O queue behind other disk read/writes.

Also, are you doing an 'ls' (which just requires reading the name/inode#
pairs), or an 'ls -l' whihc in addition requires a stat() call to read in the
inode itself)?  That makes a lot of difference.  Cache-cold on my laptop, and a
*huge* Mail/linux-kernel directory (yes, it really *is* an 11M directory,
it's got a half-million entries in it):

[~] echo 3 >| /proc/sys/vm/drop_caches
[~] cd Mail
[~/Mail] time ls linux-kernel/ | wc -l
478401

real    0m2.387s
user    0m0.500s
sys     0m0.433s
[~/Mail] ls -ld linux-kernel/
drwxr-xr-x. 2 valdis valdis 11005952 Jul 25 19:30 linux-kernel/
[~/Mail] time ls -l linux-kernel/ | wc -l
478402

real    0m32.915s
user    0m2.483s
sys     0m20.787s

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140725/e1d11ecd/attachment.bin 

Work

[Nick Krause]

On Fri, Jul 25, 2014 at 7:35 PM,  <Valdis.Kletnieks@vt.edu> wrote:
> On Fri, 25 Jul 2014 15:23:42 -0700, Arlie Stephens said:
>
>> If you want an annoying problem, explain and/or fix directory
>> performance on ext4. I've got a server where an ls of a directory took
>> 5 seconds, according to "time", even though it only has 295 entries at
>> present.
>
> I don't suppose you could get a trace of where that ls is spending its
> time with the kernel's trace facilities, or even just getting a stack trace
> of where that ls is in the kernel?
>
> I'll go out on a limb and ask if a *second* ls of the same directory runs
> quickly because it's now cache-hot.  If so, I'd start looking at whether
> there's large amounts of *other* disk activity going on, and the reads of the
> directory are getting hung in the I/O queue behind other disk read/writes.
>
> Also, are you doing an 'ls' (which just requires reading the name/inode#
> pairs), or an 'ls -l' whihc in addition requires a stat() call to read in the
> inode itself)?  That makes a lot of difference.  Cache-cold on my laptop, and a
> *huge* Mail/linux-kernel directory (yes, it really *is* an 11M directory,
> it's got a half-million entries in it):
>
> [~] echo 3 >| /proc/sys/vm/drop_caches
> [~] cd Mail
> [~/Mail] time ls linux-kernel/ | wc -l
> 478401
>
> real    0m2.387s
> user    0m0.500s
> sys     0m0.433s
> [~/Mail] ls -ld linux-kernel/
> drwxr-xr-x. 2 valdis valdis 11005952 Jul 25 19:30 linux-kernel/
> [~/Mail] time ls -l linux-kernel/ | wc -l
> 478402
>
> real    0m32.915s
> user    0m2.483s
> sys     0m20.787s
>


Would you find reading strace as I can get the system calls and
profiling the code so
so we can trace where the issue is based on timing and cpu usage of
the code running ?
If not seems the issue is either with I/O queueneeds to add support
for better reading of
multiple large disk actives.
Cheers Nick

Work (really slow directory access on ext4)

[Arlie Stephens]

On Jul 25 2014, Valdis.Kletnieks at vt.edu wrote:
> On Fri, 25 Jul 2014 15:23:42 -0700, Arlie Stephens said:
> 
> > If you want an annoying problem, explain and/or fix directory
> > performance on ext4. I've got a server where an ls of a directory took
> > 5 seconds, according to "time", even though it only has 295 entries at
> > present.
> 
> I don't suppose you could get a trace of where that ls is spending its
> time with the kernel's trace facilities, or even just getting a stack trace
> of where that ls is in the kernel?

These are all very good questions. 

To my amazement, I found that no one had yet fixed the problem by
deleting and recreating the directory, and I do have sudo access. 
This time it was only 4 seconds...
     real 0m3.992s
     user 0m0.005s
     sys  0m0.052s

> I'll go out on a limb and ask if a *second* ls of the same directory runs
> quickly because it's now cache-hot.  If so, I'd start looking at whether
> there's large amounts of *other* disk activity going on, and the reads of the
> directory are getting hung in the I/O queue behind other disk
> read/writes.

Sure enough, the cache saved me on a second read - 
     real 0m0.010s
     user 0m0.000s
     sys  0m0.010s

> Also, are you doing an 'ls' (which just requires reading the name/inode#
> pairs), or an 'ls -l' whihc in addition requires a stat() call to read in the
> inode itself)?  That makes a lot of difference.  Cache-cold on my laptop, and a
> *huge* Mail/linux-kernel directory (yes, it really *is* an 11M directory,
> it's got a half-million entries in it):

I was doing a vanilla ls. So was the original reporter, unless he has
some really strange aliases.


I'm afraid I'll be rather unpopular if I drop the caches on the system
in question, creating a burst of poor performance, so my best bet is
probably to see what I can do with ftrace on Monday, or perhaps
partway through the weekend.  

There is normally a fair amount of disk activity going on - much of it
writes. So I can expect cached blocks to age out in a reasonable time. 


> [~] echo 3 >| /proc/sys/vm/drop_caches
> [~] cd Mail
> [~/Mail] time ls linux-kernel/ | wc -l
> 478401
> 
> real    0m2.387s
> user    0m0.500s
> sys     0m0.433s
> [~/Mail] ls -ld linux-kernel/
> drwxr-xr-x. 2 valdis valdis 11005952 Jul 25 19:30 linux-kernel/

Compared to your directory, mine is microscopic

$ ls -ld xxxx
drwxr-xr-x 2 yyy yyy 36864 Jul 25 12:19 xxxx


> [~/Mail] time ls -l linux-kernel/ | wc -l
> 478402
> 
> real    0m32.915s
> user    0m2.483s
> sys     0m20.787s

-- 
Arlie

(Arlie Stephens					arlie at worldash.org)

Work (really slow directory access on ext4)

[Nick Krause]

On Fri, Jul 25, 2014 at 9:08 PM, Arlie Stephens <arlie@worldash.org> wrote:
> On Jul 25 2014, Valdis.Kletnieks at vt.edu wrote:
>> On Fri, 25 Jul 2014 15:23:42 -0700, Arlie Stephens said:
>>
>> > If you want an annoying problem, explain and/or fix directory
>> > performance on ext4. I've got a server where an ls of a directory took
>> > 5 seconds, according to "time", even though it only has 295 entries at
>> > present.
>>
>> I don't suppose you could get a trace of where that ls is spending its
>> time with the kernel's trace facilities, or even just getting a stack trace
>> of where that ls is in the kernel?
>
> These are all very good questions.
>
> To my amazement, I found that no one had yet fixed the problem by
> deleting and recreating the directory, and I do have sudo access.
> This time it was only 4 seconds...
>      real 0m3.992s
>      user 0m0.005s
>      sys  0m0.052s
>
>> I'll go out on a limb and ask if a *second* ls of the same directory runs
>> quickly because it's now cache-hot.  If so, I'd start looking at whether
>> there's large amounts of *other* disk activity going on, and the reads of the
>> directory are getting hung in the I/O queue behind other disk
>> read/writes.
>
> Sure enough, the cache saved me on a second read -
>      real 0m0.010s
>      user 0m0.000s
>      sys  0m0.010s
>
>> Also, are you doing an 'ls' (which just requires reading the name/inode#
>> pairs), or an 'ls -l' whihc in addition requires a stat() call to read in the
>> inode itself)?  That makes a lot of difference.  Cache-cold on my laptop, and a
>> *huge* Mail/linux-kernel directory (yes, it really *is* an 11M directory,
>> it's got a half-million entries in it):
>
> I was doing a vanilla ls. So was the original reporter, unless he has
> some really strange aliases.
>
>
> I'm afraid I'll be rather unpopular if I drop the caches on the system
> in question, creating a burst of poor performance, so my best bet is
> probably to see what I can do with ftrace on Monday, or perhaps
> partway through the weekend.
>
> There is normally a fair amount of disk activity going on - much of it
> writes. So I can expect cached blocks to age out in a reasonable time.
>
>
>> [~] echo 3 >| /proc/sys/vm/drop_caches
>> [~] cd Mail
>> [~/Mail] time ls linux-kernel/ | wc -l
>> 478401
>>
>> real    0m2.387s
>> user    0m0.500s
>> sys     0m0.433s
>> [~/Mail] ls -ld linux-kernel/
>> drwxr-xr-x. 2 valdis valdis 11005952 Jul 25 19:30 linux-kernel/
>
> Compared to your directory, mine is microscopic
>
> $ ls -ld xxxx
> drwxr-xr-x 2 yyy yyy 36864 Jul 25 12:19 xxxx
>
>
>> [~/Mail] time ls -l linux-kernel/ | wc -l
>> 478402
>>
>> real    0m32.915s
>> user    0m2.483s
>> sys     0m20.787s
>
> --
> Arlie
>
> (Arlie Stephens                                 arlie at worldash.org)


Arlie,
Whenever you get around to it is fine.
Just send me a log.
Cheers Nick

Work (really slow directory access on ext4)

[Nick Krause]

On Fri, Jul 25, 2014 at 9:22 PM, Nick Krause <xerofoify@gmail.com> wrote:
> On Fri, Jul 25, 2014 at 9:08 PM, Arlie Stephens <arlie@worldash.org> wrote:
>> On Jul 25 2014, Valdis.Kletnieks at vt.edu wrote:
>>> On Fri, 25 Jul 2014 15:23:42 -0700, Arlie Stephens said:
>>>
>>> > If you want an annoying problem, explain and/or fix directory
>>> > performance on ext4. I've got a server where an ls of a directory took
>>> > 5 seconds, according to "time", even though it only has 295 entries at
>>> > present.
>>>
>>> I don't suppose you could get a trace of where that ls is spending its
>>> time with the kernel's trace facilities, or even just getting a stack trace
>>> of where that ls is in the kernel?
>>
>> These are all very good questions.
>>
>> To my amazement, I found that no one had yet fixed the problem by
>> deleting and recreating the directory, and I do have sudo access.
>> This time it was only 4 seconds...
>>      real 0m3.992s
>>      user 0m0.005s
>>      sys  0m0.052s
>>
>>> I'll go out on a limb and ask if a *second* ls of the same directory runs
>>> quickly because it's now cache-hot.  If so, I'd start looking at whether
>>> there's large amounts of *other* disk activity going on, and the reads of the
>>> directory are getting hung in the I/O queue behind other disk
>>> read/writes.
>>
>> Sure enough, the cache saved me on a second read -
>>      real 0m0.010s
>>      user 0m0.000s
>>      sys  0m0.010s
>>
>>> Also, are you doing an 'ls' (which just requires reading the name/inode#
>>> pairs), or an 'ls -l' whihc in addition requires a stat() call to read in the
>>> inode itself)?  That makes a lot of difference.  Cache-cold on my laptop, and a
>>> *huge* Mail/linux-kernel directory (yes, it really *is* an 11M directory,
>>> it's got a half-million entries in it):
>>
>> I was doing a vanilla ls. So was the original reporter, unless he has
>> some really strange aliases.
>>
>>
>> I'm afraid I'll be rather unpopular if I drop the caches on the system
>> in question, creating a burst of poor performance, so my best bet is
>> probably to see what I can do with ftrace on Monday, or perhaps
>> partway through the weekend.
>>
>> There is normally a fair amount of disk activity going on - much of it
>> writes. So I can expect cached blocks to age out in a reasonable time.
>>
>>
>>> [~] echo 3 >| /proc/sys/vm/drop_caches
>>> [~] cd Mail
>>> [~/Mail] time ls linux-kernel/ | wc -l
>>> 478401
>>>
>>> real    0m2.387s
>>> user    0m0.500s
>>> sys     0m0.433s
>>> [~/Mail] ls -ld linux-kernel/
>>> drwxr-xr-x. 2 valdis valdis 11005952 Jul 25 19:30 linux-kernel/
>>
>> Compared to your directory, mine is microscopic
>>
>> $ ls -ld xxxx
>> drwxr-xr-x 2 yyy yyy 36864 Jul 25 12:19 xxxx
>>
>>
>>> [~/Mail] time ls -l linux-kernel/ | wc -l
>>> 478402
>>>
>>> real    0m32.915s
>>> user    0m2.483s
>>> sys     0m20.787s
>>
>> --
>> Arlie
>>
>> (Arlie Stephens                                 arlie at worldash.org)
>
>
> Arlie,
> Whenever you get around to it is fine.
> Just send me a log.
> Cheers Nick

Arlie,
just a friendly reminder can you try to send me the log this week.
Regards Nick

Work (really slow directory access on ext4)

[Arlie Stephens]

Hi Nick,

On Jul 29 2014, Nick Krause wrote:
> >> I was doing a vanilla ls. So was the original reporter, unless he has
> >> some really strange aliases.
> >>
> >>
> >> I'm afraid I'll be rather unpopular if I drop the caches on the system
> >> in question, creating a burst of poor performance, so my best bet is
> >> probably to see what I can do with ftrace on Monday, or perhaps
> >> partway through the weekend.
> >>
> >> There is normally a fair amount of disk activity going on - much of it
> >> writes. So I can expect cached blocks to age out in a reasonable time.
> >>
> > Arlie,
> > Whenever you get around to it is fine.
> > Just send me a log.
> > Cheers Nick
> 
> Arlie,
> just a friendly reminder can you try to send me the log this week.
> Regards Nick

I was just going to post an apology for going dark on you. I made one
attempt to capture the data yesterday, and messed up - no useful data
saved. And then half the world invaded my workspace with higher
priority tasks ;-)  

I'm going to make another attempt at it this morning.

On the good side, Vladis' observations of his mail directory have been
a great help. Now I know that simply being a large ext4 directory is
not the problem ;-)  I.e. ext4 really isn't as brain damaged as I
feared. (We had someone here who was initially sure that was it, and
he has more experience in linux server space than I do, so I took his
initial opinion at face value.) 

More soon, I hope.

-- 
Arlie

(Arlie Stephens					arlie at worldash.org)

Work (really slow directory access on ext4)

[Valdis.Kletnieks at vt.edu]

On Wed, 30 Jul 2014 10:38:13 -0700, Arlie Stephens said:

> On the good side, Vladis' observations of his mail directory have been
> a great help.

And remember, that's on a single laptop-class hard drive, no fancy raid or
anything. (Though it *is* a hybrid, with 32G of flash cache on the front end).

You throw some *real* hardware at it, it of course would go even faster.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 848 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20140730/38dc3fdf/attachment.bin 

Work (really slow directory access on ext4)

[Nick Krause]

On Wed, Jul 30, 2014 at 3:48 PM,  <Valdis.Kletnieks@vt.edu> wrote:
> On Wed, 30 Jul 2014 10:38:13 -0700, Arlie Stephens said:
>
>> On the good side, Vladis' observations of his mail directory have been
>> a great help.
>
> And remember, that's on a single laptop-class hard drive, no fancy raid or
> anything. (Though it *is* a hybrid, with 32G of flash cache on the front end).
>
> You throw some *real* hardware at it, it of course would go even faster.

Just send me the logs and anything else you think may help me.
Please note cc the ext4 mailing list as this will also let the other
ext4 developers and maintainers known about your problem.
Cheers Nick

Work (really slow directory access on ext4)

[Arlie Stephens]

Hi Nick,

[Context - directory ls taking 4-15 seconds; directory large, with
long filenames, but nowhere near as huge as Valdis' mail directory.]

I've now discovered a really bizarre pattern, and I'm inclined to stop
blaming the file system until some clarity develops. If I ever get it
to the point where I can produce a high quality bug report - with or
without patch - I will do so - but what I have now is anything but
clear and high quality. 

On Jul 30 2014, Nick Krause wrote:
> On Wed, Jul 30, 2014 at 3:48 PM,  <Valdis.Kletnieks@vt.edu> wrote:
> > On Wed, 30 Jul 2014 10:38:13 -0700, Arlie Stephens said:
> >
> >> On the good side, Vladis' observations of his mail directory have been
> >> a great help.
> >
> > And remember, that's on a single laptop-class hard drive, no fancy raid or
> > anything. (Though it *is* a hybrid, with 32G of flash cache on the front end).
> >
> > You throw some *real* hardware at it, it of course would go even faster.
> 
> Just send me the logs and anything else you think may help me.
> Please note cc the ext4 mailing list as this will also let the other
> ext4 developers and maintainers known about your problem.
> Cheers Nick

I'm now in a state of complete bafflement.  

It turns out we have a whole collection of misbehaving directories, 
making this testable without waiting for caches to clear. 

I have a couple of strace's of fast ls's, and a function ftrace that
captured about half of a 7 second ls. (The latter is huge, and
probably not suitable for posting.)

I also have a really bizarre observation, the kind that makes you
wonder whether you are actually dreaming. It appears that the
misbehaviour is strongly influenced by the choice of "time" function. 
The problem only occurs when using the shell built-in. /usr/bin/time 
always produces a fast response. 

Stranger still - flat out impossible, I'd have said before seeing it - 
a "fast" ls, run with /usr/bin/time can be followed *immediately* 
by a slow "ls", run with bash' time. It's as if the first one doesn't
warm the cache, which is completely absurd - except I've been able to
make this happen 5 times in a row, first with strace and then
without. 

# with /usr/bin/time the ls is fast
$ time -p ls bad_dir
...
real 0.21
user 0.00
sys 0.00


# with the builtin time, right *after* the strace run, the time can be 
# horrible. 
$ time -p ls bad_dir
...
real 5.60
user 0.00
sys 0.17

# run it again, and the directory is in cache as expected.
$ time -p ls bad_dir
...
real 0.11
user 0.00
sys 0.02


This is not an artefact of one or other time reporting incorrectly -
I'm noticing a long pause before output occurs, but only on the middle
test of the three. 

I can't imagine any sane way for this to be happening, short of
coincidence or user error - and I've now seen this sequence 5 times in
a row, on 5 different directories created and populated by the same
app. (Three times with strace, twice without.) 


-- 
Arlie

(Arlie Stephens					arlie at worldash.org)

Work (really slow directory access on ext4)

[Henry Hallam]

Try redirecting the ls output to /dev/null or a file, thus disabling
its color highlighting and thus removing a bunch of syscalls.  See if
it's now the same no matter what choice of 'time'.

On Thu, Jul 31, 2014 at 4:36 PM, Arlie Stephens <arlie@worldash.org> wrote:
> Hi Nick,
>
> [Context - directory ls taking 4-15 seconds; directory large, with
> long filenames, but nowhere near as huge as Valdis' mail directory.]
>
> I've now discovered a really bizarre pattern, and I'm inclined to stop
> blaming the file system until some clarity develops. If I ever get it
> to the point where I can produce a high quality bug report - with or
> without patch - I will do so - but what I have now is anything but
> clear and high quality.
>
> On Jul 30 2014, Nick Krause wrote:
>> On Wed, Jul 30, 2014 at 3:48 PM,  <Valdis.Kletnieks@vt.edu> wrote:
>> > On Wed, 30 Jul 2014 10:38:13 -0700, Arlie Stephens said:
>> >
>> >> On the good side, Vladis' observations of his mail directory have been
>> >> a great help.
>> >
>> > And remember, that's on a single laptop-class hard drive, no fancy raid or
>> > anything. (Though it *is* a hybrid, with 32G of flash cache on the front end).
>> >
>> > You throw some *real* hardware at it, it of course would go even faster.
>>
>> Just send me the logs and anything else you think may help me.
>> Please note cc the ext4 mailing list as this will also let the other
>> ext4 developers and maintainers known about your problem.
>> Cheers Nick
>
> I'm now in a state of complete bafflement.
>
> It turns out we have a whole collection of misbehaving directories,
> making this testable without waiting for caches to clear.
>
> I have a couple of strace's of fast ls's, and a function ftrace that
> captured about half of a 7 second ls. (The latter is huge, and
> probably not suitable for posting.)
>
> I also have a really bizarre observation, the kind that makes you
> wonder whether you are actually dreaming. It appears that the
> misbehaviour is strongly influenced by the choice of "time" function.
> The problem only occurs when using the shell built-in. /usr/bin/time
> always produces a fast response.
>
> Stranger still - flat out impossible, I'd have said before seeing it -
> a "fast" ls, run with /usr/bin/time can be followed *immediately*
> by a slow "ls", run with bash' time. It's as if the first one doesn't
> warm the cache, which is completely absurd - except I've been able to
> make this happen 5 times in a row, first with strace and then
> without.
>
> # with /usr/bin/time the ls is fast
> $ time -p ls bad_dir
> ...
> real 0.21
> user 0.00
> sys 0.00
>
>
> # with the builtin time, right *after* the strace run, the time can be
> # horrible.
> $ time -p ls bad_dir
> ...
> real 5.60
> user 0.00
> sys 0.17
>
> # run it again, and the directory is in cache as expected.
> $ time -p ls bad_dir
> ...
> real 0.11
> user 0.00
> sys 0.02
>
>
> This is not an artefact of one or other time reporting incorrectly -
> I'm noticing a long pause before output occurs, but only on the middle
> test of the three.
>
> I can't imagine any sane way for this to be happening, short of
> coincidence or user error - and I've now seen this sequence 5 times in
> a row, on 5 different directories created and populated by the same
> app. (Three times with strace, twice without.)
>
>
> --
> Arlie
>
> (Arlie Stephens                                 arlie at worldash.org)
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Work (really slow directory access on ext4)

[Nick Krause]

On Thu, Jul 31, 2014 at 7:41 PM, Henry Hallam <henry@pericynthion.org> wrote:
> Try redirecting the ls output to /dev/null or a file, thus disabling
> its color highlighting and thus removing a bunch of syscalls.  See if
> it's now the same no matter what choice of 'time'.
>
> On Thu, Jul 31, 2014 at 4:36 PM, Arlie Stephens <arlie@worldash.org> wrote:
>> Hi Nick,
>>
>> [Context - directory ls taking 4-15 seconds; directory large, with
>> long filenames, but nowhere near as huge as Valdis' mail directory.]
>>
>> I've now discovered a really bizarre pattern, and I'm inclined to stop
>> blaming the file system until some clarity develops. If I ever get it
>> to the point where I can produce a high quality bug report - with or
>> without patch - I will do so - but what I have now is anything but
>> clear and high quality.
>>
>> On Jul 30 2014, Nick Krause wrote:
>>> On Wed, Jul 30, 2014 at 3:48 PM,  <Valdis.Kletnieks@vt.edu> wrote:
>>> > On Wed, 30 Jul 2014 10:38:13 -0700, Arlie Stephens said:
>>> >
>>> >> On the good side, Vladis' observations of his mail directory have been
>>> >> a great help.
>>> >
>>> > And remember, that's on a single laptop-class hard drive, no fancy raid or
>>> > anything. (Though it *is* a hybrid, with 32G of flash cache on the front end).
>>> >
>>> > You throw some *real* hardware at it, it of course would go even faster.
>>>
>>> Just send me the logs and anything else you think may help me.
>>> Please note cc the ext4 mailing list as this will also let the other
>>> ext4 developers and maintainers known about your problem.
>>> Cheers Nick
>>
>> I'm now in a state of complete bafflement.
>>
>> It turns out we have a whole collection of misbehaving directories,
>> making this testable without waiting for caches to clear.
>>
>> I have a couple of strace's of fast ls's, and a function ftrace that
>> captured about half of a 7 second ls. (The latter is huge, and
>> probably not suitable for posting.)
>>
>> I also have a really bizarre observation, the kind that makes you
>> wonder whether you are actually dreaming. It appears that the
>> misbehaviour is strongly influenced by the choice of "time" function.
>> The problem only occurs when using the shell built-in. /usr/bin/time
>> always produces a fast response.
>>
>> Stranger still - flat out impossible, I'd have said before seeing it -
>> a "fast" ls, run with /usr/bin/time can be followed *immediately*
>> by a slow "ls", run with bash' time. It's as if the first one doesn't
>> warm the cache, which is completely absurd - except I've been able to
>> make this happen 5 times in a row, first with strace and then
>> without.
>>
>> # with /usr/bin/time the ls is fast
>> $ time -p ls bad_dir
>> ...
>> real 0.21
>> user 0.00
>> sys 0.00
>>
>>
>> # with the builtin time, right *after* the strace run, the time can be
>> # horrible.
>> $ time -p ls bad_dir
>> ...
>> real 5.60
>> user 0.00
>> sys 0.17
>>
>> # run it again, and the directory is in cache as expected.
>> $ time -p ls bad_dir
>> ...
>> real 0.11
>> user 0.00
>> sys 0.02
>>
>>
>> This is not an artefact of one or other time reporting incorrectly -
>> I'm noticing a long pause before output occurs, but only on the middle
>> test of the three.
>>
>> I can't imagine any sane way for this to be happening, short of
>> coincidence or user error - and I've now seen this sequence 5 times in
>> a row, on 5 different directories created and populated by the same
>> app. (Three times with strace, twice without.)
>>
>>
>> --
>> Arlie
>>
>> (Arlie Stephens                                 arlie at worldash.org)
>>
>> _______________________________________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.org
>> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

I agree with Hugo, seems right to send me the output in a file to read
to see if this actually is a bug with ext4.
Regards Nick