Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Tue, 15 Mar 2011 15:55:52 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=31142
> 
>            Summary: Large write to USB stick freezes unrelated tasks for a
>                     long time
>            Product: IO/Storage
>            Version: 2.5
>     Kernel Version: 2.6.38-rc8
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Block Layer
>         AssignedTo: axboe@kernel.dk
>         ReportedBy: avillaci@ceibo.fiec.espol.edu.ec
>         Regression: No
> 
> 
> Created an attachment (id=50902)
>  --> (https://bugzilla.kernel.org/attachment.cgi?id=50902)
> kernel backtraces from hung Eclipse task while writing to usb stick
> 
> System is Fedora 14 x86_64 with 4 GB RAM, running vanilla kernel 2.6.38-rc8.
> 
> I have a USB 2.0 high-speed memory stick with around 7.5 GB of space. Whenenver
> I write a large amount of data (several GBs of files) through any means (cp,
> nautilus GUI, etc), I notice some large applications that I consider unrelated
> to the I/O operation (Firefox web browser, Thunderbird email viewer, Eclipse
> IDE) may randomly freeze whenever I try to interact with them. I use Compiz,
> and I notice the apps getting grayed out, but I have also seen the freeze
> happening with Metacity and Gnome-shell, so I believe the window manager is
> irrelevant. Sometimes other smaller tasks (gnome-terminal, gedit) also freeze.
> For Eclipse, the hang also cause a series of kernel backtraces, attached to
> this report. The hang usually lasts for several tens of seconds, and may freeze
> and unfreeze several times while the file copying to USB takes place. All of
> the hung applications unfreeze themselves after write activity (as seen from
> the LED in the memory stick) ceases.
> 
> Reproducibility: always (with sufficiently large bulk write)
> To reproduce: 
> 1) have an usb stick with several GB of free space, with any filesystem (tried
> vfat and udf)
> 2) prepare several gb of files to copy from hard disk to usb stick
> 3) start large application (firefox, eclipse, or thunderbird)
> 4) check that application is responsive before file copy starts
> 5) insert usb stick and (auto)mount it. Previously started app is still
> responsive.
> 6) start file copy to usb stick with any command
> 7) attempt to interact with chosen application during the entirety of the file
> write
> Expected result: I/O to usb stick takes place in background, unrelated apps
> continue to be responsive in foreground.
> Actual result: some large tasks freeze for tens of seconds while write takes
> place.
> 
> Feel free to reassign this bug to a different category. It involves I/O, block,
> USB, and mmap.

rofl, will we ever fix this.

Please enable sysrq and do a sysrq-w when the tasks are blocked so we
can find where things are getting stuck.  Please avoid email client
wordwrapping when sending us the sysrq output.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 15/03/11 15:53, Andrew Morton escribio:
>
> rofl, will we ever fix this.
Does this mean there is already a duplicate of this issue? If so, which one?
> Please enable sysrq and do a sysrq-w when the tasks are blocked so we
> can find where things are getting stuck.  Please avoid email client
> wordwrapping when sending us the sysrq output.
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrew Morton]

On Tue, 15 Mar 2011 17:53:16 -0500
Alex Villac____s Lasso <avillaci@fiec.espol.edu.ec> wrote:

> El 15/03/11 15:53, Andrew Morton escribi__:
> >
> > rofl, will we ever fix this.
> Does this mean there is already a duplicate of this issue? If so, which one?

Nothing specific.  Nonsense like this has been happening for at least a
decade and it never seems to get a lot better.

> > Please enable sysrq and do a sysrq-w when the tasks are blocked so we
> > can find where things are getting stuck.  Please avoid email client
> > wordwrapping when sending us the sysrq output.
> >

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 15/03/11 18:19, Andrew Morton escribio:
> On Tue, 15 Mar 2011 17:53:16 -0500
> Alex Villac____s Lasso<avillaci@fiec.espol.edu.ec>  wrote:
>
>> El 15/03/11 15:53, Andrew Morton escribi__:
>>> rofl, will we ever fix this.
>> Does this mean there is already a duplicate of this issue? If so, which one?
> Nothing specific.  Nonsense like this has been happening for at least a
> decade and it never seems to get a lot better.
>
>>> Please enable sysrq and do a sysrq-w when the tasks are blocked so we
>>> can find where things are getting stuck.  Please avoid email client
>>> wordwrapping when sending us the sysrq output.
>>>
Posted sysrq-w report into original bug report to avoid email word-wrap.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrew Morton]

On Wed, 16 Mar 2011 10:25:16 -0500
Alex Villac____s Lasso <avillaci@fiec.espol.edu.ec> wrote:

> El 15/03/11 18:19, Andrew Morton escribi__:
> > On Tue, 15 Mar 2011 17:53:16 -0500
> > Alex Villac____s Lasso<avillaci@fiec.espol.edu.ec>  wrote:
> >
> >> El 15/03/11 15:53, Andrew Morton escribi__:
> >>> rofl, will we ever fix this.
> >> Does this mean there is already a duplicate of this issue? If so, which one?
> > Nothing specific.  Nonsense like this has been happening for at least a
> > decade and it never seems to get a lot better.
> >
> >>> Please enable sysrq and do a sysrq-w when the tasks are blocked so we
> >>> can find where things are getting stuck.  Please avoid email client
> >>> wordwrapping when sending us the sysrq output.
> >>>
> Posted sysrq-w report into original bug report to avoid email word-wrap.

https://bugzilla.kernel.org/attachment.cgi?id=50952

Interesting bits:

[70874.969550] thunderbird-bin D 000000010434e04e     0 32283  32279 0x00000080
[70874.969553]  ffff88011ba91838 0000000000000086 ffff880100000000 0000000000013880
[70874.969557]  0000000000013880 ffff88010a231780 ffff88011ba91fd8 0000000000013880
[70874.969560]  0000000000013880 0000000000013880 0000000000013880 ffff88011ba91fd8
[70874.969564] Call Trace:
[70874.969567]  [<ffffffff810d7ed3>] ? sync_page+0x0/0x4d
[70874.969569]  [<ffffffff810d7ed3>] ? sync_page+0x0/0x4d
[70874.969572]  [<ffffffff8147cedb>] io_schedule+0x47/0x62
[70874.969575]  [<ffffffff810d7f1c>] sync_page+0x49/0x4d
[70874.969577]  [<ffffffff8147d36a>] __wait_on_bit+0x48/0x7b
[70874.969580]  [<ffffffff810d8100>] wait_on_page_bit+0x72/0x79
[70874.969583]  [<ffffffff8106cdb4>] ? wake_bit_function+0x0/0x31
[70874.969586]  [<ffffffff8111833c>] migrate_pages+0x1ac/0x38d
[70874.969589]  [<ffffffff8110d6d7>] ? compaction_alloc+0x0/0x2a4
[70874.969592]  [<ffffffff8110ddfd>] compact_zone+0x3f4/0x60e
[70874.969595]  [<ffffffff8110e1dc>] compact_zone_order+0xc2/0xd1
[70874.969599]  [<ffffffff8110e27f>] try_to_compact_pages+0x94/0xea
[70874.969602]  [<ffffffff810de916>] __alloc_pages_direct_compact+0xa9/0x1a5
[70874.969605]  [<ffffffff810ddbb8>] ? drain_local_pages+0x0/0x17
[70874.969607]  [<ffffffff810df0b0>] __alloc_pages_nodemask+0x69e/0x766
[70874.969610]  [<ffffffff81113701>] ? __slab_free+0x6d/0xf6
[70874.969614]  [<ffffffff8110c0de>] alloc_pages_vma+0xec/0xf1
[70874.969617]  [<ffffffff8111be1c>] do_huge_pmd_anonymous_page+0xbf/0x267
[70874.969620]  [<ffffffff810f2497>] ? pmd_offset+0x19/0x40
[70874.969623]  [<ffffffff810f5c70>] handle_mm_fault+0x15d/0x20f
[70874.969626]  [<ffffffff8100f26a>] ? arch_get_unmapped_area_topdown+0x195/0x28f
[70874.969629]  [<ffffffff8148178c>] do_page_fault+0x33b/0x35d
[70874.969632]  [<ffffffff810fb07d>] ? do_mmap_pgoff+0x29a/0x2f4
[70874.969635]  [<ffffffff8112dcee>] ? path_put+0x22/0x27
[70874.969638]  [<ffffffff8147f145>] page_fault+0x25/0x30


[70874.969731] gedit           D 000000010434dfb0     0 32356      1 0x00000080
[70874.969734]  ffff8800982ab558 0000000000000082 ffff880102400001 0000000000013880
[70874.969737]  0000000000013880 ffff880117408000 ffff8800982abfd8 0000000000013880
[70874.969741]  0000000000013880 0000000000013880 0000000000013880 ffff8800982abfd8
[70874.969744] Call Trace:
[70874.969747]  [<ffffffff8147cedb>] io_schedule+0x47/0x62
[70874.969750]  [<ffffffff8121c403>] get_request_wait+0x10a/0x197
[70874.969753]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
[70874.969756]  [<ffffffff8121ccc4>] __make_request+0x2c8/0x3e0
[70874.969759]  [<ffffffff8111487d>] ? kmem_cache_alloc+0x73/0xeb
[70874.969762]  [<ffffffff8121bb67>] generic_make_request+0x2bc/0x336
[70874.969765]  [<ffffffff811213a5>] ? lookup_page_cgroup+0x36/0x4c
[70874.969768]  [<ffffffff8121bcc1>] submit_bio+0xe0/0xff
[70874.969770]  [<ffffffff8114d72d>] ? bio_alloc_bioset+0x4d/0xc4
[70874.969773]  [<ffffffff810edf1f>] ? inc_zone_page_state+0x2d/0x2f
[70874.969776]  [<ffffffff81149274>] submit_bh+0xe8/0x10e
[70874.969779]  [<ffffffff8114b9fa>] __block_write_full_page+0x1ea/0x2da
[70874.969782]  [<ffffffffa0689202>] ? udf_get_block+0x0/0x115 [udf]
[70874.969785]  [<ffffffff8114a640>] ? end_buffer_async_write+0x0/0x12d
[70874.969788]  [<ffffffff8114a640>] ? end_buffer_async_write+0x0/0x12d
[70874.969791]  [<ffffffffa0689202>] ? udf_get_block+0x0/0x115 [udf]
[70874.969794]  [<ffffffff8114bb76>] block_write_full_page_endio+0x8c/0x98
[70874.969796]  [<ffffffff8114bb97>] block_write_full_page+0x15/0x17
[70874.969800]  [<ffffffffa0686027>] udf_writepage+0x18/0x1a [udf]
[70874.969803]  [<ffffffff81117fed>] move_to_new_page+0x106/0x195
[70874.969806]  [<ffffffff811183de>] migrate_pages+0x24e/0x38d
[70874.969809]  [<ffffffff8110d6d7>] ? compaction_alloc+0x0/0x2a4
[70874.969812]  [<ffffffff8110ddfd>] compact_zone+0x3f4/0x60e
[70874.969815]  [<ffffffff81049c78>] ? load_balance+0xcb/0x6b0
[70874.969818]  [<ffffffff8110e1dc>] compact_zone_order+0xc2/0xd1
[70874.969821]  [<ffffffff8110e27f>] try_to_compact_pages+0x94/0xea
[70874.969824]  [<ffffffff810de916>] __alloc_pages_direct_compact+0xa9/0x1a5
[70874.969827]  [<ffffffff810dee79>] __alloc_pages_nodemask+0x467/0x766
[70874.969830]  [<ffffffff810fcfd3>] ? anon_vma_alloc+0x1a/0x1c
[70874.969833]  [<ffffffff81048a30>] ? get_parent_ip+0x11/0x41
[70874.969833]  [<ffffffff8110c0de>] alloc_pages_vma+0xec/0xf1
[70874.969833]  [<ffffffff8123430e>] ? rb_insert_color+0x66/0xe1
[70874.969833]  [<ffffffff8111be1c>] do_huge_pmd_anonymous_page+0xbf/0x267
[70874.969833]  [<ffffffff810f2497>] ? pmd_offset+0x19/0x40
[70874.969833]  [<ffffffff810f5c70>] handle_mm_fault+0x15d/0x20f
[70874.969833]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
[70874.969833]  [<ffffffff8148178c>] do_page_fault+0x33b/0x35d
[70874.969833]  [<ffffffff810fb07d>] ? do_mmap_pgoff+0x29a/0x2f4
[70874.969833]  [<ffffffff8112dcee>] ? path_put+0x22/0x27
[70874.969833]  [<ffffffff8147f145>] page_fault+0x25/0x30

So it appears that the system is full of dirty pages against a slow
device and your foreground processes have got stuck in direct reclaim
-> compaction -> migration.   That's Mel ;)

What happened to the plans to eliminate direct reclaim?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 16/03/11 17:02, Andrew Morton escribio:
> On Wed, 16 Mar 2011 10:25:16 -0500
> Alex Villac____s Lasso<avillaci@fiec.espol.edu.ec>  wrote:
>
>> El 15/03/11 18:19, Andrew Morton escribi__:
>>> On Tue, 15 Mar 2011 17:53:16 -0500
>>> Alex Villac____s Lasso<avillaci@fiec.espol.edu.ec>   wrote:
>>>
>>>> El 15/03/11 15:53, Andrew Morton escribi__:
>>>>> rofl, will we ever fix this.
>>>> Does this mean there is already a duplicate of this issue? If so, which one?
>>> Nothing specific.  Nonsense like this has been happening for at least a
>>> decade and it never seems to get a lot better.
>>>
>>>>> Please enable sysrq and do a sysrq-w when the tasks are blocked so we
>>>>> can find where things are getting stuck.  Please avoid email client
>>>>> wordwrapping when sending us the sysrq output.
>>>>>
>> Posted sysrq-w report into original bug report to avoid email word-wrap.
> https://bugzilla.kernel.org/attachment.cgi?id=50952
>
> Interesting bits:
>
> [70874.969550] thunderbird-bin D 000000010434e04e     0 32283  32279 0x00000080
> [70874.969553]  ffff88011ba91838 0000000000000086 ffff880100000000 0000000000013880
> [70874.969557]  0000000000013880 ffff88010a231780 ffff88011ba91fd8 0000000000013880
> [70874.969560]  0000000000013880 0000000000013880 0000000000013880 ffff88011ba91fd8
> [70874.969564] Call Trace:
> [70874.969567]  [<ffffffff810d7ed3>] ? sync_page+0x0/0x4d
> [70874.969569]  [<ffffffff810d7ed3>] ? sync_page+0x0/0x4d
> [70874.969572]  [<ffffffff8147cedb>] io_schedule+0x47/0x62
> [70874.969575]  [<ffffffff810d7f1c>] sync_page+0x49/0x4d
> [70874.969577]  [<ffffffff8147d36a>] __wait_on_bit+0x48/0x7b
> [70874.969580]  [<ffffffff810d8100>] wait_on_page_bit+0x72/0x79
> [70874.969583]  [<ffffffff8106cdb4>] ? wake_bit_function+0x0/0x31
> [70874.969586]  [<ffffffff8111833c>] migrate_pages+0x1ac/0x38d
> [70874.969589]  [<ffffffff8110d6d7>] ? compaction_alloc+0x0/0x2a4
> [70874.969592]  [<ffffffff8110ddfd>] compact_zone+0x3f4/0x60e
> [70874.969595]  [<ffffffff8110e1dc>] compact_zone_order+0xc2/0xd1
> [70874.969599]  [<ffffffff8110e27f>] try_to_compact_pages+0x94/0xea
> [70874.969602]  [<ffffffff810de916>] __alloc_pages_direct_compact+0xa9/0x1a5
> [70874.969605]  [<ffffffff810ddbb8>] ? drain_local_pages+0x0/0x17
> [70874.969607]  [<ffffffff810df0b0>] __alloc_pages_nodemask+0x69e/0x766
> [70874.969610]  [<ffffffff81113701>] ? __slab_free+0x6d/0xf6
> [70874.969614]  [<ffffffff8110c0de>] alloc_pages_vma+0xec/0xf1
> [70874.969617]  [<ffffffff8111be1c>] do_huge_pmd_anonymous_page+0xbf/0x267
> [70874.969620]  [<ffffffff810f2497>] ? pmd_offset+0x19/0x40
> [70874.969623]  [<ffffffff810f5c70>] handle_mm_fault+0x15d/0x20f
> [70874.969626]  [<ffffffff8100f26a>] ? arch_get_unmapped_area_topdown+0x195/0x28f
> [70874.969629]  [<ffffffff8148178c>] do_page_fault+0x33b/0x35d
> [70874.969632]  [<ffffffff810fb07d>] ? do_mmap_pgoff+0x29a/0x2f4
> [70874.969635]  [<ffffffff8112dcee>] ? path_put+0x22/0x27
> [70874.969638]  [<ffffffff8147f145>] page_fault+0x25/0x30
>
>
> [70874.969731] gedit           D 000000010434dfb0     0 32356      1 0x00000080
> [70874.969734]  ffff8800982ab558 0000000000000082 ffff880102400001 0000000000013880
> [70874.969737]  0000000000013880 ffff880117408000 ffff8800982abfd8 0000000000013880
> [70874.969741]  0000000000013880 0000000000013880 0000000000013880 ffff8800982abfd8
> [70874.969744] Call Trace:
> [70874.969747]  [<ffffffff8147cedb>] io_schedule+0x47/0x62
> [70874.969750]  [<ffffffff8121c403>] get_request_wait+0x10a/0x197
> [70874.969753]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
> [70874.969756]  [<ffffffff8121ccc4>] __make_request+0x2c8/0x3e0
> [70874.969759]  [<ffffffff8111487d>] ? kmem_cache_alloc+0x73/0xeb
> [70874.969762]  [<ffffffff8121bb67>] generic_make_request+0x2bc/0x336
> [70874.969765]  [<ffffffff811213a5>] ? lookup_page_cgroup+0x36/0x4c
> [70874.969768]  [<ffffffff8121bcc1>] submit_bio+0xe0/0xff
> [70874.969770]  [<ffffffff8114d72d>] ? bio_alloc_bioset+0x4d/0xc4
> [70874.969773]  [<ffffffff810edf1f>] ? inc_zone_page_state+0x2d/0x2f
> [70874.969776]  [<ffffffff81149274>] submit_bh+0xe8/0x10e
> [70874.969779]  [<ffffffff8114b9fa>] __block_write_full_page+0x1ea/0x2da
> [70874.969782]  [<ffffffffa0689202>] ? udf_get_block+0x0/0x115 [udf]
> [70874.969785]  [<ffffffff8114a640>] ? end_buffer_async_write+0x0/0x12d
> [70874.969788]  [<ffffffff8114a640>] ? end_buffer_async_write+0x0/0x12d
> [70874.969791]  [<ffffffffa0689202>] ? udf_get_block+0x0/0x115 [udf]
> [70874.969794]  [<ffffffff8114bb76>] block_write_full_page_endio+0x8c/0x98
> [70874.969796]  [<ffffffff8114bb97>] block_write_full_page+0x15/0x17
> [70874.969800]  [<ffffffffa0686027>] udf_writepage+0x18/0x1a [udf]
> [70874.969803]  [<ffffffff81117fed>] move_to_new_page+0x106/0x195
> [70874.969806]  [<ffffffff811183de>] migrate_pages+0x24e/0x38d
> [70874.969809]  [<ffffffff8110d6d7>] ? compaction_alloc+0x0/0x2a4
> [70874.969812]  [<ffffffff8110ddfd>] compact_zone+0x3f4/0x60e
> [70874.969815]  [<ffffffff81049c78>] ? load_balance+0xcb/0x6b0
> [70874.969818]  [<ffffffff8110e1dc>] compact_zone_order+0xc2/0xd1
> [70874.969821]  [<ffffffff8110e27f>] try_to_compact_pages+0x94/0xea
> [70874.969824]  [<ffffffff810de916>] __alloc_pages_direct_compact+0xa9/0x1a5
> [70874.969827]  [<ffffffff810dee79>] __alloc_pages_nodemask+0x467/0x766
> [70874.969830]  [<ffffffff810fcfd3>] ? anon_vma_alloc+0x1a/0x1c
> [70874.969833]  [<ffffffff81048a30>] ? get_parent_ip+0x11/0x41
> [70874.969833]  [<ffffffff8110c0de>] alloc_pages_vma+0xec/0xf1
> [70874.969833]  [<ffffffff8123430e>] ? rb_insert_color+0x66/0xe1
> [70874.969833]  [<ffffffff8111be1c>] do_huge_pmd_anonymous_page+0xbf/0x267
> [70874.969833]  [<ffffffff810f2497>] ? pmd_offset+0x19/0x40
> [70874.969833]  [<ffffffff810f5c70>] handle_mm_fault+0x15d/0x20f
> [70874.969833]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
> [70874.969833]  [<ffffffff8148178c>] do_page_fault+0x33b/0x35d
> [70874.969833]  [<ffffffff810fb07d>] ? do_mmap_pgoff+0x29a/0x2f4
> [70874.969833]  [<ffffffff8112dcee>] ? path_put+0x22/0x27
> [70874.969833]  [<ffffffff8147f145>] page_fault+0x25/0x30
>
> So it appears that the system is full of dirty pages against a slow
> device and your foreground processes have got stuck in direct reclaim
> ->  compaction ->  migration.   That's Mel ;)
>
> What happened to the plans to eliminate direct reclaim?
>
>
Browsing around bugzilla, I believe that bug 12309 looks very similar to the issue I am experiencing, especially from comment #525 onwards. Am I correct in this?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrew Morton]

On Thu, 17 Mar 2011 16:27:29 -0500
Alex Villac____s Lasso <avillaci@fiec.espol.edu.ec> wrote:

> > So it appears that the system is full of dirty pages against a slow
> > device and your foreground processes have got stuck in direct reclaim
> > ->  compaction ->  migration.   That's Mel ;)
> >
> > What happened to the plans to eliminate direct reclaim?
> >
> >
> Browsing around bugzilla, I believe that bug 12309 looks very similar to the issue I am experiencing, especially from comment #525 onwards. Am I correct in this?

ah, the epic 12309.  https://bugzilla.kernel.org/show_bug.cgi?id=12309.
If you're ever wondering how much we suck, go read that one.

I think what we're seeing in 31142 is a large amount of dirty data
buffered against a slow device.  Innocent processes enter page reclaim
and end up getting stuck trying to write to that heavily-queued and
slow device.

If so, that's probably what some of the 12309 participants are seeing. 
But there are lots of other things in that report too.


Now, the problem you're seeing in 31142 isn't really supposed to
happen.  In the direct-reclaim case the code will try to avoid
initiation of blocking I/O against a congested device, via the
bdi_write_congested() test in may_write_to_queue().  Although that code
now looks a bit busted for the order>PAGE_ALLOC_COSTLY_ORDER case,
whodidthat.

However in the case of the new(ish) compaction/migration code I don't
think we're performing that test.  migrate_pages()->unmap_and_move()
will get stuck behind that large&slow IO queue if page reclaim decided
to pass it down sync==true, as it apparently has done.

IOW, Mel broke it ;)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 17/03/11 16:47, Andrew Morton escribio:
>
> ah, the epic 12309.  https://bugzilla.kernel.org/show_bug.cgi?id=12309.
> If you're ever wondering how much we suck, go read that one.
>
> I think what we're seeing in 31142 is a large amount of dirty data
> buffered against a slow device.  Innocent processes enter page reclaim
> and end up getting stuck trying to write to that heavily-queued and
> slow device.
>
> If so, that's probably what some of the 12309 participants are seeing.
> But there are lots of other things in that report too.
>
>
> Now, the problem you're seeing in 31142 isn't really supposed to
> happen.  In the direct-reclaim case the code will try to avoid
> initiation of blocking I/O against a congested device, via the
> bdi_write_congested() test in may_write_to_queue().  Although that code
> now looks a bit busted for the order>PAGE_ALLOC_COSTLY_ORDER case,
> whodidthat.
>
> However in the case of the new(ish) compaction/migration code I don't
> think we're performing that test.  migrate_pages()->unmap_and_move()
> will get stuck behind that large&slow IO queue if page reclaim decided
> to pass it down sync==true, as it apparently has done.
>
> IOW, Mel broke it ;)
>
I don't quite follow. In my case, the congested device is the USB stick, but the affected processes should be reading/writing on the hard disk. What kind of queue(s) implementation results in pending writes to the USB stick interfering with I/O to the hard 
disk? Or am I misunderstanding? I had the (possibly incorrect) impression that each block device had its own I/O queue.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrew Morton]

On Thu, 17 Mar 2011 17:11:05 -0500
Alex Villac____s Lasso <avillaci@fiec.espol.edu.ec> wrote:

> El 17/03/11 16:47, Andrew Morton escribi__:
> >
> > ah, the epic 12309.  https://bugzilla.kernel.org/show_bug.cgi?id=12309.
> > If you're ever wondering how much we suck, go read that one.
> >
> > I think what we're seeing in 31142 is a large amount of dirty data
> > buffered against a slow device.  Innocent processes enter page reclaim
> > and end up getting stuck trying to write to that heavily-queued and
> > slow device.
> >
> > If so, that's probably what some of the 12309 participants are seeing.
> > But there are lots of other things in that report too.
> >
> >
> > Now, the problem you're seeing in 31142 isn't really supposed to
> > happen.  In the direct-reclaim case the code will try to avoid
> > initiation of blocking I/O against a congested device, via the
> > bdi_write_congested() test in may_write_to_queue().  Although that code
> > now looks a bit busted for the order>PAGE_ALLOC_COSTLY_ORDER case,
> > whodidthat.
> >
> > However in the case of the new(ish) compaction/migration code I don't
> > think we're performing that test.  migrate_pages()->unmap_and_move()
> > will get stuck behind that large&slow IO queue if page reclaim decided
> > to pass it down sync==true, as it apparently has done.
> >
> > IOW, Mel broke it ;)
> >
> I don't quite follow. In my case, the congested device is the USB stick, but the affected processes should be reading/writing on the hard disk. What kind of queue(s) implementation results in pending writes to the USB stick interfering with I/O to the hard 
> disk? Or am I misunderstanding? I had the (possibly incorrect) impression that each block device had its own I/O queue.

Your web browser is just trying to allocate some memory.  As part of
that operation it entered the kernel's page reclaim and while scanning
for memory to free, page reclaim encountered a page which was queued
for IO.  Then page reclaim waited for the IO to complete against that
page.  So the browser got stuck.

Page reclaim normally tries to avoid this situation by not waiting on
such pages, unless the calling processes was itself involved in writing
to the page's device (stored in current->backing_dev_info).  But afaict
the new compaction/migration code forgot to do this.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Mel Gorman]

On Thu, Mar 17, 2011 at 02:47:27PM -0700, Andrew Morton wrote:
> On Thu, 17 Mar 2011 16:27:29 -0500
> Alex Villac____s Lasso <avillaci@fiec.espol.edu.ec> wrote:
> 
> > > So it appears that the system is full of dirty pages against a slow
> > > device and your foreground processes have got stuck in direct reclaim
> > > ->  compaction ->  migration.   That's Mel ;)
> > >
> > > What happened to the plans to eliminate direct reclaim?
> > >
> > >
> > Browsing around bugzilla, I believe that bug 12309 looks very similar to the issue I am experiencing, especially from comment #525 onwards. Am I correct in this?
> 
> ah, the epic 12309.  https://bugzilla.kernel.org/show_bug.cgi?id=12309.
> If you're ever wondering how much we suck, go read that one.
> 

I'm reasonably sure over the last few series that we've taken a number of
steps to mitigate the problems described in #12309 although it's been a while
since I double checked. When I last stopped looking at it, we had reached
the stage where the dirty pages encountered by writeback was greatly reduced
which should have affected the stalls reported in that bug. I stopped working
on it further to see see how the IO-less dirty balancing being worked on by
Wu and Jan worked out because the next reasonable step was making sure the
flusher threads were behaving as expected. That is still a work in progress.

> I think what we're seeing in 31142 is a large amount of dirty data
> buffered against a slow device.  Innocent processes enter page reclaim
> and end up getting stuck trying to write to that heavily-queued and
> slow device.
> 
> If so, that's probably what some of the 12309 participants are seeing. 
> But there are lots of other things in that report too.
> 
> 
> Now, the problem you're seeing in 31142 isn't really supposed to
> happen.  In the direct-reclaim case the code will try to avoid
> initiation of blocking I/O against a congested device, via the
> bdi_write_congested() test in may_write_to_queue().  Although that code
> now looks a bit busted for the order>PAGE_ALLOC_COSTLY_ORDER case,
> whodidthat.
> 
> However in the case of the new(ish) compaction/migration code I don't
> think we're performing that test.  migrate_pages()->unmap_and_move()
> will get stuck behind that large&slow IO queue if page reclaim decided
> to pass it down sync==true, as it apparently has done.
> 
> IOW, Mel broke it ;)
> 

\o/ ... no wait, it's the other one - :(

If you look at the stack traces though, all of them had called
do_huge_pmd_anonymous_page() so while it looks similar to 12309, the trigger
is new because it's THP triggering compaction that is causing the stalls
rather than page reclaim doing direct writeback which was the culprit in
the past.

To confirm if this is the case, I'd be very interested in hearing if this
problem persists in the following cases

1. 2.6.38-rc8 with defrag disabled by
   echo never >/sys/kernel/mm/transparent_hugepage/defrag
   (this will stop THP allocations calling into compaction)
2. 2.6.38-rc8 with THP disabled by
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
   (if the problem still persists, then page reclaim is still a problem
    but we should still stop THP doing sync writes)
3. 2.6.37 vanilla
   (in case this is a new regression introduced since then)

Migration can do sync writes on dirty pages which is why it looks so similar
to page reclaim but this can be controlled by the value of sync_migration
passed into try_to_compact_pages(). If we find that option 1 above makes
the regression go away or at least helps a lot, then a reasonable fix may
be to never set sync_migration if __GFP_NO_KSWAPD which is always set for
THP allocations. I've added Andrea to the cc to see what he thinks.

Thanks for the report.

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

On Fri, Mar 18, 2011 at 11:13:00AM +0000, Mel Gorman wrote:
> To confirm if this is the case, I'd be very interested in hearing if this
> problem persists in the following cases
> 
> 1. 2.6.38-rc8 with defrag disabled by
>    echo never >/sys/kernel/mm/transparent_hugepage/defrag
>    (this will stop THP allocations calling into compaction)
> 2. 2.6.38-rc8 with THP disabled by
>    echo never > /sys/kernel/mm/transparent_hugepage/enabled
>    (if the problem still persists, then page reclaim is still a problem
>     but we should still stop THP doing sync writes)
> 3. 2.6.37 vanilla
>    (in case this is a new regression introduced since then)
> 
> Migration can do sync writes on dirty pages which is why it looks so similar
> to page reclaim but this can be controlled by the value of sync_migration
> passed into try_to_compact_pages(). If we find that option 1 above makes
> the regression go away or at least helps a lot, then a reasonable fix may
> be to never set sync_migration if __GFP_NO_KSWAPD which is always set for
> THP allocations. I've added Andrea to the cc to see what he thinks.

I agree. Forcing sync=0 when __GFP_NO_KSWAPD is set, sounds good to
me, if it is proven to resolve these I/O waits.

Also note that 2.6.38 upstream still misses a couple of important
compaction fixes that are in aa.git (everything relevant is already
queued in -mm but it was a bit late for 2.6.38), so I'd also be
interested to know if you can reproduce in current aa.git
origin/master branch.

If it's a __GFP_NO_KSWAPD allocation (do_huge_pmd_anonymous_page())
that is present in the hanging stack traces, I strongly doubt any of
the changes in aa.git is going to help at all, but it worth a try to
be sure.

http://git.kernel.org/?p=linux/kernel/git/andrea/aa.git;a=shortlog
http://git.kernel.org/?p=linux/kernel/git/andrea/aa.git;a=commit;h=48ad57f498835621d8bad83b972ee6e6c395523a
http://git.kernel.org/?p=linux/kernel/git/andrea/aa.git;a=commit;h=8f6854f7cbf71bc61758bcd92497378e1f677552
http://git.kernel.org/?p=linux/kernel/git/andrea/aa.git;a=commit;h=8ff6d16eb15d2b328bbe715fcaf453b6fedb2cf9
http://git.kernel.org/?p=linux/kernel/git/andrea/aa.git;a=commit;h=e31adb46cd8c4f331cfb02c938e88586d5846bf8

This is the implementation of Mel's idea that you can apply to
upstream or aa.git to see what happens...

===
Subject: compaction: use async migrate for __GFP_NO_KSWAPD

From: Andrea Arcangeli <aarcange@redhat.com>

__GFP_NO_KSWAPD allocations are usually very expensive and not mandatory to
succeed (they have graceful fallback). Waiting for I/O in those, tends to be
overkill in terms of latencies, so we can reduce their latency by disabling
sync migrate.

Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
---

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index bd76256..36d1c79 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2085,7 +2085,7 @@ rebalance:
 					sync_migration);
 	if (page)
 		goto got_pg;
-	sync_migration = true;
+	sync_migration = !(gfp_mask & __GFP_NO_KSWAPD);
 
 	/* Try direct reclaim and then allocating */
 	page = __alloc_pages_direct_reclaim(gfp_mask, order,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 18/03/11 06:13, Mel Gorman escribio:
>
> \o/ ... no wait, it's the other one - :(
>
> If you look at the stack traces though, all of them had called
> do_huge_pmd_anonymous_page() so while it looks similar to 12309, the trigger
> is new because it's THP triggering compaction that is causing the stalls
> rather than page reclaim doing direct writeback which was the culprit in
> the past.
>
> To confirm if this is the case, I'd be very interested in hearing if this
> problem persists in the following cases
>
> 1. 2.6.38-rc8 with defrag disabled by
>     echo never>/sys/kernel/mm/transparent_hugepage/defrag
>     (this will stop THP allocations calling into compaction)
> 2. 2.6.38-rc8 with THP disabled by
>     echo never>
> /sys/kernel/mm/transparent_hugepage/enabled
>     (if the problem still persists, then page reclaim is still a problem
>      but we should still stop THP doing sync writes)
> 3. 2.6.37 vanilla
>     (in case this is a new regression introduced since then)
>
> Migration can do sync writes on dirty pages which is why it looks so similar
> to page reclaim but this can be controlled by the value of sync_migration
> passed into try_to_compact_pages(). If we find that option 1 above makes
> the regression go away or at least helps a lot, then a reasonable fix may
> be to never set sync_migration if __GFP_NO_KSWAPD which is always set for
> THP allocations. I've added Andrea to the cc to see what he thinks.
>
> Thanks for the report.
>
I have just done tests 1 and 2 on 2.6.38 (final, not -rc8), and I have verified that echoing "never" on either /sys/kernel/mm/transparent_hugepage/defrag or /sys/kernel/mm/transparent_hugepage/enabled does allow the file copy to USB to proceed smoothly 
(copying 4GB of data). Just to verify, I later wrote "always" to both files, and sure enough, some applications stalled when I repeated the same file copy. So I have at least a workaround for the issue. Given this evidence, will the patch at comment #14 
fix the issue for good?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Mel Gorman]

On Fri, Mar 18, 2011 at 01:05:15PM -0500, Alex Villac??s Lasso wrote:
> El 18/03/11 06:13, Mel Gorman escribio:
> >
> >\o/ ... no wait, it's the other one - :(
> >
> >If you look at the stack traces though, all of them had called
> >do_huge_pmd_anonymous_page() so while it looks similar to 12309, the trigger
> >is new because it's THP triggering compaction that is causing the stalls
> >rather than page reclaim doing direct writeback which was the culprit in
> >the past.
> >
> >To confirm if this is the case, I'd be very interested in hearing if this
> >problem persists in the following cases
> >
> >1. 2.6.38-rc8 with defrag disabled by
> >    echo never>/sys/kernel/mm/transparent_hugepage/defrag
> >    (this will stop THP allocations calling into compaction)
> >2. 2.6.38-rc8 with THP disabled by
> >    echo never>
> >/sys/kernel/mm/transparent_hugepage/enabled
> >    (if the problem still persists, then page reclaim is still a problem
> >     but we should still stop THP doing sync writes)
> >3. 2.6.37 vanilla
> >    (in case this is a new regression introduced since then)
> >
> >Migration can do sync writes on dirty pages which is why it looks so similar
> >to page reclaim but this can be controlled by the value of sync_migration
> >passed into try_to_compact_pages(). If we find that option 1 above makes
> >the regression go away or at least helps a lot, then a reasonable fix may
> >be to never set sync_migration if __GFP_NO_KSWAPD which is always set for
> >THP allocations. I've added Andrea to the cc to see what he thinks.
> >
> >Thanks for the report.
> >
>
> I have just done tests 1 and 2 on 2.6.38 (final, not -rc8), and I
> have verified that echoing "never" on either
> /sys/kernel/mm/transparent_hugepage/defrag or
> /sys/kernel/mm/transparent_hugepage/enabled does allow the file copy
> to USB to proceed smoothly (copying 4GB of data). Just to verify, I
> later wrote "always" to both files, and sure enough, some
> applications stalled when I repeated the same file copy. So I have
> at least a workaround for the issue. Given this evidence, will the
> patch at comment #14 fix the issue for good?
> 

Thanks for testing and reporting, it's very helpful. Based on that that
report the patch should help. Can you test it to be absolutly sure please?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 19/03/11 08:46, Mel Gorman escribio:
> On Fri, Mar 18, 2011 at 01:05:15PM -0500, Alex Villac??s Lasso wrote:
>> El 18/03/11 06:13, Mel Gorman escribio:
>>> \o/ ... no wait, it's the other one - :(
>>>
>>> If you look at the stack traces though, all of them had called
>>> do_huge_pmd_anonymous_page() so while it looks similar to 12309, the trigger
>>> is new because it's THP triggering compaction that is causing the stalls
>>> rather than page reclaim doing direct writeback which was the culprit in
>>> the past.
>>>
>>> To confirm if this is the case, I'd be very interested in hearing if this
>>> problem persists in the following cases
>>>
>>> 1. 2.6.38-rc8 with defrag disabled by
>>>     echo never>/sys/kernel/mm/transparent_hugepage/defrag
>>>     (this will stop THP allocations calling into compaction)
>>> 2. 2.6.38-rc8 with THP disabled by
>>>     echo never>
>>> /sys/kernel/mm/transparent_hugepage/enabled
>>>     (if the problem still persists, then page reclaim is still a problem
>>>      but we should still stop THP doing sync writes)
>>> 3. 2.6.37 vanilla
>>>     (in case this is a new regression introduced since then)
>>>
>>> Migration can do sync writes on dirty pages which is why it looks so similar
>>> to page reclaim but this can be controlled by the value of sync_migration
>>> passed into try_to_compact_pages(). If we find that option 1 above makes
>>> the regression go away or at least helps a lot, then a reasonable fix may
>>> be to never set sync_migration if __GFP_NO_KSWAPD which is always set for
>>> THP allocations. I've added Andrea to the cc to see what he thinks.
>>>
>>> Thanks for the report.
>>>
>> I have just done tests 1 and 2 on 2.6.38 (final, not -rc8), and I
>> have verified that echoing "never" on either
>> /sys/kernel/mm/transparent_hugepage/defrag or
>> /sys/kernel/mm/transparent_hugepage/enabled does allow the file copy
>> to USB to proceed smoothly (copying 4GB of data). Just to verify, I
>> later wrote "always" to both files, and sure enough, some
>> applications stalled when I repeated the same file copy. So I have
>> at least a workaround for the issue. Given this evidence, will the
>> patch at comment #14 fix the issue for good?
>>
> Thanks for testing and reporting, it's very helpful. Based on that that
> report the patch should help. Can you test it to be absolutly sure please?
>
>
The patch did not help. I have attached a sysrq-w trace with the patch applied in the bug report.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

On Sat, Mar 19, 2011 at 11:04:02AM -0500, Alex Villaci-s Lasso wrote:
> The patch did not help. I have attached a sysrq-w trace with the patch applied in the bug report.

Most processes are stuck in udf_writepage. That's because migrate is
calling ->writepage on dirty pages even when sync=0.

This may do better, can you test it in replacement of the previous
patch?

Thanks,
Andrea

===
Subject: compaction: use async migrate for __GFP_NO_KSWAPD

From: Andrea Arcangeli <aarcange@redhat.com>

__GFP_NO_KSWAPD allocations are usually very expensive and not mandatory to
succeed (they have graceful fallback). Waiting for I/O in those, tends to be
overkill in terms of latencies, so we can reduce their latency by disabling
sync migrate.

Stop calling ->writepage on dirty cache when migrate sync mode is not set.

Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
---
 mm/migrate.c    |   35 ++++++++++++++++++++++++++---------
 mm/page_alloc.c |    2 +-
 2 files changed, 27 insertions(+), 10 deletions(-)

--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2085,7 +2085,7 @@ rebalance:
 					sync_migration);
 	if (page)
 		goto got_pg;
-	sync_migration = true;
+	sync_migration = !(gfp_mask & __GFP_NO_KSWAPD);
 
 	/* Try direct reclaim and then allocating */
 	page = __alloc_pages_direct_reclaim(gfp_mask, order,
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -536,10 +536,15 @@ static int writeout(struct address_space
  * Default handling if a filesystem does not provide a migration function.
  */
 static int fallback_migrate_page(struct address_space *mapping,
-	struct page *newpage, struct page *page)
+				 struct page *newpage, struct page *page,
+				 int sync)
 {
-	if (PageDirty(page))
-		return writeout(mapping, page);
+	if (PageDirty(page)) {
+		if (sync)
+			return writeout(mapping, page);
+		else
+			return -EBUSY;
+	}
 
 	/*
 	 * Buffers may be managed in a filesystem specific way.
@@ -564,7 +569,7 @@ static int fallback_migrate_page(struct 
  *  == 0 - success
  */
 static int move_to_new_page(struct page *newpage, struct page *page,
-						int remap_swapcache)
+			    int remap_swapcache, int sync)
 {
 	struct address_space *mapping;
 	int rc;
@@ -597,7 +602,7 @@ static int move_to_new_page(struct page 
 		rc = mapping->a_ops->migratepage(mapping,
 						newpage, page);
 	else
-		rc = fallback_migrate_page(mapping, newpage, page);
+		rc = fallback_migrate_page(mapping, newpage, page, sync);
 
 	if (rc) {
 		newpage->mapping = NULL;
@@ -641,6 +646,10 @@ static int unmap_and_move(new_page_t get
 	rc = -EAGAIN;
 
 	if (!trylock_page(page)) {
+		if (!sync) {
+			rc = -EBUSY;
+			goto move_newpage;
+		}
 		if (!force)
 			goto move_newpage;
 
@@ -686,7 +695,11 @@ static int unmap_and_move(new_page_t get
 	BUG_ON(charge);
 
 	if (PageWriteback(page)) {
-		if (!force || !sync)
+		if (!sync) {
+			rc = -EBUSY;
+			goto uncharge;
+		}
+		if (!force)
 			goto uncharge;
 		wait_on_page_writeback(page);
 	}
@@ -757,7 +770,7 @@ static int unmap_and_move(new_page_t get
 
 skip_unmap:
 	if (!page_mapped(page))
-		rc = move_to_new_page(newpage, page, remap_swapcache);
+		rc = move_to_new_page(newpage, page, remap_swapcache, sync);
 
 	if (rc && remap_swapcache)
 		remove_migration_ptes(page, page);
@@ -834,7 +847,11 @@ static int unmap_and_move_huge_page(new_
 	rc = -EAGAIN;
 
 	if (!trylock_page(hpage)) {
-		if (!force || !sync)
+		if (!sync) {
+			rc = -EBUSY;
+			goto out;
+		}
+		if (!force)
 			goto out;
 		lock_page(hpage);
 	}
@@ -850,7 +867,7 @@ static int unmap_and_move_huge_page(new_
 	try_to_unmap(hpage, TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);
 
 	if (!page_mapped(hpage))
-		rc = move_to_new_page(new_hpage, hpage, 1);
+		rc = move_to_new_page(new_hpage, hpage, 1, sync);
 
 	if (rc)
 		remove_migration_ptes(hpage, hpage);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Mel Gorman]

On Sun, Mar 20, 2011 at 12:51:44AM +0100, Andrea Arcangeli wrote:
> On Sat, Mar 19, 2011 at 11:04:02AM -0500, Alex Villaci-s Lasso wrote:
> > The patch did not help. I have attached a sysrq-w trace with the patch applied in the bug report.
> 
> Most processes are stuck in udf_writepage. That's because migrate is
> calling ->writepage on dirty pages even when sync=0.
> 
> This may do better, can you test it in replacement of the previous
> patch?
> 
> ===
> Subject: compaction: use async migrate for __GFP_NO_KSWAPD
> 
> From: Andrea Arcangeli <aarcange@redhat.com>
> 
> __GFP_NO_KSWAPD allocations are usually very expensive and not mandatory to
> succeed (they have graceful fallback). Waiting for I/O in those, tends to be
> overkill in terms of latencies, so we can reduce their latency by disabling
> sync migrate.
> 
> Stop calling ->writepage on dirty cache when migrate sync mode is not set.
> 
> Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
> ---
>  mm/migrate.c    |   35 ++++++++++++++++++++++++++---------
>  mm/page_alloc.c |    2 +-
>  2 files changed, 27 insertions(+), 10 deletions(-)
> 
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2085,7 +2085,7 @@ rebalance:
>  					sync_migration);
>  	if (page)
>  		goto got_pg;
> -	sync_migration = true;
> +	sync_migration = !(gfp_mask & __GFP_NO_KSWAPD);
>  
>  	/* Try direct reclaim and then allocating */
>  	page = __alloc_pages_direct_reclaim(gfp_mask, order,
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -536,10 +536,15 @@ static int writeout(struct address_space
>   * Default handling if a filesystem does not provide a migration function.
>   */
>  static int fallback_migrate_page(struct address_space *mapping,
> -	struct page *newpage, struct page *page)
> +				 struct page *newpage, struct page *page,
> +				 int sync)
>  {
> -	if (PageDirty(page))
> -		return writeout(mapping, page);
> +	if (PageDirty(page)) {
> +		if (sync)
> +			return writeout(mapping, page);
> +		else
> +			return -EBUSY;
> +	}
>  
>  	/*
>  	 * Buffers may be managed in a filesystem specific way.

The check is at the wrong level I believe because it misses NFS pages which
will still get queued for IO which can block waiting on a request to complete.

> @@ -564,7 +569,7 @@ static int fallback_migrate_page(struct 
>   *  == 0 - success
>   */
>  static int move_to_new_page(struct page *newpage, struct page *page,
> -						int remap_swapcache)
> +			    int remap_swapcache, int sync)

sync should be bool.

>  {
>  	struct address_space *mapping;
>  	int rc;
> @@ -597,7 +602,7 @@ static int move_to_new_page(struct page 
>  		rc = mapping->a_ops->migratepage(mapping,
>  						newpage, page);
>  	else
> -		rc = fallback_migrate_page(mapping, newpage, page);
> +		rc = fallback_migrate_page(mapping, newpage, page, sync);
>  
>  	if (rc) {
>  		newpage->mapping = NULL;
> @@ -641,6 +646,10 @@ static int unmap_and_move(new_page_t get
>  	rc = -EAGAIN;
>  
>  	if (!trylock_page(page)) {
> +		if (!sync) {
> +			rc = -EBUSY;
> +			goto move_newpage;
> +		}

It's overkill to return EBUSY just because we failed to get a lock which could
be released very quickly. If we left rc as -EAGAIN it would retry again.
The worst case scenario is that the current process is the holder of the
lock and the loop is pointless but this is a relatively rare situation
(other than Hugh's loopback test aside which seems to be particularly good
at triggering that situation).

>  		if (!force)
>  			goto move_newpage;
>  
> @@ -686,7 +695,11 @@ static int unmap_and_move(new_page_t get
>  	BUG_ON(charge);
>  
>  	if (PageWriteback(page)) {
> -		if (!force || !sync)
> +		if (!sync) {
> +			rc = -EBUSY;
> +			goto uncharge;
> +		}
> +		if (!force)
>  			goto uncharge;

Where as this is ok because if the page is being written back, it's fairly
unlikely it'll get cleared quickly enough for the retry loop to make sense.

>  		wait_on_page_writeback(page);
>  	}
> @@ -757,7 +770,7 @@ static int unmap_and_move(new_page_t get
>  
>  skip_unmap:
>  	if (!page_mapped(page))
> -		rc = move_to_new_page(newpage, page, remap_swapcache);
> +		rc = move_to_new_page(newpage, page, remap_swapcache, sync);
>  
>  	if (rc && remap_swapcache)
>  		remove_migration_ptes(page, page);
> @@ -834,7 +847,11 @@ static int unmap_and_move_huge_page(new_
>  	rc = -EAGAIN;
>  
>  	if (!trylock_page(hpage)) {
> -		if (!force || !sync)
> +		if (!sync) {
> +			rc = -EBUSY;
> +			goto out;
> +		}
> +		if (!force)
>  			goto out;
>  		lock_page(hpage);
>  	}

As before, it's worth retrying to get the lock as it could be released
very shortly.

> @@ -850,7 +867,7 @@ static int unmap_and_move_huge_page(new_
>  	try_to_unmap(hpage, TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);
>  
>  	if (!page_mapped(hpage))
> -		rc = move_to_new_page(new_hpage, hpage, 1);
> +		rc = move_to_new_page(new_hpage, hpage, 1, sync);
>  
>  	if (rc)
>  		remove_migration_ptes(hpage, hpage);
> 

Because of the NFS pages and being a bit aggressive about using -EBUSY,
how about the following instead? (build tested only unfortunately)

==== CUT HERE ====
mm: compaction: Use async migration for __GFP_NO_KSWAPD and enforce no writeback

From: Andrea Arcangeli <aarcange@redhat.com>

__GFP_NO_KSWAPD allocations are usually very expensive and not mandatory
to succeed as they have graceful fallback. Waiting for I/O in those, tends
to be overkill in terms of latencies, so we can reduce their latency by
disabling sync migrate.

Unfortunately, even with async migration it's still possible for the
process to be blocked waiting for a request slot (e.g. get_request_wait
in the block layer) when ->writepage is called. To prevent __GFP_NO_KSWAPD
blocking, this patch prevents ->writepage being called on dirty page cache
for asynchronous migration.

[mel@csn.ul.ie: Avoid writebacks for NFS, retry locked pages, use bool]
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
---
 mm/migrate.c    |   47 ++++++++++++++++++++++++++++++-----------------
 mm/page_alloc.c |    2 +-
 2 files changed, 31 insertions(+), 18 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index 352de555..1b45508 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -564,7 +564,7 @@ static int fallback_migrate_page(struct address_space *mapping,
  *  == 0 - success
  */
 static int move_to_new_page(struct page *newpage, struct page *page,
-						int remap_swapcache)
+					int remap_swapcache, bool sync)
 {
 	struct address_space *mapping;
 	int rc;
@@ -586,18 +586,23 @@ static int move_to_new_page(struct page *newpage, struct page *page,
 	mapping = page_mapping(page);
 	if (!mapping)
 		rc = migrate_page(mapping, newpage, page);
-	else if (mapping->a_ops->migratepage)
-		/*
-		 * Most pages have a mapping and most filesystems
-		 * should provide a migration function. Anonymous
-		 * pages are part of swap space which also has its
-		 * own migration function. This is the most common
-		 * path for page migration.
-		 */
-		rc = mapping->a_ops->migratepage(mapping,
-						newpage, page);
-	else
-		rc = fallback_migrate_page(mapping, newpage, page);
+	else {
+		/* Do not writeback pages if !sync */
+		if (PageDirty(page) && !sync)
+			rc = -EBUSY;
+		else if (mapping->a_ops->migratepage)
+			/*
+		 	* Most pages have a mapping and most filesystems
+		 	* should provide a migration function. Anonymous
+		 	* pages are part of swap space which also has its
+		 	* own migration function. This is the most common
+		 	* path for page migration.
+		 	*/
+			rc = mapping->a_ops->migratepage(mapping,
+							newpage, page);
+		else
+			rc = fallback_migrate_page(mapping, newpage, page);
+	}
 
 	if (rc) {
 		newpage->mapping = NULL;
@@ -641,7 +646,7 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
 	rc = -EAGAIN;
 
 	if (!trylock_page(page)) {
-		if (!force)
+		if (!force || !sync)
 			goto move_newpage;
 
 		/*
@@ -686,7 +691,15 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
 	BUG_ON(charge);
 
 	if (PageWriteback(page)) {
-		if (!force || !sync)
+		/*
+		 * For !sync, there is no point retrying as the retry loop
+		 * is expected to be too short for PageWriteback to be cleared
+		 */
+		if (!sync) {
+			rc = -EBUSY;
+			goto uncharge;
+		}
+		if (!force)
 			goto uncharge;
 		wait_on_page_writeback(page);
 	}
@@ -757,7 +770,7 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
 
 skip_unmap:
 	if (!page_mapped(page))
-		rc = move_to_new_page(newpage, page, remap_swapcache);
+		rc = move_to_new_page(newpage, page, remap_swapcache, sync);
 
 	if (rc && remap_swapcache)
 		remove_migration_ptes(page, page);
@@ -850,7 +863,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,
 	try_to_unmap(hpage, TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);
 
 	if (!page_mapped(hpage))
-		rc = move_to_new_page(new_hpage, hpage, 1);
+		rc = move_to_new_page(new_hpage, hpage, 1, sync);
 
 	if (rc)
 		remove_migration_ptes(hpage, hpage);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index cdef1d4..ce6d601 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2085,7 +2085,7 @@ rebalance:
 					sync_migration);
 	if (page)
 		goto got_pg;
-	sync_migration = true;
+	sync_migration = !(gfp_mask & __GFP_NO_KSWAPD);
 
 	/* Try direct reclaim and then allocating */
 	page = __alloc_pages_direct_reclaim(gfp_mask, order,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

On Mon, Mar 21, 2011 at 09:41:49AM +0000, Mel Gorman wrote:
> The check is at the wrong level I believe because it misses NFS pages which
> will still get queued for IO which can block waiting on a request to complete.

But for example ->migratepage won't block at all for swapcache... it's
just a pointer for migrate_page... so I didnt' want to skip what could
be nonblocking, it just makes migrate less reliable for no good in
some case. The fallback case is very likely blocking instead so I only
returned -EBUSY there.

Best would be to pass a sync/nonblock param to migratepage(nonblock)
so that nfs_migrate_page can pass "nonblock" instead of "false" to
nfs_find_and_lock_request.

> sync should be bool.

That's better thanks.

> It's overkill to return EBUSY just because we failed to get a lock which could
> be released very quickly. If we left rc as -EAGAIN it would retry again.
> The worst case scenario is that the current process is the holder of the
> lock and the loop is pointless but this is a relatively rare situation
> (other than Hugh's loopback test aside which seems to be particularly good
> at triggering that situation).

This change was only meant to possibly avoid some cpu waste in the
tight loop, not really "blocking" related so I'm sure ok to drop it
for now. The page lock holder better to be quick because with sync=0
the tight loop will retry real fast. If the holder blocks we're not so
smart at retrying in a tight loop but for now it's ok.

> >  			goto move_newpage;
> >  
> > @@ -686,7 +695,11 @@ static int unmap_and_move(new_page_t get
> >  	BUG_ON(charge);
> >  
> >  	if (PageWriteback(page)) {
> > -		if (!force || !sync)
> > +		if (!sync) {
> > +			rc = -EBUSY;
> > +			goto uncharge;
> > +		}
> > +		if (!force)
> >  			goto uncharge;
> 
> Where as this is ok because if the page is being written back, it's fairly
> unlikely it'll get cleared quickly enough for the retry loop to make sense.

Agreed.

> Because of the NFS pages and being a bit aggressive about using -EBUSY,
> how about the following instead? (build tested only unfortunately)

I tested my version below but I think one needs udf with lots of dirty
pages plus the usb to trigger this which I don't have setup
immediately.

> @@ -586,18 +586,23 @@ static int move_to_new_page(struct page *newpage, struct page *page,
>  	mapping = page_mapping(page);
>  	if (!mapping)
>  		rc = migrate_page(mapping, newpage, page);
> -	else if (mapping->a_ops->migratepage)
> -		/*
> -		 * Most pages have a mapping and most filesystems
> -		 * should provide a migration function. Anonymous
> -		 * pages are part of swap space which also has its
> -		 * own migration function. This is the most common
> -		 * path for page migration.
> -		 */
> -		rc = mapping->a_ops->migratepage(mapping,
> -						newpage, page);
> -	else
> -		rc = fallback_migrate_page(mapping, newpage, page);
> +	else {
> +		/* Do not writeback pages if !sync */
> +		if (PageDirty(page) && !sync)
> +			rc = -EBUSY;

I think it's better to at least change it to:

if (PageDirty(page) && !sync && mapping->a_ops->migratepage != migrate_page))

I wasn't sure how to handle noblocking ->migratepage for swapcache and
tmpfs but probably the above check is a good enough approximation.

Before sending my patch I thought of adding a "sync" parameter to
->migratepage(..., sync/nonblock) but then the patch become
bigger... and I just wanted to know if this was the problem or not so
I deferred it.

If we're sure that all migratepage blocks except for things like
swapcache/tmpfs or other not-filebacked things that defines it to
migrate_page, we're pretty well covered by adding a check like above
migratepage == migrate_page and maybe we don't need to add a
"sync/nonblock" parameter to ->migratepage(). For example the
buffer_migrate_page can block too in lock_buffer.

This is the patch I'm trying with the addition of the above check and
some comment space/tab issue cleanup.

===
Subject: mm: compaction: Use async migration for __GFP_NO_KSWAPD and enforce no writeback

From: Andrea Arcangeli <aarcange@redhat.com>

__GFP_NO_KSWAPD allocations are usually very expensive and not mandatory
to succeed as they have graceful fallback. Waiting for I/O in those, tends
to be overkill in terms of latencies, so we can reduce their latency by
disabling sync migrate.

Unfortunately, even with async migration it's still possible for the
process to be blocked waiting for a request slot (e.g. get_request_wait
in the block layer) when ->writepage is called. To prevent __GFP_NO_KSWAPD
blocking, this patch prevents ->writepage being called on dirty page cache
for asynchronous migration.

[mel@csn.ul.ie: Avoid writebacks for NFS, retry locked pages, use bool]
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
---
 mm/migrate.c    |   48 +++++++++++++++++++++++++++++++++---------------
 mm/page_alloc.c |    2 +-
 2 files changed, 34 insertions(+), 16 deletions(-)

--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -564,7 +564,7 @@ static int fallback_migrate_page(struct 
  *  == 0 - success
  */
 static int move_to_new_page(struct page *newpage, struct page *page,
-						int remap_swapcache)
+					int remap_swapcache, bool sync)
 {
 	struct address_space *mapping;
 	int rc;
@@ -586,18 +586,28 @@ static int move_to_new_page(struct page 
 	mapping = page_mapping(page);
 	if (!mapping)
 		rc = migrate_page(mapping, newpage, page);
-	else if (mapping->a_ops->migratepage)
+	else {
 		/*
-		 * Most pages have a mapping and most filesystems
-		 * should provide a migration function. Anonymous
-		 * pages are part of swap space which also has its
-		 * own migration function. This is the most common
-		 * path for page migration.
+		 * Do not writeback pages if !sync and migratepage is
+		 * not pointing to migrate_page() which is nonblocking
+		 * (swapcache/tmpfs uses migratepage = migrate_page).
 		 */
-		rc = mapping->a_ops->migratepage(mapping,
-						newpage, page);
-	else
-		rc = fallback_migrate_page(mapping, newpage, page);
+		if (PageDirty(page) && !sync &&
+		    mapping->a_ops->migratepage != migrate_page)
+			rc = -EBUSY;
+		else if (mapping->a_ops->migratepage)
+			/*
+			 * Most pages have a mapping and most filesystems
+			 * should provide a migration function. Anonymous
+			 * pages are part of swap space which also has its
+			 * own migration function. This is the most common
+			 * path for page migration.
+			 */
+			rc = mapping->a_ops->migratepage(mapping,
+							newpage, page);
+		else
+			rc = fallback_migrate_page(mapping, newpage, page);
+	}
 
 	if (rc) {
 		newpage->mapping = NULL;
@@ -641,7 +651,7 @@ static int unmap_and_move(new_page_t get
 	rc = -EAGAIN;
 
 	if (!trylock_page(page)) {
-		if (!force)
+		if (!force || !sync)
 			goto move_newpage;
 
 		/*
@@ -686,7 +696,15 @@ static int unmap_and_move(new_page_t get
 	BUG_ON(charge);
 
 	if (PageWriteback(page)) {
-		if (!force || !sync)
+		/*
+		 * For !sync, there is no point retrying as the retry loop
+		 * is expected to be too short for PageWriteback to be cleared
+		 */
+		if (!sync) {
+			rc = -EBUSY;
+			goto uncharge;
+		}
+		if (!force)
 			goto uncharge;
 		wait_on_page_writeback(page);
 	}
@@ -757,7 +775,7 @@ static int unmap_and_move(new_page_t get
 
 skip_unmap:
 	if (!page_mapped(page))
-		rc = move_to_new_page(newpage, page, remap_swapcache);
+		rc = move_to_new_page(newpage, page, remap_swapcache, sync);
 
 	if (rc && remap_swapcache)
 		remove_migration_ptes(page, page);
@@ -850,7 +868,7 @@ static int unmap_and_move_huge_page(new_
 	try_to_unmap(hpage, TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);
 
 	if (!page_mapped(hpage))
-		rc = move_to_new_page(new_hpage, hpage, 1);
+		rc = move_to_new_page(new_hpage, hpage, 1, sync);
 
 	if (rc)
 		remove_migration_ptes(hpage, hpage);
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2085,7 +2085,7 @@ rebalance:
 					sync_migration);
 	if (page)
 		goto got_pg;
-	sync_migration = true;
+	sync_migration = !(gfp_mask & __GFP_NO_KSWAPD);
 
 	/* Try direct reclaim and then allocating */
 	page = __alloc_pages_direct_reclaim(gfp_mask, order,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 21/03/11 08:48, Andrea Arcangeli escribio:
>
> ===
> Subject: mm: compaction: Use async migration for __GFP_NO_KSWAPD and enforce no writeback
>
> From: Andrea Arcangeli<aarcange@redhat.com>
>
> __GFP_NO_KSWAPD allocations are usually very expensive and not mandatory
> to succeed as they have graceful fallback. Waiting for I/O in those, tends
> to be overkill in terms of latencies, so we can reduce their latency by
> disabling sync migrate.
>
> Unfortunately, even with async migration it's still possible for the
> process to be blocked waiting for a request slot (e.g. get_request_wait
> in the block layer) when ->writepage is called. To prevent __GFP_NO_KSWAPD
> blocking, this patch prevents ->writepage being called on dirty page cache
> for asynchronous migration.
>
> [mel@csn.ul.ie: Avoid writebacks for NFS, retry locked pages, use bool]
> Signed-off-by: Andrea Arcangeli<aarcange@redhat.com>
> Signed-off-by: Mel Gorman<mel@csn.ul.ie>
> ---
>   mm/migrate.c    |   48 +++++++++++++++++++++++++++++++++---------------
>   mm/page_alloc.c |    2 +-
>   2 files changed, 34 insertions(+), 16 deletions(-)
The latest patch fails to apply in vanilla 2.6.38:

[alex@srv64 linux-2.6.38]$ patch -p1 --dry-run < ../\[Bug\ 31142\]\ Large\ write\ to\ USB\ stick\ freezes\ unrelated\ tasks\ for\ a\ long\ time.eml
(Stripping trailing CRs from patch.)
patching file mm/migrate.c
Hunk #1 FAILED at 564.
Hunk #2 FAILED at 586.
Hunk #3 FAILED at 641.
Hunk #4 FAILED at 686.
Hunk #5 FAILED at 757.
Hunk #6 FAILED at 850.
6 out of 6 hunks FAILED -- saving rejects to file mm/migrate.c.rej
(Stripping trailing CRs from patch.)
patching file mm/page_alloc.c
Hunk #1 FAILED at 2085.
1 out of 1 hunk FAILED -- saving rejects to file mm/page_alloc.c.rej

I will try to apply the patch manually.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 21/03/11 10:22, Alex Villaci-s Lasso escribio:
> El 21/03/11 08:48, Andrea Arcangeli escribio:
>>
>> ===
>> Subject: mm: compaction: Use async migration for __GFP_NO_KSWAPD and enforce no writeback
>>
>> From: Andrea Arcangeli<aarcange@redhat.com>
>>
>> __GFP_NO_KSWAPD allocations are usually very expensive and not mandatory
>> to succeed as they have graceful fallback. Waiting for I/O in those, tends
>> to be overkill in terms of latencies, so we can reduce their latency by
>> disabling sync migrate.
>>
>> Unfortunately, even with async migration it's still possible for the
>> process to be blocked waiting for a request slot (e.g. get_request_wait
>> in the block layer) when ->writepage is called. To prevent __GFP_NO_KSWAPD
>> blocking, this patch prevents ->writepage being called on dirty page cache
>> for asynchronous migration.
>>
>> [mel@csn.ul.ie: Avoid writebacks for NFS, retry locked pages, use bool]
>> Signed-off-by: Andrea Arcangeli<aarcange@redhat.com>
>> Signed-off-by: Mel Gorman<mel@csn.ul.ie>
>> ---
>>   mm/migrate.c    |   48 +++++++++++++++++++++++++++++++++---------------
>>   mm/page_alloc.c |    2 +-
>>   2 files changed, 34 insertions(+), 16 deletions(-)
> The latest patch fails to apply in vanilla 2.6.38:
>
> [alex@srv64 linux-2.6.38]$ patch -p1 --dry-run < ../\[Bug\ 31142\]\ Large\ write\ to\ USB\ stick\ freezes\ unrelated\ tasks\ for\ a\ long\ time.eml
> (Stripping trailing CRs from patch.)
> patching file mm/migrate.c
> Hunk #1 FAILED at 564.
> Hunk #2 FAILED at 586.
> Hunk #3 FAILED at 641.
> Hunk #4 FAILED at 686.
> Hunk #5 FAILED at 757.
> Hunk #6 FAILED at 850.
> 6 out of 6 hunks FAILED -- saving rejects to file mm/migrate.c.rej
> (Stripping trailing CRs from patch.)
> patching file mm/page_alloc.c
> Hunk #1 FAILED at 2085.
> 1 out of 1 hunk FAILED -- saving rejects to file mm/page_alloc.c.rej
>
> I will try to apply the patch manually.
>
After some massaging, I succeeded in applying the patch. Compiling now.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

On Mon, Mar 21, 2011 at 10:22:43AM -0500, Alex Villaci-s Lasso wrote:
> I will try to apply the patch manually.

Hmm, I checked that this latest version below applies clean both
against vanilla 2.6.38 and current current git. It should apply clean
if you run:

     "git fetch; git checkout -f origin/master"

or "git checkout -f v2.6.38"

I got this and another migrate patch from Hugh both applied in aa.git
so you may use that one too if you want:

     "git clone --reference linux-2.6 git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git"

(with --reference it should clone very fast, linux-2.6 must be a clone
of the upstream linux-2.6.git tree)

Thanks,
Andrea

===
Subject: mm: compaction: Use async migration for __GFP_NO_KSWAPD and enforce no writeback

From: Andrea Arcangeli <aarcange@redhat.com>

__GFP_NO_KSWAPD allocations are usually very expensive and not mandatory
to succeed as they have graceful fallback. Waiting for I/O in those, tends
to be overkill in terms of latencies, so we can reduce their latency by
disabling sync migrate.

Unfortunately, even with async migration it's still possible for the
process to be blocked waiting for a request slot (e.g. get_request_wait
in the block layer) when ->writepage is called. To prevent __GFP_NO_KSWAPD
blocking, this patch prevents ->writepage being called on dirty page cache
for asynchronous migration.

[mel@csn.ul.ie: Avoid writebacks for NFS, retry locked pages, use bool]
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
---
 mm/migrate.c    |   48 +++++++++++++++++++++++++++++++++---------------
 mm/page_alloc.c |    2 +-
 2 files changed, 34 insertions(+), 16 deletions(-)

--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -564,7 +564,7 @@ static int fallback_migrate_page(struct 
  *  == 0 - success
  */
 static int move_to_new_page(struct page *newpage, struct page *page,
-						int remap_swapcache)
+					int remap_swapcache, bool sync)
 {
 	struct address_space *mapping;
 	int rc;
@@ -586,18 +586,28 @@ static int move_to_new_page(struct page 
 	mapping = page_mapping(page);
 	if (!mapping)
 		rc = migrate_page(mapping, newpage, page);
-	else if (mapping->a_ops->migratepage)
+	else {
 		/*
-		 * Most pages have a mapping and most filesystems
-		 * should provide a migration function. Anonymous
-		 * pages are part of swap space which also has its
-		 * own migration function. This is the most common
-		 * path for page migration.
+		 * Do not writeback pages if !sync and migratepage is
+		 * not pointing to migrate_page() which is nonblocking
+		 * (swapcache/tmpfs uses migratepage = migrate_page).
 		 */
-		rc = mapping->a_ops->migratepage(mapping,
-						newpage, page);
-	else
-		rc = fallback_migrate_page(mapping, newpage, page);
+		if (PageDirty(page) && !sync &&
+		    mapping->a_ops->migratepage != migrate_page)
+			rc = -EBUSY;
+		else if (mapping->a_ops->migratepage)
+			/*
+			 * Most pages have a mapping and most filesystems
+			 * should provide a migration function. Anonymous
+			 * pages are part of swap space which also has its
+			 * own migration function. This is the most common
+			 * path for page migration.
+			 */
+			rc = mapping->a_ops->migratepage(mapping,
+							newpage, page);
+		else
+			rc = fallback_migrate_page(mapping, newpage, page);
+	}
 
 	if (rc) {
 		newpage->mapping = NULL;
@@ -641,7 +651,7 @@ static int unmap_and_move(new_page_t get
 	rc = -EAGAIN;
 
 	if (!trylock_page(page)) {
-		if (!force)
+		if (!force || !sync)
 			goto move_newpage;
 
 		/*
@@ -686,7 +696,15 @@ static int unmap_and_move(new_page_t get
 	BUG_ON(charge);
 
 	if (PageWriteback(page)) {
-		if (!force || !sync)
+		/*
+		 * For !sync, there is no point retrying as the retry loop
+		 * is expected to be too short for PageWriteback to be cleared
+		 */
+		if (!sync) {
+			rc = -EBUSY;
+			goto uncharge;
+		}
+		if (!force)
 			goto uncharge;
 		wait_on_page_writeback(page);
 	}
@@ -757,7 +775,7 @@ static int unmap_and_move(new_page_t get
 
 skip_unmap:
 	if (!page_mapped(page))
-		rc = move_to_new_page(newpage, page, remap_swapcache);
+		rc = move_to_new_page(newpage, page, remap_swapcache, sync);
 
 	if (rc && remap_swapcache)
 		remove_migration_ptes(page, page);
@@ -850,7 +868,7 @@ static int unmap_and_move_huge_page(new_
 	try_to_unmap(hpage, TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);
 
 	if (!page_mapped(hpage))
-		rc = move_to_new_page(new_hpage, hpage, 1);
+		rc = move_to_new_page(new_hpage, hpage, 1, sync);
 
 	if (rc)
 		remove_migration_ptes(hpage, hpage);
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2085,7 +2085,7 @@ rebalance:
 					sync_migration);
 	if (page)
 		goto got_pg;
-	sync_migration = true;
+	sync_migration = !(gfp_mask & __GFP_NO_KSWAPD);
 
 	/* Try direct reclaim and then allocating */
 	page = __alloc_pages_direct_reclaim(gfp_mask, order,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Mel Gorman]

On Mon, Mar 21, 2011 at 02:48:32PM +0100, Andrea Arcangeli wrote:
> On Mon, Mar 21, 2011 at 09:41:49AM +0000, Mel Gorman wrote:
> > The check is at the wrong level I believe because it misses NFS pages which
> > will still get queued for IO which can block waiting on a request to complete.
> 
> But for example ->migratepage won't block at all for swapcache... it's
> just a pointer for migrate_page... so I didnt' want to skip what could
> be nonblocking, it just makes migrate less reliable for no good in
> some case. The fallback case is very likely blocking instead so I only
> returned -EBUSY there.
> 

Fair point.

> Best would be to pass a sync/nonblock param to migratepage(nonblock)
> so that nfs_migrate_page can pass "nonblock" instead of "false" to
> nfs_find_and_lock_request.
> 

I had considered this but thought passing in sync to things like
migrate_page() that ignored it looked a little ugly.

> > sync should be bool.
> 
> That's better thanks.
> 
> > It's overkill to return EBUSY just because we failed to get a lock which could
> > be released very quickly. If we left rc as -EAGAIN it would retry again.
> > The worst case scenario is that the current process is the holder of the
> > lock and the loop is pointless but this is a relatively rare situation
> > (other than Hugh's loopback test aside which seems to be particularly good
> > at triggering that situation).
> 
> This change was only meant to possibly avoid some cpu waste in the
> tight loop, not really "blocking" related so I'm sure ok to drop it
> for now. The page lock holder better to be quick because with sync=0
> the tight loop will retry real fast. If the holder blocks we're not so
> smart at retrying in a tight loop but for now it's ok.
> 

Ok.

> > >  			goto move_newpage;
> > >  
> > > @@ -686,7 +695,11 @@ static int unmap_and_move(new_page_t get
> > >  	BUG_ON(charge);
> > >  
> > >  	if (PageWriteback(page)) {
> > > -		if (!force || !sync)
> > > +		if (!sync) {
> > > +			rc = -EBUSY;
> > > +			goto uncharge;
> > > +		}
> > > +		if (!force)
> > >  			goto uncharge;
> > 
> > Where as this is ok because if the page is being written back, it's fairly
> > unlikely it'll get cleared quickly enough for the retry loop to make sense.
> 
> Agreed.
> 
> > Because of the NFS pages and being a bit aggressive about using -EBUSY,
> > how about the following instead? (build tested only unfortunately)
> 
> I tested my version below but I think one needs udf with lots of dirty
> pages plus the usb to trigger this which I don't have setup
> immediately.
> 
> > @@ -586,18 +586,23 @@ static int move_to_new_page(struct page *newpage, struct page *page,
> >  	mapping = page_mapping(page);
> >  	if (!mapping)
> >  		rc = migrate_page(mapping, newpage, page);
> > -	else if (mapping->a_ops->migratepage)
> > -		/*
> > -		 * Most pages have a mapping and most filesystems
> > -		 * should provide a migration function. Anonymous
> > -		 * pages are part of swap space which also has its
> > -		 * own migration function. This is the most common
> > -		 * path for page migration.
> > -		 */
> > -		rc = mapping->a_ops->migratepage(mapping,
> > -						newpage, page);
> > -	else
> > -		rc = fallback_migrate_page(mapping, newpage, page);
> > +	else {
> > +		/* Do not writeback pages if !sync */
> > +		if (PageDirty(page) && !sync)
> > +			rc = -EBUSY;
> 
> I think it's better to at least change it to:
> 
> if (PageDirty(page) && !sync && mapping->a_ops->migratepage != migrate_page))
> 
> I wasn't sure how to handle noblocking ->migratepage for swapcache and
> tmpfs but probably the above check is a good enough approximation.
> 

It's a good enough approximation. It's a little ugly but I don't think
it's much uglier than passing in unused parameters to migrate_page().

> Before sending my patch I thought of adding a "sync" parameter to
> ->migratepage(..., sync/nonblock) but then the patch become
> bigger... and I just wanted to know if this was the problem or not so
> I deferred it.
> 

I deferred it for similar reasons. It was becoming a much larger change
than should be necessary for the fix.

> If we're sure that all migratepage blocks except for things like
> swapcache/tmpfs or other not-filebacked things that defines it to
> migrate_page, we're pretty well covered by adding a check like above
> migratepage == migrate_page and maybe we don't need to add a
> "sync/nonblock" parameter to ->migratepage(). For example the
> buffer_migrate_page can block too in lock_buffer.
> 

Agreed.

> This is the patch I'm trying with the addition of the above check and
> some comment space/tab issue cleanup.
> 

Nothing bad jumped out at me. Lets see how it gets on with testing.

Thanks

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 21/03/11 11:37, Mel Gorman escribio:
> On Mon, Mar 21, 2011 at 02:48:32PM +0100, Andrea Arcangeli wrote:
>
> Nothing bad jumped out at me. Lets see how it gets on with testing.
>
> Thanks
>
As with the previous patch, this one did not completely solve the freezing tasks issue. However, as with the previous patch, the freezes took longer to appear, and now lasted less (10 to 12 seconds instead of freezing until the end of the usb copy).

I have attached the new sysrq-w trace to the bug report.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

On Mon, Mar 21, 2011 at 12:05:40PM -0500, Alex Villaci-s Lasso wrote:
> El 21/03/11 11:37, Mel Gorman escribio:
> > On Mon, Mar 21, 2011 at 02:48:32PM +0100, Andrea Arcangeli wrote:
> >
> > Nothing bad jumped out at me. Lets see how it gets on with testing.
> >
> > Thanks
> >
> As with the previous patch, this one did not completely solve the freezing tasks issue. However, as with the previous patch, the freezes took longer to appear, and now lasted less (10 to 12 seconds instead of freezing until the end of the usb copy).
> 
> I have attached the new sysrq-w trace to the bug report.

migrate and compaction disappeared from the traces as we hoped
for. The THP allocations left throttles on writeback during reclaim
like any 4k allocation would do:

[ 2629.256809]  [<ffffffff810e43c3>] wait_on_page_writeback+0x1b/0x1d
[ 2629.256812]  [<ffffffff810e5992>] shrink_page_list+0x134/0x478
[ 2629.256815]  [<ffffffff810e614f>] shrink_inactive_list+0x29f/0x39a
[ 2629.256818]  [<ffffffff810dbd55>] ? zone_watermark_ok+0x1f/0x21
[ 2629.256820]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
[ 2629.256823]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
[ 2629.256827]  [<ffffffff810e6c87>] do_try_to_free_pages+0xdd/0x2e3
[ 2629.256830]  [<ffffffff810e70eb>] try_to_free_pages+0xaa/0xef
[ 2629.256833]  [<ffffffff810deede>] __alloc_pages_nodemask+0x4cc/0x772
[ 2629.256837]  [<ffffffff8110c0ea>] alloc_pages_vma+0xec/0xf1
[ 2629.256840]  [<ffffffff8111be94>] do_huge_pmd_anonymous_page+0xbf/0x267
[ 2629.256844]  [<ffffffff810f24a3>] ? pmd_offset+0x19/0x40
[ 2629.256846]  [<ffffffff810f5c7c>] handle_mm_fault+0x15d/0x20f
[ 2629.256850]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
[ 2629.256853]  [<ffffffff814818cc>] do_page_fault+0x33b/0x35d
[ 2629.256856]  [<ffffffff810fb089>] ? do_mmap_pgoff+0x29a/0x2f4
[ 2629.256859]  [<ffffffff8112dd66>] ? path_put+0x22/0x27
[ 2629.256861]  [<ffffffff8147f285>] page_fault+0x25/0x30

They throttle on writeback I/O completion like kswapd too:

[ 2849.098751]  [<ffffffff8147d00b>] io_schedule+0x47/0x62
[ 2849.098756]  [<ffffffff8121c47b>] get_request_wait+0x10a/0x197
[ 2849.098760]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
[ 2849.098763]  [<ffffffff8121cd3c>] __make_request+0x2c8/0x3e0
[ 2849.098767]  [<ffffffff81114889>] ? kmem_cache_alloc+0x73/0xeb
[ 2849.098771]  [<ffffffff8121bbdf>] generic_make_request+0x2bc/0x336
[ 2849.098774]  [<ffffffff8121bd39>] submit_bio+0xe0/0xff
[ 2849.098777]  [<ffffffff8114d7a5>] ? bio_alloc_bioset+0x4d/0xc4
[ 2849.098781]  [<ffffffff810edf2b>] ? inc_zone_page_state+0x2d/0x2f
[ 2849.098785]  [<ffffffff811492ec>] submit_bh+0xe8/0x10e
[ 2849.098788]  [<ffffffff8114ba72>] __block_write_full_page+0x1ea/0x2da
[ 2849.098793]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
[ 2849.098796]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
[ 2849.098799]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
[ 2849.098802]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
[ 2849.098805]  [<ffffffff8114bbee>] block_write_full_page_endio+0x8c/0x98
[ 2849.098808]  [<ffffffff8114bc0f>] block_write_full_page+0x15/0x17
[ 2849.098811]  [<ffffffffa06e2027>] udf_writepage+0x18/0x1a [udf]
[ 2849.098814]  [<ffffffff810e44fd>] pageout+0x138/0x255
[ 2849.098817]  [<ffffffff810e5ad7>] shrink_page_list+0x279/0x478
[ 2849.098820]  [<ffffffff810e60ec>] shrink_inactive_list+0x23c/0x39a
[ 2849.098824]  [<ffffffff81481a46>] ? add_preempt_count+0xae/0xb2
[ 2849.098828]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
[ 2849.098831]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
[ 2849.098834]  [<ffffffff810dbdf8>] ? zone_watermark_ok_safe+0xa1/0xae
[ 2849.098837]  [<ffffffff810e773f>] kswapd+0x51c/0x89f

I'm unsure if there's any other problem left that can be attributed to
compaction/migrate (especially considering the THP allocations have no
__GFP_REPEAT set and should_continue_reclaim should break the loop if
nr_reclaim is zero, plus compaction_suitable requires not much more
memory to be reclaimed if compared to no-compaction).

I'd suggest to try a few more times with "echo never >
/sys/kernel/mm/transparent_hugepage/enabled" and see if it still makes
a difference.

I doubt udf is as optimal as other fs in optimizing locks and
writebacks but that may not be relevant to this issue (or at least it
most certainly wasn't until now and the same we had so far probably
applied to all fs, from this point things aren't as clear as before if
responsiveness is still worse than without THP). Anther thing that I
find confusing is the udf vs USB stick, I'd expect udf on a usb dvd
not a usbstick flash drive, I'd use vfat or ext4 on a flash drive.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 21/03/11 15:16, Andrea Arcangeli escribio:
> On Mon, Mar 21, 2011 at 12:05:40PM -0500, Alex Villaci-s Lasso wrote:
>> El 21/03/11 11:37, Mel Gorman escribio:
>>> On Mon, Mar 21, 2011 at 02:48:32PM +0100, Andrea Arcangeli wrote:
>>>
>>> Nothing bad jumped out at me. Lets see how it gets on with testing.
>>>
>>> Thanks
>>>
>> As with the previous patch, this one did not completely solve the freezing tasks issue. However, as with the previous patch, the freezes took longer to appear, and now lasted less (10 to 12 seconds instead of freezing until the end of the usb copy).
>>
>> I have attached the new sysrq-w trace to the bug report.
> migrate and compaction disappeared from the traces as we hoped
> for. The THP allocations left throttles on writeback during reclaim
> like any 4k allocation would do:
>
> [ 2629.256809]  [<ffffffff810e43c3>] wait_on_page_writeback+0x1b/0x1d
> [ 2629.256812]  [<ffffffff810e5992>] shrink_page_list+0x134/0x478
> [ 2629.256815]  [<ffffffff810e614f>] shrink_inactive_list+0x29f/0x39a
> [ 2629.256818]  [<ffffffff810dbd55>] ? zone_watermark_ok+0x1f/0x21
> [ 2629.256820]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
> [ 2629.256823]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
> [ 2629.256827]  [<ffffffff810e6c87>] do_try_to_free_pages+0xdd/0x2e3
> [ 2629.256830]  [<ffffffff810e70eb>] try_to_free_pages+0xaa/0xef
> [ 2629.256833]  [<ffffffff810deede>] __alloc_pages_nodemask+0x4cc/0x772
> [ 2629.256837]  [<ffffffff8110c0ea>] alloc_pages_vma+0xec/0xf1
> [ 2629.256840]  [<ffffffff8111be94>] do_huge_pmd_anonymous_page+0xbf/0x267
> [ 2629.256844]  [<ffffffff810f24a3>] ? pmd_offset+0x19/0x40
> [ 2629.256846]  [<ffffffff810f5c7c>] handle_mm_fault+0x15d/0x20f
> [ 2629.256850]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
> [ 2629.256853]  [<ffffffff814818cc>] do_page_fault+0x33b/0x35d
> [ 2629.256856]  [<ffffffff810fb089>] ? do_mmap_pgoff+0x29a/0x2f4
> [ 2629.256859]  [<ffffffff8112dd66>] ? path_put+0x22/0x27
> [ 2629.256861]  [<ffffffff8147f285>] page_fault+0x25/0x30
>
> They throttle on writeback I/O completion like kswapd too:
>
> [ 2849.098751]  [<ffffffff8147d00b>] io_schedule+0x47/0x62
> [ 2849.098756]  [<ffffffff8121c47b>] get_request_wait+0x10a/0x197
> [ 2849.098760]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
> [ 2849.098763]  [<ffffffff8121cd3c>] __make_request+0x2c8/0x3e0
> [ 2849.098767]  [<ffffffff81114889>] ? kmem_cache_alloc+0x73/0xeb
> [ 2849.098771]  [<ffffffff8121bbdf>] generic_make_request+0x2bc/0x336
> [ 2849.098774]  [<ffffffff8121bd39>] submit_bio+0xe0/0xff
> [ 2849.098777]  [<ffffffff8114d7a5>] ? bio_alloc_bioset+0x4d/0xc4
> [ 2849.098781]  [<ffffffff810edf2b>] ? inc_zone_page_state+0x2d/0x2f
> [ 2849.098785]  [<ffffffff811492ec>] submit_bh+0xe8/0x10e
> [ 2849.098788]  [<ffffffff8114ba72>] __block_write_full_page+0x1ea/0x2da
> [ 2849.098793]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
> [ 2849.098796]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
> [ 2849.098799]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
> [ 2849.098802]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
> [ 2849.098805]  [<ffffffff8114bbee>] block_write_full_page_endio+0x8c/0x98
> [ 2849.098808]  [<ffffffff8114bc0f>] block_write_full_page+0x15/0x17
> [ 2849.098811]  [<ffffffffa06e2027>] udf_writepage+0x18/0x1a [udf]
> [ 2849.098814]  [<ffffffff810e44fd>] pageout+0x138/0x255
> [ 2849.098817]  [<ffffffff810e5ad7>] shrink_page_list+0x279/0x478
> [ 2849.098820]  [<ffffffff810e60ec>] shrink_inactive_list+0x23c/0x39a
> [ 2849.098824]  [<ffffffff81481a46>] ? add_preempt_count+0xae/0xb2
> [ 2849.098828]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
> [ 2849.098831]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
> [ 2849.098834]  [<ffffffff810dbdf8>] ? zone_watermark_ok_safe+0xa1/0xae
> [ 2849.098837]  [<ffffffff810e773f>] kswapd+0x51c/0x89f
>
> I'm unsure if there's any other problem left that can be attributed to
> compaction/migrate (especially considering the THP allocations have no
> __GFP_REPEAT set and should_continue_reclaim should break the loop if
> nr_reclaim is zero, plus compaction_suitable requires not much more
> memory to be reclaimed if compared to no-compaction).
>
> I'd suggest to try a few more times with "echo never>
> /sys/kernel/mm/transparent_hugepage/enabled" and see if it still makes
> a difference.
>
> I doubt udf is as optimal as other fs in optimizing locks and
> writebacks but that may not be relevant to this issue (or at least it
> most certainly wasn't until now and the same we had so far probably
> applied to all fs, from this point things aren't as clear as before if
> responsiveness is still worse than without THP). Anther thing that I
> find confusing is the udf vs USB stick, I'd expect udf on a usb dvd
> not a usbstick flash drive, I'd use vfat or ext4 on a flash drive.
>
By echoing "never" to /sys/kernel/mm/transparent_hugepage/enabled , the copy still goes smoothly without any application freezes, even with the recent patch applied.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Mel Gorman]

On Mon, Mar 21, 2011 at 09:16:41PM +0100, Andrea Arcangeli wrote:
> On Mon, Mar 21, 2011 at 12:05:40PM -0500, Alex Villaci-s Lasso wrote:
> > El 21/03/11 11:37, Mel Gorman escribio:
> > > On Mon, Mar 21, 2011 at 02:48:32PM +0100, Andrea Arcangeli wrote:
> > >
> > > Nothing bad jumped out at me. Lets see how it gets on with testing.
> > >
> > > Thanks
> > >
> > As with the previous patch, this one did not completely solve the freezing tasks issue. However, as with the previous patch, the freezes took longer to appear, and now lasted less (10 to 12 seconds instead of freezing until the end of the usb copy).
> > 
> > I have attached the new sysrq-w trace to the bug report.
> 
> migrate and compaction disappeared from the traces as we hoped
> for. The THP allocations left throttles on writeback during reclaim
> like any 4k allocation would do:
> 
> [ 2629.256809]  [<ffffffff810e43c3>] wait_on_page_writeback+0x1b/0x1d
> [ 2629.256812]  [<ffffffff810e5992>] shrink_page_list+0x134/0x478
> [ 2629.256815]  [<ffffffff810e614f>] shrink_inactive_list+0x29f/0x39a
> [ 2629.256818]  [<ffffffff810dbd55>] ? zone_watermark_ok+0x1f/0x21
> [ 2629.256820]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
> [ 2629.256823]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
> [ 2629.256827]  [<ffffffff810e6c87>] do_try_to_free_pages+0xdd/0x2e3
> [ 2629.256830]  [<ffffffff810e70eb>] try_to_free_pages+0xaa/0xef
> [ 2629.256833]  [<ffffffff810deede>] __alloc_pages_nodemask+0x4cc/0x772
> [ 2629.256837]  [<ffffffff8110c0ea>] alloc_pages_vma+0xec/0xf1
> [ 2629.256840]  [<ffffffff8111be94>] do_huge_pmd_anonymous_page+0xbf/0x267
> [ 2629.256844]  [<ffffffff810f24a3>] ? pmd_offset+0x19/0x40
> [ 2629.256846]  [<ffffffff810f5c7c>] handle_mm_fault+0x15d/0x20f
> [ 2629.256850]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
> [ 2629.256853]  [<ffffffff814818cc>] do_page_fault+0x33b/0x35d
> [ 2629.256856]  [<ffffffff810fb089>] ? do_mmap_pgoff+0x29a/0x2f4
> [ 2629.256859]  [<ffffffff8112dd66>] ? path_put+0x22/0x27
> [ 2629.256861]  [<ffffffff8147f285>] page_fault+0x25/0x30
> 

There is an important difference between THP and generic order-0 reclaim
though. Once defrag is enabled in THP, it can enter direct reclaim for
reclaim/compaction where more pages may be claimed than for a base page
fault thereby encountering more dirty pages and stalling.

> They throttle on writeback I/O completion like kswapd too:
> 
> [ 2849.098751]  [<ffffffff8147d00b>] io_schedule+0x47/0x62
> [ 2849.098756]  [<ffffffff8121c47b>] get_request_wait+0x10a/0x197
> [ 2849.098760]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
> [ 2849.098763]  [<ffffffff8121cd3c>] __make_request+0x2c8/0x3e0
> [ 2849.098767]  [<ffffffff81114889>] ? kmem_cache_alloc+0x73/0xeb
> [ 2849.098771]  [<ffffffff8121bbdf>] generic_make_request+0x2bc/0x336
> [ 2849.098774]  [<ffffffff8121bd39>] submit_bio+0xe0/0xff
> [ 2849.098777]  [<ffffffff8114d7a5>] ? bio_alloc_bioset+0x4d/0xc4
> [ 2849.098781]  [<ffffffff810edf2b>] ? inc_zone_page_state+0x2d/0x2f
> [ 2849.098785]  [<ffffffff811492ec>] submit_bh+0xe8/0x10e
> [ 2849.098788]  [<ffffffff8114ba72>] __block_write_full_page+0x1ea/0x2da
> [ 2849.098793]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
> [ 2849.098796]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
> [ 2849.098799]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
> [ 2849.098802]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
> [ 2849.098805]  [<ffffffff8114bbee>] block_write_full_page_endio+0x8c/0x98
> [ 2849.098808]  [<ffffffff8114bc0f>] block_write_full_page+0x15/0x17
> [ 2849.098811]  [<ffffffffa06e2027>] udf_writepage+0x18/0x1a [udf]
> [ 2849.098814]  [<ffffffff810e44fd>] pageout+0x138/0x255
> [ 2849.098817]  [<ffffffff810e5ad7>] shrink_page_list+0x279/0x478
> [ 2849.098820]  [<ffffffff810e60ec>] shrink_inactive_list+0x23c/0x39a
> [ 2849.098824]  [<ffffffff81481a46>] ? add_preempt_count+0xae/0xb2
> [ 2849.098828]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
> [ 2849.098831]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
> [ 2849.098834]  [<ffffffff810dbdf8>] ? zone_watermark_ok_safe+0xa1/0xae
> [ 2849.098837]  [<ffffffff810e773f>] kswapd+0x51c/0x89f
> 
> I'm unsure if there's any other problem left that can be attributed to
> compaction/migrate (especially considering the THP allocations have no
> __GFP_REPEAT set and should_continue_reclaim should break the loop if
> nr_reclaim is zero, plus compaction_suitable requires not much more
> memory to be reclaimed if compared to no-compaction).
> 

I think we are breaking out because the report says the stalls aren't as
bad but not before we have waited on writeback of a few dirty pages. This
could be addressed in a number of ways but all of them impact THP in some way.

1. We could disable defrag by default. This will avoid the stalling at
   the cost of fewer pages being promoted even when plenty of clean pages
   were available.

2. We could redefine __GFP_NO_KSWAPD as __GFP_ASYNC to mean a) do not
   wake up kswapd that generates IO possibly causing syncs later b) does
   not queue any pages for IO itself and c) never waits on page writeback.
   This would also avoid stalls but it would disrupt LRU ordering by
   reclaiming younger pages than would otherwise have been reclaimed.

3. Again redefine __GFP_NO_KSWAPD but abort allocation if any dirty or
   writeback page is encountered during reclaim. This makes the assumption
   that dirty pages at the end of the LRU implies memory is under enough
   pressure to not care about promotion. This will also result in THP
   promoting fewer pages but has less impact on LRU ordering.

Which would you prefer? Other suggestions?

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

On Tue, Mar 22, 2011 at 11:20:32AM +0000, Mel Gorman wrote:
> On Mon, Mar 21, 2011 at 09:16:41PM +0100, Andrea Arcangeli wrote:
> > On Mon, Mar 21, 2011 at 12:05:40PM -0500, Alex Villaci-s Lasso wrote:
> > > El 21/03/11 11:37, Mel Gorman escribio:
> > > > On Mon, Mar 21, 2011 at 02:48:32PM +0100, Andrea Arcangeli wrote:
> > > >
> > > > Nothing bad jumped out at me. Lets see how it gets on with testing.
> > > >
> > > > Thanks
> > > >
> > > As with the previous patch, this one did not completely solve the freezing tasks issue. However, as with the previous patch, the freezes took longer to appear, and now lasted less (10 to 12 seconds instead of freezing until the end of the usb copy).
> > > 
> > > I have attached the new sysrq-w trace to the bug report.
> > 
> > migrate and compaction disappeared from the traces as we hoped
> > for. The THP allocations left throttles on writeback during reclaim
> > like any 4k allocation would do:
> > 
> > [ 2629.256809]  [<ffffffff810e43c3>] wait_on_page_writeback+0x1b/0x1d
> > [ 2629.256812]  [<ffffffff810e5992>] shrink_page_list+0x134/0x478
> > [ 2629.256815]  [<ffffffff810e614f>] shrink_inactive_list+0x29f/0x39a
> > [ 2629.256818]  [<ffffffff810dbd55>] ? zone_watermark_ok+0x1f/0x21
> > [ 2629.256820]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
> > [ 2629.256823]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
> > [ 2629.256827]  [<ffffffff810e6c87>] do_try_to_free_pages+0xdd/0x2e3
> > [ 2629.256830]  [<ffffffff810e70eb>] try_to_free_pages+0xaa/0xef
> > [ 2629.256833]  [<ffffffff810deede>] __alloc_pages_nodemask+0x4cc/0x772
> > [ 2629.256837]  [<ffffffff8110c0ea>] alloc_pages_vma+0xec/0xf1
> > [ 2629.256840]  [<ffffffff8111be94>] do_huge_pmd_anonymous_page+0xbf/0x267
> > [ 2629.256844]  [<ffffffff810f24a3>] ? pmd_offset+0x19/0x40
> > [ 2629.256846]  [<ffffffff810f5c7c>] handle_mm_fault+0x15d/0x20f
> > [ 2629.256850]  [<ffffffff8100f298>] ? arch_get_unmapped_area_topdown+0x1c3/0x28f
> > [ 2629.256853]  [<ffffffff814818cc>] do_page_fault+0x33b/0x35d
> > [ 2629.256856]  [<ffffffff810fb089>] ? do_mmap_pgoff+0x29a/0x2f4
> > [ 2629.256859]  [<ffffffff8112dd66>] ? path_put+0x22/0x27
> > [ 2629.256861]  [<ffffffff8147f285>] page_fault+0x25/0x30
> > 
> 
> There is an important difference between THP and generic order-0 reclaim
> though. Once defrag is enabled in THP, it can enter direct reclaim for
> reclaim/compaction where more pages may be claimed than for a base page
> fault thereby encountering more dirty pages and stalling.

Ok but then 2M gets allocated. With 4k pages it's true there's less
work done for each 4k page but reclaim runs several more times to
allocate 512 4k pages instead of a single 2M page. So I'm unsure if
that should create a noticeable difference.

> > They throttle on writeback I/O completion like kswapd too:
> > 
> > [ 2849.098751]  [<ffffffff8147d00b>] io_schedule+0x47/0x62
> > [ 2849.098756]  [<ffffffff8121c47b>] get_request_wait+0x10a/0x197
> > [ 2849.098760]  [<ffffffff8106cd77>] ? autoremove_wake_function+0x0/0x3d
> > [ 2849.098763]  [<ffffffff8121cd3c>] __make_request+0x2c8/0x3e0
> > [ 2849.098767]  [<ffffffff81114889>] ? kmem_cache_alloc+0x73/0xeb
> > [ 2849.098771]  [<ffffffff8121bbdf>] generic_make_request+0x2bc/0x336
> > [ 2849.098774]  [<ffffffff8121bd39>] submit_bio+0xe0/0xff
> > [ 2849.098777]  [<ffffffff8114d7a5>] ? bio_alloc_bioset+0x4d/0xc4
> > [ 2849.098781]  [<ffffffff810edf2b>] ? inc_zone_page_state+0x2d/0x2f
> > [ 2849.098785]  [<ffffffff811492ec>] submit_bh+0xe8/0x10e
> > [ 2849.098788]  [<ffffffff8114ba72>] __block_write_full_page+0x1ea/0x2da
> > [ 2849.098793]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
> > [ 2849.098796]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
> > [ 2849.098799]  [<ffffffff8114a6b8>] ? end_buffer_async_write+0x0/0x12d
> > [ 2849.098802]  [<ffffffffa06e5202>] ? udf_get_block+0x0/0x115 [udf]
> > [ 2849.098805]  [<ffffffff8114bbee>] block_write_full_page_endio+0x8c/0x98
> > [ 2849.098808]  [<ffffffff8114bc0f>] block_write_full_page+0x15/0x17
> > [ 2849.098811]  [<ffffffffa06e2027>] udf_writepage+0x18/0x1a [udf]
> > [ 2849.098814]  [<ffffffff810e44fd>] pageout+0x138/0x255
> > [ 2849.098817]  [<ffffffff810e5ad7>] shrink_page_list+0x279/0x478
> > [ 2849.098820]  [<ffffffff810e60ec>] shrink_inactive_list+0x23c/0x39a
> > [ 2849.098824]  [<ffffffff81481a46>] ? add_preempt_count+0xae/0xb2
> > [ 2849.098828]  [<ffffffff810dfe81>] ? determine_dirtyable_memory+0x1d/0x27
> > [ 2849.098831]  [<ffffffff810e6849>] shrink_zone+0x362/0x464
> > [ 2849.098834]  [<ffffffff810dbdf8>] ? zone_watermark_ok_safe+0xa1/0xae
> > [ 2849.098837]  [<ffffffff810e773f>] kswapd+0x51c/0x89f
> > 
> > I'm unsure if there's any other problem left that can be attributed to
> > compaction/migrate (especially considering the THP allocations have no
> > __GFP_REPEAT set and should_continue_reclaim should break the loop if
> > nr_reclaim is zero, plus compaction_suitable requires not much more
> > memory to be reclaimed if compared to no-compaction).
> > 
> 
> I think we are breaking out because the report says the stalls aren't as
> bad but not before we have waited on writeback of a few dirty pages. This
> could be addressed in a number of ways but all of them impact THP in some way.
> 
> 1. We could disable defrag by default. This will avoid the stalling at
>    the cost of fewer pages being promoted even when plenty of clean pages
>    were available.
> 
> 2. We could redefine __GFP_NO_KSWAPD as __GFP_ASYNC to mean a) do not
>    wake up kswapd that generates IO possibly causing syncs later b) does
>    not queue any pages for IO itself and c) never waits on page writeback.
>    This would also avoid stalls but it would disrupt LRU ordering by
>    reclaiming younger pages than would otherwise have been reclaimed.
> 
> 3. Again redefine __GFP_NO_KSWAPD but abort allocation if any dirty or
>    writeback page is encountered during reclaim. This makes the assumption
>    that dirty pages at the end of the LRU implies memory is under enough
>    pressure to not care about promotion. This will also result in THP
>    promoting fewer pages but has less impact on LRU ordering.
> 
> Which would you prefer? Other suggestions?

I'm not particularly excited by any of the above options first because
it'd decreases the reliability of allocations. And more important
because it won't solve anything for the no-THP related allocations
that would still create the same problem considering that compaction
is the issue as he verified that setting defrag=no solves the problem
(think SLUB that even for a radix tree allocation uses order 2 first,
just like THP tries order 9 before order 0, slub or the kernel stack
allocation aren't using anything like __GFP_ASYNC). So I'm afraid
we're more likely to hide the issue by tackling it entirely on the THP
side.

I asked yesterday by PM to Alex if the mouse pointer moves or not
during the stalls (if it doesn't that may be a scheduler issue with
the compaction irq disabled and lack of cond_resched) and to try
aa.git. Upstream still misses several compaction improvements that we
did over the last weeks and that I've in my queue and that are in -mm
as well. So before making more changes, considering the stack traces
looks very healthy now, I'd wait to be sure the hangs aren't already
solved by any of the other scheduling/irq latency fixes. I guess they
aren't going to help but it worth a try. Verifying if this happens
with a more optimal filesystem like ext4 I think is also interesting,
it may be something in udf internal locking that gets in the way of
compaction.

If we still have a problem with current aa.git and ext4, then I'd hope
we can find some other more genuine bit to improve like the bits we've
improved so far, but if there's nothing wrong and it gets unfixable,
then my preference would be to either create a defrag mode that is in
between "yes/no", or alternatively to be simpler and make the default
between defrag yes|no configurable at build time and through a command
line in grub, and hope that SLUB doesn't clashes on it too. The
current "default" is optimal for several server environments where we
know most of the allocations are long lived. So we want to still have
an option to be as reliable as we are toady for those.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 22/03/11 10:03, Andrea Arcangeli escribio:
>
> I asked yesterday by PM to Alex if the mouse pointer moves or not
> during the stalls (if it doesn't that may be a scheduler issue with
> the compaction irq disabled and lack of cond_resched) and to try
> aa.git. Upstream still misses several compaction improvements that we
> did over the last weeks and that I've in my queue and that are in -mm
> as well. So before making more changes, considering the stack traces
> looks very healthy now, I'd wait to be sure the hangs aren't already
> solved by any of the other scheduling/irq latency fixes. I guess they
> aren't going to help but it worth a try. Verifying if this happens
> with a more optimal filesystem like ext4 I think is also interesting,
> it may be something in udf internal locking that gets in the way of
> compaction.
>
> If we still have a problem with current aa.git and ext4, then I'd hope
> we can find some other more genuine bit to improve like the bits we've
> improved so far, but if there's nothing wrong and it gets unfixable,
> then my preference would be to either create a defrag mode that is in
> between "yes/no", or alternatively to be simpler and make the default
> between defrag yes|no configurable at build time and through a command
> line in grub, and hope that SLUB doesn't clashes on it too. The
> current "default" is optimal for several server environments where we
> know most of the allocations are long lived. So we want to still have
> an option to be as reliable as we are toady for those.
>
I have just tested aa.git as of today, with the USB stick formatted as FAT32. I could no longer reproduce the stalls. There was no need to format as ext4. No /proc workarounds required.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

On Tue, Mar 22, 2011 at 03:34:10PM -0500, Alex Villaci-s Lasso wrote:
> I have just tested aa.git as of today, with the USB stick formatted
> as FAT32. I could no longer reproduce the stalls. There was no need
> to format as ext4. No /proc workarounds required.

Sounds good.

So Andrew the patches to apply to solve this most certainly are:

http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-compaction-minimise-the-time-irqs-are-disabled-while-isolating-pages-for-migration.patch
http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-compaction-minimise-the-time-irqs-are-disabled-while-isolating-pages-for-migration-fix.patch
http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-compaction-minimise-the-time-irqs-are-disabled-while-isolating-free-pages.patch
http://git.kernel.org/?p=linux/kernel/git/andrea/aa.git;a=patch;h=6daf7ff3adc1a243aa9f5a77c7bde2b713a3188a (not in -mm, posted to linux-mm Message-ID 20110321134832.GC5719)

Very likely it's the combination of all the above that is equally
important and needed for this specific compaction issue.

==== rest of aa.git not relevant for this bugreport below ====

http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-compaction-prevent-kswapd-compacting-memory-to-reduce-cpu-usage.patch
http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-vmscan-kswapd-should-not-free-an-excessive-number-of-pages-when-balancing-small-zones.patch

http://git.kernel.org/?p=linux/kernel/git/andrea/aa.git;a=patch;h=cb107ebbb7541e5442fd897436440e71835b6496 (not in -mm posted to linux-mm Message-ID: alpine.LSU.2.00.1103192318100.1877)

http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-add-__gfp_other_node-flag.patch
http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-add-__gfp_other_node-flag-checkpatch-fixes.patch
http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-use-__gfp_other_node-for-transparent-huge-pages.patch
http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-use-__gfp_other_node-for-transparent-huge-pages-checkpatch-fixes.patch
http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-add-vm-counters-for-transparent-hugepages.patch
http://userweb.kernel.org/~akpm/mmotm/broken-out/mm-add-vm-counters-for-transparent-hugepages-checkpatch-fixes.patch

smaps* (5 patches in -mm)


This is one experimental new feature that should improve mremap
significantly regardless of THP on or off (but bigger boost with THP
on). I'm using it for weeks without problem, I'd suggest it for
inclusion too. The version in the below link is the most uptodate. It
fixes a build trouble (s/__split_huge_page_pmd/split_huge_page_pmd/ in
move_page_tables) with CONFIG_TRANSPARENT_HUGEPAGE=n compared to the
last version I posted to linux-mm. But this is low priority.

http://git.kernel.org/?p=linux/kernel/git/andrea/aa.git;a=patch;h=0e6f8bd8802c3309195d3e1a7af50093ed488f2d

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

Hi Alex,

could you also try to reverse this below bit (not the whole previous
patch: only the bit below quoted below) with "patch -p1 -R < thismail"
on top of your current aa.git tree, and see if you notice any
regression compared to the previous aa.git build that worked well?

This is part of the fix, but I'd need to be sure this really makes a
difference before sticking to it for long. I'm not concerned by
keeping it, but it adds dirt, and the closer THP allocations are to
any other high order allocation the better. So the less
__GFP_NO_KSWAPD affects the better. The hint about not telling kswapd
to insist in the background for order 9 allocations with fallback
(like THP) is the maximum I consider clean because there's khugepaged
with its alloc_sleep_millisecs that replaces the kswapd task for THP
allocations. So that is clean enough, but when __GFP_NO_KSWAPD starts
to make compaction behave slightly different from a SLUB order 2
allocation I don't like it (especially because if you later enable
SLUB or some driver you may run into the same compaction issue again
if the below change is making a difference).

If things works fine even after you reverse the below, we can safely
undo this change and also feel safer for all other high order
allocations, so it'll make life easier. (plus we don't want
unnecessary special changes, we need to be sure this makes a
difference to keep it for long)

--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2085,7 +2085,7 @@ rebalance:
 					sync_migration);
 	if (page)
 		goto got_pg;
-	sync_migration = true;
+	sync_migration = !(gfp_mask & __GFP_NO_KSWAPD);
 
 	/* Try direct reclaim and then allocating */
 	page = __alloc_pages_direct_reclaim(gfp_mask, order,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

[-- Attachment #1: Type: text/plain, Size: 2834 bytes --]

El 22/03/11 19:37, Andrea Arcangeli escribio:
> Hi Alex,
>
> could you also try to reverse this below bit (not the whole previous
> patch: only the bit below quoted below) with "patch -p1 -R<  thismail"
> on top of your current aa.git tree, and see if you notice any
> regression compared to the previous aa.git build that worked well?
>
> This is part of the fix, but I'd need to be sure this really makes a
> difference before sticking to it for long. I'm not concerned by
> keeping it, but it adds dirt, and the closer THP allocations are to
> any other high order allocation the better. So the less
> __GFP_NO_KSWAPD affects the better. The hint about not telling kswapd
> to insist in the background for order 9 allocations with fallback
> (like THP) is the maximum I consider clean because there's khugepaged
> with its alloc_sleep_millisecs that replaces the kswapd task for THP
> allocations. So that is clean enough, but when __GFP_NO_KSWAPD starts
> to make compaction behave slightly different from a SLUB order 2
> allocation I don't like it (especially because if you later enable
> SLUB or some driver you may run into the same compaction issue again
> if the below change is making a difference).
>
> If things works fine even after you reverse the below, we can safely
> undo this change and also feel safer for all other high order
> allocations, so it'll make life easier. (plus we don't want
> unnecessary special changes, we need to be sure this makes a
> difference to keep it for long)
>
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2085,7 +2085,7 @@ rebalance:
>   					sync_migration);
>   	if (page)
>   		goto got_pg;
> -	sync_migration = true;
> +	sync_migration = !(gfp_mask&  __GFP_NO_KSWAPD);
>
>   	/* Try direct reclaim and then allocating */
>   	page = __alloc_pages_direct_reclaim(gfp_mask, order,
>

> On Tue, Mar 22, 2011 at 03:34:10PM -0500, Alex Villaci-s Lasso wrote:
>> >  I have just tested aa.git as of today, with the USB stick formatted
>> >  as FAT32. I could no longer reproduce the stall
> Probably udf is not optimized enough but I wonder if maybe the
> udf->vfat change helped more than the other patches. We need the other
> patches anyway to provide responsive behavior including the one you
> tested before aa.git so it's not very important if udf was the
> problem, but it might have been.
>
I tried to reformat the stick as UDF to check whether the stall was filesystem-sensitive. Apparently it is. I managed to induce the freeze on firefox while performing the same copy on the aa.git kernel. Then I reformatted the stick as FAT32 and repeated 
the test, and it also induced freezes, although they were a bit shorter and occurred late in the copy progress. I have attached the traces in the bug report. All of this is with the kernel before reversing the quoted patch.

[-- Attachment #2: Type: text/html, Size: 3601 bytes --]

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

Latest update: with 2.6.39-rc1 the stalls only last for a few seconds, but they are still there. So, as I said before, they are reduced but not eliminated.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

Hi Alex,

On Mon, Apr 04, 2011 at 10:37:44AM -0500, Alex Villaci-s Lasso wrote:
> Latest update: with 2.6.39-rc1 the stalls only last for a few
> seconds, but they are still there. So, as I said before, they are
> reduced but not eliminated.

Ok from a complete stall for the whole duration of the I/O to a few
second stall we're clearly going into the right direction.

The few second stalls happen with udf? And vfat doesn't stall?

Minchan rightly pointed out that the (panik) change we made in
page_alloc.c changes the semantics of the __GFP_NO_KSWAPD bit. I also
talked with Mel about it. We think it's nicer if we can keep THP
allocations as close as any other high order allocation as
possible. There are already plenty of __GFP bits with complex
semantics. __GFP_NO_KSWAPD is simple and it's nice to stay simple: it
means the allocation relies on a different kernel daemon for the
background work (which is khugepaged instead of kswapd in the THP
case, where khugepaged uses a non intrusive alloc_sleep_millisec
throttling in case of MM congestion, unlike kswapd would do).

So this is no solution to your problem (if vfat already works I think
that might be a better solution), but we'd like to know if you get any
_worse_ stall compared to current 2.6.39-rc, by applying the below
patch on top of 2.6.39-rc. If this doesn't make any difference, we can
safely apply it to remove unnecessary complications.

Thanks,
Andrea

===
Subject: compaction: reverse the change that forbid sync migraton with __GFP_NO_KSWAPD

From: Andrea Arcangeli <aarcange@redhat.com>

It's uncertain this has been beneficial, so it's safer to undo it. All other
compaction users would still go in synchronous mode if a first attempt of async
compaction failed. Hopefully we don't need to force special behavior for THP
(which is the only __GFP_NO_KSWAPD user so far and it's the easier to exercise
and to be noticeable). This also make __GFP_NO_KSWAPD return to its original
strict semantics specific to bypass kswapd, as THP allocations have khugepaged
for the async THP allocations/compactions.

Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
---
 mm/page_alloc.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2105,7 +2105,7 @@ rebalance:
 					sync_migration);
 	if (page)
 		goto got_pg;
-	sync_migration = !(gfp_mask & __GFP_NO_KSWAPD);
+	sync_migration = true;
 
 	/* Try direct reclaim and then allocating */
 	page = __alloc_pages_direct_reclaim(gfp_mask, order,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 08/04/11 14:09, Andrea Arcangeli escribio:
> Hi Alex,
>
> On Mon, Apr 04, 2011 at 10:37:44AM -0500, Alex Villaci-s Lasso wrote:
>> Latest update: with 2.6.39-rc1 the stalls only last for a few
>> seconds, but they are still there. So, as I said before, they are
>> reduced but not eliminated.
> Ok from a complete stall for the whole duration of the I/O to a few
> second stall we're clearly going into the right direction.
>
> The few second stalls happen with udf? And vfat doesn't stall?
>
> Minchan rightly pointed out that the (panik) change we made in
> page_alloc.c changes the semantics of the __GFP_NO_KSWAPD bit. I also
> talked with Mel about it. We think it's nicer if we can keep THP
> allocations as close as any other high order allocation as
> possible. There are already plenty of __GFP bits with complex
> semantics. __GFP_NO_KSWAPD is simple and it's nice to stay simple: it
> means the allocation relies on a different kernel daemon for the
> background work (which is khugepaged instead of kswapd in the THP
> case, where khugepaged uses a non intrusive alloc_sleep_millisec
> throttling in case of MM congestion, unlike kswapd would do).
>
> So this is no solution to your problem (if vfat already works I think
> that might be a better solution), but we'd like to know if you get any
> _worse_ stall compared to current 2.6.39-rc, by applying the below
> patch on top of 2.6.39-rc. If this doesn't make any difference, we can
> safely apply it to remove unnecessary complications.
>
> Thanks,
> Andrea
>
> ===
> Subject: compaction: reverse the change that forbid sync migraton with __GFP_NO_KSWAPD
>
> From: Andrea Arcangeli<aarcange@redhat.com>
>
> It's uncertain this has been beneficial, so it's safer to undo it. All other
> compaction users would still go in synchronous mode if a first attempt of async
> compaction failed. Hopefully we don't need to force special behavior for THP
> (which is the only __GFP_NO_KSWAPD user so far and it's the easier to exercise
> and to be noticeable). This also make __GFP_NO_KSWAPD return to its original
> strict semantics specific to bypass kswapd, as THP allocations have khugepaged
> for the async THP allocations/compactions.
>
> Signed-off-by: Andrea Arcangeli<aarcange@redhat.com>
> ---
>   mm/page_alloc.c |    2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2105,7 +2105,7 @@ rebalance:
>   					sync_migration);
>   	if (page)
>   		goto got_pg;
> -	sync_migration = !(gfp_mask&  __GFP_NO_KSWAPD);
> +	sync_migration = true;
>
>   	/* Try direct reclaim and then allocating */
>   	page = __alloc_pages_direct_reclaim(gfp_mask, order,
>
The stalls occur even with vfat. I am no longer using udf, since (right now) it is not necessary. I will test this patch now.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 08/04/11 15:06, Alex Villaci-s Lasso escribio:
> El 08/04/11 14:09, Andrea Arcangeli escribio:
>> Hi Alex,
>>
>> On Mon, Apr 04, 2011 at 10:37:44AM -0500, Alex Villaci-s Lasso wrote:
>>> Latest update: with 2.6.39-rc1 the stalls only last for a few
>>> seconds, but they are still there. So, as I said before, they are
>>> reduced but not eliminated.
>> Ok from a complete stall for the whole duration of the I/O to a few
>> second stall we're clearly going into the right direction.
>>
>> The few second stalls happen with udf? And vfat doesn't stall?
>>
>> Minchan rightly pointed out that the (panik) change we made in
>> page_alloc.c changes the semantics of the __GFP_NO_KSWAPD bit. I also
>> talked with Mel about it. We think it's nicer if we can keep THP
>> allocations as close as any other high order allocation as
>> possible. There are already plenty of __GFP bits with complex
>> semantics. __GFP_NO_KSWAPD is simple and it's nice to stay simple: it
>> means the allocation relies on a different kernel daemon for the
>> background work (which is khugepaged instead of kswapd in the THP
>> case, where khugepaged uses a non intrusive alloc_sleep_millisec
>> throttling in case of MM congestion, unlike kswapd would do).
>>
>> So this is no solution to your problem (if vfat already works I think
>> that might be a better solution), but we'd like to know if you get any
>> _worse_ stall compared to current 2.6.39-rc, by applying the below
>> patch on top of 2.6.39-rc. If this doesn't make any difference, we can
>> safely apply it to remove unnecessary complications.
>>
>> Thanks,
>> Andrea
>>
>> ===
>> Subject: compaction: reverse the change that forbid sync migraton with __GFP_NO_KSWAPD
>>
>> From: Andrea Arcangeli<aarcange@redhat.com>
>>
>> It's uncertain this has been beneficial, so it's safer to undo it. All other
>> compaction users would still go in synchronous mode if a first attempt of async
>> compaction failed. Hopefully we don't need to force special behavior for THP
>> (which is the only __GFP_NO_KSWAPD user so far and it's the easier to exercise
>> and to be noticeable). This also make __GFP_NO_KSWAPD return to its original
>> strict semantics specific to bypass kswapd, as THP allocations have khugepaged
>> for the async THP allocations/compactions.
>>
>> Signed-off-by: Andrea Arcangeli<aarcange@redhat.com>
>> ---
>>   mm/page_alloc.c |    2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> --- a/mm/page_alloc.c
>> +++ b/mm/page_alloc.c
>> @@ -2105,7 +2105,7 @@ rebalance:
>>                       sync_migration);
>>       if (page)
>>           goto got_pg;
>> -    sync_migration = !(gfp_mask&  __GFP_NO_KSWAPD);
>> +    sync_migration = true;
>>
>>       /* Try direct reclaim and then allocating */
>>       page = __alloc_pages_direct_reclaim(gfp_mask, order,
>>
> The stalls occur even with vfat. I am no longer using udf, since (right now) it is not necessary. I will test this patch now.
>
 From preliminary tests, I feel that the patch actually eliminates the stalls. I have just copied nearly 6 GB of data into my USB stick and noticed no application freezes.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Alex Villací­s Lasso]

El 12/04/11 11:27, Alex Villaci-s Lasso escribio:
>>> ===
>>> Subject: compaction: reverse the change that forbid sync migraton with __GFP_NO_KSWAPD
>>>
>>> From: Andrea Arcangeli<aarcange@redhat.com>
>>>
>>> It's uncertain this has been beneficial, so it's safer to undo it. All other
>>> compaction users would still go in synchronous mode if a first attempt of async
>>> compaction failed. Hopefully we don't need to force special behavior for THP
>>> (which is the only __GFP_NO_KSWAPD user so far and it's the easier to exercise
>>> and to be noticeable). This also make __GFP_NO_KSWAPD return to its original
>>> strict semantics specific to bypass kswapd, as THP allocations have khugepaged
>>> for the async THP allocations/compactions.
>>>
>>> Signed-off-by: Andrea Arcangeli<aarcange@redhat.com>
>>> ---
>>>   mm/page_alloc.c |    2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> --- a/mm/page_alloc.c
>>> +++ b/mm/page_alloc.c
>>> @@ -2105,7 +2105,7 @@ rebalance:
>>>                       sync_migration);
>>>       if (page)
>>>           goto got_pg;
>>> -    sync_migration = !(gfp_mask&  __GFP_NO_KSWAPD);
>>> +    sync_migration = true;
>>>
>>>       /* Try direct reclaim and then allocating */
>>>       page = __alloc_pages_direct_reclaim(gfp_mask, order,
>>>
>> The stalls occur even with vfat. I am no longer using udf, since (right now) it is not necessary. I will test this patch now.
>>
>
> From preliminary tests, I feel that the patch actually eliminates the stalls. I have just copied nearly 6 GB of data into my USB stick and noticed no application freezes.
>
I retract that. I have tested 2.6.39-rc3 after a day of having several heavy applications loaded in memory, and the stalls do get worse when reversing the patch.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bugme-new] [Bug 31142] New: Large write to USB stick freezes unrelated tasks for a long time

[Andrea Arcangeli]

Hello Alex,

On Thu, Apr 14, 2011 at 12:25:48PM -0500, Alex Villaci-s Lasso wrote:
> I retract that. I have tested 2.6.39-rc3 after a day of having
> several heavy applications loaded in memory, and the stalls do get
> worse when reversing the patch.

Well I was already afraid your stalling wasn't 100%
reproducible. Depends on background load like you said. I think if it
never happens when you didn't start the heavy applications yet is good
enough result for now. When we throttle on I/O and there are heavy
apps things may stall even without usb drive because the more memory
pressure the more every little write() or memory allocation, may stall
too regardless of compaction, we've no perfection in that area yet
(the best way to tackle this are the per-process write throttling
levels).

Bottom line is that the additional congestion created by the heavy app
is by far not easy to quantify or to assume as a reproducible, so it
is very possible it was the same as before and in general if we want
to apply that change it's cleaner to do it unconditionally for all
allocation orders and not in function of __GFP_NO_KSWAPD. So I think
the patch is still good to go.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>