* Re: XFS problem
[not found] <CAAA8XhOXuszcyCaMOcVWb-erGAJQdhBHCX-gsJ4KpH+Td7+bPQ@mail.gmail.com>
@ 2012-01-24 14:05 ` Jan Kara
2012-01-24 21:39 ` Christoph Hellwig
1 sibling, 0 replies; 7+ messages in thread
From: Jan Kara @ 2012-01-24 14:05 UTC (permalink / raw)
To: Kelbel Junior; +Cc: linux-kernel, xfs
Just adding XFS list to catch more attention...
On Fri 20-01-12 15:55:18, Kelbel Junior wrote:
> Hello again ...
>
> So ... after changing the memory allocator SLUB to SLAB in the Linux
> kernel and apply the patch mentioned below, the system was stable for
> 3 days and today it happened again delay to deliver content on the
> disc.
>
> When I went to see if there was something wrong in /var/log/messages
> appear several times this block:
>
> server012 kernel: XFS: possible memory allocation deadlock in
> kmem_alloc (mode:0x250)
> server012 kernel: Pid: 24885, comm: kworker/0:3 Tainted: G W 2.6.38.8-cve #2
> server012 kernel: Call Trace:
> server012 kernel: [<ffffffff811cfc65>] ? kmem_alloc+0xa6/0xb4
> server012 kernel: [<ffffffff811cfc91>] ? kmem_realloc+0x1e/0x51
> server012 kernel: [<ffffffff811b7b4e>] ?
> xfs_iext_realloc_indirect+0x3e/0x46
> server012 kernel: [<ffffffff811b7cf6>] ? xfs_iext_irec_new+0x36/0xcf
> server012 kernel: [<ffffffff811cfc26>] ? kmem_alloc+0x67/0xb4
> server012 kernel: [<ffffffff811b8014>] ?
> xfs_iext_add_indirect_multi+0x195/0x1de
> server012 kernel: [<ffffffff811b8a66>] ? xfs_iext_add+0x1a6/0x1e2
> server012 kernel: [<ffffffff811a4523>] ? xfs_btree_update+0x68/0xb5
> server012 kernel: [<ffffffff811b8b36>] ? xfs_iext_insert+0x34/0x5f
> server012 kernel: [<ffffffff8119de57>] ?
> xfs_bmap_add_extent_unwritten_real+0xace/0xc47
> server012 kernel: [<ffffffff811b80e2>] ? xfs_iext_bno_to_irec+0x85/0xae
> server012 kernel: [<ffffffff8119f367>] ? xfs_bmap_add_extent+0x1fd/0x326
> server012 kernel: [<ffffffff811cfb13>] ? kmem_zone_alloc+0x67/0xb4
> server012 kernel: [<ffffffff811a30b7>] ? xfs_bmbt_init_cursor+0x3a/0x110
> server012 kernel: [<ffffffff811a1d9b>] ? xfs_bmapi+0x9ff/0xd3e
> server012 kernel: [<ffffffff811bc4a1>] ?
> xfs_iomap_write_unwritten+0x1d8/0x27d
> server012 kernel: [<ffffffff8120a7c6>] ? kobject_put+0x47/0x4c
> server012 kernel: [<ffffffff812dda57>] ? put_device+0x12/0x14
> server012 kernel: [<ffffffff811d02b0>] ? xfs_end_io+0x42/0x91
> server012 kernel: [<ffffffff81048789>] ? process_one_work+0x189/0x25e
> server012 kernel: [<ffffffff811d026e>] ? xfs_end_io+0x0/0x91
> server012 kernel: [<ffffffff81048b14>] ? worker_thread+0x132/0x24f
> server012 kernel: [<ffffffff810489e2>] ? worker_thread+0x0/0x24f
> server012 kernel: [<ffffffff8104d55f>] ? kthread+0x81/0x89
> server012 kernel: [<ffffffff81003754>] ? kernel_thread_helper+0x4/0x10
> server012 kernel: [<ffffffff8104d4de>] ? kthread+0x0/0x89
> server012 kernel: [<ffffffff81003750>] ? kernel_thread_helper+0x0/0x10
>
> Has anyone seen or had this problem?
>
> below "uname" on host:
> Linux server012 2.6.38.8 #2 SMP Tue Jan 17 10:17:05 BRST 2012 x86_64
> Intel(R) Core(TM) i3 CPU 540 @ 3.07GHz GenuineIntel GNU/Linux
>
>
>
> Thanks.
> Kelbel Junior
>
>
>
>
> 2012/1/16 Kelbel Junior <jymmyjr@gmail.com>:
> > Hello, everybody.
> > It appeared that same error message when using kernel version 2.6.38.8 ...
> >
> > Several lines in /var/log/messages with:
> > kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode: 0x250)
> >
> > Does anyone know know if this bug has been fixed in newer versions of Linux
> > kernel? 3.x maybe?
> >
> >
> > 2011/5/11 Christoph Hellwig <>
> >>
> >> And here's the actual patch, sorry:
> >>
> >>
> >> Index: linux-2.6/fs/xfs/linux-2.6/kmem.c
> >> ===================================================================
> >> --- linux-2.6.orig/fs/xfs/linux-2.6/kmem.c 2011-05-11 17:29:51.729191621
> >> +0200
> >>
> >> +++ linux-2.6/fs/xfs/linux-2.6/kmem.c 2011-05-11 17:30:22.915689382 +0200
> >> @@ -56,10 +56,12 @@ kmem_alloc(size_t size, unsigned int __n
> >> ptr = kmalloc(size, lflags);
> >> if (ptr || (flags & (KM_MAYFAIL|KM_NOSLEEP)))
> >>
> >> return ptr;
> >> - if (!(++retries % 100))
> >> + if (!(++retries % 100)) {
> >> xfs_err(NULL,
> >> "possible memory allocation deadlock in %s (mode:0x%x)",
> >> __func__, lflags);
> >> + dump_stack();
> >>
> >> + }
> >> congestion_wait(BLK_RW_ASYNC, HZ/50);
> >> } while (1);
> >> }
> >> @@ -112,10 +114,12 @@ kmem_zone_alloc(kmem_zone_t *zone, unsig
> >> ptr = kmem_cache_alloc(zone, lflags);
> >> if (ptr || (flags & (KM_MAYFAIL|KM_NOSLEEP)))
> >>
> >> return ptr;
> >> - if (!(++retries % 100))
> >> + if (!(++retries % 100)) {
> >> xfs_err(NULL,
> >> "possible memory allocation deadlock in %s (mode:0x%x)",
> >> __func__, lflags);
> >> + dump_stack();
> >>
> >> + }
> >> congestion_wait(BLK_RW_ASYNC, HZ/50);
> >> } while (1);
> >> }
> >>
> >>
> >> 2011/5/11 Christoph Hellwig <>
> >>>
> >>> You're probably running into a different issue. Any chance you can try
> >>> the attached patch to see where you're stuck exactly?
> >>>
> >>> Also in case you're using slub (CONFIG_SLUB) can you check if it still
> >>>
> >>> happens when using slab (CONFIG_SLAB) instead?
> >>>
> >>>
> >>> 2011/5/10 Felipe Wilhelms Damasio - Taghos <>
> >>>>
> >>>> Hi Mr. Hellwig,
> >>>>
> >>>> I saw the XFS deadlock thread regarding 2.6.38:
> >>>>
> >>>> http://www.linux.sgi.com/archives/xfs/2011-03/msg00185.html
> >>>>
> >>>>
> >>>> I'm seeing too in 2.6.38.4. A lot of
> >>>>
> >>>> XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
> >>>>
> >>>> I checked the changelog for 2.6.38.6 and can't be sure if the fix
> >>>> got
> >>>> in...could you tell me if it did?
> >>>>
> >>>>
> >>>> Thanks,
> >>>>
> >>>> --
> >>>>
> >>>> Felipe Wilhelms Damasio
> >>>>
> >>>>
> >>>> TAGHOS - Tecnologia
> >>>> Rua Prof. Alvaro Alvim, 211
> >>>> Porto Alegre - RS - (51) 3239-3180
> >>>> www.taghos.com.br <http://www.taghos.com.br/>
> >>>>
> >>>
> >>
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: XFS problem
[not found] <CAAA8XhOXuszcyCaMOcVWb-erGAJQdhBHCX-gsJ4KpH+Td7+bPQ@mail.gmail.com>
2012-01-24 14:05 ` XFS problem Jan Kara
@ 2012-01-24 21:39 ` Christoph Hellwig
2012-01-26 15:57 ` Kelbel Junior
1 sibling, 1 reply; 7+ messages in thread
From: Christoph Hellwig @ 2012-01-24 21:39 UTC (permalink / raw)
To: Kelbel Junior; +Cc: linux-kernel, xfs
On Fri, Jan 20, 2012 at 03:55:18PM -0200, Kelbel Junior wrote:
> Hello again ...
>
> So ... after changing the memory allocator SLUB to SLAB in the Linux
> kernel and apply the patch mentioned below, the system was stable for
> 3 days and today it happened again delay to deliver content on the
> disc.
>
> When I went to see if there was something wrong in /var/log/messages
> appear several times this block:
Can you try the patch below to make the extent indirection array
reallocation more efficient? It's still working around the symptoms,
though.
What kind of workload are you running to get this extremely high number
of extents?
Index: xfs/fs/xfs/kmem.c
===================================================================
--- xfs.orig/fs/xfs/kmem.c 2011-10-17 09:28:57.159149025 +0200
+++ xfs/fs/xfs/kmem.c 2012-01-24 21:04:50.405954910 +0100
@@ -86,19 +86,22 @@ kmem_free(const void *ptr)
}
void *
-kmem_realloc(const void *ptr, size_t newsize, size_t oldsize,
- unsigned int __nocast flags)
+kmem_realloc(const void *old, size_t newsize, unsigned int __nocast flags)
{
+ gfp_t lflags = kmem_flags_convert(flags);
+ int retries = 0;
void *new;
- new = kmem_alloc(newsize, flags);
- if (ptr) {
- if (new)
- memcpy(new, ptr,
- ((oldsize < newsize) ? oldsize : newsize));
- kmem_free(ptr);
- }
- return new;
+ do {
+ new = krealloc(old, newsize, lflags);
+ if (new || (flags & (KM_MAYFAIL|KM_NOSLEEP)))
+ return new;
+ if (!(++retries % 100))
+ xfs_err(NULL,
+ "possible memory allocation deadlock in %s (mode:0x%x)",
+ __func__, lflags);
+ congestion_wait(BLK_RW_ASYNC, HZ/50);
+ } while (1);
}
void *
Index: xfs/fs/xfs/kmem.h
===================================================================
--- xfs.orig/fs/xfs/kmem.h 2011-12-18 23:35:30.893167926 +0100
+++ xfs/fs/xfs/kmem.h 2012-01-24 21:00:07.865956600 +0100
@@ -56,7 +56,7 @@ kmem_flags_convert(unsigned int __nocast
extern void *kmem_alloc(size_t, unsigned int __nocast);
extern void *kmem_zalloc(size_t, unsigned int __nocast);
-extern void *kmem_realloc(const void *, size_t, size_t, unsigned int __nocast);
+extern void *kmem_realloc(const void *, size_t, unsigned int __nocast);
extern void kmem_free(const void *);
static inline void *kmem_zalloc_large(size_t size)
Index: xfs/fs/xfs/xfs_inode.c
===================================================================
--- xfs.orig/fs/xfs/xfs_inode.c 2012-01-24 20:57:27.079290895 +0100
+++ xfs/fs/xfs/xfs_inode.c 2012-01-24 21:02:48.145955643 +0100
@@ -1826,7 +1826,6 @@ xfs_iroot_realloc(
new_max = cur_max + rec_diff;
new_size = (size_t)XFS_BMAP_BROOT_SPACE_CALC(new_max);
ifp->if_broot = kmem_realloc(ifp->if_broot, new_size,
- (size_t)XFS_BMAP_BROOT_SPACE_CALC(cur_max), /* old size */
KM_SLEEP | KM_NOFS);
op = (char *)XFS_BMAP_BROOT_PTR_ADDR(mp, ifp->if_broot, 1,
ifp->if_broot_bytes);
@@ -1968,7 +1967,6 @@ xfs_idata_realloc(
ifp->if_u1.if_data =
kmem_realloc(ifp->if_u1.if_data,
real_size,
- ifp->if_real_bytes,
KM_SLEEP | KM_NOFS);
}
} else {
@@ -3254,8 +3252,7 @@ xfs_iext_realloc_direct(
if (rnew_size != ifp->if_real_bytes) {
ifp->if_u1.if_extents =
kmem_realloc(ifp->if_u1.if_extents,
- rnew_size,
- ifp->if_real_bytes, KM_NOFS);
+ rnew_size, KM_NOFS);
}
if (rnew_size > ifp->if_real_bytes) {
memset(&ifp->if_u1.if_extents[ifp->if_bytes /
@@ -3333,20 +3330,15 @@ xfs_iext_realloc_indirect(
xfs_ifork_t *ifp, /* inode fork pointer */
int new_size) /* new indirection array size */
{
- int nlists; /* number of irec's (ex lists) */
- int size; /* current indirection array size */
-
ASSERT(ifp->if_flags & XFS_IFEXTIREC);
- nlists = ifp->if_real_bytes / XFS_IEXT_BUFSZ;
- size = nlists * sizeof(xfs_ext_irec_t);
ASSERT(ifp->if_real_bytes);
- ASSERT((new_size >= 0) && (new_size != size));
+ ASSERT(new_size >= 0);
+
if (new_size == 0) {
xfs_iext_destroy(ifp);
} else {
- ifp->if_u1.if_ext_irec = (xfs_ext_irec_t *)
- kmem_realloc(ifp->if_u1.if_ext_irec,
- new_size, size, KM_NOFS);
+ ifp->if_u1.if_ext_irec =
+ kmem_realloc(ifp->if_u1.if_ext_irec, new_size, KM_NOFS);
}
}
Index: xfs/fs/xfs/xfs_log_recover.c
===================================================================
--- xfs.orig/fs/xfs/xfs_log_recover.c 2011-12-18 23:35:30.906501259 +0100
+++ xfs/fs/xfs/xfs_log_recover.c 2012-01-24 21:02:20.352622478 +0100
@@ -1489,7 +1489,7 @@ xlog_recover_add_to_cont_trans(
old_ptr = item->ri_buf[item->ri_cnt-1].i_addr;
old_len = item->ri_buf[item->ri_cnt-1].i_len;
- ptr = kmem_realloc(old_ptr, len+old_len, old_len, 0u);
+ ptr = kmem_realloc(old_ptr, len+old_len, KM_SLEEP);
memcpy(&ptr[old_len], dp, len); /* d, s, l */
item->ri_buf[item->ri_cnt-1].i_len += len;
item->ri_buf[item->ri_cnt-1].i_addr = ptr;
Index: xfs/fs/xfs/xfs_mount.c
===================================================================
--- xfs.orig/fs/xfs/xfs_mount.c 2011-12-18 23:35:30.906501259 +0100
+++ xfs/fs/xfs/xfs_mount.c 2012-01-24 21:05:29.699288013 +0100
@@ -147,7 +147,6 @@ xfs_uuid_mount(
if (hole < 0) {
xfs_uuid_table = kmem_realloc(xfs_uuid_table,
(xfs_uuid_table_size + 1) * sizeof(*xfs_uuid_table),
- xfs_uuid_table_size * sizeof(*xfs_uuid_table),
KM_SLEEP);
hole = xfs_uuid_table_size++;
}
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 7+ messages in thread