* [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1
@ 2009-01-09 22:22 Nicholas A. Bellinger
2009-01-12 2:02 ` Mike Christie
0 siblings, 1 reply; 6+ messages in thread
From: Nicholas A. Bellinger @ 2009-01-09 22:22 UTC (permalink / raw)
To: Open iSCSI, linux-scsi; +Cc: Mike Christie, Hannes Reinecke
Greetings Mike, Hannes and Co,
During some recent testing using the Open/iSCSI Initiator v2.0-870.1,
against the LIO-Target v3.0 tree, I noticed that while running the
following script:
while [ 1 ]; do
iscsiadm -m node -T $TARGETNAME -p $PORTAL --login
iscsiadm -m node -T $TARGETNAME -p $PORTAL --logout
done
for an extended period of time that I started getting OOM failures on
the VMs running Open/iSCSI. Upon closer examination, this is what I
found:
<Open-iSCSI Node 1>
Linux ubuntu 2.6.27.10 #2 SMP Tue Jan 6 18:33:00 PST 2009 i686 GNU/Linux
Using open-iscsi-2.0-870.1:
[78196.520214] scsi7981 : iSCSI Initiator over TCP/IP
[78284.175307] scsi7982 : iSCSI Initiator over TCP/IP
[78338.568656] scsi7983 : iSCSI Initiator over TCP/IP
[78405.888822] scsi7984 : iSCSI Initiator over TCP/IP
Output from slaptop:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1037001 1036598 99% 0.03K 9177 113 36708K size-32
-------------------------------------------------------------------------
<Open-iSCSI Node 2>
Linux opensuse 2.6.22.5-31-default #1 SMP 2007/09/21 22:29:00 UTC i686 i686 i386 GNU/Linux
scsi7046 : iSCSI Initiator over TCP/IP
scsi7047 : iSCSI Initiator over TCP/IP
scsi7048 : iSCSI Initiator over TCP/IP
scsi7049 : iSCSI Initiator over TCP/IP
Output from slabtop:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
914057 913581 99% 0.03K 8089 113 32356K size-32
-------------------------------------------------------------------------
So it appears that memory is getting leaked in the size-32 range with
each --login + --logout invocation. I also tried the same test with the
shipping Open/iSCSI code in Debian v4 and OpenSuse 10.3 and these also
suffer from the same issue.
Also of interest is that running the following script for Discovery
SendTargets *DOES NOT* reproduce the leak.
while [ 1 ]; do
iscsiadm -m discovery -t sendtargets -p $PORTAL
done
Please let me know if there is anything else I can do to help diagnose
the issue.
Many thanks for your most valuable of time,
--nab
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1
2009-01-09 22:22 [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1 Nicholas A. Bellinger
@ 2009-01-12 2:02 ` Mike Christie
2009-01-13 0:27 ` Nicholas A. Bellinger
0 siblings, 1 reply; 6+ messages in thread
From: Mike Christie @ 2009-01-12 2:02 UTC (permalink / raw)
To: Nicholas A. Bellinger; +Cc: Open iSCSI, linux-scsi, Hannes Reinecke
Nicholas A. Bellinger wrote:
> Greetings Mike, Hannes and Co,
>
> During some recent testing using the Open/iSCSI Initiator v2.0-870.1,
> against the LIO-Target v3.0 tree, I noticed that while running the
> following script:
>
> while [ 1 ]; do
> iscsiadm -m node -T $TARGETNAME -p $PORTAL --login
> iscsiadm -m node -T $TARGETNAME -p $PORTAL --logout
> done
>
> for an extended period of time that I started getting OOM failures on
> the VMs running Open/iSCSI. Upon closer examination, this is what I
> found:
>
> <Open-iSCSI Node 1>
>
> Linux ubuntu 2.6.27.10 #2 SMP Tue Jan 6 18:33:00 PST 2009 i686 GNU/Linux
>
> Using open-iscsi-2.0-870.1:
>
> [78196.520214] scsi7981 : iSCSI Initiator over TCP/IP
> [78284.175307] scsi7982 : iSCSI Initiator over TCP/IP
> [78338.568656] scsi7983 : iSCSI Initiator over TCP/IP
> [78405.888822] scsi7984 : iSCSI Initiator over TCP/IP
>
Hey, so are there any devices on the target? I do not see the normal
type/size info we see when scsi disks are found. Just checking. That
rules a lot of places out.
If there are disks, but they just are not gettting logged could you
remove them from the target so we can take some structs out of the mix?
> Output from slaptop:
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 1037001 1036598 99% 0.03K 9177 113 36708K size-32
>
> -------------------------------------------------------------------------
>
> <Open-iSCSI Node 2>
>
> Linux opensuse 2.6.22.5-31-default #1 SMP 2007/09/21 22:29:00 UTC i686 i686 i386 GNU/Linux
>
> scsi7046 : iSCSI Initiator over TCP/IP
> scsi7047 : iSCSI Initiator over TCP/IP
> scsi7048 : iSCSI Initiator over TCP/IP
> scsi7049 : iSCSI Initiator over TCP/IP
>
> Output from slabtop:
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 914057 913581 99% 0.03K 8089 113 32356K size-32
>
> -------------------------------------------------------------------------
>
> So it appears that memory is getting leaked in the size-32 range with
> each --login + --logout invocation. I also tried the same test with the
> shipping Open/iSCSI code in Debian v4 and OpenSuse 10.3 and these also
> suffer from the same issue.
>
> Also of interest is that running the following script for Discovery
> SendTargets *DOES NOT* reproduce the leak.
>
> while [ 1 ]; do
> iscsiadm -m discovery -t sendtargets -p $PORTAL
> done
The leak in the size-32 slab would be a kernel object right? if so the
sendtargets test not leaking means that this is a problem in the
session/connection kernel struct setup/destruction. The sendtargets code
is all in userspace so it would not leak in those objects.
I was out of the office sick last week, so let me catch up on work stuff
then I will try to send a patch. If you want you could try to stick
printks in iscsi driver model object release functions to make sure they
are getting fired, but that gets nasty.
>
> Please let me know if there is anything else I can do to help diagnose
> the issue.
>
> Many thanks for your most valuable of time,
>
> --nab
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1
2009-01-12 2:02 ` Mike Christie
@ 2009-01-13 0:27 ` Nicholas A. Bellinger
2009-01-13 1:24 ` Mike Christie
0 siblings, 1 reply; 6+ messages in thread
From: Nicholas A. Bellinger @ 2009-01-13 0:27 UTC (permalink / raw)
To: Mike Christie; +Cc: Open iSCSI, linux-scsi, Hannes Reinecke
On Sun, 2009-01-11 at 20:02 -0600, Mike Christie wrote:
> Nicholas A. Bellinger wrote:
> > Greetings Mike, Hannes and Co,
> >
> > During some recent testing using the Open/iSCSI Initiator v2.0-870.1,
> > against the LIO-Target v3.0 tree, I noticed that while running the
> > following script:
> >
> > while [ 1 ]; do
> > iscsiadm -m node -T $TARGETNAME -p $PORTAL --login
> > iscsiadm -m node -T $TARGETNAME -p $PORTAL --logout
> > done
> >
> > for an extended period of time that I started getting OOM failures on
> > the VMs running Open/iSCSI. Upon closer examination, this is what I
> > found:
> >
> > <Open-iSCSI Node 1>
> >
> > Linux ubuntu 2.6.27.10 #2 SMP Tue Jan 6 18:33:00 PST 2009 i686 GNU/Linux
> >
> > Using open-iscsi-2.0-870.1:
> >
> > [78196.520214] scsi7981 : iSCSI Initiator over TCP/IP
> > [78284.175307] scsi7982 : iSCSI Initiator over TCP/IP
> > [78338.568656] scsi7983 : iSCSI Initiator over TCP/IP
> > [78405.888822] scsi7984 : iSCSI Initiator over TCP/IP
> >
>
>
> Hey, so are there any devices on the target? I do not see the normal
> type/size info we see when scsi disks are found. Just checking. That
> rules a lot of places out.
>
> If there are disks, but they just are not gettting logged could you
> remove them from the target so we can take some structs out of the mix?
>
I tried the same test both WITH and WITHOUT iSCSI LUNs getting
registered on the Open/iSCSI side. The leak in size-32 is still present
in both cases.
>
> > Output from slaptop:
> >
> > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> > 1037001 1036598 99% 0.03K 9177 113 36708K size-32
> >
> > -------------------------------------------------------------------------
> >
> > <Open-iSCSI Node 2>
> >
> > Linux opensuse 2.6.22.5-31-default #1 SMP 2007/09/21 22:29:00 UTC i686 i686 i386 GNU/Linux
> >
> > scsi7046 : iSCSI Initiator over TCP/IP
> > scsi7047 : iSCSI Initiator over TCP/IP
> > scsi7048 : iSCSI Initiator over TCP/IP
> > scsi7049 : iSCSI Initiator over TCP/IP
> >
> > Output from slabtop:
> >
> > OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> > 914057 913581 99% 0.03K 8089 113 32356K size-32
> >
> > -------------------------------------------------------------------------
> >
> > So it appears that memory is getting leaked in the size-32 range with
> > each --login + --logout invocation. I also tried the same test with the
> > shipping Open/iSCSI code in Debian v4 and OpenSuse 10.3 and these also
> > suffer from the same issue.
> >
> > Also of interest is that running the following script for Discovery
> > SendTargets *DOES NOT* reproduce the leak.
> >
> > while [ 1 ]; do
> > iscsiadm -m discovery -t sendtargets -p $PORTAL
> > done
>
>
> The leak in the size-32 slab would be a kernel object right? if so the
> sendtargets test not leaking means that this is a problem in the
> session/connection kernel struct setup/destruction. The sendtargets code
> is all in userspace so it would not leak in those objects.
>
>
> I was out of the office sick last week, so let me catch up on work stuff
> then I will try to send a patch. If you want you could try to stick
> printks in iscsi driver model object release functions to make sure they
> are getting fired, but that gets nasty.
>
>
Ok, I still have my hands full with LIO v3.0 code atm, so I am not sure
how soon I could get to this..
--nab
> >
> > Please let me know if there is anything else I can do to help diagnose
> > the issue.
> >
> > Many thanks for your most valuable of time,
> >
> > --nab
> >
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1
2009-01-13 0:27 ` Nicholas A. Bellinger
@ 2009-01-13 1:24 ` Mike Christie
2009-01-13 2:41 ` Mike Christie
0 siblings, 1 reply; 6+ messages in thread
From: Mike Christie @ 2009-01-13 1:24 UTC (permalink / raw)
To: Nicholas A. Bellinger; +Cc: Open iSCSI, linux-scsi, Hannes Reinecke
Nicholas A. Bellinger wrote:
> Ok, I still have my hands full with LIO v3.0 code atm, so I am not sure
> how soon I could get to this..
>
Don't worry about it. I can replicate it here.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1
2009-01-13 1:24 ` Mike Christie
@ 2009-01-13 2:41 ` Mike Christie
2009-01-13 18:32 ` Nicholas A. Bellinger
0 siblings, 1 reply; 6+ messages in thread
From: Mike Christie @ 2009-01-13 2:41 UTC (permalink / raw)
To: open-iscsi; +Cc: Nicholas A. Bellinger, linux-scsi, Hannes Reinecke
[-- Attachment #1: Type: text/plain, Size: 433 bytes --]
Mike Christie wrote:
> Nicholas A. Bellinger wrote:
>
>> Ok, I still have my hands full with LIO v3.0 code atm, so I am not sure
>> how soon I could get to this..
>>
>
> Don't worry about it. I can replicate it here.
>
Crazy, I guess it is a leak we always had (at least back to 2.6.22 (did
not look firther)), and no one noticed. Here is a patch. I will send it
to James with some other patches I had.
Thanks for the report.
[-- Attachment #2: libiscsi-fix-pool-queue-leak.patch --]
[-- Type: text/plain, Size: 335 bytes --]
diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c
index 7225b6e..257c241 100644
--- a/drivers/scsi/libiscsi.c
+++ b/drivers/scsi/libiscsi.c
@@ -1981,6 +1981,7 @@ void iscsi_pool_free(struct iscsi_pool *q)
kfree(q->pool[i]);
if (q->pool)
kfree(q->pool);
+ kfree(q->queue);
}
EXPORT_SYMBOL_GPL(iscsi_pool_free);
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1
2009-01-13 2:41 ` Mike Christie
@ 2009-01-13 18:32 ` Nicholas A. Bellinger
0 siblings, 0 replies; 6+ messages in thread
From: Nicholas A. Bellinger @ 2009-01-13 18:32 UTC (permalink / raw)
To: open-iscsi; +Cc: linux-scsi, Hannes Reinecke
On Mon, 2009-01-12 at 20:41 -0600, Mike Christie wrote:
> Mike Christie wrote:
> > Nicholas A. Bellinger wrote:
> >
> >> Ok, I still have my hands full with LIO v3.0 code atm, so I am not sure
> >> how soon I could get to this..
> >>
> >
> > Don't worry about it. I can replicate it here.
> >
>
> Crazy, I guess it is a leak we always had (at least back to 2.6.22 (did
> not look firther)), and no one noticed. Here is a patch. I will send it
> to James with some other patches I had.
>
> Thanks for the report.
>
Thanks Mike!!!
--nab
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "open-iscsi" group.
> To post to this group, send email to open-iscsi@googlegroups.com
> To unsubscribe from this group, send email to open-iscsi+unsubscribe@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/open-iscsi
> -~----------~----~----~----~------~----~------~--~---
>
> plain text document attachment (libiscsi-fix-pool-queue-leak.patch)
> diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c
> index 7225b6e..257c241 100644
> --- a/drivers/scsi/libiscsi.c
> +++ b/drivers/scsi/libiscsi.c
> @@ -1981,6 +1981,7 @@ void iscsi_pool_free(struct iscsi_pool *q)
> kfree(q->pool[i]);
> if (q->pool)
> kfree(q->pool);
> + kfree(q->queue);
> }
> EXPORT_SYMBOL_GPL(iscsi_pool_free);
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-01-13 18:33 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-01-09 22:22 [Open/iSCSI] Memory leak in repetitive --login/--logout with v2.0-870.1 Nicholas A. Bellinger
2009-01-12 2:02 ` Mike Christie
2009-01-13 0:27 ` Nicholas A. Bellinger
2009-01-13 1:24 ` Mike Christie
2009-01-13 2:41 ` Mike Christie
2009-01-13 18:32 ` Nicholas A. Bellinger
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox