Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

[FUJITA Tomonori]

From: Arjan van de Ven <arjan@infradead.org>
Subject: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software
Date: Tue, 01 Mar 2005 10:46:03 +0100

> fsync or msync() ? I would imagine the target mmaping it's backend in
> userspace and using msync() to kick off IO. At which point it's not that
> much different from the control you do of the pagecache from inside the
> kernel...

Can we avoid calling mmap() and munmap() repeatedly with large disk?


> > When the target calls fsync(), dirty page cache is supposed to be
> > committed to disk. However, the disk drive uses write-back policy, it
> > is not. The data is still in disk drive cache. There is no system call
> > to control disk drive cache. So the target (in the user space) cannot
> > make good use of it.
> 
> fsync() (and I suppose msync()) nowadays send a "flush cache" command to
> the physical disk as well. This is new since 2.6.9 or so.
> 
> > The current code also assumes the disk drive uses write-through
> > policy. This is because no handy vfs interface for controlling disk
> > drive cache. I think that there is some room for further improvement
> > in the Linux kernel for storage systems.
> 
> that's already present since 2.6.9.....

Thanks a lot. I've not noticed these changes. I'll see the code later.

Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

[Arjan van de Ven]

On Tue, 2005-03-01 at 19:22 +0900, FUJITA Tomonori wrote:
> From: Arjan van de Ven <arjan@infradead.org>
> Subject: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software
> Date: Tue, 01 Mar 2005 10:46:03 +0100
> 
> > fsync or msync() ? I would imagine the target mmaping it's backend in
> > userspace and using msync() to kick off IO. At which point it's not that
> > much different from the control you do of the pagecache from inside the
> > kernel...
> 
> Can we avoid calling mmap() and munmap() repeatedly with large disk?

my server has 512Gb address space with 2.6.9/2.6.10, and a lot more than
that with the 2.6.11 kernel (4 level page tables rock). So the answer
would be yes.

(and on old servers without 64 bit, you indeed need to mmap/munmap
lazily to create a window, but I suspect that the 3 Gb of address space
you have there can be managed smart to minimize the number of unmaps if
you really try)

Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

[Arjan van de Ven]

On Tue, 2005-03-01 at 11:33 +0100, Arjan van de Ven wrote:
> On Tue, 2005-03-01 at 19:22 +0900, FUJITA Tomonori wrote:
> > From: Arjan van de Ven <arjan@infradead.org>
> > Subject: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software
> > Date: Tue, 01 Mar 2005 10:46:03 +0100
> > 
> > > fsync or msync() ? I would imagine the target mmaping it's backend in
> > > userspace and using msync() to kick off IO. At which point it's not that
> > > much different from the control you do of the pagecache from inside the
> > > kernel...
> > 
> > Can we avoid calling mmap() and munmap() repeatedly with large disk?
> 
> my server has 512Gb address space with 2.6.9/2.6.10, and a lot more than
> that with the 2.6.11 kernel (4 level page tables rock). So the answer
> would be yes.
> 
> (and on old servers without 64 bit, you indeed need to mmap/munmap
> lazily to create a window, but I suspect that the 3 Gb of address space
> you have there can be managed smart to minimize the number of unmaps if
> you really try)

note that on 32 bit servers the kernel side needs to do kmap() on the
pages anyway, and that a kmap/kunmap series is very much equivalent to a
mmap/munmap series in lots of ways, so I doubt that has many additional
savings for doing it in kernel space.

Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

[Libor Vanek]

Arjan van de Ven wrote:

>On Tue, 2005-03-01 at 19:22 +0900, FUJITA Tomonori wrote:
>  
>
>>From: Arjan van de Ven <arjan@infradead.org>
>>Subject: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software
>>Date: Tue, 01 Mar 2005 10:46:03 +0100
>>
>>    
>>
>>>fsync or msync() ? I would imagine the target mmaping it's backend in
>>>userspace and using msync() to kick off IO. At which point it's not that
>>>much different from the control you do of the pagecache from inside the
>>>kernel...
>>>      
>>>
>>Can we avoid calling mmap() and munmap() repeatedly with large disk?
>>    
>>
>
>my server has 512Gb address space with 2.6.9/2.6.10, and a lot more than
>that with the 2.6.11 kernel (4 level page tables rock). So the answer
>would be yes.
>
>(and on old servers without 64 bit, you indeed need to mmap/munmap
>lazily to create a window, but I suspect that the 3 Gb of address space
>you have there can be managed smart to minimize the number of unmaps if
>you really try)
>  
>
I don't know in detail what are you talking about (if whole disk must 
fit address space) but please consider we're speaking about TBs (10-20 
TB RAID is quite cheap nowadays with 400 GB SATA disks).

-- 
Best regards,
Libor Vanek

Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

[Arjan van de Ven]

On Tue, 2005-03-01 at 11:48 +0100, Libor Vanek wrote:
> >
> I don't know in detail what are you talking about (if whole disk must 
> fit address space) but please consider we're speaking about TBs (10-20 
> TB RAID is quite cheap nowadays with 400 GB SATA disks).

so? if you need one map/unmap per terabyte, the cost of that is like
zero.

Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

[FUJITA Tomonori]

From: Arjan van de Ven <arjan@infradead.org>
Subject: Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target	software
Date: Tue, 01 Mar 2005 11:46:32 +0100

> note that on 32 bit servers the kernel side needs to do kmap() on the
> pages anyway, and that a kmap/kunmap series is very much equivalent to a
> mmap/munmap series in lots of ways, so I doubt that has many additional
> savings for doing it in kernel space.

The code uses the vfs interface, kmap_atomic() is used instead of
kmap(). kmap_atomic() is much faster kmap().

Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

[Ming Zhang]

On Tue, 2005-03-01 at 14:01, Vikas Aggarwal wrote:
> And If  future IET be visioned as an Enterprise Class Array(Multiple
> Host-Side Adapters ie., FAs + Multiple Device Side Adapters ie., DAs),
> should better be in direct control of all the system-resources without
> being pushed out the kernel.
> 
of course this is the reason why we use that name.

ming

> Bryan Henderson <hbryan@us.ibm.com> wrote:
>         One thing that's implicit in your reasons for wanting to be in
>         the kernel 
>         is that you've chosen to exploit the kernel's page cache. As a
>         user of 
>         the page cache, you have more control from inside the kernel
>         than from 
>         user space. The page cache was designed to be fundamentally
>         invisible to 
>         user space.
>         
>         A pure user space implementation of an ISCSI target would use
>         process 
>         virtual memory for a cache and manage it itself. It would
>         access the 
>         storage with direct I/O. It looks to me like this is aimed at
>         a 
>         single-application Linux system (the whole system is just an
>         ISCSI 
>         target), which means there's not much need for a kernel to
>         manage shared 
>         resources.
>         
>         --
>         Bryan Henderson IBM Almaden Research Center
>         San Jose CA Filesystems
>         
>         
>         -------------------------------------------------------
>         SF email is sponsored by - The IT Pro duct Guide
>         Read honest & candid reviews on hundreds of IT Products from
>         real users.
>         Discover which products truly live up to the hype. Start
>         reading now.
>         http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
>         _______________________________________________
>         Iscsitarget-devel mailing list
>         Iscsitarget-devel@lists.sourceforge.net
>         https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel
> 
> ______________________________________________________________________
> Do you Yahoo!?
> Read only the mail you want - Yahoo! Mail SpamGuard.

Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

[FUJITA Tomonori]

From: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Subject: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software
Date: Tue, 01 Mar 2005 18:35:49 +0900

> The last reason is that user-space cost like memory copy. With 1Gbs
> Ethernet, is is not critical. However, with 10G, it is critical, I
> expect. I've been setting up 10G experimental infrastructure to
> evaluate iSCSI performance.

If we try to build high-performance iSCSI target software on 10Gbs
Ethernet, I think that we need to implement it in the kernel
space, although this topic is still in the research stage.

IICR, Intel provides open-source iSCSI target software, which uses
mmap() in the user space, as you suggested. However, they chose a
different approach to implement iSCSI target software for 10Gbs. They
modified the Linux kernel TCP stack to fully integrate iSCSI
functionality with it. It is still not clear how to get the best
performance out of 10GBs Ethernet, however, it seems that using just
the socket interface is not good enough. So I think that we need to
control all the system-resources to do it.

We've not optimized our code yet. The current code simply uses the
socket interface, however, moving some functionality to the TCP stack
can improve the performance, although I don't plan to modify the TCP
stack. A possible approach to do it is exploiting sk_data_ready().

In addition, you can find several papers (e.g., Chelsio technical
papers) saying that UP kernel can achieve better throughput than SMP
kernels with 10Gbs. There is some room for further improvement in the
Linux kernel.

Re: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software

[Dmitry Yusupov]

On Wed, 2005-03-02 at 14:04 +0900, FUJITA Tomonori wrote:
> From: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
> Subject: [Iscsitarget-devel] Re: [ANNOUNCE] iSCSI enterprise target software
> Date: Tue, 01 Mar 2005 18:35:49 +0900
> 
> > The last reason is that user-space cost like memory copy. With 1Gbs
> > Ethernet, is is not critical. However, with 10G, it is critical, I
> > expect. I've been setting up 10G experimental infrastructure to
> > evaluate iSCSI performance.
> 
> If we try to build high-performance iSCSI target software on 10Gbs
> Ethernet, I think that we need to implement it in the kernel
> space, although this topic is still in the research stage.
> 
> IICR, Intel provides open-source iSCSI target software, which uses
> mmap() in the user space, as you suggested. However, they chose a
> different approach to implement iSCSI target software for 10Gbs. They
> modified the Linux kernel TCP stack to fully integrate iSCSI
> functionality with it. It is still not clear how to get the best
> performance out of 10GBs Ethernet, however, it seems that using just
> the socket interface is not good enough. So I think that we need to
> control all the system-resources to do it.
> 
> We've not optimized our code yet. The current code simply uses the
> socket interface, however, moving some functionality to the TCP stack
> can improve the performance, although I don't plan to modify theu TCP
> stack. A possible approach to do it is exploiting sk_data_ready().

www.open-iscsi.org project uses this technique on receiving path and
shows very nice performance numbers for Read operations comparable with
HW accelerators.

To add such a functionality to an existing project will probably require
total re-design.

Regards,
Dima