Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Vladislav Bolkhovitin]

(My apologies that I'm late to this discussion. I wasn't subscribed to 
ksummit-2008-discuss, so wasn't aware about it. Big thanks to Nicholas 
who pointed me on it. For the same reason I will reply to all related 
messages at once. Sorry, I wasn't able to restore all the CC's. I hope, 
nobody would object that I added scst-devel and linux-scsi mailing lists 
on CC. I added them, because I understand that ksummit-2008-discuss 
mailing list isn't the best place for such discussion.)

> The main complaint about STGT is that it's mostly based in user space.
> With the big objection being that this makes it hard to share the
> in-kernel packet processing pieces between the in-kernel initiators and
> the in-user targets.  (Conversely, this in userspace thing is one reason
> why developers like STGT: it's easy to get at the protocol stacks and
> inject things).

Unfortunately, you keep ignoring the essential point that STGT is 
fundamentally based on microkernel-like distributed processing 
architecture, from which Linux gone from the very beginning, because it 
makes things a lot more complicated for virtually no gain. See below 
about buffers memory manager problem as an example of such 
overcomplication. As soon as there is a boundary in processing, there 
are immediately all that boundary related troubles.

> The solution that seemed to secure agreement at the storage summit was
> to enhance STGT with the in-kernel pieces of SCST to allow for both
> in-user and in-kernel targets depending on what the stack user
> wanted ... but I've seen no patches for this.

No surprises, because it is practically impossible. A SCSI target core, 
basically, consist from 3 parts: (1) SCSI target state machine, (2) 
buffers memory manager and (3) commands execution backend. The 
fundamental difference between SCST and STGT, which makes them so widely 
incompatible is the buffers memory manager. In SCST it's fully in the 
kernel, but for STGT it's in the user space. Direct consequence of this 
that the user space must be able to see buffers, so many optimizations, 
like putting pages directly to page cache, are impossible for security 
reasons. Additionally, the buffers memory manager is a central piece of 
code around which the whole SCSI target engine is built. Parts of code 
from SCST would be compatible with STGT only when the whole memory 
manager would be moved in STGT from user space to kernel space together 
with all related SCSI target state machine states, which in turn carry 
almost all other target states to the kernel. But then the questions 
would arise: "What's left for user space?" and "What's the difference 
with SCST"?

> The problem is that there are two separate constituencies for a target
> mode implementation
> 
>      1. People who need target mode to hack on storage fabrics. For them
>         a user mode implementation is much more easily hackable than a
>         kernel one.
>      2. People who want maximal efficiency in the target.  For them,
>         shoving everything into the kernel seems to be the best thing to
>         do.
> 
> The problem is that the currently upstream STGT implementation satisfies
> the first constituency.  The loud clamour for upstreaming SCST is to
> satisfy the second.  Amazingly enough, I think both constituencies have
> a right to a solution.  I also don't think two separate solutions with
> non overlapping code bases in the kernel is the right thing to do, so
> what I'd like is for us to evolve the existing solution into something
> that will work for both (by taking pieces of the out of SCST and placing
> them into STGT).
> 
> The problem is that proponents of constituency number 2 term the code
> for constituency 1 "flawed" because they don't see anything in user
> space satisfying their efficiency need.  Until we can get past this
> either 1 or 2 mentality there is going to be no progress because
> "progress" is offered as take it or leave it solutions for either
> constituency.
> 
> The solution I offered that I think might satisfy both is the ability to
> run pieces of the current STGT in-kernel to get the efficiency gains.  I
> haven't heard any other solutions proposed that would satisfy both ...
> unless I've missed something?

James, you missed that SCST for a long time has support for implementing 
backstorage devices in the user space. This is provided through 
backstorage device handler called scst_user. You can find its interface 
description in http://scst.sourceforge.net/scst_user_spec.txt and the 
example code in http://scst.sourceforge.net/handler_fileio_tgt.html. And 
one interesting point about it that it's more efficient than STGT, 
because SCST has memory manager in kernel, which eliminates the need in 
constant malloc()/get_user_pages()/free() calls on each SCSI command and 
allows on average only one context switch and one user-kernel switch per 
command. At least two companies sell VTLs based on SCST/scst_user and 
several developing their products.

Thus, SCST satisfies both kinds of people: ones who need maximal 
efficiency and ones who want more easily hackable implementation of 
their devices.

If all necessary pieces of STGT moved in the kernel, it would become 
pretty much the same as SCST at the moment. Then, for what all that 
effort would be? How many years would it take? Wouldn't it be simpler to 
grab already existing program, than to solve already solved problems one 
more time?

> The basic comment is that since we've already discussed it at the
> storage summit, and there's been no code progress at all since that
> time, what benefit is to be gained by airing it with a wider audience
> (particularly a less storage centric one)?

Related to SCST, I'm going to submit it together with 3 target drivers 
(for iSCSI, FC and SRPT transports) for review and inclusion after I 
return from vacation in the second half of September. Almost all is 
already ready for that, I simply don't want to disappear in the middle 
of discussion.

> Should the generic target mode be based on SCSI (like SCST), or, should
> it sit _ABOVE_ drivers/scsi, and allow for a generic method for both
> SCSI and non SCSI based transports. 

I proposed for a SCSI target a separate subsystem, which I described in 
http://lkml.org/lkml/2008/5/2/348 and http://lkml.org/lkml/2008/5/4/160. 
There is almost no common between regular (initiator) SCSI and target 
subsystems. Relations between them similar to relations between regular 
SCSI and block subsystems. Hence, there is no point to keep them 
together. It would be better if they are separated with separate 
maintainers.

> I believe the LIO-Core algoritims
> for allocation of linked list pages subsystem dependent memory are ideal
> for the latter, which is what should be the princaple topic discussion
> regardless kernel-level generic target mode infrastructure.   That is,
> common code that can be used for both target mode iSCSI, FC, FCoE,
> PSCSI, AoE and SATA.

I've also thought how to make SCST core useful for non-SCSI transports, 
namely AoE. It seems to me that the best way would be if for all AoE 
commands AoE target driver (let's use SCSI terms) would submit to SCST 
core the corresponding generated by it SCSI CDBs. Since ATA can be 
considered as a functional subset of SCSI, such approach should work 
pretty well. Other transports (FCoE and SATA) are, basically, SCSI, so 
they can be used directly.

> Hmm, really? STGT development is more active IET.

Tomo-san, be honest. It was *you*, who stopped the IET development, 
forcing people to switch to STGT. There is no point to prepare patches 
for IET, if they will never get merged.

> That sounds like a perfect topic for the scsi mailing list, why not take
> it there right now and get it resolved this week?

I agree. Simply there are too few people who simply understand what 
"SCSI target" means and there are no big chances to have sufficient 
amount of them there.

Vlad

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[FUJITA Tomonori]

On Thu, 21 Aug 2008 23:42:51 +0400
Vladislav Bolkhovitin <vst@vlnb.net> wrote:

> > Hmm, really? STGT development is more active IET.
> 
> Tomo-san, be honest. It was *you*, who stopped the IET development, 
> forcing people to switch to STGT. There is no point to prepare patches 
> for IET, if they will never get merged.

I stopped working on IET but it doesn't mean that I force others to
stop working on IET. Arne said that he merges patches.

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Vladislav Bolkhovitin]

FUJITA Tomonori wrote:
> On Thu, 21 Aug 2008 23:42:51 +0400
> Vladislav Bolkhovitin <vst@vlnb.net> wrote:
> 
>>> Hmm, really? STGT development is more active IET.
>> Tomo-san, be honest. It was *you*, who stopped the IET development, 
>> forcing people to switch to STGT. There is no point to prepare patches 
>> for IET, if they will never get merged.
> 
> I stopped working on IET but it doesn't mean that I force others to
> stop working on IET. Arne said that he merges patches.

Well, how about declaring unlimited code freeze for all core parts? Has 
anybody, except you, had a right to merge patches to IET? Which %% of 
coming non-trivial patches have you merged in the last few years? Near 
zero? For instance, how many not too bad patches from Ross S. W. Walker 
were silently ignored? He didn't received even small comments, which 
made him so frustrated, so he stopped all the development activities?

Vlad

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[FUJITA Tomonori]

On Fri, 22 Aug 2008 23:01:51 +0400
Vladislav Bolkhovitin <vst@vlnb.net> wrote:

> FUJITA Tomonori wrote:
> > On Thu, 21 Aug 2008 23:42:51 +0400
> > Vladislav Bolkhovitin <vst@vlnb.net> wrote:
> > 
> >>> Hmm, really? STGT development is more active IET.
> >> Tomo-san, be honest. It was *you*, who stopped the IET development, 
> >> forcing people to switch to STGT. There is no point to prepare patches 
> >> for IET, if they will never get merged.
> > 
> > I stopped working on IET but it doesn't mean that I force others to
> > stop working on IET. Arne said that he merges patches.
> 
> Well, how about declaring unlimited code freeze for all core parts?

Why is it difficult for you to understand what I wrote? I wrote again:

I stopped working on IET but it doesn't mean that I force others to
stop working on IET.


> Has anybody, except you, had a right to merge patches to IET?

Nobody asked me to give a right to merge patches to IET. When Arne
asked me, I was happy to give it him because he has a good knowledge
of IET code.


> Which %% of 
> coming non-trivial patches have you merged in the last few years? Near 
> zero?

Probably, near zero because I stopped working on IET. What does 'stop
working on something' mean for you?


> For instance, how many not too bad patches from Ross S. W. Walker 
> were silently ignored? He didn't received even small comments, which 
> made him so frustrated, so he stopped all the development activities?

Several days ago on the mailing list, I saw that Ross works on IET
with Arne.


I have no idea why you think that you know better than me about the
project that I had maintained. But can you please stop talking about
wrong information? I can just ignore your wrong statements (I often
did) but I don't like to see other people misunderstanding.

Thanks,

Re: Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Vladislav Bolkhovitin]

FUJITA Tomonori wrote:
>>>>> Hmm, really? STGT development is more active IET.
>>>> Tomo-san, be honest. It was *you*, who stopped the IET development, 
>>>> forcing people to switch to STGT. There is no point to prepare patches 
>>>> for IET, if they will never get merged.
>>> I stopped working on IET but it doesn't mean that I force others to
>>> stop working on IET. Arne said that he merges patches.
>> Well, how about declaring unlimited code freeze for all core parts?
> 
> Why is it difficult for you to understand what I wrote? I wrote again:
> 
> I stopped working on IET but it doesn't mean that I force others to
> stop working on IET.

You stopped working on it, but you didn't stop maintaining it, did you? 
The were neither announcement of it, nor request for somebody else to do 
the maintainership work. So, for everybody you remained the maintainer 
and the only person who can accept or reject patches.

>> Has anybody, except you, had a right to merge patches to IET?
> 
> Nobody asked me to give a right to merge patches to IET. When Arne
> asked me, I was happy to give it him because he has a good knowledge
> of IET code.

Great position! You put yourself as misunderstood. But the problem that 
everybody misunderstood you and it's really hard to believe that you 
wasn't aware of it.

In fact, it was *you* who should have asked about the replacement. It's 
pretty regular practice in Open Source, when people get bored of some 
project and gracefully pass its maintainership to someone other.

>> Which %% of 
>> coming non-trivial patches have you merged in the last few years? Near 
>> zero?
> 
> Probably, near zero because I stopped working on IET. What does 'stop
> working on something' mean for you?
> 
> 
>> For instance, how many not too bad patches from Ross S. W. Walker 
>> were silently ignored? He didn't received even small comments, which 
>> made him so frustrated, so he stopped all the development activities?
> 
> Several days ago on the mailing list, I saw that Ross works on IET
> with Arne.
> 
> 
> I have no idea why you think that you know better than me about the
> project that I had maintained. But can you please stop talking about
> wrong information? I can just ignore your wrong statements (I often
> did) but I don't like to see other people misunderstanding.

This isn't only my impression, this is rather what people using IET 
think. Everyone after digging in IET mailing list archive can find the 
evidences. For instance, see this e-mail: 
http://www.nabble.com/forum/ViewPost.jtp?post=13776875&framed=y
You didn't disprove it, although definitely read, because you replied on 
the later message in that thread. So, what one should think after that 
and after seeing a lot of good patches going to nowhere and only trivial 
ones merged?

> Thanks,
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Scst-devel mailing list
> Scst-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scst-devel
> 

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Mike Christie]

snip

>> non overlapping code bases in the kernel is the right thing to do, so
>> what I'd like is for us to evolve the existing solution into something
>> that will work for both (by taking pieces of the out of SCST and placing
>> them into STGT).
>>

snip

> 
> James, you missed that SCST for a long time has support for implementing 
> backstorage devices in the user space. This is provided through 
> backstorage device handler called scst_user. You can find its interface 

Does scst support moving the scsi state machine there? I think that is 
what James meant by some of his comments, but I do not think this detail 
is too important. If you can, great one less todo item.

> 
> If all necessary pieces of STGT moved in the kernel, it would become 
> pretty much the same as SCST at the moment. 

In the top paragraph James is saying to move scst into stgt, so it seems 
like we can do the following:

1. You and with Nick battle each other about what are the best pieces of 
scst and his target and what should go upstream.

2. Do a ibm vscsi target for the new framework since that is the only 
upstream kernel target right now. For qla2xxx, we can forget the patches 
we sent qlogic and use what comes with the common framework since scst 
has one already.

3. Start sending patches for common code like scatterlist improvements 
and scatterlist memory reservations.

4. Send patches for new target infrastructure core code for review and 
cleanup. And send ibm vscsi target driver for an example and to make 
sure there are no functionality regressions. The latter should not be 
too hard because stgt does not have many features right? :)

5. When common code is merged and new core target infrastructure is 
through the review process we can just swap out the new code for stgt 
and name it whatever you want.

Tomo and I can handle trying to modify the new framework to support 
putting a scsi state machine in userspace and sharing that with kernel 
code. Right now the primary targets for stgt that users are deploying 
are are completely in userspace, so we do not have much to worry about. 
There are ibm vscsi users, but they are a lot smaller in number compared 
to iscsi. I would bet the ibmvscsi would prefer to use the kernel target 
we make too.

Our userspace tools will also be able to support both kernel space and 
userspace targets with little trouble so distros that have stgt will not 
notice any differences.

Surely this will be faster than writing all these mails and wasting time 
on this :)

I think James also said something about moving STGT in-kernel to get 
performance gains, but I do not think it means that we have to push 
exact code that sits in Tomo's git tree from usrspace into the kernel. 
If along the way we replace it with scst or Nick's code and we end up 
with a variant of scst or Nicks code that can still support userspace 
targets then I do not think any one is going to make long threads like 
these have resulted in :)

Will this work for everyone?

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[James Bottomley]

On Thu, 2008-08-21 at 18:31 -0500, Mike Christie wrote:
> I think James also said something about moving STGT in-kernel to get 
> performance gains, but I do not think it means that we have to push 
> exact code that sits in Tomo's git tree from usrspace into the kernel. 
> If along the way we replace it with scst or Nick's code and we end up 
> with a variant of scst or Nicks code that can still support userspace 
> targets then I do not think any one is going to make long threads like 
> these have resulted in :)

I meant actually allowing performance critical pieces to work either
in-user or in-kernel.  How, I'm not sure ... if we could use the same
code for both, that would be brilliant ... if we have to have separate
pieces, that will be OK.

The error injection and transport debug people think it's important to
have the state machine in user space for fast prototyping and debugging,
so I'm not going to take this away from them.

> Will this work for everyone?

Sounds like a plan.  (However, it also sounds suspiciously like the last
plan we had from the storage summit which didn't actually attract any
implementers ...)

James

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Vladislav Bolkhovitin]

James Bottomley wrote:
> On Thu, 2008-08-21 at 18:31 -0500, Mike Christie wrote:
>> I think James also said something about moving STGT in-kernel to get 
>> performance gains, but I do not think it means that we have to push 
>> exact code that sits in Tomo's git tree from usrspace into the kernel. 
>> If along the way we replace it with scst or Nick's code and we end up 
>> with a variant of scst or Nicks code that can still support userspace 
>> targets then I do not think any one is going to make long threads like 
>> these have resulted in :)
> 
> I meant actually allowing performance critical pieces to work either
> in-user or in-kernel.  How, I'm not sure ... if we could use the same
> code for both, that would be brilliant ... if we have to have separate
> pieces, that will be OK.
> 
> The error injection and transport debug people think it's important to
> have the state machine in user space for fast prototyping and debugging,
> so I'm not going to take this away from them.

Nobody has been asking you about that. Simply, there's no need in it.

>> Will this work for everyone?
> 
> Sounds like a plan.  (However, it also sounds suspiciously like the last
> plan we had from the storage summit which didn't actually attract any
> implementers ...)

I at that time heard nothing about it. Nobody asked me, nobody let me 
know about it and nobody asked me to participate. Nothing about it was 
in linux-scsi. It was completely behind my back.

Vlad

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[James Bottomley]

On Fri, 2008-08-22 at 23:00 +0400, Vladislav Bolkhovitin wrote:
> James Bottomley wrote:
> > On Thu, 2008-08-21 at 18:31 -0500, Mike Christie wrote:
> >> I think James also said something about moving STGT in-kernel to get 
> >> performance gains, but I do not think it means that we have to push 
> >> exact code that sits in Tomo's git tree from usrspace into the kernel. 
> >> If along the way we replace it with scst or Nick's code and we end up 
> >> with a variant of scst or Nicks code that can still support userspace 
> >> targets then I do not think any one is going to make long threads like 
> >> these have resulted in :)
> > 
> > I meant actually allowing performance critical pieces to work either
> > in-user or in-kernel.  How, I'm not sure ... if we could use the same
> > code for both, that would be brilliant ... if we have to have separate
> > pieces, that will be OK.
> > 
> > The error injection and transport debug people think it's important to
> > have the state machine in user space for fast prototyping and debugging,
> > so I'm not going to take this away from them.
> 
> Nobody has been asking you about that. Simply, there's no need in it.

I wasn't aware you were privy to my conversations ... however, I suggest
that you must have missed it.

Quite a few people who want to work on transports don't have the budget
for the hardware.  Emulators are things they use to get around this
problem.  To them, therefore, it's a definite need.

> >> Will this work for everyone?
> > 
> > Sounds like a plan.  (However, it also sounds suspiciously like the last
> > plan we had from the storage summit which didn't actually attract any
> > implementers ...)
> 
> I at that time heard nothing about it. Nobody asked me, nobody let me 
> know about it and nobody asked me to participate. Nothing about it was 
> in linux-scsi. It was completely behind my back.

http://www.usenix.org/events/lsf08/

summaries page 6.

The invitation to participate was here:

http://marc.info/?t=119325401900042

James

Re: Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Vladislav Bolkhovitin]

James Bottomley wrote:
> On Fri, 2008-08-22 at 23:00 +0400, Vladislav Bolkhovitin wrote:
>> James Bottomley wrote:
>>> On Thu, 2008-08-21 at 18:31 -0500, Mike Christie wrote:
>>>> I think James also said something about moving STGT in-kernel to get 
>>>> performance gains, but I do not think it means that we have to push 
>>>> exact code that sits in Tomo's git tree from usrspace into the kernel. 
>>>> If along the way we replace it with scst or Nick's code and we end up 
>>>> with a variant of scst or Nicks code that can still support userspace 
>>>> targets then I do not think any one is going to make long threads like 
>>>> these have resulted in :)
>>> I meant actually allowing performance critical pieces to work either
>>> in-user or in-kernel.  How, I'm not sure ... if we could use the same
>>> code for both, that would be brilliant ... if we have to have separate
>>> pieces, that will be OK.
>>>
>>> The error injection and transport debug people think it's important to
>>> have the state machine in user space for fast prototyping and debugging,
>>> so I'm not going to take this away from them.
>> Nobody has been asking you about that. Simply, there's no need in it.
> 
> I wasn't aware you were privy to my conversations ... however, I suggest
> that you must have missed it.
> 
> Quite a few people who want to work on transports don't have the budget
> for the hardware.  Emulators are things they use to get around this
> problem.  To them, therefore, it's a definite need.

Seems, there is a confusion and we are writing about different things. I 
mean backstorage device emulation in user space (this is what scst_user 
provides), but seems you mean the opposite side of SCSI commands 
processing in target: target transports, i.e. target drivers, in user 
space, which emulate some SCSI transport, correct?

>>>> Will this work for everyone?
>>> Sounds like a plan.  (However, it also sounds suspiciously like the last
>>> plan we had from the storage summit which didn't actually attract any
>>> implementers ...)
>> I at that time heard nothing about it. Nobody asked me, nobody let me 
>> know about it and nobody asked me to participate. Nothing about it was 
>> in linux-scsi. It was completely behind my back.
> 
> http://www.usenix.org/events/lsf08/
> 
> summaries page 6.
> 
> The invitation to participate was here:
> 
> http://marc.info/?t=119325401900042

I was aware about the event, although for obvious reasons wasn't able to 
participate. I meant not it, but the decision and the plan. Neither one 
of them was published in any public place, wasn't it?

> James
> 
> 
> 

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Vladislav Bolkhovitin]

Mike Christie wrote:
> snip
> 
>>> non overlapping code bases in the kernel is the right thing to do, so
>>> what I'd like is for us to evolve the existing solution into something
>>> that will work for both (by taking pieces of the out of SCST and placing
>>> them into STGT).
>>>
> 
> snip
> 
>> James, you missed that SCST for a long time has support for implementing 
>> backstorage devices in the user space. This is provided through 
>> backstorage device handler called scst_user. You can find its interface 
> 
> Does scst support moving the scsi state machine there? I think that is 
> what James meant by some of his comments, but I do not think this detail 
> is too important. If you can, great one less todo item.

Scst_user provides hooks for all the SCST core states. Some of them are 
optional, like for commands parsing, some are required, like for 
commands execution. Is it sufficient?

I would suggest you to look at fileio_tgt. It's a full functional SBC 
device emulator (more than full, actually, since it's also a test program).

>> If all necessary pieces of STGT moved in the kernel, it would become 
>> pretty much the same as SCST at the moment. 
> 
> In the top paragraph James is saying to move scst into stgt, so it seems 
> like we can do the following:
> 
> 1. You and with Nick battle each other about what are the best pieces of 
> scst and his target and what should go upstream.

We generally already agreed in Nick's iSCSI target over SCST core engine 
with some necessary modifications in it taken from LIO core.

> 2. Do a ibm vscsi target for the new framework since that is the only 
> upstream kernel target right now. For qla2xxx, we can forget the patches 
> we sent qlogic and use what comes with the common framework since scst 
> has one already.
> 
> 3. Start sending patches for common code like scatterlist improvements 
> and scatterlist memory reservations.

What do you mean?

> 4. Send patches for new target infrastructure core code for review and 
> cleanup. And send ibm vscsi target driver for an example and to make 
> sure there are no functionality regressions. The latter should not be 
> too hard because stgt does not have many features right? :)
> 
> 5. When common code is merged and new core target infrastructure is 
> through the review process we can just swap out the new code for stgt 
> and name it whatever you want.
> 
> Tomo and I can handle trying to modify the new framework to support 
> putting a scsi state machine in userspace and sharing that with kernel 
> code. Right now the primary targets for stgt that users are deploying 
> are are completely in userspace, so we do not have much to worry about. 
> There are ibm vscsi users, but they are a lot smaller in number compared 
> to iscsi. I would bet the ibmvscsi would prefer to use the kernel target 
> we make too.
> 
> Our userspace tools will also be able to support both kernel space and 
> userspace targets with little trouble so distros that have stgt will not 
> notice any differences.

So, do you propose to start converting STGT to SCST? Do I understand you 
correctly?

> Surely this will be faster than writing all these mails and wasting time 
> on this :)
> 
> I think James also said something about moving STGT in-kernel to get 
> performance gains, but I do not think it means that we have to push 
> exact code that sits in Tomo's git tree from usrspace into the kernel. 
> If along the way we replace it with scst or Nick's code and we end up 
> with a variant of scst or Nicks code that can still support userspace 
> targets then I do not think any one is going to make long threads like 
> these have resulted in :)
> 
> Will this work for everyone?

We need to agree at first which target's architecture we are going to 
use as a base in which we will incorporate the necessary additions from 
other targets. We with Nick have chosen SCST, now it's your, James and 
other interested people's turn. As I already wrote, STGT and its kernel 
part is too widely incompatible with SCST and choosing it as a base will 
lead to unneeded effort.

I would propose to return to this question after I submit SCST patches. 
Then we would see the code and decide, what's good in it and what's bad.

Vlad

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Arjan van de Ven]

Vladislav Bolkhovitin wrote:
> We need to agree at first which target's architecture we are going to 

what we need is for this thread to move to a more useful place than the kernel summit list ..
a polite request to everyone on the thread to remove the kernel summit list from the CC...

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Mike Christie]

Vladislav Bolkhovitin wrote:
>> 3. Start sending patches for common code like scatterlist improvements 
>> and scatterlist memory reservations.
> 
> What do you mean?

There were some nice functionality that Nick's target had and I thought 
scst had too that could be more generic like the ability to reservre 
memory for a scatterlist or basically make sure you can have a 
scatterlist that is within a drivers/hardwares limits and can fullfill 
the request. For exmaple if the target driver could only do X segments, 
but the command was 8 MBs then we need a scatterlist with big segments, 
and I thought you guys had nice to code to handle this. It is the same 
thing that is needed by the SCSI upper layer drives when chaining is not 
supported.

You should rip that out of the scst patch set or whatever Nicsk patchset 
is called and get it in right away to make it easier to review the 
target patches (less target code to argue about :)).


> 
>> 4. Send patches for new target infrastructure core code for review and 
>> cleanup. And send ibm vscsi target driver for an example and to make 
>> sure there are no functionality regressions. The latter should not be 
>> too hard because stgt does not have many features right? :)
>>
>> 5. When common code is merged and new core target infrastructure is 
>> through the review process we can just swap out the new code for stgt 
>> and name it whatever you want.
>>
>> Tomo and I can handle trying to modify the new framework to support 
>> putting a scsi state machine in userspace and sharing that with kernel 
>> code. Right now the primary targets for stgt that users are deploying 
>> are are completely in userspace, so we do not have much to worry 
>> about. There are ibm vscsi users, but they are a lot smaller in number 
>> compared to iscsi. I would bet the ibmvscsi would prefer to use the 
>> kernel target we make too.
>>
>> Our userspace tools will also be able to support both kernel space and 
>> userspace targets with little trouble so distros that have stgt will 
>> not notice any differences.
> 
> So, do you propose to start converting STGT to SCST? Do I understand you 
> correctly?

Will we ever finish these threads if we go the other way where we 
convert scst to stgt :) Yes, we will convert stgt to scst if that is how 
you want to word it. I think when people are saying evolve the code they 
do not mean that we have to force scst into stgt or the reverse. With 
James allowing performance critical stuff in the kernel we can add 
whatever is best. For the stgt functionalty that may not be support by 
scst or whatever is going to go in, then yeah, Tomo and I can handle that.

Re: Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Vladislav Bolkhovitin]

Mike Christie wrote:
> Vladislav Bolkhovitin wrote:
>>> 3. Start sending patches for common code like scatterlist improvements 
>>> and scatterlist memory reservations.
>> What do you mean?
> 
> There were some nice functionality that Nick's target had and I thought 
> scst had too that could be more generic like the ability to reservre 
> memory for a scatterlist or basically make sure you can have a 
> scatterlist that is within a drivers/hardwares limits and can fullfill 
> the request. For exmaple if the target driver could only do X segments, 
> but the command was 8 MBs then we need a scatterlist with big segments, 
> and I thought you guys had nice to code to handle this. It is the same 
> thing that is needed by the SCSI upper layer drives when chaining is not 
> supported.
> 
> You should rip that out of the scst patch set or whatever Nicsk patchset 
> is called and get it in right away to make it easier to review the 
> target patches (less target code to argue about :)).

Yes, there is such code. Sure, it can be extracted and posted as a 
separate patch, so other drivers would be able to use it as well (it 
isn't SCSI specific, so, I think, should be put somewhere in drivers/base/).

But, in fact, so far we have not argued a bit of code. We argue 
fundamental principles, so extracting SGV cache or Nick's analog won't 
actually change anything.

>>> 4. Send patches for new target infrastructure core code for review and 
>>> cleanup. And send ibm vscsi target driver for an example and to make 
>>> sure there are no functionality regressions. The latter should not be 
>>> too hard because stgt does not have many features right? :)
>>>
>>> 5. When common code is merged and new core target infrastructure is 
>>> through the review process we can just swap out the new code for stgt 
>>> and name it whatever you want.
>>>
>>> Tomo and I can handle trying to modify the new framework to support 
>>> putting a scsi state machine in userspace and sharing that with kernel 
>>> code. Right now the primary targets for stgt that users are deploying 
>>> are are completely in userspace, so we do not have much to worry 
>>> about. There are ibm vscsi users, but they are a lot smaller in number 
>>> compared to iscsi. I would bet the ibmvscsi would prefer to use the 
>>> kernel target we make too.
>>>
>>> Our userspace tools will also be able to support both kernel space and 
>>> userspace targets with little trouble so distros that have stgt will 
>>> not notice any differences.
>> So, do you propose to start converting STGT to SCST? Do I understand you 
>> correctly?
> 
> Will we ever finish these threads if we go the other way where we 
> convert scst to stgt :) Yes, we will convert stgt to scst if that is how 
> you want to word it. I think when people are saying evolve the code they 
> do not mean that we have to force scst into stgt or the reverse. With 
> James allowing performance critical stuff in the kernel we can add 
> whatever is best. For the stgt functionalty that may not be support by 
> scst or whatever is going to go in, then yeah, Tomo and I can handle that.

What not clear to me are the reasons behind decision to use STGT as a 
base point for future development. AFAIK, the only valuable advantage it 
has over SCST is ibmvscsi target driver. And that's all. ISER target 
driver doesn't need any kernel support, so can happily live without 
STGT. Correct me, if I wrong.

Usually people, choosing the starting point of some consolidating 
project among several similar ones, choose one, which is the closest to 
what they ultimately want to see in the finish. Apparently, SCST is a 
lot closer to that, than STGT, correct? So, then, why the choice was 
done to start from STGT? Definitely, this choice will add a lot of 
effort without any return. Or, am I missing something? There is nothing 
personal in my questions, I'm caring only about amount of needed effort.

Vlad

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Nicholas A. Bellinger]

On Thu, 2008-08-21 at 18:31 -0500, Mike Christie wrote:
> snip
> 

Ok, I took in the weekend to let things settle for a few moments.. 

> > 
> > If all necessary pieces of STGT moved in the kernel, it would become 
> > pretty much the same as SCST at the moment. 
> 
> In the top paragraph James is saying to move scst into stgt, so it seems 
> like we can do the following:
> 
> 1. You and with Nick battle each other about what are the best pieces of 
> scst and his target and what should go upstream.
> 

Ok, Vlad and I have discussed and determined that the process will be
something roughly along the lines of

*) Vlad releases upstream patches for SCST Core in the late v2.6.27 time
frame.
*) /me will cherry picks from lio-core-2.6.git specific code (namely the
memory allocation and mapping algorithms) and begin to integrate these
into SCST-Core upstream and get bulk I/O moving between SCST Core
kernel/user CDB generation API.
*) Around the same time, I will begin the porting process of moving
LIO-Target to use the new upstream target infrastructure, whatever it
shall be called.. :-)


> 2. Do a ibm vscsi target for the new framework since that is the only 
> upstream kernel target right now. For qla2xxx, we can forget the patches 
> we sent qlogic and use what comes with the common framework since scst 
> has one already.

This would be fun.. :-)  What hardware do I need to do this again..?

> 
> 3. Start sending patches for common code like scatterlist improvements 
> and scatterlist memory reservations.
> 

Ok, I have been thinking about how to go about this, and it seems to
make sense to incorporate these improvements from LIO-Core into SCST
Core during the first phase.  This codepath in LIO is historically known
as ICF_SCSI_DATA_SG_IO_CDB which handles the 1 -> N allocation and
mapping for the generic linked list pages -> contigious scatterlist
array memory mapping case.

This is also piece that can be generic for handling WRITE/READ I/Os with
LBA + SECTOR_COUNT + SECTOR_SIZE across the "SCSI-less" target cases.  I
guess putting it into SCST Core for now makes sense, espically if we
move SCST Core into supporting the SCSI-less target fabrics..

> 4. Send patches for new target infrastructure core code for review and 
> cleanup. And send ibm vscsi target driver for an example and to make 
> sure there are no functionality regressions. The latter should not be 
> too hard because stgt does not have many features right? :)
> 

Understood.

> 5. When common code is merged and new core target infrastructure is 
> through the review process we can just swap out the new code for stgt 
> and name it whatever you want.
> 

Heh, looks like we have to start thinking of a new name. :-)

> Tomo and I can handle trying to modify the new framework to support 
> putting a scsi state machine in userspace and sharing that with kernel 
> code. Right now the primary targets for stgt that users are deploying 
> are are completely in userspace, so we do not have much to worry about. 
> There are ibm vscsi users, but they are a lot smaller in number compared 
> to iscsi. I would bet the ibmvscsi would prefer to use the kernel target 
> we make too.
> 

Understood.

> Our userspace tools will also be able to support both kernel space and 
> userspace targets with little trouble so distros that have stgt will not 
> notice any differences.
> 

So this is one of the items that I mentioned to Vlad as being a
potential stumbling block..  I stated that I have no problem dumping the
LIO IOCTL in favor of something better, and having to break backwards
compat with LIO userspace tools (well, until I have to emulate the
backwards compat wrt target-ctl ;-)) is something I am ready to deal
with.

So I definately see your point here, having the exist CLI logic in place
into distros is something that we are going to have to deal with (even
if the core behind them is completely different).  

I should also mention that virtually all of the configuration logic done
from the LIO-Core and LIO-Target IOCTLs today is intended (minus a bug
or two :-) to be done without having to shutdown/stop anything other
than the referenced object for the target-ctl operation.  Obviously, I
really want to keep the same functionality with the CLI interface the
LIO pieces are ported to.

I will be sure to take a look at the STGT control interface and see how
I can fit the current functionality in.  I believe SCST uses an IOCTL
today as well, which would put them in a similar situation.


> Surely this will be faster than writing all these mails and wasting time 
> on this :)
> 
> I think James also said something about moving STGT in-kernel to get 
> performance gains, but I do not think it means that we have to push 
> exact code that sits in Tomo's git tree from usrspace into the kernel. 
> If along the way we replace it with scst or Nick's code and we end up 
> with a variant of scst or Nicks code that can still support userspace 
> targets then I do not think any one is going to make long threads like 
> these have resulted in :)
> 
> Will this work for everyone?
> 

Sounds good.  Many thanks for putting things into perspective mnc.

:-)

--nab

Re: Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Vladislav Bolkhovitin]

Nicholas A. Bellinger wrote:
> On Thu, 2008-08-21 at 18:31 -0500, Mike Christie wrote:
>> snip
>>
> 
> Ok, I took in the weekend to let things settle for a few moments.. 
> 
>>> If all necessary pieces of STGT moved in the kernel, it would become 
>>> pretty much the same as SCST at the moment. 
>> In the top paragraph James is saying to move scst into stgt, so it seems 
>> like we can do the following:
>>
>> 1. You and with Nick battle each other about what are the best pieces of 
>> scst and his target and what should go upstream.
>>
> 
> Ok, Vlad and I have discussed and determined that the process will be
> something roughly along the lines of
> 
> *) Vlad releases upstream patches for SCST Core in the late v2.6.27 time
> frame.
> *) /me will cherry picks from lio-core-2.6.git specific code (namely the
> memory allocation and mapping algorithms) and begin to integrate these
> into SCST-Core upstream and get bulk I/O moving between SCST Core
> kernel/user CDB generation API.
> *) Around the same time, I will begin the porting process of moving
> LIO-Target to use the new upstream target infrastructure, whatever it
> shall be called.. :-)

I agree

>> Our userspace tools will also be able to support both kernel space and 
>> userspace targets with little trouble so distros that have stgt will not 
>> notice any differences.
> 
> So this is one of the items that I mentioned to Vlad as being a
> potential stumbling block..  I stated that I have no problem dumping the
> LIO IOCTL in favor of something better, and having to break backwards
> compat with LIO userspace tools (well, until I have to emulate the
> backwards compat wrt target-ctl ;-)) is something I am ready to deal
> with.
> 
> So I definately see your point here, having the exist CLI logic in place
> into distros is something that we are going to have to deal with (even
> if the core behind them is completely different).  
> 
> I should also mention that virtually all of the configuration logic done
> from the LIO-Core and LIO-Target IOCTLs today is intended (minus a bug
> or two :-) to be done without having to shutdown/stop anything other
> than the referenced object for the target-ctl operation.  Obviously, I
> really want to keep the same functionality with the CLI interface the
> LIO pieces are ported to.
> 
> I will be sure to take a look at the STGT control interface and see how
> I can fit the current functionality in.  I believe SCST uses an IOCTL
> today as well, which would put them in a similar situation.

I, personally, don't like an interface, like tgtadm, which tries to do 
all non-trivial configuration work from a single command line tool, 
because of the following 3 reasons:

1. It's a lot of effort to write and maintain such a tool, because it 
needs to be extensible to work with new modules and modes. For instance, 
iptables tool is 86K lines long. The whole netfilter code for all 
network protocols in the kernel is less that 50K lines long. Do we want 
such a code bloat (170% of code to configure) and dedicated team of 
maintainers to solve a fundamentally simple configuration task?

2. It assumes the stateless type of configuration, when each call 
configures exactly one thing without any side effects on already 
configured or future entries. This approach is good for cases like 
iptables, but for SCSI targets it's possible that several configuration 
steps require to be done in an atomic manner, like adding an iSCSI 
target and configuring its parameters.

3. It's hard to read 5+ parameters in one command line, so it's a lot 
easier to make a mistake there.

So, I believe, a configuration interface should be rather /proc or /sys 
interface based. I don't think we should care much about backward 
compatibility with tgtadm, because the existing interface doesn't reach 
the state of being widely used. As I already mentioned, only ibmvscsi at 
the moment uses it, hardware for which is pretty rare.

Thus, I would suggest that before making the further move we should also 
consider configuration interfaces of SCST and LIO and choose the best 
things from all 3.

Vlad

Re: [Scst-devel] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Ming Zhang]

On Wed, 2008-08-27 at 21:56 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger wrote:
> > On Thu, 2008-08-21 at 18:31 -0500, Mike Christie wrote:
> >> snip
> >>
> > 
> > Ok, I took in the weekend to let things settle for a few moments.. 
> > 
> >>> If all necessary pieces of STGT moved in the kernel, it would become 
> >>> pretty much the same as SCST at the moment. 
> >> In the top paragraph James is saying to move scst into stgt, so it seems 
> >> like we can do the following:
> >>
> >> 1. You and with Nick battle each other about what are the best pieces of 
> >> scst and his target and what should go upstream.
> >>
> > 
> > Ok, Vlad and I have discussed and determined that the process will be
> > something roughly along the lines of
> > 
> > *) Vlad releases upstream patches for SCST Core in the late v2.6.27 time
> > frame.
> > *) /me will cherry picks from lio-core-2.6.git specific code (namely the
> > memory allocation and mapping algorithms) and begin to integrate these
> > into SCST-Core upstream and get bulk I/O moving between SCST Core
> > kernel/user CDB generation API.
> > *) Around the same time, I will begin the porting process of moving
> > LIO-Target to use the new upstream target infrastructure, whatever it
> > shall be called.. :-)
> 
> I agree
> 
> >> Our userspace tools will also be able to support both kernel space and 
> >> userspace targets with little trouble so distros that have stgt will not 
> >> notice any differences.
> > 
> > So this is one of the items that I mentioned to Vlad as being a
> > potential stumbling block..  I stated that I have no problem dumping the
> > LIO IOCTL in favor of something better, and having to break backwards
> > compat with LIO userspace tools (well, until I have to emulate the
> > backwards compat wrt target-ctl ;-)) is something I am ready to deal
> > with.
> > 
> > So I definately see your point here, having the exist CLI logic in place
> > into distros is something that we are going to have to deal with (even
> > if the core behind them is completely different).  
> > 
> > I should also mention that virtually all of the configuration logic done
> > from the LIO-Core and LIO-Target IOCTLs today is intended (minus a bug
> > or two :-) to be done without having to shutdown/stop anything other
> > than the referenced object for the target-ctl operation.  Obviously, I
> > really want to keep the same functionality with the CLI interface the
> > LIO pieces are ported to.
> > 
> > I will be sure to take a look at the STGT control interface and see how
> > I can fit the current functionality in.  I believe SCST uses an IOCTL
> > today as well, which would put them in a similar situation.
> 
> I, personally, don't like an interface, like tgtadm, which tries to do 
> all non-trivial configuration work from a single command line tool, 
> because of the following 3 reasons:
> 
> 1. It's a lot of effort to write and maintain such a tool, because it 
> needs to be extensible to work with new modules and modes. For instance, 
> iptables tool is 86K lines long. The whole netfilter code for all 
> network protocols in the kernel is less that 50K lines long. Do we want 
> such a code bloat (170% of code to configure) and dedicated team of 
> maintainers to solve a fundamentally simple configuration task?
> 
> 2. It assumes the stateless type of configuration, when each call 
> configures exactly one thing without any side effects on already 
> configured or future entries. This approach is good for cases like 
> iptables, but for SCSI targets it's possible that several configuration 
> steps require to be done in an atomic manner, like adding an iSCSI 
> target and configuring its parameters.
> 
> 3. It's hard to read 5+ parameters in one command line, so it's a lot 
> easier to make a mistake there.
> 
> So, I believe, a configuration interface should be rather /proc or /sys 
> interface based. I don't think we should care much about backward 
> compatibility with tgtadm, because the existing interface doesn't reach 
> the state of being widely used. As I already mentioned, only ibmvscsi at 
> the moment uses it, hardware for which is pretty rare.
> 

forget about proc. configfs is better. but problem is how u configure
user space target with configfs?


> Thus, I would suggest that before making the further move we should also 
> consider configuration interfaces of SCST and LIO and choose the best 
> things from all 3.


> 
> Vlad
> 
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Scst-devel mailing list
> Scst-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/scst-devel

Re: Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Vladislav Bolkhovitin]

Ming Zhang wrote:
>> I, personally, don't like an interface, like tgtadm, which tries to do 
>> all non-trivial configuration work from a single command line tool, 
>> because of the following 3 reasons:
>>
>> 1. It's a lot of effort to write and maintain such a tool, because it 
>> needs to be extensible to work with new modules and modes. For instance, 
>> iptables tool is 86K lines long. The whole netfilter code for all 
>> network protocols in the kernel is less that 50K lines long. Do we want 
>> such a code bloat (170% of code to configure) and dedicated team of 
>> maintainers to solve a fundamentally simple configuration task?
>>
>> 2. It assumes the stateless type of configuration, when each call 
>> configures exactly one thing without any side effects on already 
>> configured or future entries. This approach is good for cases like 
>> iptables, but for SCSI targets it's possible that several configuration 
>> steps require to be done in an atomic manner, like adding an iSCSI 
>> target and configuring its parameters.
>>
>> 3. It's hard to read 5+ parameters in one command line, so it's a lot 
>> easier to make a mistake there.
>>
>> So, I believe, a configuration interface should be rather /proc or /sys 
>> interface based. I don't think we should care much about backward 
>> compatibility with tgtadm, because the existing interface doesn't reach 
>> the state of being widely used. As I already mentioned, only ibmvscsi at 
>> the moment uses it, hardware for which is pretty rare.
> 
> forget about proc. configfs is better. but problem is how u configure
> user space target with configfs?

It doesn't matter much for me procfs or configfs or sysfs, although I 
definitely would prefer procfs, because it's already used by SCST, so no 
additional effort is needed. (The choice of procfs was purely for 
historical reasons; SCST was originally made for 2.4 kernels, where 
there were no other alternatives). But it doesn't relate to the choice 
of the fundamental approach "single utility for all possible parameters" 
vs "single entry for each parameter".

>> Thus, I would suggest that before making the further move we should also 
>> consider configuration interfaces of SCST and LIO and choose the best 
>> things from all 3.
> 
> 
>> Vlad
>>
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> Scst-devel mailing list
>> Scst-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/scst-devel
> 
> 

Re: Kernel Level Generic Target Mode control path

[Nicholas A. Bellinger]

On Wed, 2008-08-27 at 21:56 +0400, Vladislav Bolkhovitin wrote:
> >> Our userspace tools will also be able to support both kernel space and 
> >> userspace targets with little trouble so distros that have stgt will not 
> >> notice any differences.
> > 
> > So this is one of the items that I mentioned to Vlad as being a
> > potential stumbling block..  I stated that I have no problem dumping the
> > LIO IOCTL in favor of something better, and having to break backwards
> > compat with LIO userspace tools (well, until I have to emulate the
> > backwards compat wrt target-ctl ;-)) is something I am ready to deal
> > with.
> > 
> > So I definately see your point here, having the exist CLI logic in place
> > into distros is something that we are going to have to deal with (even
> > if the core behind them is completely different).  
> > 
> > I should also mention that virtually all of the configuration logic done
> > from the LIO-Core and LIO-Target IOCTLs today is intended (minus a bug
> > or two :-) to be done without having to shutdown/stop anything other
> > than the referenced object for the target-ctl operation.  Obviously, I
> > really want to keep the same functionality with the CLI interface the
> > LIO pieces are ported to.
> > 
> > I will be sure to take a look at the STGT control interface and see how
> > I can fit the current functionality in.  I believe SCST uses an IOCTL
> > today as well, which would put them in a similar situation.
> 
> I, personally, don't like an interface, like tgtadm, which tries to do 
> all non-trivial configuration work from a single command line tool, 
> because of the following 3 reasons:
> 
> 1. It's a lot of effort to write and maintain such a tool, because it 
> needs to be extensible to work with new modules and modes. For instance, 
> iptables tool is 86K lines long. The whole netfilter code for all 
> network protocols in the kernel is less that 50K lines long. Do we want 
> such a code bloat (170% of code to configure) and dedicated team of 
> maintainers to solve a fundamentally simple configuration task?
> 

Hrrmmm, yes I could see the same type of problem with the amount of
modularity requires for having plugins registering their own sets of
parameters for their control path code..

> 2. It assumes the stateless type of configuration, when each call 
> configures exactly one thing without any side effects on already 
> configured or future entries.
>  This approach is good for cases like 
> iptables, but for SCSI targets it's possible that several configuration 
> steps require to be done in an atomic manner, like adding an iSCSI 
> target and configuring its parameters.
> 

Well, the ability for an admin to force an LIO-Core related action, say
removing an HBA and all associated storage object with lots of exported
LUNs and running I/O, or an LIO-Target related action, say removing an
entire iSCSI Target Node with targetname=) at any time..  This obviously
require precision interaction between Target Fabric I/O Paths <-> Target
Core and Target Core <-> Control Interface to Admin.

That control interface needs to be protected in object contexts.  In
LIO-Core this is on a per HBA (be it physical or virtual) context.  With
LIO-Target this is on a iSCSI Target Node by targetname -> Target Portal
Group Tag context.  Obviously doing this from an IOCTL is the only real
choice I had when this code started in 2001, but I wonder how configfs
would work for something like this.

In any event, I will keep in mind that I might have to disable some of
my control path during the LIO-Target <-> SCST Core migration until we
can get resolved on the shutdown control path side..

> 3. It's hard to read 5+ parameters in one command line, so it's a lot 
> easier to make a mistake there.

No, I completely agree.  But I honestly think the actual target CLI
interface and parameters to admin need to do alot of pre-execution
script logic in userspace to reference different interested objects,
without the admin have to provide all of stuff.  I do this today to
determine the major/minor for lvm_uuid= (from lvs -v), md_uuid= (from
mdadm -D) and udev_path= (from /dev/disk)..

Same goes for real SCSI devices that we are exporting directly from
drivers/scsi.  We want to use EVPD Unit Serial or Device Identificaton
where ever able to reference said storage object.

> So, I believe, a configuration interface should be rather /proc or /sys 
> interface based. I don't think we should care much about backward 
> compatibility with tgtadm, because the existing interface doesn't reach 
> the state of being widely used.

I would definately vote against proc here for the fancy stuff I
mentioned above.  I have experience enabled core-iscsi to use sysfs for
RO data, but nothing along the lines of what would be required for a
generic target mode RW control path.  Does anyone with sysfs experience
have any comments on thing..?

>  As I already mentioned, only ibmvscsi at 
> the moment uses it, hardware for which is pretty rare.
> 

I am sure we could get some time this hardware if we wanted.. :-)

> Thus, I would suggest that before making the further move we should also 
> consider configuration interfaces of SCST and LIO and choose the best 
> things from all 3.
> 

So, Ming mentioned configfs for this...  /me reads about configfs

--nab

Re: Kernel Level Generic Target Mode control path

[Nicholas A. Bellinger]

On Wed, 2008-08-27 at 15:13 -0700, Nicholas A. Bellinger wrote:
> On Wed, 2008-08-27 at 21:56 +0400, Vladislav Bolkhovitin wrote:
> > >> Our userspace tools will also be able to support both kernel space and 
> > >> userspace targets with little trouble so distros that have stgt will not 
> > >> notice any differences.
> > > 
> > > So this is one of the items that I mentioned to Vlad as being a
> > > potential stumbling block..  I stated that I have no problem dumping the
> > > LIO IOCTL in favor of something better, and having to break backwards
> > > compat with LIO userspace tools (well, until I have to emulate the
> > > backwards compat wrt target-ctl ;-)) is something I am ready to deal
> > > with.
> > > 
> > > So I definately see your point here, having the exist CLI logic in place
> > > into distros is something that we are going to have to deal with (even
> > > if the core behind them is completely different).  
> > > 
> > > I should also mention that virtually all of the configuration logic done
> > > from the LIO-Core and LIO-Target IOCTLs today is intended (minus a bug
> > > or two :-) to be done without having to shutdown/stop anything other
> > > than the referenced object for the target-ctl operation.  Obviously, I
> > > really want to keep the same functionality with the CLI interface the
> > > LIO pieces are ported to.
> > > 
> > > I will be sure to take a look at the STGT control interface and see how
> > > I can fit the current functionality in.  I believe SCST uses an IOCTL
> > > today as well, which would put them in a similar situation.
> > 
> > I, personally, don't like an interface, like tgtadm, which tries to do 
> > all non-trivial configuration work from a single command line tool, 
> > because of the following 3 reasons:
> > 
> > 1. It's a lot of effort to write and maintain such a tool, because it 
> > needs to be extensible to work with new modules and modes. For instance, 
> > iptables tool is 86K lines long. The whole netfilter code for all 
> > network protocols in the kernel is less that 50K lines long. Do we want 
> > such a code bloat (170% of code to configure) and dedicated team of 
> > maintainers to solve a fundamentally simple configuration task?
> > 
> 
> Hrrmmm, yes I could see the same type of problem with the amount of
> modularity requires for having plugins registering their own sets of
> parameters for their control path code..
> 
> > 2. It assumes the stateless type of configuration, when each call 
> > configures exactly one thing without any side effects on already 
> > configured or future entries.
> >  This approach is good for cases like 
> > iptables, but for SCSI targets it's possible that several configuration 
> > steps require to be done in an atomic manner, like adding an iSCSI 
> > target and configuring its parameters.
> > 
> 
> Well, the ability for an admin to force an LIO-Core related action, say
> removing an HBA and all associated storage object with lots of exported
> LUNs and running I/O, or an LIO-Target related action, say removing an
> entire iSCSI Target Node with targetname=) at any time..  This obviously
> require precision interaction between Target Fabric I/O Paths <-> Target
> Core and Target Core <-> Control Interface to Admin.
> 
> That control interface needs to be protected in object contexts.  In
> LIO-Core this is on a per HBA (be it physical or virtual) context.  With
> LIO-Target this is on a iSCSI Target Node by targetname -> Target Portal
> Group Tag context.  Obviously doing this from an IOCTL is the only real
> choice I had when this code started in 2001, but I wonder how configfs
> would work for something like this.
> 
> In any event, I will keep in mind that I might have to disable some of
> my control path during the LIO-Target <-> SCST Core migration until we
> can get resolved on the shutdown control path side..
> 
> > 3. It's hard to read 5+ parameters in one command line, so it's a lot 
> > easier to make a mistake there.
> 
> No, I completely agree.  But I honestly think the actual target CLI
> interface and parameters to admin need to do alot of pre-execution
> script logic in userspace to reference different interested objects,
> without the admin have to provide all of stuff.  I do this today to
> determine the major/minor for lvm_uuid= (from lvs -v), md_uuid= (from
> mdadm -D) and udev_path= (from /dev/disk)..
> 
> Same goes for real SCSI devices that we are exporting directly from
> drivers/scsi.  We want to use EVPD Unit Serial or Device Identificaton
> where ever able to reference said storage object.
> 
> > So, I believe, a configuration interface should be rather /proc or /sys 
> > interface based. I don't think we should care much about backward 
> > compatibility with tgtadm, because the existing interface doesn't reach 
> > the state of being widely used.
> 
> I would definately vote against proc here for the fancy stuff I
> mentioned above.  I have experience enabled core-iscsi to use sysfs for
> RO data, but nothing along the lines of what would be required for a
> generic target mode RW control path.  Does anyone with sysfs experience
> have any comments on thing..?
> 
> >  As I already mentioned, only ibmvscsi at 
> > the moment uses it, hardware for which is pretty rare.
> > 
> 
> I am sure we could get some time this hardware if we wanted.. :-)
> 
> > Thus, I would suggest that before making the further move we should also 
> > consider configuration interfaces of SCST and LIO and choose the best 
> > things from all 3.
> > 
> 
> So, Ming mentioned configfs for this...  /me reads about configfs
> 

Ah yes, I remember now, this is part of the OCFS2 patchset and it uses
it for /config/cluster/ocfs2/[heartbeat,node] information, as well as
other tunables related to millisecond timeout values.  These are
basically the values from /etc/ocfs2/cluster.conf..

Looking at http://lwn.net/Articles/148973/, this looks like a strong
contender for what we would require with a generic kernel level target
engine.

--nab

Re: Kernel Level Generic Target Mode control path

[Vladislav Bolkhovitin]

Nicholas A. Bellinger wrote:
>>> Thus, I would suggest that before making the further move we should also 
>>> consider configuration interfaces of SCST and LIO and choose the best 
>>> things from all 3.
>>>
>> So, Ming mentioned configfs for this...  /me reads about configfs
>>
> 
> Ah yes, I remember now, this is part of the OCFS2 patchset and it uses
> it for /config/cluster/ocfs2/[heartbeat,node] information, as well as
> other tunables related to millisecond timeout values.  These are
> basically the values from /etc/ocfs2/cluster.conf..
> 
> Looking at http://lwn.net/Articles/148973/, this looks like a strong
> contender for what we would require with a generic kernel level target
> engine.

For our task I don't think configfs has any advantage over sysfs.

Vlad

Re: Kernel Level Generic Target Mode control path

[Vladislav Bolkhovitin]

Nicholas A. Bellinger wrote:
> On Wed, 2008-08-27 at 21:56 +0400, Vladislav Bolkhovitin wrote:
>>>> Our userspace tools will also be able to support both kernel space and 
>>>> userspace targets with little trouble so distros that have stgt will not 
>>>> notice any differences.
>>> So this is one of the items that I mentioned to Vlad as being a
>>> potential stumbling block..  I stated that I have no problem dumping the
>>> LIO IOCTL in favor of something better, and having to break backwards
>>> compat with LIO userspace tools (well, until I have to emulate the
>>> backwards compat wrt target-ctl ;-)) is something I am ready to deal
>>> with.
>>>
>>> So I definately see your point here, having the exist CLI logic in place
>>> into distros is something that we are going to have to deal with (even
>>> if the core behind them is completely different).  
>>>
>>> I should also mention that virtually all of the configuration logic done
>>> from the LIO-Core and LIO-Target IOCTLs today is intended (minus a bug
>>> or two :-) to be done without having to shutdown/stop anything other
>>> than the referenced object for the target-ctl operation.  Obviously, I
>>> really want to keep the same functionality with the CLI interface the
>>> LIO pieces are ported to.
>>>
>>> I will be sure to take a look at the STGT control interface and see how
>>> I can fit the current functionality in.  I believe SCST uses an IOCTL
>>> today as well, which would put them in a similar situation.
>> I, personally, don't like an interface, like tgtadm, which tries to do 
>> all non-trivial configuration work from a single command line tool, 
>> because of the following 3 reasons:
>>
>> 1. It's a lot of effort to write and maintain such a tool, because it 
>> needs to be extensible to work with new modules and modes. For instance, 
>> iptables tool is 86K lines long. The whole netfilter code for all 
>> network protocols in the kernel is less that 50K lines long. Do we want 
>> such a code bloat (170% of code to configure) and dedicated team of 
>> maintainers to solve a fundamentally simple configuration task?
> 
> Hrrmmm, yes I could see the same type of problem with the amount of
> modularity requires for having plugins registering their own sets of
> parameters for their control path code..

I should add in this context that in the UNIX world the choice between a 
huge application, capable of doing all possible kinds of work, and a set 
of small utilities, doing exactly one simple thing each, was done 
decades ago in favor of the set of small utilities: find, grep, sort, 
etc. And I believe now nobody would claim that it was a wrong choice.

>> 2. It assumes the stateless type of configuration, when each call 
>> configures exactly one thing without any side effects on already 
>> configured or future entries.
>>  This approach is good for cases like 
>> iptables, but for SCSI targets it's possible that several configuration 
>> steps require to be done in an atomic manner, like adding an iSCSI 
>> target and configuring its parameters.
> 
> Well, the ability for an admin to force an LIO-Core related action, say
> removing an HBA and all associated storage object with lots of exported
> LUNs and running I/O, or an LIO-Target related action, say removing an
> entire iSCSI Target Node with targetname=) at any time..  This obviously
> require precision interaction between Target Fabric I/O Paths <-> Target
> Core and Target Core <-> Control Interface to Admin.
> 
> That control interface needs to be protected in object contexts.  In
> LIO-Core this is on a per HBA (be it physical or virtual) context.  With
> LIO-Target this is on a iSCSI Target Node by targetname -> Target Portal
> Group Tag context.  Obviously doing this from an IOCTL is the only real
> choice I had when this code started in 2001, but I wonder how configfs
> would work for something like this.

My favorite configuration interface would have 2 levels.

1. The lowest level would be /proc, /sys, etc. based and allow to 
configure exactly one parameter or set of related parameters with 
corresponding subparameters necessary to provide required atomicity, if 
needed. For instance, this is how a new virtual read-only SBC device 
with 4K block size is added in SCST vdisk handler:

# echo "open disk_4K /disk_4K 4096 READ_ONLY" >/proc/scst/vdisk/vdisk

2. The higher level interface(s) would allow people to not bother with 
low level commands, but use regular text config file(s). See scstadmin 
utility for example. It allows to do all necessary configuration of SCST 
core from /etc/scst.conf file. Such interface must have an important 
property: it must be able to detect changes in the config file and apply 
them to the running system. That property would allow to have system 
configuration always persistent: if one needs to change something, he 
would edit the config file and rerun the corresponding utility 
(scstadmin in this example; it really can do that, though with some 
limitations). Although this interface level would completely belong to 
user space, we in kernel need to provide for it a convenient interface.

Target drivers and backstorage device handlers, who need advanced 
configuration, would have own low and high level interfaces, as needed. 
For instance, an iSCSI target must not start serving clients until all 
its targets fully configured. Otherwise, initiators can get rejected for 
not yet configured target and erroneously consider it dead. In 
iSCSI-SCST the user space part of the target doesn't start accepting 
connection until it finishes reading /etc/iscsi-scst.conf file.

>> 3. It's hard to read 5+ parameters in one command line, so it's a lot 
>> easier to make a mistake there.
> 
> No, I completely agree.  But I honestly think the actual target CLI
> interface and parameters to admin need to do alot of pre-execution
> script logic in userspace to reference different interested objects,
> without the admin have to provide all of stuff.  I do this today to
> determine the major/minor for lvm_uuid= (from lvs -v), md_uuid= (from
> mdadm -D) and udev_path= (from /dev/disk)..
> 
> Same goes for real SCSI devices that we are exporting directly from
> drivers/scsi.  We want to use EVPD Unit Serial or Device Identificaton
> where ever able to reference said storage object.

Yes, this is why we need the high level interface. Otherwise for complex 
targets the configuration task quickly grows up to a nightmare.

>> So, I believe, a configuration interface should be rather /proc or /sys 
>> interface based. I don't think we should care much about backward 
>> compatibility with tgtadm, because the existing interface doesn't reach 
>> the state of being widely used.
> 
> I would definately vote against proc here for the fancy stuff I
> mentioned above.  I have experience enabled core-iscsi to use sysfs for
> RO data, but nothing along the lines of what would be required for a
> generic target mode RW control path.  Does anyone with sysfs experience
> have any comments on thing..?

Sysfs as well as configfs have one big disadvantage. They limit each 
file to only 4KB. This would force us for to create a subdirectory for 
each device and for each connected initiator. I don't like seeing 
thousands subdirectories. Additionally, such layout is a lot less 
convenient for parsing for the high level configuration tool, which 
needs to find out the difference between the current configuration and 
content of the corresponding config file.

Currently, with procfs SCST can list in /proc/scst/sessions virtually 
unlimited amount of connected initiators in a simple for parsing manner. 
It was done using seq interface well and simply. Neither sysfs, nor 
configfs support seq interface. This would introduce significant effort 
in both kernel and user spaces.

Debugfs supports seq interface, but, because of the name, I doubt we can 
use it ;)

Thus, looks like we'd better stay with /proc. After all, networking and 
VM widely use /proc for internal configuration. Why SCSI target is worse?

>>  As I already mentioned, only ibmvscsi at 
>> the moment uses it, hardware for which is pretty rare.
>>
> 
> I am sure we could get some time this hardware if we wanted.. :-)
> 
>> Thus, I would suggest that before making the further move we should also 
>> consider configuration interfaces of SCST and LIO and choose the best 
>> things from all 3.
>>
> 
> So, Ming mentioned configfs for this...  /me reads about configfs
> 
> --nab
> 
> 
> 

Re: Kernel Level Generic Target Mode control path

[James Bottomley]

On Thu, 2008-08-28 at 21:52 +0400, Vladislav Bolkhovitin wrote:
> >> So, I believe, a configuration interface should be rather /proc or /sys 
> >> interface based. I don't think we should care much about backward 
> >> compatibility with tgtadm, because the existing interface doesn't reach 
> >> the state of being widely used.
> > 
> > I would definately vote against proc here for the fancy stuff I
> > mentioned above.  I have experience enabled core-iscsi to use sysfs for
> > RO data, but nothing along the lines of what would be required for a
> > generic target mode RW control path.  Does anyone with sysfs experience
> > have any comments on thing..?
> 
> Sysfs as well as configfs have one big disadvantage. They limit each 
> file to only 4KB. This would force us for to create a subdirectory for 
> each device and for each connected initiator. I don't like seeing 
> thousands subdirectories. Additionally, such layout is a lot less 
> convenient for parsing for the high level configuration tool, which 
> needs to find out the difference between the current configuration and 
> content of the corresponding config file.

That's because each of those interfaces is designed around one value per
file.

> Currently, with procfs SCST can list in /proc/scst/sessions virtually 
> unlimited amount of connected initiators in a simple for parsing manner. 
> It was done using seq interface well and simply. Neither sysfs, nor 
> configfs support seq interface. This would introduce significant effort 
> in both kernel and user spaces.
> 
> Debugfs supports seq interface, but, because of the name, I doubt we can 
> use it ;)
> 
> Thus, looks like we'd better stay with /proc. After all, networking and 
> VM widely use /proc for internal configuration. Why SCSI target is worse?

I wouldn't do that.  /proc has been deprecated for several years for
large config interfaces (indeed it's trying to be scaled back only to
deal with processes).  Nothing with a huge /proc interface would pass a
review for 2.6 nowadays.

James

Re: Kernel Level Generic Target Mode control path

[Nicholas A. Bellinger]

On Thu, 2008-08-28 at 21:52 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger wrote:

> 
> >> 2. It assumes the stateless type of configuration, when each call 
> >> configures exactly one thing without any side effects on already 
> >> configured or future entries.
> >>  This approach is good for cases like 
> >> iptables, but for SCSI targets it's possible that several configuration 
> >> steps require to be done in an atomic manner, like adding an iSCSI 
> >> target and configuring its parameters.
> > 
> > Well, the ability for an admin to force an LIO-Core related action, say
> > removing an HBA and all associated storage object with lots of exported
> > LUNs and running I/O, or an LIO-Target related action, say removing an
> > entire iSCSI Target Node with targetname=) at any time..  This obviously
> > require precision interaction between Target Fabric I/O Paths <-> Target
> > Core and Target Core <-> Control Interface to Admin.
> > 
> > That control interface needs to be protected in object contexts.  In
> > LIO-Core this is on a per HBA (be it physical or virtual) context.  With
> > LIO-Target this is on a iSCSI Target Node by targetname -> Target Portal
> > Group Tag context.  Obviously doing this from an IOCTL is the only real
> > choice I had when this code started in 2001, but I wonder how configfs
> > would work for something like this.
> 
> My favorite configuration interface would have 2 levels.
> 
> 1. The lowest level would be /proc, /sys, etc. based and allow to 
> configure exactly one parameter or set of related parameters with 
> corresponding subparameters necessary to provide required atomicity, if 
> needed. For instance, this is how a new virtual read-only SBC device 
> with 4K block size is added in SCST vdisk handler:
> 
> # echo "open disk_4K /disk_4K 4096 READ_ONLY" >/proc/scst/vdisk/vdisk
> 

So, I would consider proc and sysfs both medieval for RW data, with the
latter being slightly sharper instrument than the former, but both not
very effective..

Configfs on the other hand, is quite sharp in the battlefield..  Here is
what I am thinking for LIO-Target after it was loaded to an configfs
enabled generic target engine:

# This is the iSCSI Qualifed Target Name we will be creating
MY_TARGETIQN=/config/target/iscsi/iqn.superturbodiskarray
# From 'lvs -v'
LVM_UUID=y2sbeD-insM-xykn-s3SV-3tge-VWhn-xB4FMv
# From 'lsscsi' or '/proc/scsi/scsi'
SCSI_HCTL_LOCATION=1:0:0:0

# Same as 'target-ctl coreaddtiqn targetname=$IQN' with LIO today
mkdir $MY_TARGETIQN

# Make TPGT=1 on iqn.superturbodiskarray
mkdir $MY_TARGETIQN/tpgt_1

# Make TPGT=1 run in Demo Mode for this example (no Initiator ACLs)
echo 1 > $MY_TARGETIQN/tpgt_1/attribs/generate_node_acls

# Create network portal mapping to TPGT=1
mkdir $MY_TARGETIQN/tpgt_1/np/192.168.100.10

# DEPENDS ON GENERIC TARGET CORE
# Create TPGT=1,LUN=0 from Linux LVM Block Device
mkdir $MY_TARGETIQN/tpgt_1/lun_0
echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location

# DEPENDS ON GENERIC TARGET CORE
# Create TPGT=1,LUN=1 from SCSI Layer
mkdir $MY_TARGETIQN/tpgt_1/lun_1
echo $SCSI_HCTL_LOCATION > $MY_TARGETIQN/tpgt_1/lun_1/location

# This is the atomic part, once we throw this flag iSCSI Initiators will
be allowed to login to this TPGT
echo 1 > $MY_TARGETIQN/tpgt_1/enable_tpg

# The equivliant of 'target-ctl coredeltiqn targetname=$IQN' today
rm -rf $MY_TARGETIQN

I think this would make a very useful and extremely flexiable interface
for my purposes...  What do you think about the potential..?

> 2. The higher level interface(s) would allow people to not bother with 
> low level commands, but use regular text config file(s). See scstadmin 
> utility for example. It allows to do all necessary configuration of SCST 
> core from /etc/scst.conf file. Such interface must have an important 
> property: it must be able to detect changes in the config file and apply 
> them to the running system. That property would allow to have system 
> configuration always persistent: if one needs to change something, he 
> would edit the config file and rerun the corresponding utility 
> (scstadmin in this example; it really can do that, though with some 
> limitations). Although this interface level would completely belong to 
> user space, we in kernel need to provide for it a convenient interface.
> 
> Target drivers and backstorage device handlers, who need advanced 
> configuration, would have own low and high level interfaces, as needed. 
> For instance, an iSCSI target must not start serving clients until all 
> its targets fully configured. Otherwise, initiators can get rejected for 
> not yet configured target and erroneously consider it dead. In 
> iSCSI-SCST the user space part of the target doesn't start accepting 
> connection until it finishes reading /etc/iscsi-scst.conf file.
> 
> >> 3. It's hard to read 5+ parameters in one command line, so it's a lot 
> >> easier to make a mistake there.
> > 
> > No, I completely agree.  But I honestly think the actual target CLI
> > interface and parameters to admin need to do alot of pre-execution
> > script logic in userspace to reference different interested objects,
> > without the admin have to provide all of stuff.  I do this today to
> > determine the major/minor for lvm_uuid= (from lvs -v), md_uuid= (from
> > mdadm -D) and udev_path= (from /dev/disk)..
> > 
> > Same goes for real SCSI devices that we are exporting directly from
> > drivers/scsi.  We want to use EVPD Unit Serial or Device Identificaton
> > where ever able to reference said storage object.
> 
> Yes, this is why we need the high level interface. Otherwise for complex 
> targets the configuration task quickly grows up to a nightmare.
> 
> >> So, I believe, a configuration interface should be rather /proc or /sys 
> >> interface based. I don't think we should care much about backward 
> >> compatibility with tgtadm, because the existing interface doesn't reach 
> >> the state of being widely used.
> > 
> > I would definately vote against proc here for the fancy stuff I
> > mentioned above.  I have experience enabled core-iscsi to use sysfs for
> > RO data, but nothing along the lines of what would be required for a
> > generic target mode RW control path.  Does anyone with sysfs experience
> > have any comments on thing..?
> 
> Sysfs as well as configfs have one big disadvantage. They limit each 
> file to only 4KB. This would force us for to create a subdirectory for 
> each device and for each connected initiator. I don't like seeing 
> thousands subdirectories. Additionally, such layout is a lot less 
> convenient for parsing for the high level configuration tool, which 
> needs to find out the difference between the current configuration and 
> content of the corresponding config file.
> 

So yeah, the output with configfs is limited to PAGE_SIZE as well, but
for the R/W cases we don't expect that data sets to exceed this per
configfs mkdir invocation.. 

> Currently, with procfs SCST can list in /proc/scst/sessions virtually 
> unlimited amount of connected initiators in a simple for parsing manner. 
> It was done using seq interface well and simply. Neither sysfs, nor 
> configfs support seq interface. This would introduce significant effort 
> in both kernel and user spaces.
> 

Same for me with all of the target-ctl IOCTL commands.   No one ever
said upstream target code was not going to require significant
effort. :-)

> Debugfs supports seq interface, but, because of the name, I doubt we can 
> use it ;)
>
> Thus, looks like we'd better stay with /proc. After all, networking and 
> VM widely use /proc for internal configuration. Why SCSI target is worse?
> 

So yeah, RW configuration data going with /proc is completely
unacceptable for my purposes.  However since all of the LIO iSCSI and
SCSI MIB code is procfs + seq_file based (and is read-only) I figure
that considering there are other MIBs related procfs code in other
subsystems, that this would not be too much of a stumbling point.

So this configfs stuff is really starting to grow on me, I am surprised
that I have missed it for so long, I know the author (and alot of the
OCFSv2 team) and I believe their reasons for creating and moving ocfs2
to use configfs (even though a production ocfs2 create has much *FEWER*
configuration information respresented in configfs directories and
entries than a production iSCSI target would) provide compelling
evidence to move our efforts for a generic kernel target engine in this
direction.  So yes, I do feel your pain wrt to existing code, but I
believe that using procfs from my current IOCTL would quite honestly be
a step back.  

How about this, I will begin to implement the LIO-Target pieces in
configfs for lio-core-2.6.git, and leave LIO-Core in IOCTL for now, and
once I have some running code, I will look at the process to begin to
incorporate the requirements to preform the "dumb registration" with a
generic target engine.  From there, that would give me a good idea of
what would be required with SCST and configfs.  From the above example
with configfs, these would be:

# Create TPGT=1,LUN=0 from Linux LVM Block Device using UUID from
# 'lvs -v' output
mkdir $MY_TARGETIQN/tpgt_1/lun_0
echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location

# Create TPGT=1,LUN=1 from SCSI Layer h/c/t/l Parameters from lsscsi
# or /proc/scsi/scsi output
mkdir $MY_TARGETIQN/tpgt_1/lun_1
echo $SCSI_HCTL_LOCATION > $MY_TARGETIQN/tpgt_1/lun_1/location

Of course, we could do poke directly at Target-Core storage objects via
configfs as well!

# Preform a LUN reset on $LVM_UUID on all mapped target PORTs/LUNs
echo 1 > /config/target/core/$LVM_UUID/lun_reset
# Remove the SCSI device from the target, and all mapped PORTs/LUNs
# and fall all outstanding I/Os.
rm -rf /config/target/core/$SCSI_HTL_LOCATION

Again, I think this would be extremely flexible and easily extendable
for new uses. Also, providing some level of backwards compatibility with
our project's existing CLI nomenclature would not been that hard if our
repsective projects really require it.

I would also be happy to help with configfs and SCST if you are
interested..

--nab

Re: Kernel Level Generic Target Mode control path

[Nicholas A. Bellinger]

On Thu, 2008-08-28 at 16:08 -0700, Nicholas A. Bellinger wrote:

<snip>

> > Thus, looks like we'd better stay with /proc. After all, networking and 
> > VM widely use /proc for internal configuration. Why SCSI target is worse?
> > 
> 
> So yeah, RW configuration data going with /proc is completely
> unacceptable for my purposes.  However since all of the LIO iSCSI and
> SCSI MIB code is procfs + seq_file based (and is read-only) I figure
> that considering there are other MIBs related procfs code in other
> subsystems, that this would not be too much of a stumbling point.
> 
> So this configfs stuff is really starting to grow on me, I am surprised
> that I have missed it for so long, I know the author (and alot of the
> OCFSv2 team) and I believe their reasons for creating and moving ocfs2
> to use configfs (even though a production ocfs2 create has much *FEWER*
> configuration information respresented in configfs directories and
> entries than a production iSCSI target would) provide compelling
> evidence to move our efforts for a generic kernel target engine in this
> direction.  So yes, I do feel your pain wrt to existing code, but I
> believe that using procfs from my current IOCTL would quite honestly be
> a step back.  
> 
> How about this, I will begin to implement the LIO-Target pieces in
> configfs for lio-core-2.6.git, and leave LIO-Core in IOCTL for now, and
> once I have some running code, I will look at the process to begin to
> incorporate the requirements to preform the "dumb registration" with a
> generic target engine.  From there, that would give me a good idea of
> what would be required with SCST and configfs.  From the above example
> with configfs, these would be:
> 
> # Create TPGT=1,LUN=0 from Linux LVM Block Device using UUID from
> # 'lvs -v' output
> mkdir $MY_TARGETIQN/tpgt_1/lun_0
> echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location
> 
> # Create TPGT=1,LUN=1 from SCSI Layer h/c/t/l Parameters from lsscsi
> # or /proc/scsi/scsi output
> mkdir $MY_TARGETIQN/tpgt_1/lun_1
> echo $SCSI_HCTL_LOCATION > $MY_TARGETIQN/tpgt_1/lun_1/location
> 

I should mention in this example that the registration of $LVM_UUID and
$SCSI_HCTL_LOCATION storage objects would be done behind the scenes with
these two 'addluntotpg' configfs mkdir's are called.

This means that it would be possible to do storage object registration
before hand and tweak storage object settings / attributes:

# Import $LVM_UUID into Target-Core
mkdir /config/target/core/$LVM_UUID
# Make all LUN Mappings READ-ONLY to $LVM_UUID
echo 0 > /config/target/core/$LVM_UUID/lun_access

... <run other /config/target/$IQN calls>

# Create TPGT=1,LUN=0 from Linux LVM Block Device using UUID from
> # 'lvs -v' output
mkdir $MY_TARGETIQN/tpgt_1/lun_0
# If $LVM_UUID storage object is not already registered with generic 
# target core, automatic registration will occur
echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location

# Finally, allow iSCSI Initiators login to the TargetName+TPGT endpoint
echo 1 > $MY_TARGETIQN/tpgt_1/enable_tpg

--nab

Re: Kernel Level Generic Target Mode control path

[Vladislav Bolkhovitin]

Nicholas A. Bellinger wrote:
> On Thu, 2008-08-28 at 21:52 +0400, Vladislav Bolkhovitin wrote:
>> Nicholas A. Bellinger wrote:
> 
>>>> 2. It assumes the stateless type of configuration, when each call 
>>>> configures exactly one thing without any side effects on already 
>>>> configured or future entries.
>>>>  This approach is good for cases like 
>>>> iptables, but for SCSI targets it's possible that several configuration 
>>>> steps require to be done in an atomic manner, like adding an iSCSI 
>>>> target and configuring its parameters.
>>> Well, the ability for an admin to force an LIO-Core related action, say
>>> removing an HBA and all associated storage object with lots of exported
>>> LUNs and running I/O, or an LIO-Target related action, say removing an
>>> entire iSCSI Target Node with targetname=) at any time..  This obviously
>>> require precision interaction between Target Fabric I/O Paths <-> Target
>>> Core and Target Core <-> Control Interface to Admin.
>>>
>>> That control interface needs to be protected in object contexts.  In
>>> LIO-Core this is on a per HBA (be it physical or virtual) context.  With
>>> LIO-Target this is on a iSCSI Target Node by targetname -> Target Portal
>>> Group Tag context.  Obviously doing this from an IOCTL is the only real
>>> choice I had when this code started in 2001, but I wonder how configfs
>>> would work for something like this.
>> My favorite configuration interface would have 2 levels.
>>
>> 1. The lowest level would be /proc, /sys, etc. based and allow to 
>> configure exactly one parameter or set of related parameters with 
>> corresponding subparameters necessary to provide required atomicity, if 
>> needed. For instance, this is how a new virtual read-only SBC device 
>> with 4K block size is added in SCST vdisk handler:
>>
>> # echo "open disk_4K /disk_4K 4096 READ_ONLY" >/proc/scst/vdisk/vdisk
>>
> 
> So, I would consider proc and sysfs both medieval for RW data, with the
> latter being slightly sharper instrument than the former, but both not
> very effective..
> 
> Configfs on the other hand, is quite sharp in the battlefield..  Here is
> what I am thinking for LIO-Target after it was loaded to an configfs
> enabled generic target engine:
> 
> # This is the iSCSI Qualifed Target Name we will be creating
> MY_TARGETIQN=/config/target/iscsi/iqn.superturbodiskarray
> # From 'lvs -v'
> LVM_UUID=y2sbeD-insM-xykn-s3SV-3tge-VWhn-xB4FMv
> # From 'lsscsi' or '/proc/scsi/scsi'
> SCSI_HCTL_LOCATION=1:0:0:0
> 
> # Same as 'target-ctl coreaddtiqn targetname=$IQN' with LIO today
> mkdir $MY_TARGETIQN
> 
> # Make TPGT=1 on iqn.superturbodiskarray
> mkdir $MY_TARGETIQN/tpgt_1
> 
> # Make TPGT=1 run in Demo Mode for this example (no Initiator ACLs)
> echo 1 > $MY_TARGETIQN/tpgt_1/attribs/generate_node_acls
> 
> # Create network portal mapping to TPGT=1
> mkdir $MY_TARGETIQN/tpgt_1/np/192.168.100.10
> 
> # DEPENDS ON GENERIC TARGET CORE
> # Create TPGT=1,LUN=0 from Linux LVM Block Device
> mkdir $MY_TARGETIQN/tpgt_1/lun_0
> echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location
> 
> # DEPENDS ON GENERIC TARGET CORE
> # Create TPGT=1,LUN=1 from SCSI Layer
> mkdir $MY_TARGETIQN/tpgt_1/lun_1
> echo $SCSI_HCTL_LOCATION > $MY_TARGETIQN/tpgt_1/lun_1/location
> 
> # This is the atomic part, once we throw this flag iSCSI Initiators will
> be allowed to login to this TPGT
> echo 1 > $MY_TARGETIQN/tpgt_1/enable_tpg
> 
> # The equivliant of 'target-ctl coredeltiqn targetname=$IQN' today
> rm -rf $MY_TARGETIQN
> 
> I think this would make a very useful and extremely flexiable interface
> for my purposes...  What do you think about the potential..?

It has a big problem with atomicity of changes. Configuration of each 
iSCSI target should be atomic and on the target driver's start 
configuration of *all* targets as a whole should be atomic as well. How 
are you going to solve this issue?

Plus:

1. As I already wrote, separate subdir for each target and device is 
harder to parse

2. The high level interface needs to lock somehow the low level 
interface during updates to protect from simultaneous actions. I don't 
see a simple way to do that in configfs.

>> 2. The higher level interface(s) would allow people to not bother with 
>> low level commands, but use regular text config file(s). See scstadmin 
>> utility for example. It allows to do all necessary configuration of SCST 
>> core from /etc/scst.conf file. Such interface must have an important 
>> property: it must be able to detect changes in the config file and apply 
>> them to the running system. That property would allow to have system 
>> configuration always persistent: if one needs to change something, he 
>> would edit the config file and rerun the corresponding utility 
>> (scstadmin in this example; it really can do that, though with some 
>> limitations). Although this interface level would completely belong to 
>> user space, we in kernel need to provide for it a convenient interface.
>>
>> Target drivers and backstorage device handlers, who need advanced 
>> configuration, would have own low and high level interfaces, as needed. 
>> For instance, an iSCSI target must not start serving clients until all 
>> its targets fully configured. Otherwise, initiators can get rejected for 
>> not yet configured target and erroneously consider it dead. In 
>> iSCSI-SCST the user space part of the target doesn't start accepting 
>> connection until it finishes reading /etc/iscsi-scst.conf file.
>>
>>>> 3. It's hard to read 5+ parameters in one command line, so it's a lot 
>>>> easier to make a mistake there.
>>> No, I completely agree.  But I honestly think the actual target CLI
>>> interface and parameters to admin need to do alot of pre-execution
>>> script logic in userspace to reference different interested objects,
>>> without the admin have to provide all of stuff.  I do this today to
>>> determine the major/minor for lvm_uuid= (from lvs -v), md_uuid= (from
>>> mdadm -D) and udev_path= (from /dev/disk)..
>>>
>>> Same goes for real SCSI devices that we are exporting directly from
>>> drivers/scsi.  We want to use EVPD Unit Serial or Device Identificaton
>>> where ever able to reference said storage object.
>> Yes, this is why we need the high level interface. Otherwise for complex 
>> targets the configuration task quickly grows up to a nightmare.
>>
>>>> So, I believe, a configuration interface should be rather /proc or /sys 
>>>> interface based. I don't think we should care much about backward 
>>>> compatibility with tgtadm, because the existing interface doesn't reach 
>>>> the state of being widely used.
>>> I would definately vote against proc here for the fancy stuff I
>>> mentioned above.  I have experience enabled core-iscsi to use sysfs for
>>> RO data, but nothing along the lines of what would be required for a
>>> generic target mode RW control path.  Does anyone with sysfs experience
>>> have any comments on thing..?
>> Sysfs as well as configfs have one big disadvantage. They limit each 
>> file to only 4KB. This would force us for to create a subdirectory for 
>> each device and for each connected initiator. I don't like seeing 
>> thousands subdirectories. Additionally, such layout is a lot less 
>> convenient for parsing for the high level configuration tool, which 
>> needs to find out the difference between the current configuration and 
>> content of the corresponding config file.
>>
> 
> So yeah, the output with configfs is limited to PAGE_SIZE as well, but
> for the R/W cases we don't expect that data sets to exceed this per
> configfs mkdir invocation.. 
> 
>> Currently, with procfs SCST can list in /proc/scst/sessions virtually 
>> unlimited amount of connected initiators in a simple for parsing manner. 
>> It was done using seq interface well and simply. Neither sysfs, nor 
>> configfs support seq interface. This would introduce significant effort 
>> in both kernel and user spaces.
>>
> 
> Same for me with all of the target-ctl IOCTL commands.   No one ever
> said upstream target code was not going to require significant
> effort. :-)

I don't think we should create the additional one :-)

>> Debugfs supports seq interface, but, because of the name, I doubt we can 
>> use it ;)
>>
>> Thus, looks like we'd better stay with /proc. After all, networking and 
>> VM widely use /proc for internal configuration. Why SCSI target is worse?
>>
> 
> So yeah, RW configuration data going with /proc is completely
> unacceptable for my purposes.  However since all of the LIO iSCSI and
> SCSI MIB code is procfs + seq_file based (and is read-only) I figure
> that considering there are other MIBs related procfs code in other
> subsystems, that this would not be too much of a stumbling point.
> 
> So this configfs stuff is really starting to grow on me, I am surprised
> that I have missed it for so long, I know the author (and alot of the
> OCFSv2 team) and I believe their reasons for creating and moving ocfs2
> to use configfs (even though a production ocfs2 create has much *FEWER*
> configuration information respresented in configfs directories and
> entries than a production iSCSI target would) provide compelling
> evidence to move our efforts for a generic kernel target engine in this
> direction.  So yes, I do feel your pain wrt to existing code, but I
> believe that using procfs from my current IOCTL would quite honestly be
> a step back.  
> 
> How about this, I will begin to implement the LIO-Target pieces in
> configfs for lio-core-2.6.git, and leave LIO-Core in IOCTL for now, and
> once I have some running code, I will look at the process to begin to
> incorporate the requirements to preform the "dumb registration" with a
> generic target engine.  From there, that would give me a good idea of
> what would be required with SCST and configfs.  From the above example
> with configfs, these would be:
> 
> # Create TPGT=1,LUN=0 from Linux LVM Block Device using UUID from
> # 'lvs -v' output
> mkdir $MY_TARGETIQN/tpgt_1/lun_0
> echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location
> 
> # Create TPGT=1,LUN=1 from SCSI Layer h/c/t/l Parameters from lsscsi
> # or /proc/scsi/scsi output
> mkdir $MY_TARGETIQN/tpgt_1/lun_1
> echo $SCSI_HCTL_LOCATION > $MY_TARGETIQN/tpgt_1/lun_1/location
> 
> Of course, we could do poke directly at Target-Core storage objects via
> configfs as well!
> 
> # Preform a LUN reset on $LVM_UUID on all mapped target PORTs/LUNs
> echo 1 > /config/target/core/$LVM_UUID/lun_reset
> # Remove the SCSI device from the target, and all mapped PORTs/LUNs
> # and fall all outstanding I/Os.
> rm -rf /config/target/core/$SCSI_HTL_LOCATION
> 
> Again, I think this would be extremely flexible and easily extendable
> for new uses. Also, providing some level of backwards compatibility with
> our project's existing CLI nomenclature would not been that hard if our
> repsective projects really require it.
> 
> I would also be happy to help with configfs and SCST if you are
> interested..

Thanks, but more I'm thinking, more I'm coming to an IOCTL's based 
interface, where each submodule has own set of IOCTL's, multiplexed over 
a single device, and for each module there would be a simple dedicated 
utility. For example, to configure virtual disk backstorage handler: 
scst_vdisk_ctl, which would allow to create, delete and manage virtual 
FILEIO/BLOCKIO disks. Or for managing security groups there would be 
scst_acl_ctl utility. Each such utility would be very simple, would 
offload kernel from parsing character strings or converting to them, as 
it is necessary for configfs, each new future module would come with own 
management tool. Git uses a similar approach: a dedicated program for 
each action.

Data of big lists, like list of devices, would be returned at once in an 
easy to parse form, which would considerably simplify creation of the 
high level interface.

Such approach would be pretty simple for implementation and maintenance 
(I love simplicity) as well as solve all other configfs problems, namely:

1. The atomicity would be natural. An iscsi_tgt_ctl utility would parse 
the input or corresponding config file and create a target with all 
parameters in a single IOCTL call.

2. Locking would be automatic via forced exclusive open without any 
additional actions.

Also SCST /proc helpers, which target drivers and backstorage handlers 
intensively use, would fit in this approach quite well.

How's that look?

Vlad

Re: Kernel Level Generic Target Mode control path

[Nicholas A. Bellinger]

On Fri, 2008-08-29 at 20:28 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger wrote:
> > On Thu, 2008-08-28 at 21:52 +0400, Vladislav Bolkhovitin wrote:
> >> Nicholas A. Bellinger wrote:
> > 
> >>>> 2. It assumes the stateless type of configuration, when each call 
> >>>> configures exactly one thing without any side effects on already 
> >>>> configured or future entries.
> >>>>  This approach is good for cases like 
> >>>> iptables, but for SCSI targets it's possible that several configuration 
> >>>> steps require to be done in an atomic manner, like adding an iSCSI 
> >>>> target and configuring its parameters.
> >>> Well, the ability for an admin to force an LIO-Core related action, say
> >>> removing an HBA and all associated storage object with lots of exported
> >>> LUNs and running I/O, or an LIO-Target related action, say removing an
> >>> entire iSCSI Target Node with targetname=) at any time..  This obviously
> >>> require precision interaction between Target Fabric I/O Paths <-> Target
> >>> Core and Target Core <-> Control Interface to Admin.
> >>>
> >>> That control interface needs to be protected in object contexts.  In
> >>> LIO-Core this is on a per HBA (be it physical or virtual) context.  With
> >>> LIO-Target this is on a iSCSI Target Node by targetname -> Target Portal
> >>> Group Tag context.  Obviously doing this from an IOCTL is the only real
> >>> choice I had when this code started in 2001, but I wonder how configfs
> >>> would work for something like this.
> >> My favorite configuration interface would have 2 levels.
> >>
> >> 1. The lowest level would be /proc, /sys, etc. based and allow to 
> >> configure exactly one parameter or set of related parameters with 
> >> corresponding subparameters necessary to provide required atomicity, if 
> >> needed. For instance, this is how a new virtual read-only SBC device 
> >> with 4K block size is added in SCST vdisk handler:
> >>
> >> # echo "open disk_4K /disk_4K 4096 READ_ONLY" >/proc/scst/vdisk/vdisk
> >>
> > 
> > So, I would consider proc and sysfs both medieval for RW data, with the
> > latter being slightly sharper instrument than the former, but both not
> > very effective..
> > 
> > Configfs on the other hand, is quite sharp in the battlefield..  Here is
> > what I am thinking for LIO-Target after it was loaded to an configfs
> > enabled generic target engine:
> > 
> > # This is the iSCSI Qualifed Target Name we will be creating
> > MY_TARGETIQN=/config/target/iscsi/iqn.superturbodiskarray
> > # From 'lvs -v'
> > LVM_UUID=y2sbeD-insM-xykn-s3SV-3tge-VWhn-xB4FMv
> > # From 'lsscsi' or '/proc/scsi/scsi'
> > SCSI_HCTL_LOCATION=1:0:0:0
> > 
> > # Same as 'target-ctl coreaddtiqn targetname=$IQN' with LIO today
> > mkdir $MY_TARGETIQN
> > 
> > # Make TPGT=1 on iqn.superturbodiskarray
> > mkdir $MY_TARGETIQN/tpgt_1
> > 
> > # Make TPGT=1 run in Demo Mode for this example (no Initiator ACLs)
> > echo 1 > $MY_TARGETIQN/tpgt_1/attribs/generate_node_acls
> > 
> > # Create network portal mapping to TPGT=1
> > mkdir $MY_TARGETIQN/tpgt_1/np/192.168.100.10
> > 
> > # DEPENDS ON GENERIC TARGET CORE
> > # Create TPGT=1,LUN=0 from Linux LVM Block Device
> > mkdir $MY_TARGETIQN/tpgt_1/lun_0
> > echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location
> > 
> > # DEPENDS ON GENERIC TARGET CORE
> > # Create TPGT=1,LUN=1 from SCSI Layer
> > mkdir $MY_TARGETIQN/tpgt_1/lun_1
> > echo $SCSI_HCTL_LOCATION > $MY_TARGETIQN/tpgt_1/lun_1/location
> > 
> > # This is the atomic part, once we throw this flag iSCSI Initiators will
> > be allowed to login to this TPGT
> > echo 1 > $MY_TARGETIQN/tpgt_1/enable_tpg
> > 
> > # The equivliant of 'target-ctl coredeltiqn targetname=$IQN' today
> > rm -rf $MY_TARGETIQN
> > 
> > I think this would make a very useful and extremely flexiable interface
> > for my purposes...  What do you think about the potential..?
> 
> It has a big problem with atomicity of changes. Configuration of each 
> iSCSI target should be atomic 

Vlad, they are atomic.  No iSCSI Initiator are allowed to login to the
TargetName+TPGT until:

> # This is the atomic part, once we throw this flag iSCSI Initiatorswill
> > be allowed to login to this TPGT
> > echo 1 > $MY_TARGETIQN/tpgt_1/enable_tpg

Each TargetName+TPGT is protected when a 'target-ctl' IOCTL op related
to TargetName+TPGT is called.  The same is true for configfs, any time
anything under $MY_TARGETIQN is accessed or created, we take the
iscsi_tiqn_t mutex to protect it.    When anything under
$MY_TARGETIQN/tpgt_# is accessed, the iscsi_portal_group_t mutex is
protecting it.

With until enable_tpg to let initiators login to that TargetName+TPGT, I
honestly do not see your concern here.

> and on the target driver's start 
> configuration of *all* targets as a whole should be atomic as well. How 
> are you going to solve this issue?
> 

In what example would ALL iSCSI TargetName context configuration need to
be atomic to each other..?  Having the LIO-Target design above I have
never ran into a requirement like this, what requirement do you have in
mind..?  

> Plus:
> 
> 1. As I already wrote, separate subdir for each target and device is 
> harder to parse
> 
> 2. The high level interface needs to lock somehow the low level 
> interface during updates to protect from simultaneous actions. I don't 
> see a simple way to do that in configfs.
> 

Not having problem locking/mutex are in place is going to cause problems
regardless of configfs is used.  Converting LIO-Target from IOCTL ->
configfs is really easy because all of the target-ctl IOCTL ops are
already protected, so using things like a configfs trigger are simple
because I do not have to add any additional locking considerations
because the ops are already protected in the IOCTL context.

> >> 2. The higher level interface(s) would allow people to not bother with 
> >> low level commands, but use regular text config file(s). See scstadmin 
> >> utility for example. It allows to do all necessary configuration of SCST 
> >> core from /etc/scst.conf file. Such interface must have an important 
> >> property: it must be able to detect changes in the config file and apply 
> >> them to the running system. That property would allow to have system 
> >> configuration always persistent: if one needs to change something, he 
> >> would edit the config file and rerun the corresponding utility 
> >> (scstadmin in this example; it really can do that, though with some 
> >> limitations). Although this interface level would completely belong to 
> >> user space, we in kernel need to provide for it a convenient interface.
> >>
> >> Target drivers and backstorage device handlers, who need advanced 
> >> configuration, would have own low and high level interfaces, as needed. 
> >> For instance, an iSCSI target must not start serving clients until all 
> >> its targets fully configured. Otherwise, initiators can get rejected for 
> >> not yet configured target and erroneously consider it dead. In 
> >> iSCSI-SCST the user space part of the target doesn't start accepting 
> >> connection until it finishes reading /etc/iscsi-scst.conf file.
> >>
> >>>> 3. It's hard to read 5+ parameters in one command line, so it's a lot 
> >>>> easier to make a mistake there.
> >>> No, I completely agree.  But I honestly think the actual target CLI
> >>> interface and parameters to admin need to do alot of pre-execution
> >>> script logic in userspace to reference different interested objects,
> >>> without the admin have to provide all of stuff.  I do this today to
> >>> determine the major/minor for lvm_uuid= (from lvs -v), md_uuid= (from
> >>> mdadm -D) and udev_path= (from /dev/disk)..
> >>>
> >>> Same goes for real SCSI devices that we are exporting directly from
> >>> drivers/scsi.  We want to use EVPD Unit Serial or Device Identificaton
> >>> where ever able to reference said storage object.
> >> Yes, this is why we need the high level interface. Otherwise for complex 
> >> targets the configuration task quickly grows up to a nightmare.
> >>
> >>>> So, I believe, a configuration interface should be rather /proc or /sys 
> >>>> interface based. I don't think we should care much about backward 
> >>>> compatibility with tgtadm, because the existing interface doesn't reach 
> >>>> the state of being widely used.
> >>> I would definately vote against proc here for the fancy stuff I
> >>> mentioned above.  I have experience enabled core-iscsi to use sysfs for
> >>> RO data, but nothing along the lines of what would be required for a
> >>> generic target mode RW control path.  Does anyone with sysfs experience
> >>> have any comments on thing..?
> >> Sysfs as well as configfs have one big disadvantage. They limit each 
> >> file to only 4KB. This would force us for to create a subdirectory for 
> >> each device and for each connected initiator. I don't like seeing 
> >> thousands subdirectories. Additionally, such layout is a lot less 
> >> convenient for parsing for the high level configuration tool, which 
> >> needs to find out the difference between the current configuration and 
> >> content of the corresponding config file.
> >>
> > 
> > So yeah, the output with configfs is limited to PAGE_SIZE as well, but
> > for the R/W cases we don't expect that data sets to exceed this per
> > configfs mkdir invocation.. 
> > 
> >> Currently, with procfs SCST can list in /proc/scst/sessions virtually 
> >> unlimited amount of connected initiators in a simple for parsing manner. 
> >> It was done using seq interface well and simply. Neither sysfs, nor 
> >> configfs support seq interface. This would introduce significant effort 
> >> in both kernel and user spaces.
> >>
> > 
> > Same for me with all of the target-ctl IOCTL commands.   No one ever
> > said upstream target code was not going to require significant
> > effort. :-)
> 
> I don't think we should create the additional one :-)
> 

We are not creating a new one, we are using one that already exists
upstream that was made for exactly the type of problem we are looking to
solve.  Using procfs or and IOCTL for anything serious upstream is not
an option, not because they upstream maintainers like to make our life
hard, but because they are poor interfaces for what we want to do.

Vlad, please consider configfs.  After evaluating my requirements with
LIO-Target, there is no technical hangups or major gotchas I can see for
implementing the above example.  I know that LIO-Target with the example
above there are *NO* "atomicity of changes" issues I have from simply
converting IOCTL -> configfs, because the LIO-Target code called from
IOCTL already does the protection for the different contexts provided
for the example, and $MY_TARGETIQN/tpgt_#/enable_tpg protects iSCSI
Initiators from logging into that endpoint to access storage object
LUNs.

How you give an problem case where you think a generic target engine
configuration scenario would not work with configfs, and I will explain
how with LIO-Target/LIO-Core it would and does work..?

--nab

> >> Debugfs supports seq interface, but, because of the name, I doubt we can 
> >> use it ;)
> >>
> >> Thus, looks like we'd better stay with /proc. After all, networking and 
> >> VM widely use /proc for internal configuration. Why SCSI target is worse?
> >>
> > 
> > So yeah, RW configuration data going with /proc is completely
> > unacceptable for my purposes.  However since all of the LIO iSCSI and
> > SCSI MIB code is procfs + seq_file based (and is read-only) I figure
> > that considering there are other MIBs related procfs code in other
> > subsystems, that this would not be too much of a stumbling point.
> > 
> > So this configfs stuff is really starting to grow on me, I am surprised
> > that I have missed it for so long, I know the author (and alot of the
> > OCFSv2 team) and I believe their reasons for creating and moving ocfs2
> > to use configfs (even though a production ocfs2 create has much *FEWER*
> > configuration information respresented in configfs directories and
> > entries than a production iSCSI target would) provide compelling
> > evidence to move our efforts for a generic kernel target engine in this
> > direction.  So yes, I do feel your pain wrt to existing code, but I
> > believe that using procfs from my current IOCTL would quite honestly be
> > a step back.  
> > 
> > How about this, I will begin to implement the LIO-Target pieces in
> > configfs for lio-core-2.6.git, and leave LIO-Core in IOCTL for now, and
> > once I have some running code, I will look at the process to begin to
> > incorporate the requirements to preform the "dumb registration" with a
> > generic target engine.  From there, that would give me a good idea of
> > what would be required with SCST and configfs.  From the above example
> > with configfs, these would be:
> > 
> > # Create TPGT=1,LUN=0 from Linux LVM Block Device using UUID from
> > # 'lvs -v' output
> > mkdir $MY_TARGETIQN/tpgt_1/lun_0
> > echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location
> > 
> > # Create TPGT=1,LUN=1 from SCSI Layer h/c/t/l Parameters from lsscsi
> > # or /proc/scsi/scsi output
> > mkdir $MY_TARGETIQN/tpgt_1/lun_1
> > echo $SCSI_HCTL_LOCATION > $MY_TARGETIQN/tpgt_1/lun_1/location
> > 
> > Of course, we could do poke directly at Target-Core storage objects via
> > configfs as well!
> > 
> > # Preform a LUN reset on $LVM_UUID on all mapped target PORTs/LUNs
> > echo 1 > /config/target/core/$LVM_UUID/lun_reset
> > # Remove the SCSI device from the target, and all mapped PORTs/LUNs
> > # and fall all outstanding I/Os.
> > rm -rf /config/target/core/$SCSI_HTL_LOCATION
> > 
> > Again, I think this would be extremely flexible and easily extendable
> > for new uses. Also, providing some level of backwards compatibility with
> > our project's existing CLI nomenclature would not been that hard if our
> > repsective projects really require it.
> > 
> > I would also be happy to help with configfs and SCST if you are
> > interested..
> 
> Thanks, but more I'm thinking, more I'm coming to an IOCTL's based 
> interface, where each submodule has own set of IOCTL's, multiplexed over 
> a single device, and for each module there would be a simple dedicated 
> utility. For example, to configure virtual disk backstorage handler: 
> scst_vdisk_ctl, which would allow to create, delete and manage virtual 
> FILEIO/BLOCKIO disks. Or for managing security groups there would be 
> scst_acl_ctl utility. Each such utility would be very simple, would 
> offload kernel from parsing character strings or converting to them, as 
> it is necessary for configfs, each new future module would come with own 
> management tool. Git uses a similar approach: a dedicated program for 
> each action.
> 
> Data of big lists, like list of devices, would be returned at once in an 
> easy to parse form, which would considerably simplify creation of the 
> high level interface.
> 
> Such approach would be pretty simple for implementation and maintenance 
> (I love simplicity) as well as solve all other configfs problems, namely:
> 
> 1. The atomicity would be natural. An iscsi_tgt_ctl utility would parse 
> the input or corresponding config file and create a target with all 
> parameters in a single IOCTL call.
> 
> 2. Locking would be automatic via forced exclusive open without any 
> additional actions.
> 
> Also SCST /proc helpers, which target drivers and backstorage handlers 
> intensively use, would fit in this approach quite well.
> 
> How's that look?
> 
> Vlad
> 

Re: Kernel Level Generic Target Mode control path

[Vladislav Bolkhovitin]

Nicholas A. Bellinger wrote:
> On Fri, 2008-08-29 at 20:28 +0400, Vladislav Bolkhovitin wrote:
>> Nicholas A. Bellinger wrote:
>>> On Thu, 2008-08-28 at 21:52 +0400, Vladislav Bolkhovitin wrote:
>>>> Nicholas A. Bellinger wrote:
>>>>>> 2. It assumes the stateless type of configuration, when each call 
>>>>>> configures exactly one thing without any side effects on already 
>>>>>> configured or future entries.
>>>>>>  This approach is good for cases like 
>>>>>> iptables, but for SCSI targets it's possible that several configuration 
>>>>>> steps require to be done in an atomic manner, like adding an iSCSI 
>>>>>> target and configuring its parameters.
>>>>> Well, the ability for an admin to force an LIO-Core related action, say
>>>>> removing an HBA and all associated storage object with lots of exported
>>>>> LUNs and running I/O, or an LIO-Target related action, say removing an
>>>>> entire iSCSI Target Node with targetname=) at any time..  This obviously
>>>>> require precision interaction between Target Fabric I/O Paths <-> Target
>>>>> Core and Target Core <-> Control Interface to Admin.
>>>>>
>>>>> That control interface needs to be protected in object contexts.  In
>>>>> LIO-Core this is on a per HBA (be it physical or virtual) context.  With
>>>>> LIO-Target this is on a iSCSI Target Node by targetname -> Target Portal
>>>>> Group Tag context.  Obviously doing this from an IOCTL is the only real
>>>>> choice I had when this code started in 2001, but I wonder how configfs
>>>>> would work for something like this.
>>>> My favorite configuration interface would have 2 levels.
>>>>
>>>> 1. The lowest level would be /proc, /sys, etc. based and allow to 
>>>> configure exactly one parameter or set of related parameters with 
>>>> corresponding subparameters necessary to provide required atomicity, if 
>>>> needed. For instance, this is how a new virtual read-only SBC device 
>>>> with 4K block size is added in SCST vdisk handler:
>>>>
>>>> # echo "open disk_4K /disk_4K 4096 READ_ONLY" >/proc/scst/vdisk/vdisk
>>>>
>>> So, I would consider proc and sysfs both medieval for RW data, with the
>>> latter being slightly sharper instrument than the former, but both not
>>> very effective..
>>>
>>> Configfs on the other hand, is quite sharp in the battlefield..  Here is
>>> what I am thinking for LIO-Target after it was loaded to an configfs
>>> enabled generic target engine:
>>>
>>> # This is the iSCSI Qualifed Target Name we will be creating
>>> MY_TARGETIQN=/config/target/iscsi/iqn.superturbodiskarray
>>> # From 'lvs -v'
>>> LVM_UUID=y2sbeD-insM-xykn-s3SV-3tge-VWhn-xB4FMv
>>> # From 'lsscsi' or '/proc/scsi/scsi'
>>> SCSI_HCTL_LOCATION=1:0:0:0
>>>
>>> # Same as 'target-ctl coreaddtiqn targetname=$IQN' with LIO today
>>> mkdir $MY_TARGETIQN
>>>
>>> # Make TPGT=1 on iqn.superturbodiskarray
>>> mkdir $MY_TARGETIQN/tpgt_1
>>>
>>> # Make TPGT=1 run in Demo Mode for this example (no Initiator ACLs)
>>> echo 1 > $MY_TARGETIQN/tpgt_1/attribs/generate_node_acls
>>>
>>> # Create network portal mapping to TPGT=1
>>> mkdir $MY_TARGETIQN/tpgt_1/np/192.168.100.10
>>>
>>> # DEPENDS ON GENERIC TARGET CORE
>>> # Create TPGT=1,LUN=0 from Linux LVM Block Device
>>> mkdir $MY_TARGETIQN/tpgt_1/lun_0
>>> echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location
>>>
>>> # DEPENDS ON GENERIC TARGET CORE
>>> # Create TPGT=1,LUN=1 from SCSI Layer
>>> mkdir $MY_TARGETIQN/tpgt_1/lun_1
>>> echo $SCSI_HCTL_LOCATION > $MY_TARGETIQN/tpgt_1/lun_1/location
>>>
>>> # This is the atomic part, once we throw this flag iSCSI Initiators will
>>> be allowed to login to this TPGT
>>> echo 1 > $MY_TARGETIQN/tpgt_1/enable_tpg
>>>
>>> # The equivliant of 'target-ctl coredeltiqn targetname=$IQN' today
>>> rm -rf $MY_TARGETIQN
>>>
>>> I think this would make a very useful and extremely flexiable interface
>>> for my purposes...  What do you think about the potential..?
>> It has a big problem with atomicity of changes. Configuration of each 
>> iSCSI target should be atomic 
> 
> Vlad, they are atomic.  No iSCSI Initiator are allowed to login to the
> TargetName+TPGT until:
> 
>> # This is the atomic part, once we throw this flag iSCSI Initiatorswill
>>> be allowed to login to this TPGT
>>> echo 1 > $MY_TARGETIQN/tpgt_1/enable_tpg
> 
> Each TargetName+TPGT is protected when a 'target-ctl' IOCTL op related
> to TargetName+TPGT is called.  The same is true for configfs, any time
> anything under $MY_TARGETIQN is accessed or created, we take the
> iscsi_tiqn_t mutex to protect it.    When anything under
> $MY_TARGETIQN/tpgt_# is accessed, the iscsi_portal_group_t mutex is
> protecting it.
> 
> With until enable_tpg to let initiators login to that TargetName+TPGT, I
> honestly do not see your concern here.
> 
>> and on the target driver's start 
>> configuration of *all* targets as a whole should be atomic as well. How 
>> are you going to solve this issue?
>>
> 
> In what example would ALL iSCSI TargetName context configuration need to
> be atomic to each other..?  Having the LIO-Target design above I have
> never ran into a requirement like this, what requirement do you have in
> mind..?  

I've already written it: when target restarted. Disconnected initiators 
can reconnect on the wrong time and can see "no target", then consider 
it dead. The same is true for target shutdown: it should be atomic too.

>> Plus:
>>
>> 1. As I already wrote, separate subdir for each target and device is 
>> harder to parse
>>
>> 2. The high level interface needs to lock somehow the low level 
>> interface during updates to protect from simultaneous actions. I don't 
>> see a simple way to do that in configfs.
>>
> 
> Not having problem locking/mutex are in place is going to cause problems
> regardless of configfs is used.  Converting LIO-Target from IOCTL ->
> configfs is really easy because all of the target-ctl IOCTL ops are
> already protected, so using things like a configfs trigger are simple
> because I do not have to add any additional locking considerations
> because the ops are already protected in the IOCTL context.

What would be if a program, who taken that mutex get dead before 
releasing it? You wouldn't receive a notification about its death, 
although with IOCTL's you would receive it. Will you invent a motex 
revoking mechanism for that.

>>>> 2. The higher level interface(s) would allow people to not bother with 
>>>> low level commands, but use regular text config file(s). See scstadmin 
>>>> utility for example. It allows to do all necessary configuration of SCST 
>>>> core from /etc/scst.conf file. Such interface must have an important 
>>>> property: it must be able to detect changes in the config file and apply 
>>>> them to the running system. That property would allow to have system 
>>>> configuration always persistent: if one needs to change something, he 
>>>> would edit the config file and rerun the corresponding utility 
>>>> (scstadmin in this example; it really can do that, though with some 
>>>> limitations). Although this interface level would completely belong to 
>>>> user space, we in kernel need to provide for it a convenient interface.
>>>>
>>>> Target drivers and backstorage device handlers, who need advanced 
>>>> configuration, would have own low and high level interfaces, as needed. 
>>>> For instance, an iSCSI target must not start serving clients until all 
>>>> its targets fully configured. Otherwise, initiators can get rejected for 
>>>> not yet configured target and erroneously consider it dead. In 
>>>> iSCSI-SCST the user space part of the target doesn't start accepting 
>>>> connection until it finishes reading /etc/iscsi-scst.conf file.
>>>>
>>>>>> 3. It's hard to read 5+ parameters in one command line, so it's a lot 
>>>>>> easier to make a mistake there.
>>>>> No, I completely agree.  But I honestly think the actual target CLI
>>>>> interface and parameters to admin need to do alot of pre-execution
>>>>> script logic in userspace to reference different interested objects,
>>>>> without the admin have to provide all of stuff.  I do this today to
>>>>> determine the major/minor for lvm_uuid= (from lvs -v), md_uuid= (from
>>>>> mdadm -D) and udev_path= (from /dev/disk)..
>>>>>
>>>>> Same goes for real SCSI devices that we are exporting directly from
>>>>> drivers/scsi.  We want to use EVPD Unit Serial or Device Identificaton
>>>>> where ever able to reference said storage object.
>>>> Yes, this is why we need the high level interface. Otherwise for complex 
>>>> targets the configuration task quickly grows up to a nightmare.
>>>>
>>>>>> So, I believe, a configuration interface should be rather /proc or /sys 
>>>>>> interface based. I don't think we should care much about backward 
>>>>>> compatibility with tgtadm, because the existing interface doesn't reach 
>>>>>> the state of being widely used.
>>>>> I would definately vote against proc here for the fancy stuff I
>>>>> mentioned above.  I have experience enabled core-iscsi to use sysfs for
>>>>> RO data, but nothing along the lines of what would be required for a
>>>>> generic target mode RW control path.  Does anyone with sysfs experience
>>>>> have any comments on thing..?
>>>> Sysfs as well as configfs have one big disadvantage. They limit each 
>>>> file to only 4KB. This would force us for to create a subdirectory for 
>>>> each device and for each connected initiator. I don't like seeing 
>>>> thousands subdirectories. Additionally, such layout is a lot less 
>>>> convenient for parsing for the high level configuration tool, which 
>>>> needs to find out the difference between the current configuration and 
>>>> content of the corresponding config file.
>>>>
>>> So yeah, the output with configfs is limited to PAGE_SIZE as well, but
>>> for the R/W cases we don't expect that data sets to exceed this per
>>> configfs mkdir invocation.. 
>>>
>>>> Currently, with procfs SCST can list in /proc/scst/sessions virtually 
>>>> unlimited amount of connected initiators in a simple for parsing manner. 
>>>> It was done using seq interface well and simply. Neither sysfs, nor 
>>>> configfs support seq interface. This would introduce significant effort 
>>>> in both kernel and user spaces.
>>>>
>>> Same for me with all of the target-ctl IOCTL commands.   No one ever
>>> said upstream target code was not going to require significant
>>> effort. :-)
>> I don't think we should create the additional one :-)
>>
> 
> We are not creating a new one, we are using one that already exists
> upstream that was made for exactly the type of problem we are looking to
> solve.  Using procfs or and IOCTL for anything serious upstream is not
> an option, not because they upstream maintainers like to make our life
> hard, but because they are poor interfaces for what we want to do.
> 
> Vlad, please consider configfs.  After evaluating my requirements with
> LIO-Target, there is no technical hangups or major gotchas I can see for
> implementing the above example.  I know that LIO-Target with the example
> above there are *NO* "atomicity of changes" issues I have from simply
> converting IOCTL -> configfs, because the LIO-Target code called from
> IOCTL already does the protection for the different contexts provided
> for the example, and $MY_TARGETIQN/tpgt_#/enable_tpg protects iSCSI
> Initiators from logging into that endpoint to access storage object
> LUNs.
> 
> How you give an problem case where you think a generic target engine
> configuration scenario would not work with configfs, and I will explain
> how with LIO-Target/LIO-Core it would and does work..?

I have the only thing against configfs: I feel that using it could be 
harder than using well thought IOCTL interface. What's definite, that 
amount of in-kernel code for configfs will be considerably bigger, than 
for IOCTLs, but this is thing about what kernel developers do carefully 
care. In my observations

But if you so excited about configfs and willing to take care about all 
the headaches of moving to it, I won't object.

I only always thought that considering possible options means 
considering them all, but I feel you didn't read what I wrote below 
about IOCTL interface ;)

I'm leaving now, so let's return to the discussion in two weeks time.

> --nab
> 
>>>> Debugfs supports seq interface, but, because of the name, I doubt we can 
>>>> use it ;)
>>>>
>>>> Thus, looks like we'd better stay with /proc. After all, networking and 
>>>> VM widely use /proc for internal configuration. Why SCSI target is worse?
>>>>
>>> So yeah, RW configuration data going with /proc is completely
>>> unacceptable for my purposes.  However since all of the LIO iSCSI and
>>> SCSI MIB code is procfs + seq_file based (and is read-only) I figure
>>> that considering there are other MIBs related procfs code in other
>>> subsystems, that this would not be too much of a stumbling point.
>>>
>>> So this configfs stuff is really starting to grow on me, I am surprised
>>> that I have missed it for so long, I know the author (and alot of the
>>> OCFSv2 team) and I believe their reasons for creating and moving ocfs2
>>> to use configfs (even though a production ocfs2 create has much *FEWER*
>>> configuration information respresented in configfs directories and
>>> entries than a production iSCSI target would) provide compelling
>>> evidence to move our efforts for a generic kernel target engine in this
>>> direction.  So yes, I do feel your pain wrt to existing code, but I
>>> believe that using procfs from my current IOCTL would quite honestly be
>>> a step back.  
>>>
>>> How about this, I will begin to implement the LIO-Target pieces in
>>> configfs for lio-core-2.6.git, and leave LIO-Core in IOCTL for now, and
>>> once I have some running code, I will look at the process to begin to
>>> incorporate the requirements to preform the "dumb registration" with a
>>> generic target engine.  From there, that would give me a good idea of
>>> what would be required with SCST and configfs.  From the above example
>>> with configfs, these would be:
>>>
>>> # Create TPGT=1,LUN=0 from Linux LVM Block Device using UUID from
>>> # 'lvs -v' output
>>> mkdir $MY_TARGETIQN/tpgt_1/lun_0
>>> echo $LVM_UUID > $MY_TARGETIQN/tpgt_1/lun_0/location
>>>
>>> # Create TPGT=1,LUN=1 from SCSI Layer h/c/t/l Parameters from lsscsi
>>> # or /proc/scsi/scsi output
>>> mkdir $MY_TARGETIQN/tpgt_1/lun_1
>>> echo $SCSI_HCTL_LOCATION > $MY_TARGETIQN/tpgt_1/lun_1/location
>>>
>>> Of course, we could do poke directly at Target-Core storage objects via
>>> configfs as well!
>>>
>>> # Preform a LUN reset on $LVM_UUID on all mapped target PORTs/LUNs
>>> echo 1 > /config/target/core/$LVM_UUID/lun_reset
>>> # Remove the SCSI device from the target, and all mapped PORTs/LUNs
>>> # and fall all outstanding I/Os.
>>> rm -rf /config/target/core/$SCSI_HTL_LOCATION
>>>
>>> Again, I think this would be extremely flexible and easily extendable
>>> for new uses. Also, providing some level of backwards compatibility with
>>> our project's existing CLI nomenclature would not been that hard if our
>>> repsective projects really require it.
>>>
>>> I would also be happy to help with configfs and SCST if you are
>>> interested..
>> Thanks, but more I'm thinking, more I'm coming to an IOCTL's based 
>> interface, where each submodule has own set of IOCTL's, multiplexed over 
>> a single device, and for each module there would be a simple dedicated 
>> utility. For example, to configure virtual disk backstorage handler: 
>> scst_vdisk_ctl, which would allow to create, delete and manage virtual 
>> FILEIO/BLOCKIO disks. Or for managing security groups there would be 
>> scst_acl_ctl utility. Each such utility would be very simple, would 
>> offload kernel from parsing character strings or converting to them, as 
>> it is necessary for configfs, each new future module would come with own 
>> management tool. Git uses a similar approach: a dedicated program for 
>> each action.
>>
>> Data of big lists, like list of devices, would be returned at once in an 
>> easy to parse form, which would considerably simplify creation of the 
>> high level interface.
>>
>> Such approach would be pretty simple for implementation and maintenance 
>> (I love simplicity) as well as solve all other configfs problems, namely:
>>
>> 1. The atomicity would be natural. An iscsi_tgt_ctl utility would parse 
>> the input or corresponding config file and create a target with all 
>> parameters in a single IOCTL call.
>>
>> 2. Locking would be automatic via forced exclusive open without any 
>> additional actions.
>>
>> Also SCST /proc helpers, which target drivers and backstorage handlers 
>> intensively use, would fit in this approach quite well.
>>
>> How's that look?
>>
>> Vlad
>>
> 
> 

Re: Kernel Level Generic Target Mode control path

[Bart Van Assche]

On Sat, Aug 30, 2008 at 10:53 PM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
> What would be if a program, who taken that mutex get dead before releasing
> it? You wouldn't receive a notification about its death, although with
> IOCTL's you would receive it. Will you invent a motex revoking mechanism for
> that.

Any mutex that has been locked by kernel code should be unlocked
before control is given back to userspace. Making kernel mutexes
visible to userspace would be very bad design.

Bart.

Re: Kernel Level Generic Target Mode control path

[Nicholas A. Bellinger]

On Sun, 2008-08-31 at 00:53 +0400, Vladislav Bolkhovitin wrote:
> Nicholas A. Bellinger wrote:
> > On Fri, 2008-08-29 at 20:28 +0400, Vladislav Bolkhovitin wrote:
> >> Nicholas A. Bellinger wrote:
> >
> > Vlad, they are atomic.  No iSCSI Initiator are allowed to login to the
> > TargetName+TPGT until:
> > 
> >> # This is the atomic part, once we throw this flag iSCSI Initiatorswill
> >>> be allowed to login to this TPGT
> >>> echo 1 > $MY_TARGETIQN/tpgt_1/enable_tpg
> > 
> > Each TargetName+TPGT is protected when a 'target-ctl' IOCTL op related
> > to TargetName+TPGT is called.  The same is true for configfs, any time
> > anything under $MY_TARGETIQN is accessed or created, we take the
> > iscsi_tiqn_t mutex to protect it.    When anything under
> > $MY_TARGETIQN/tpgt_# is accessed, the iscsi_portal_group_t mutex is
> > protecting it.
> > 
> > With until enable_tpg to let initiators login to that TargetName+TPGT, I
> > honestly do not see your concern here.
> > 
> >> and on the target driver's start 
> >> configuration of *all* targets as a whole should be atomic as well. How 
> >> are you going to solve this issue?
> >>
> > 
> > In what example would ALL iSCSI TargetName context configuration need to
> > be atomic to each other..?  Having the LIO-Target design above I have
> > never ran into a requirement like this, what requirement do you have in
> > mind..?  
> 
> I've already written it: when target restarted. Disconnected initiators 
> can reconnect on the wrong time and can see "no target", then consider 
> it dead.

Yes, this case you describe is exactly what I am talking about with the
"enable_tpg" flag for LIO-Target under configfs.  Each fabric can define
it's own configfs groups and items to fit its own requirements, which
means, that a global "allow initiators to login" flag would like look:

FABRIC=pscsi

# Allow all initiators to login to all Parallel SCSI endpoints
echo 1 > /sys/kernel/config/target/$FABRIC/allow_initiators_to_login

Or, just enable one endpoint on another fabric:

FABRIC=sas

# Only allow initiators to login to sas_endpoint_name 
echo 1 > /sys/kernel/config/target/$FABRIC/sas_endpoint_name/enable_login

Anyways, you get the idea.  The great part of course is that this
requires zero changes to userspace code!!

>  The same is true for target shutdown: it should be atomic too.
> 

I am not debating these fundementals, I am mearly saying that depending
upon configfs for target configuration would entail the same types of
logic and mutual exclusion requirements in procfs, sysfs, or an IOCTL.

> >> 2. The high level interface needs to lock somehow the low level 
> >> interface during updates to protect from simultaneous actions. I don't 
> >> see a simple way to do that in configfs.
> >>
> > 
> > Not having problem locking/mutex are in place is going to cause problems
> > regardless of configfs is used.  Converting LIO-Target from IOCTL ->
> > configfs is really easy because all of the target-ctl IOCTL ops are
> > already protected, so using things like a configfs trigger are simple
> > because I do not have to add any additional locking considerations
> > because the ops are already protected in the IOCTL context.
> 
> What would be if a program, who taken that mutex get dead before 
> releasing it? You wouldn't receive a notification about its death, 
> although with IOCTL's you would receive it. Will you invent a motex 
> revoking mechanism for that.
> 

Hrrmm, why would an IOCTL process context be any different than a
configfs process context..?  Just because a userspace process is running
kernel code and user hits CTRL-C, or the userspace code segfaults, does
not mean the kernel code stops running, it just means that those SIGINT
or SIGSEGVs are pending for that process and down_interruptible()
semaphores will not sleep and signal_pending(current) will return TRUE,
etc.

I still don't see how the case of a userspace process accessing kernel
code would be different between an IOCTL and configfs process context..?


> >>>> Sysfs as well as configfs have one big disadvantage. They limit each 
> >>>> file to only 4KB. This would force us for to create a subdirectory for 
> >>>> each device and for each connected initiator. I don't like seeing 
> >>>> thousands subdirectories. Additionally, such layout is a lot less 
> >>>> convenient for parsing for the high level configuration tool, which 
> >>>> needs to find out the difference between the current configuration and 
> >>>> content of the corresponding config file.
> >>>>
> >>> So yeah, the output with configfs is limited to PAGE_SIZE as well, but
> >>> for the R/W cases we don't expect that data sets to exceed this per
> >>> configfs mkdir invocation.. 
> >>>
> >>>> Currently, with procfs SCST can list in /proc/scst/sessions virtually 
> >>>> unlimited amount of connected initiators in a simple for parsing manner. 
> >>>> It was done using seq interface well and simply. Neither sysfs, nor 
> >>>> configfs support seq interface. This would introduce significant effort 
> >>>> in both kernel and user spaces.
> >>>>
> >>> Same for me with all of the target-ctl IOCTL commands.   No one ever
> >>> said upstream target code was not going to require significant
> >>> effort. :-)
> >> I don't think we should create the additional one :-)
> >>
> > 
> > We are not creating a new one, we are using one that already exists
> > upstream that was made for exactly the type of problem we are looking to
> > solve.  Using procfs or and IOCTL for anything serious upstream is not
> > an option, not because they upstream maintainers like to make our life
> > hard, but because they are poor interfaces for what we want to do.
> > 
> > Vlad, please consider configfs.  After evaluating my requirements with
> > LIO-Target, there is no technical hangups or major gotchas I can see for
> > implementing the above example.  I know that LIO-Target with the example
> > above there are *NO* "atomicity of changes" issues I have from simply
> > converting IOCTL -> configfs, because the LIO-Target code called from
> > IOCTL already does the protection for the different contexts provided
> > for the example, and $MY_TARGETIQN/tpgt_#/enable_tpg protects iSCSI
> > Initiators from logging into that endpoint to access storage object
> > LUNs.
> > 
> > How you give an problem case where you think a generic target engine
> > configuration scenario would not work with configfs, and I will explain
> > how with LIO-Target/LIO-Core it would and does work..?
> 
> I have the only thing against configfs: I feel that using it could be 
> harder than using well thought IOCTL interface. What's definite, that 
> amount of in-kernel code for configfs will be considerably bigger, than 
> for IOCTLs, but this is thing about what kernel developers do carefully 
> care. In my observations
> 

I think the kernel code would be a bit larger using configfs, but equal
logic and complexity with an IOCTL.  The great benefit with configfs of
course is that you never have to break compatibility from having to
change your struct foo that is passed with the IOCTL between
kernel/user.  Also, not having to maintain a mess of C userspace code is
always a benefit.  With configfs the target CLI that I have in mind
would basically be all shell scripts reference admin defined human
readable aliases mapped to targetname, target portals, storage objects,
LUNs, etc.

> But if you so excited about configfs and willing to take care about all 
> the headaches of moving to it, I won't object.
> 

:-)

> I only always thought that considering possible options means 
> considering them all, but I feel you didn't read what I wrote below 
> about IOCTL interface ;)
> 

Well, since I have been using an IOCTL for so long with LIO, I am
painfully fimilar with the benefits and limitations..

> I'm leaving now, so let's return to the discussion in two weeks time.
> 

Sounds good!

--nab

Re: Kernel Level Generic Target Mode control path

[Vladislav Bolkhovitin]

Nicholas A. Bellinger wrote:
> On Sun, 2008-08-31 at 00:53 +0400, Vladislav Bolkhovitin wrote:
>> Nicholas A. Bellinger wrote:
>>> On Fri, 2008-08-29 at 20:28 +0400, Vladislav Bolkhovitin wrote:
>>>> Nicholas A. Bellinger wrote:
>>> Vlad, they are atomic.  No iSCSI Initiator are allowed to login to the
>>> TargetName+TPGT until:
>>>
>>>> # This is the atomic part, once we throw this flag iSCSI Initiatorswill
>>>>> be allowed to login to this TPGT
>>>>> echo 1 > $MY_TARGETIQN/tpgt_1/enable_tpg
>>> Each TargetName+TPGT is protected when a 'target-ctl' IOCTL op related
>>> to TargetName+TPGT is called.  The same is true for configfs, any time
>>> anything under $MY_TARGETIQN is accessed or created, we take the
>>> iscsi_tiqn_t mutex to protect it.    When anything under
>>> $MY_TARGETIQN/tpgt_# is accessed, the iscsi_portal_group_t mutex is
>>> protecting it.
>>>
>>> With until enable_tpg to let initiators login to that TargetName+TPGT, I
>>> honestly do not see your concern here.
>>>
>>>> and on the target driver's start 
>>>> configuration of *all* targets as a whole should be atomic as well. How 
>>>> are you going to solve this issue?
>>>>
>>> In what example would ALL iSCSI TargetName context configuration need to
>>> be atomic to each other..?  Having the LIO-Target design above I have
>>> never ran into a requirement like this, what requirement do you have in
>>> mind..?  
>> I've already written it: when target restarted. Disconnected initiators 
>> can reconnect on the wrong time and can see "no target", then consider 
>> it dead.
> 
> Yes, this case you describe is exactly what I am talking about with the
> "enable_tpg" flag for LIO-Target under configfs.  Each fabric can define
> it's own configfs groups and items to fit its own requirements, which
> means, that a global "allow initiators to login" flag would like look:
> 
> FABRIC=pscsi
> 
> # Allow all initiators to login to all Parallel SCSI endpoints
> echo 1 > /sys/kernel/config/target/$FABRIC/allow_initiators_to_login
> 
> Or, just enable one endpoint on another fabric:
> 
> FABRIC=sas
> 
> # Only allow initiators to login to sas_endpoint_name 
> echo 1 > /sys/kernel/config/target/$FABRIC/sas_endpoint_name/enable_login
> 
> Anyways, you get the idea.  The great part of course is that this
> requires zero changes to userspace code!!
> 
>>  The same is true for target shutdown: it should be atomic too.
>>
> 
> I am not debating these fundementals, I am mearly saying that depending
> upon configfs for target configuration would entail the same types of
> logic and mutual exclusion requirements in procfs, sysfs, or an IOCTL.

Basically, yes, but there are small differences, which *might* be 
important.

1. If an application, holding IOCTL device fd, dies, the IOCTL handler 
receives notification, on which it can clean all associated resources. 
This seems isn't possible with configfs. Or do I miss something?

2. In IOCTL you can make a bigger piece of work at once, i.e. atomically.

>>>> 2. The high level interface needs to lock somehow the low level 
>>>> interface during updates to protect from simultaneous actions. I don't 
>>>> see a simple way to do that in configfs.
>>>>
>>> Not having problem locking/mutex are in place is going to cause problems
>>> regardless of configfs is used.  Converting LIO-Target from IOCTL ->
>>> configfs is really easy because all of the target-ctl IOCTL ops are
>>> already protected, so using things like a configfs trigger are simple
>>> because I do not have to add any additional locking considerations
>>> because the ops are already protected in the IOCTL context.
>> What would be if a program, who taken that mutex get dead before 
>> releasing it? You wouldn't receive a notification about its death, 
>> although with IOCTL's you would receive it. Will you invent a motex 
>> revoking mechanism for that.
>>
> 
> Hrrmm, why would an IOCTL process context be any different than a
> configfs process context..?  Just because a userspace process is running
> kernel code and user hits CTRL-C, or the userspace code segfaults, does
> not mean the kernel code stops running, it just means that those SIGINT
> or SIGSEGVs are pending for that process and down_interruptible()
> semaphores will not sleep and signal_pending(current) will return TRUE,
> etc.
> 
> I still don't see how the case of a userspace process accessing kernel
> code would be different between an IOCTL and configfs process context..?

See (1) above.

Anyway, as I wrote before, if you like configfs and there are no 
objections from other people, let's go with configfs.

>>>>>> Sysfs as well as configfs have one big disadvantage. They limit each 
>>>>>> file to only 4KB. This would force us for to create a subdirectory for 
>>>>>> each device and for each connected initiator. I don't like seeing 
>>>>>> thousands subdirectories. Additionally, such layout is a lot less 
>>>>>> convenient for parsing for the high level configuration tool, which 
>>>>>> needs to find out the difference between the current configuration and 
>>>>>> content of the corresponding config file.
>>>>>>
>>>>> So yeah, the output with configfs is limited to PAGE_SIZE as well, but
>>>>> for the R/W cases we don't expect that data sets to exceed this per
>>>>> configfs mkdir invocation.. 
>>>>>
>>>>>> Currently, with procfs SCST can list in /proc/scst/sessions virtually 
>>>>>> unlimited amount of connected initiators in a simple for parsing manner. 
>>>>>> It was done using seq interface well and simply. Neither sysfs, nor 
>>>>>> configfs support seq interface. This would introduce significant effort 
>>>>>> in both kernel and user spaces.
>>>>>>
>>>>> Same for me with all of the target-ctl IOCTL commands.   No one ever
>>>>> said upstream target code was not going to require significant
>>>>> effort. :-)
>>>> I don't think we should create the additional one :-)
>>>>
>>> We are not creating a new one, we are using one that already exists
>>> upstream that was made for exactly the type of problem we are looking to
>>> solve.  Using procfs or and IOCTL for anything serious upstream is not
>>> an option, not because they upstream maintainers like to make our life
>>> hard, but because they are poor interfaces for what we want to do.
>>>
>>> Vlad, please consider configfs.  After evaluating my requirements with
>>> LIO-Target, there is no technical hangups or major gotchas I can see for
>>> implementing the above example.  I know that LIO-Target with the example
>>> above there are *NO* "atomicity of changes" issues I have from simply
>>> converting IOCTL -> configfs, because the LIO-Target code called from
>>> IOCTL already does the protection for the different contexts provided
>>> for the example, and $MY_TARGETIQN/tpgt_#/enable_tpg protects iSCSI
>>> Initiators from logging into that endpoint to access storage object
>>> LUNs.
>>>
>>> How you give an problem case where you think a generic target engine
>>> configuration scenario would not work with configfs, and I will explain
>>> how with LIO-Target/LIO-Core it would and does work..?
>> I have the only thing against configfs: I feel that using it could be 
>> harder than using well thought IOCTL interface. What's definite, that 
>> amount of in-kernel code for configfs will be considerably bigger, than 
>> for IOCTLs, but this is thing about what kernel developers do carefully 
>> care. In my observations
> 
> I think the kernel code would be a bit larger using configfs, but equal
> logic and complexity with an IOCTL.  The great benefit with configfs of
> course is that you never have to break compatibility from having to
> change your struct foo that is passed with the IOCTL between
> kernel/user.  Also, not having to maintain a mess of C userspace code is
> always a benefit.  With configfs the target CLI that I have in mind
> would basically be all shell scripts reference admin defined human
> readable aliases mapped to targetname, target portals, storage objects,
> LUNs, etc.

Have you considered an application, like scstadmin, which needs to parse 
all existing configuration to figure out the difference between it and 
the corresponding config file? The layout should be friendly for such 
applications.

Vlad

Re: Kernel Level Generic Target Mode control path

[Bart Van Assche]

On Fri, Aug 29, 2008 at 6:28 PM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
> It has a big problem with atomicity of changes. Configuration of each iSCSI
> target should be atomic and on the target driver's start configuration of
> *all* targets as a whole should be atomic as well. How are you going to
> solve this issue?

One way to solve this (I'm not saying it's the best way to solve this)
is to implement a transaction interface through configfs. A configfs
interface for transactions could e.g. be implemented as follows:
1. Provide one integer variable representing the sequence number of
the ongoing transaction. This variable is incremented before its value
is returned to userspace.
2. Provide as many variables as needed to allow entry of all the data
involved in the transaction.
3. Provide one variable for requesting a commit. A commit is requested
by writing a number to this variable. If and only if the value written
equals the current value of counter (1), a commit is performed.
Whether or not this action triggered a commit must be reported to
userspace, e.g. through errno.

User space processes must use this interface as follows:
(a) Read counter (1).
(b) Write all relevant data to (2).
(c) Write the value read in step (a) to variable (3).
(d) If step (c) reported that the transaction failed, go back to step (a).

This is an optimistic locking scheme that can be used by several
processes simultaneously and that survives processes that are killed
while performing a transaction.

Bart.

Re: Kernel Level Generic Target Mode control path

[Vladislav Bolkhovitin]

Bart Van Assche wrote:
> On Fri, Aug 29, 2008 at 6:28 PM, Vladislav Bolkhovitin <vst@vlnb.net> wrote:
>> It has a big problem with atomicity of changes. Configuration of each iSCSI
>> target should be atomic and on the target driver's start configuration of
>> *all* targets as a whole should be atomic as well. How are you going to
>> solve this issue?
> 
> One way to solve this (I'm not saying it's the best way to solve this)
> is to implement a transaction interface through configfs. A configfs
> interface for transactions could e.g. be implemented as follows:
> 1. Provide one integer variable representing the sequence number of
> the ongoing transaction. This variable is incremented before its value
> is returned to userspace.
> 2. Provide as many variables as needed to allow entry of all the data
> involved in the transaction.
> 3. Provide one variable for requesting a commit. A commit is requested
> by writing a number to this variable. If and only if the value written
> equals the current value of counter (1), a commit is performed.
> Whether or not this action triggered a commit must be reported to
> userspace, e.g. through errno.
> 
> User space processes must use this interface as follows:
> (a) Read counter (1).
> (b) Write all relevant data to (2).
> (c) Write the value read in step (a) to variable (3).
> (d) If step (c) reported that the transaction failed, go back to step (a).
> 
> This is an optimistic locking scheme that can be used by several
> processes simultaneously and that survives processes that are killed
> while performing a transaction.

Unfortunately, this seems to be a lot more complicated, than I would 
want to have...

> Bart.
> 

Re: [Ksummit-2008-discuss] Kernel Summit Request for Discussion: The Future of Target mode and Cloud storage on Linux

[Arjan van de Ven]

Vladislav Bolkhovitin wrote:
> (My apologies that I'm late to this discussion. I wasn't subscribed to 
> ksummit-2008-discuss, so wasn't aware about it. Big thanks to Nicholas 
> 

no need to apologize; it's not the right list for this to be discussed anyway,
and it's increasingly clear that this isn't a topic for the Kernel Summit... it's
a topic where the stakeholders really need to get their act together and in a storage
forum come up with some agreement.....