Re: Why using configfs as the only interface is wrong for a storage target

Re: Why using configfs as the only interface is wrong for a storage target

[Bart Van Assche]

On Fri, Feb 4, 2011 at 7:45 AM, Nicholas A. Bellinger
<nab@linux-iscsi.org> wrote:
> Please consider the following patch series for mainline target code.
> It consists of predominately configfs bugfixes uncovered with recent SLUB
> poison testing, and proper removal of legacy procfs target_core_mib.c code.
> Note that the complete set of fabric independent statistics (SCSI MIBs) and
> fabric dependent statistics will be included as native configfs group context
> 'per value' attribute series during the .39 time frame.

I'm still not convinced that using configfs in a storage target as the
only interface between kernel space and user space is a good idea.
While configfs may satisfy all the needs of an iSCSI target, the use
of configfs in combination with hot-pluggable HCAs is really awkward.
Whenever a HCA is plugged in, the user has to issue mkdir commands to
make these interfaces appear in configfs. And whenever a HCA is
removed, stale information will remain present in configfs until the
user issues an rmdir command. As we all know, it is not possible for a
storage target to make these directories appear / disappear
automatically in configfs because of basic configfs design choices.

Regarding the SCSI-MIB (RFC 4455): using configfs makes it impossible
to ensure that the information in the scsiTgtPortEntry table is
current. If a HCA is plugged in, it won't appear in that table until
the user or some software issues an mkdir command. If a HCA is
removed, it won't appear in that table until an rmdir command is
issued. All these mkdir / rmdir commands are redundant since the
target core already knows which HCAs are available - a natural way to
implement a storage target driver that has to handle HCA hot-plugging
is to let it tell the storage target core about hot-plug events in the
bus add_one / remove_one callbacks.

Bart.

Re: Why using configfs as the only interface is wrong for a storage target

[Joel Becker]

On Mon, Feb 07, 2011 at 12:41:18PM +0100, Bart Van Assche wrote:
> On Fri, Feb 4, 2011 at 7:45 AM, Nicholas A. Bellinger
> <nab@linux-iscsi.org> wrote:
> > Please consider the following patch series for mainline target code.
> > It consists of predominately configfs bugfixes uncovered with recent SLUB
> > poison testing, and proper removal of legacy procfs target_core_mib.c code.
> > Note that the complete set of fabric independent statistics (SCSI MIBs) and
> > fabric dependent statistics will be included as native configfs group context
> > 'per value' attribute series during the .39 time frame.
> 
> I'm still not convinced that using configfs in a storage target as the
> only interface between kernel space and user space is a good idea.
> While configfs may satisfy all the needs of an iSCSI target, the use
> of configfs in combination with hot-pluggable HCAs is really awkward.
> Whenever a HCA is plugged in, the user has to issue mkdir commands to
> make these interfaces appear in configfs. And whenever a HCA is
> removed, stale information will remain present in configfs until the
> user issues an rmdir command. As we all know, it is not possible for a
> storage target to make these directories appear / disappear
> automatically in configfs because of basic configfs design choices.

	Any configuration would have to be handled.  We have plenty of
stuff that is handled by userspace hooks out of udev, etc.  That's a
normal hotplug process.
	Essentially, you're not challenging Nick's use of configfs here,
you're challenging his environment of setting up the target stack from
userspace.

Joel

-- 

"Sometimes I think the surest sign intelligent
 life exists elsewhere in the universe is that
 none of it has tried to contact us."
                                -Calvin & Hobbes

			http://www.jlbec.org/
			jlbec@evilplan.org

Re: Why using configfs as the only interface is wrong for a storage target

[Bart Van Assche]

On Mon, Feb 7, 2011 at 12:53 PM, Joel Becker <jlbec@evilplan.org> wrote:
>
> On Mon, Feb 07, 2011 at 12:41:18PM +0100, Bart Van Assche wrote:
> > On Fri, Feb 4, 2011 at 7:45 AM, Nicholas A. Bellinger
> > <nab@linux-iscsi.org> wrote:
> > > Please consider the following patch series for mainline target code.
> > > It consists of predominately configfs bugfixes uncovered with recent SLUB
> > > poison testing, and proper removal of legacy procfs target_core_mib.c code.
> > > Note that the complete set of fabric independent statistics (SCSI MIBs) and
> > > fabric dependent statistics will be included as native configfs group context
> > > 'per value' attribute series during the .39 time frame.
> >
> > I'm still not convinced that using configfs in a storage target as the
> > only interface between kernel space and user space is a good idea.
> > While configfs may satisfy all the needs of an iSCSI target, the use
> > of configfs in combination with hot-pluggable HCAs is really awkward.
> > Whenever a HCA is plugged in, the user has to issue mkdir commands to
> > make these interfaces appear in configfs. And whenever a HCA is
> > removed, stale information will remain present in configfs until the
> > user issues an rmdir command. As we all know, it is not possible for a
> > storage target to make these directories appear / disappear
> > automatically in configfs because of basic configfs design choices.
>
>        Any configuration would have to be handled.  We have plenty of
> stuff that is handled by userspace hooks out of udev, etc.  That's a
> normal hotplug process.
>        Essentially, you're not challenging Nick's use of configfs here,
> you're challenging his environment of setting up the target stack from
> userspace.

Hello Joel,

While I'm fine with using configfs where appropriate, I do not agree
with the choice of configfs as the only interface between user space
and kernel for a storage target. It seems overkill to me to depend on
user space software to make sure that the user space visible
information about HCAs is up to date while the target core could
easily ensure itself that such information would be up to date if it
would be using sysfs instead of configfs.

Furthermore, there is currently no mechanism present in the target
code to inform user space about HCA list updates. Even more: no
mechanism has been implemented yet to allow user space to find out
which HCAs are controlled by a target driver nor which names have been
assigned by a target driver to the HCAs it controls. This seems like
an oversight to me.

Regarding udev: no udev events are generated for HCAs controlled by
target drivers since the objects created by target drivers are not
regular devices.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why using configfs as the only interface is wrong for a storage target

[Nicholas A. Bellinger]

On Mon, 2011-02-07 at 13:08 +0100, Bart Van Assche wrote:
> On Mon, Feb 7, 2011 at 12:53 PM, Joel Becker <jlbec@evilplan.org> wrote:
> >
> > On Mon, Feb 07, 2011 at 12:41:18PM +0100, Bart Van Assche wrote:
> > > On Fri, Feb 4, 2011 at 7:45 AM, Nicholas A. Bellinger
> > > <nab@linux-iscsi.org> wrote:
> > > > Please consider the following patch series for mainline target code.
> > > > It consists of predominately configfs bugfixes uncovered with recent SLUB
> > > > poison testing, and proper removal of legacy procfs target_core_mib.c code.
> > > > Note that the complete set of fabric independent statistics (SCSI MIBs) and
> > > > fabric dependent statistics will be included as native configfs group context
> > > > 'per value' attribute series during the .39 time frame.
> > >
> > > I'm still not convinced that using configfs in a storage target as the
> > > only interface between kernel space and user space is a good idea.
> > > While configfs may satisfy all the needs of an iSCSI target, the use
> > > of configfs in combination with hot-pluggable HCAs is really awkward.
> > > Whenever a HCA is plugged in, the user has to issue mkdir commands to
> > > make these interfaces appear in configfs. And whenever a HCA is
> > > removed, stale information will remain present in configfs until the
> > > user issues an rmdir command. As we all know, it is not possible for a
> > > storage target to make these directories appear / disappear
> > > automatically in configfs because of basic configfs design choices.
> >
> >        Any configuration would have to be handled.  We have plenty of
> > stuff that is handled by userspace hooks out of udev, etc.  That's a
> > normal hotplug process.
> >        Essentially, you're not challenging Nick's use of configfs here,
> > you're challenging his environment of setting up the target stack from
> > userspace.
> 
> Hello Joel,
> 
> While I'm fine with using configfs where appropriate, I do not agree
> with the choice of configfs as the only interface between user space
> and kernel for a storage target. It seems overkill to me to depend on
> user space software to make sure that the user space visible
> information about HCAs is up to date while the target core could
> easily ensure itself that such information would be up to date if it
> would be using sysfs instead of configfs.
> 

I am not sure why you think someone absoulutely has to do a struct
configfs_group_operations->make_group() just to see if HW exists that is
target mode operation..  What about providing
a /sys/kernel/config/target/$FABRIC_MODE/available_targets attribute
with a list of which HW is accessable by userspace code driving creation
of struct config_group..?  What able reading existing sysfs
supported_modes information from /sys/class/scsi_host/ and determine HW
target mode specific port WWN information and using python code to drive
creation of /sys/kernel/config/target/$FABRIC_MOD/$FABRIC_WWPN..?

> Furthermore, there is currently no mechanism present in the target
> code to inform user space about HCA list updates. Even more: no
> mechanism has been implemented yet to allow user space to find out
> which HCAs are controlled by a target driver nor which names have been
> assigned by a target driver to the HCAs it controls. This seems like
> an oversight to me.
> 

Again, we use userspace code to *drive* the configfs layout, and create
explict parent/child relationships between kernel data structures
containing struct config_group members.  This allows the vast majority
of our data structure reference counting to be done using configfs/VFS
level protection instead of with stupid and clumsly ad-hoc structure
referencing code.

Just to refresh your memory, the main reasons why we are using configfs
for target mode are:

*) configfs represents the individual target data structures who's
creation/deletion are driven from enterily userspace.

*) The parent/child relationships of dependant data structures is
handled transparently to the configfs consumer (eg: no hard requirement
internal reference counting)

*) The module reference counting of target core -> fabric module is
handled transparently to configfs consumers *and* TCM fabric modules

*) The current implementation of TCM/ConfigFS contains no global locks.
Each /sys/kernel/config/target/$FABRIC_1 operates independently of
/sys/kernel/config/target/$FABRIC_2

If you think there is a geninue issue then I suggest you send out a
thread with git-format-patch'ed output for discussion instead of
high-jacking a patch series thread with your preconceived notions of how
user shell interfaction and userspace code should work with kernel level
configfs layout for HW target mode.  That would certainly be a more
effective method of getting me (or anyone) to address what you consider
to be shortcomings with code for kernel-level configfs/sysfs target code
than a mashup of items from your current TODO List.

--nab

Re: Why using configfs as the only interface is wrong for a storage target

[James Bottomley]

On Mon, 2011-02-07 at 03:53 -0800, Joel Becker wrote:
> On Mon, Feb 07, 2011 at 12:41:18PM +0100, Bart Van Assche wrote:
> > On Fri, Feb 4, 2011 at 7:45 AM, Nicholas A. Bellinger
> > <nab@linux-iscsi.org> wrote:
> > > Please consider the following patch series for mainline target code.
> > > It consists of predominately configfs bugfixes uncovered with recent SLUB
> > > poison testing, and proper removal of legacy procfs target_core_mib.c code.
> > > Note that the complete set of fabric independent statistics (SCSI MIBs) and
> > > fabric dependent statistics will be included as native configfs group context
> > > 'per value' attribute series during the .39 time frame.
> > 
> > I'm still not convinced that using configfs in a storage target as the
> > only interface between kernel space and user space is a good idea.
> > While configfs may satisfy all the needs of an iSCSI target, the use
> > of configfs in combination with hot-pluggable HCAs is really awkward.
> > Whenever a HCA is plugged in, the user has to issue mkdir commands to
> > make these interfaces appear in configfs. And whenever a HCA is
> > removed, stale information will remain present in configfs until the
> > user issues an rmdir command. As we all know, it is not possible for a
> > storage target to make these directories appear / disappear
> > automatically in configfs because of basic configfs design choices.
> 
> 	Any configuration would have to be handled.  We have plenty of
> stuff that is handled by userspace hooks out of udev, etc.  That's a
> normal hotplug process.
> 	Essentially, you're not challenging Nick's use of configfs here,
> you're challenging his environment of setting up the target stack from
> userspace.

I think the overall philosophical point here, and it's a good one
because I've heard it from several sources, is that it's not possible to
separate configuration from status completely.  The classic example is
where the kernel has to validate and adjust config information, but the
storage specific one is where events alter the topology.  In either
case, the configfs tree gets out of sync with reality if the kernel does
the adjustment..  Just saying we have to use a user space tool to fix it
up is a bit of a cop out because the kernel has already adjusted its own
configuration, so getting userspace to work out what the kernel's done
and adjust configfs is a bit sub optimal.

James

Re: Why using configfs as the only interface is wrong for a storage target

[Emmanuel Florac]

Le Mon, 07 Feb 2011 08:41:49 -0600
James Bottomley <James.Bottomley@suse.de> écrivait:

> I think the overall philosophical point here, and it's a good one
> because I've heard it from several sources, is that it's not possible
> to separate configuration from status completely. 

This would be true only if the machines weren't ever rebooted. Out of
this uncommon case, this is even worse than the previous state of the
matter : instead of having a (eventually nice, human readable)
configuration file, you need a real program to create dynamically the
devices at boot time, which is therefore your ACTUAL configuration
file. So all you're achieving with this philosophy is replacing data
files by programs -- anyone but kernel programmers will find this a
terrible idea.

Then to avoid the hassle, someone will have the clever idea to build a
way to load an INI file into configfs through a specially crafted
program living in /etc/init.d, and we'll have been going full circle.
Slow clap... This is the most brain-dead idea I've heard of in a long
time.

configfs would be a good idea only if the kernel could make it
transparently persistent. Until that happens, it's a near perfect
administor pet peeve.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why using configfs as the only interface is wrong for a storage target

[Nicholas A. Bellinger]

On Mon, 2011-02-07 at 16:02 +0100, Emmanuel Florac wrote:
> Le Mon, 07 Feb 2011 08:41:49 -0600
> James Bottomley <James.Bottomley@suse.de> écrivait:
> 
> > I think the overall philosophical point here, and it's a good one
> > because I've heard it from several sources, is that it's not possible
> > to separate configuration from status completely. 
> 
> This would be true only if the machines weren't ever rebooted. Out of
> this uncommon case, this is even worse than the previous state of the
> matter : instead of having a (eventually nice, human readable)
> configuration file, you need a real program to create dynamically the
> devices at boot time, which is therefore your ACTUAL configuration
> file. So all you're achieving with this philosophy is replacing data
> files by programs -- anyone but kernel programmers will find this a
> terrible idea.
> 
> Then to avoid the hassle, someone will have the clever idea to build a
> way to load an INI file into configfs through a specially crafted
> program living in /etc/init.d, and we'll have been going full circle.
> Slow clap... This is the most brain-dead idea I've heard of in a long
> time.
> 

Sorry, but I have no idea what you are talking about.

We use direct configfs mkdirs and symlinks in our configuration files.
There are special cases where we use lio-utils.git python code to load
additional backend device specific metadata (like T10 WWN, persistent
reservations APTPL, ALUA multipath secondary port state, etc)

The whole idea is that configfs re-presents the parent/child
relationships of data structures who's creation/deletion is driven by
userspace.  Having python code walk this layout and output the exact
running configuration is how we save -> reinstate the running config,
and it's really quite trivial with configfs.

But that is just the simple stuff, where configfs really shines is with
a object oriented library and high level interactive shell that allows
the configfs hierarchy to be fully utilized by application level
programmers and end-user administrators.

--nab

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why using configfs as the only interface is wrong for a storage target

[Emmanuel Florac]

Le Mon, 07 Feb 2011 12:09:39 -0800 vous écriviez:

> Sorry, but I have no idea what you are talking about.

Sorry too, I'm just a typical target user, and I'm pretty confused.

> We use direct configfs mkdirs and symlinks in our configuration files.

> There are special cases where we use lio-utils.git python code to load
> additional backend device specific metadata (like T10 WWN, persistent
> reservations APTPL, ALUA multipath secondary port state, etc)

The configuration files are actually programs, because even a simple
bash script is a program. I'm getting uneasy. These are not your
regular, non executable config files.

> The whole idea is that configfs re-presents the parent/child
> relationships of data structures whose creation/deletion is driven by
> userspace. 

Yes, this is the really good part, and I'll be eternally grateful for
this. Thank you. 

> Having python code walk this layout and output the exact
> running configuration is how we save -> reinstate the running config,
> and it's really quite trivial with configfs.

Please understand that at this point, anybody lacking a deep
understanding of kernel innards like me or 99.9% of linux system
admins or users, really don't give a fsck. I actually understand the
problem with using sysfs for that, but the fact is that from a
usability point of view this doesn't make a frigging difference, except
that we have another unfathomable virtual filesystem. We could do just
the same with /proc or /sys, and a startup script for either ietd or
scst or stgt, from the user POV.

So as long as the config is not really, completely automagically made
persistent by some in-kernel mechanism, this isn't any better than a
daemon loading a config file. In fact, it's worse, because it's more
unusual. 

> But that is just the simple stuff, where configfs really shines is
> with a object oriented library and high level interactive shell that
> allows the configfs hierarchy to be fully utilized by application
> level programmers and end-user administrators.

You know, my first reaction was :

Aaaaargh. Are you trying to convince me that you've done the windows
registry right this time? I don't want no stinking object oriented
library to manage my scsi targets.

But finally, OK, I see the point for infrastructure. Now tell me, could
we avoid more virtual fs creep in the future ? 

What bothers me is that I'm not sure you can make a Ferrari by bolting
new parts onto a Ford, or in this particular case importing Plan9
architectural features into the Linux kernel.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why using configfs as the only interface is wrong for a storage target

[Joel Becker]

On Mon, Feb 07, 2011 at 10:38:45PM +0100, Emmanuel Florac wrote:
> Le Mon, 07 Feb 2011 12:09:39 -0800 vous écriviez:
> The configuration files are actually programs, because even a simple
> bash script is a program. I'm getting uneasy. These are not your
> regular, non executable config files.

	This is independent of configfs vs other methods, and probably
wants to be visited on its own.  More on this below.

> > Having python code walk this layout and output the exact
> > running configuration is how we save -> reinstate the running config,
> > and it's really quite trivial with configfs.
> 
> Please understand that at this point, anybody lacking a deep
> understanding of kernel innards like me or 99.9% of linux system
> admins or users, really don't give a fsck. I actually understand the

	Nor should they.  That's kind of the point.  Again, see below.

> Aaaaargh. Are you trying to convince me that you've done the windows
> registry right this time? I don't want no stinking object oriented
> library to manage my scsi targets.
> 
> But finally, OK, I see the point for infrastructure. Now tell me, could
> we avoid more virtual fs creep in the future ? 

	Configfs is not a registry.
	Let me try to restate the issue.  The kernel doesn't persist
state; whether DM maps, iSCSI target configs, or network interface
addresses, some userspace entity has to tell the kernel about it on
each boot.  We have multiple methods of doing this: proc/sys, sysfs,
configfs, ioctl(2)s, etc.  So if you want to have your system advertise
sdb1 as an iSCSI LUN, you have to tell the target code.
	configfs was designed for exactly this case.  You want to create
a kernel object (eg a "target" or a "LUN mapping"), and userspace drives
this.  You might have multiple objects that need to be connected
("targets" and "LUNs advertised by this target"), and configfs handles
this too.
	You could do this with an ioctl(2), but you'd still have to have
some persistent configuration in userspace so that the ioctl(2) gets
called on each reboot to rebuild the mapping.  This is exactly analogous
to "mdadm --assemble" running on each boot.  mdadm(8) uses an ioctl(2),
but it still needs /etc/mdadm.conf to know what to send down the
ioctl(2).  Similarly, /etc/sysctl.conf tells sysctl(8) what to store on
boot.
	So any of these services have to provide some userspace program
that reads a configuration and knows how to send it down to the kernel.
They should be isolating the "how I send it down to the kernel" from
your average admin.  Whether that config is a simple text file or a
complex scripty mess is independent of this.  One would hope that, like
mdadm(8), you have a simple human-readable text file that a program
turns into the proper kernel invocation.
	It's great that Nick has fancy tools to fly around the target
configfs space, but that's all fluff for people writing and debugging
the tools.  Same with mdadm(8) and the MD ioctl(2)s.  You don't want to
know the details of the ioctl(2) call; you just want mdadm(8) to work.

Joel

-- 

"Here's something to think about:  How come you never see a headline
 like ``Psychic Wins Lottery''?"
	- Jay Leno

			http://www.jlbec.org/
			jlbec@evilplan.org
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why using configfs as the only interface is wrong for a storage target

[Joel Becker]

On Mon, Feb 07, 2011 at 02:44:09PM -0800, Joel Becker wrote:
> 	It's great that Nick has fancy tools to fly around the target
> configfs space, but that's all fluff for people writing and debugging

	I'm glad that Nick is excited about what configfs provides him
as a developer examining the system, but I think he goes overboard
extolling these virtues.  This distracts from the core discussion of
configfs as a method of communicating the configuration to kernelspace.

Joel

-- 

Life's Little Instruction Book #207

	"Swing for the fence."

			http://www.jlbec.org/
			jlbec@evilplan.org

Re: Why using configfs as the only interface is wrong for a storage target

[Nicholas A. Bellinger]

On Mon, 2011-02-07 at 14:44 -0800, Joel Becker wrote:
> On Mon, Feb 07, 2011 at 10:38:45PM +0100, Emmanuel Florac wrote:
> > Le Mon, 07 Feb 2011 12:09:39 -0800 vous écriviez:
> > The configuration files are actually programs, because even a simple
> > bash script is a program. I'm getting uneasy. These are not your
> > regular, non executable config files.
> 
> 	This is independent of configfs vs other methods, and probably
> wants to be visited on its own.  More on this below.
> 
> > > Having python code walk this layout and output the exact
> > > running configuration is how we save -> reinstate the running config,
> > > and it's really quite trivial with configfs.
> > 
> > Please understand that at this point, anybody lacking a deep
> > understanding of kernel innards like me or 99.9% of linux system
> > admins or users, really don't give a fsck. I actually understand the
> 
> 	Nor should they.  That's kind of the point.  Again, see below.
> 
> > Aaaaargh. Are you trying to convince me that you've done the windows
> > registry right this time? I don't want no stinking object oriented
> > library to manage my scsi targets.
> > 
> > But finally, OK, I see the point for infrastructure. Now tell me, could
> > we avoid more virtual fs creep in the future ? 
> 
> 	Configfs is not a registry.
> 	Let me try to restate the issue.  The kernel doesn't persist
> state; whether DM maps, iSCSI target configs, or network interface
> addresses, some userspace entity has to tell the kernel about it on
> each boot.  We have multiple methods of doing this: proc/sys, sysfs,
> configfs, ioctl(2)s, etc.  So if you want to have your system advertise
> sdb1 as an iSCSI LUN, you have to tell the target code.
> 	configfs was designed for exactly this case.  You want to create
> a kernel object (eg a "target" or a "LUN mapping"), and userspace drives
> this.  You might have multiple objects that need to be connected
> ("targets" and "LUNs advertised by this target"), and configfs handles
> this too.
> 	You could do this with an ioctl(2), but you'd still have to have
> some persistent configuration in userspace so that the ioctl(2) gets
> called on each reboot to rebuild the mapping.  This is exactly analogous
> to "mdadm --assemble" running on each boot.  mdadm(8) uses an ioctl(2),
> but it still needs /etc/mdadm.conf to know what to send down the
> ioctl(2).  Similarly, /etc/sysctl.conf tells sysctl(8) what to store on
> boot.
> 	So any of these services have to provide some userspace program
> that reads a configuration and knows how to send it down to the kernel.
> They should be isolating the "how I send it down to the kernel" from
> your average admin.  Whether that config is a simple text file or a
> complex scripty mess is independent of this.  One would hope that, like
> mdadm(8), you have a simple human-readable text file that a program
> turns into the proper kernel invocation.

Completely agreed.  I prefer having a /bin/sh walkable layout of target
kernel data structures using struct config_group references, and using
easily maintainable intrepreted code running in userspace to handle
changes to that layout.

IOCTLs are fine for nuts-and-bolts pieces like mdadm, but I think trying
to provide an easily expandable control interface across multiple linux
kernel modules w/ internal data structure reference counting is a pretty
painful exercise for actually adding new fabric module kernel and new
<cough> and flexiable userspace code.

> 	It's great that Nick has fancy tools to fly around the target
> configfs space, but that's all fluff for people writing and debugging
> the tools.  Same with mdadm(8) and the MD ioctl(2)s.  You don't want to
> know the details of the ioctl(2) call; you just want mdadm(8) to work.
> 

So wrt to the last point, I would like to clarify one item wrt to the
RTS userspace tools with LIO target mode v4 kernel code.

We (RisingTideSystems & Linux-iSCSI.org) will be publically releasing
our single-node high level shell+CLI (rtsadmin) and object oriented
python library (rtslib) to drive real-time management of 'for-38'
mainline /sys/kernel/config/target/ configfs code.  These are the
userspace packages that RTS/LIO customers+partners+friends have been
using in production with iscsi_target_mod and tcm_loop in v3.x
lio-core-backports.git before the mainline adoption of TCM v4 code.  The
public documentation for both of these is available here:

http://www.risingtidesystems.com/doc/rtsadmin/html/rtsadmin_reference.html
http://www.risingtidesystems.com/doc/rtslib/html/

Our code will also be including generic support for all v4 compatiable
-> target_core_fabric_configfs.c modules, and will be released in git
trees as the 'for-39' merge window opens later this spring. 

We are currently offering rtsadmin-frozen packages on v4.0-rc and v3.x
LIO kernel code on a case-by-case basis to a number of folks, and we
would be happy to provide new forward looking TCM fabric module
developers with this high level shell that can be used directly during
their own fabric port I/O bringup efforts from generic fabric level
struct config_group /sys/kernel/config/target/core/$HBA/$DEV/ backends
on for-38 code.

Best Regards,

--nab

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why using configfs as the only interface is wrong for a storage target

[Emmanuel Florac]

Le Mon, 7 Feb 2011 14:44:09 -0800 vous écriviez:

> They should be isolating the "how I send it down to the kernel" from
> your average admin.  Whether that config is a simple text file or a
> complex scripty mess is independent of this.  One would hope that,
> like mdadm(8), you have a simple human-readable text file that a
> program turns into the proper kernel invocation.

Gosh, ex-act-ly.

> 	It's great that Nick has fancy tools to fly around the target
> configfs space, but that's all fluff for people writing and debugging
> the tools.  Same with mdadm(8) and the MD ioctl(2)s.  You don't want
> to know the details of the ioctl(2) call; you just want mdadm(8) to
> work.

I understand that we can't overnight get rid of sysctl and stuff.
So far configfs looks like one more kernel configuration subsystem
usable only with ocfs and lio. What I'm afraid of is several
different configuration systems living side by side for the next 10
years in the kernel.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why using configfs as the only interface is wrong for a storage target

[Hannes Reinecke]

On 02/07/2011 03:41 PM, James Bottomley wrote:
> On Mon, 2011-02-07 at 03:53 -0800, Joel Becker wrote:
>> On Mon, Feb 07, 2011 at 12:41:18PM +0100, Bart Van Assche wrote:
>>> On Fri, Feb 4, 2011 at 7:45 AM, Nicholas A. Bellinger
>>> <nab@linux-iscsi.org> wrote:
>>>> Please consider the following patch series for mainline target code.
>>>> It consists of predominately configfs bugfixes uncovered with recent SLUB
>>>> poison testing, and proper removal of legacy procfs target_core_mib.c code.
>>>> Note that the complete set of fabric independent statistics (SCSI MIBs) and
>>>> fabric dependent statistics will be included as native configfs group context
>>>> 'per value' attribute series during the .39 time frame.
>>>
>>> I'm still not convinced that using configfs in a storage target as the
>>> only interface between kernel space and user space is a good idea.
>>> While configfs may satisfy all the needs of an iSCSI target, the use
>>> of configfs in combination with hot-pluggable HCAs is really awkward.
>>> Whenever a HCA is plugged in, the user has to issue mkdir commands to
>>> make these interfaces appear in configfs. And whenever a HCA is
>>> removed, stale information will remain present in configfs until the
>>> user issues an rmdir command. As we all know, it is not possible for a
>>> storage target to make these directories appear / disappear
>>> automatically in configfs because of basic configfs design choices.
>>
>> 	Any configuration would have to be handled.  We have plenty of
>> stuff that is handled by userspace hooks out of udev, etc.  That's a
>> normal hotplug process.
>> 	Essentially, you're not challenging Nick's use of configfs here,
>> you're challenging his environment of setting up the target stack from
>> userspace.
> 
> I think the overall philosophical point here, and it's a good one
> because I've heard it from several sources, is that it's not possible to
> separate configuration from status completely.  The classic example is
> where the kernel has to validate and adjust config information, but the
> storage specific one is where events alter the topology.  In either
> case, the configfs tree gets out of sync with reality if the kernel does
> the adjustment..  Just saying we have to use a user space tool to fix it
> up is a bit of a cop out because the kernel has already adjusted its own
> configuration, so getting userspace to work out what the kernel's done
> and adjust configfs is a bit sub optimal.
> 
This sounds exactly like the familiar problem we have on zSeries;
devices are being configured via sysfs but require a separate store
to hold the information if the device doesn't happen to be present.

What we came up with is
a) Devices which are not configured won't be activated.
   Essentially this means you have to have a configuration for
   each device (_not_ stored in sysfs, obviously), and use this
   as the primary config. If and when the device becomes available,
   udev will use this information to configure the device.
b) If the device state changes either the device is smart enough
   to handle it internally or the correct events have to be send out
   via udev to allow proper configuration here.

And yes, without udev that wouldn't work. If you have a problem
with this, you can stop reading now.

Applying this to configfs we need to define a configuration file
format, and a set of udev rules which would use this information
to configure the configfs tree.

Of course, it would be preferable to have target_core to update
the information itself (eg for device removal).
If that wouldn't be possible we need to insure that the configfs
part doesn't crash if the underlying device is removed but rather
returns sensible errors when accessed.

But I don't see any principle problem here.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why using configfs as the only interface is wrong for a storage target

[Sergiu]

Hello all,

I have an issue related to this topic. I am trying to detect via a libudev
wrapper if device based on configfs is mounted. 
I have a small usb detector based on libudev and libudev(the wapper),
"udev_enumerate_add_match_subsystem( enumerate, "usb" )" is used as a filter
and I mount the a device using 'mount none /cfg1 -t configfs'.
Linux detects the device as mounted but udev detector does not. Do you know
if what I am trying to do is possible, to mount a file based on configfs,
and edit the structure so that it is detected by udev? I edited the vendor
id, the product id, class id in the configfs structure but the detector does
not see it.

Thank in advance,
Sergiu

Re: Why using configfs as the only interface is wrong for a storage target

[Joel Becker]

On Mon, Feb 07, 2011 at 08:41:49AM -0600, James Bottomley wrote:
> On Mon, 2011-02-07 at 03:53 -0800, Joel Becker wrote:
> > On Mon, Feb 07, 2011 at 12:41:18PM +0100, Bart Van Assche wrote:
> > > I'm still not convinced that using configfs in a storage target as the
> > > only interface between kernel space and user space is a good idea.

	Two points here.  First, a design principle of configfs is that
it cannot and should not be one-stop shop for all possible userspace
interaction.  If there are things that fit better in sysfs, there is no
reason the target code can't export them via sysfs.  So any contention
that we can't add other pieces of the puzzle in an attempt to shoehorn
everything into configfs is sub-optimal. 

> I think the overall philosophical point here, and it's a good one
> because I've heard it from several sources, is that it's not possible to
> separate configuration from status completely.  The classic example is
> where the kernel has to validate and adjust config information, but the

	Nor should you try.  You can certainly update the attributes of
objects created in configfs via the kernel, and you can certainly use
configfs with sysfs, ioctl, procfs, or netlink in the same subsystem.

Joel


-- 

Life's Little Instruction Book #226

	"When someone hugs you, let them be the first to let go."

			http://www.jlbec.org/
			jlbec@evilplan.org

Re: Why using configfs as the only interface is wrong for a storage target

[Nicholas A. Bellinger]

On Mon, 2011-02-07 at 12:41 +0100, Bart Van Assche wrote:
> On Fri, Feb 4, 2011 at 7:45 AM, Nicholas A. Bellinger
> <nab@linux-iscsi.org> wrote:
> > Please consider the following patch series for mainline target code.
> > It consists of predominately configfs bugfixes uncovered with recent SLUB
> > poison testing, and proper removal of legacy procfs target_core_mib.c code.
> > Note that the complete set of fabric independent statistics (SCSI MIBs) and
> > fabric dependent statistics will be included as native configfs group context
> > 'per value' attribute series during the .39 time frame.
> 

<SNIP>

> As we all know, it is not possible for a
> storage target to make these directories appear / disappear
> automatically in configfs because of basic configfs design choices.
> 

Of course it's possible to drive configfs struct config_group creation
from kernel space.  That is a tool that was used during LIO v2.x -> v3.x
IOCTL -> CONFIGFS bringup and conversion, remember..?

However Joel as the configfs maintainer did not want this code in
mainline, and I dropped the patch from lio-core-2.6.git some time ago.
If you think this is a worth-while discussion to revist perhaps you
should re-create the logic, and address the issues that Joel had with
the original code.

--nab