QoS for iSCSI target?

QoS for iSCSI target?

[Chris Friesen]

Hi,

I'm looking for information on whether the iSCSI target in the kernel offers any 
way to do QoS between traffic driven by different initiators.

I'm trying to make sure that one initiator can't do a denial-of-service attack 
against others.

Does the kernel target have this sort of thing built-in, or do I need to look at 
network traffic-shaping to achieve this?

Thanks,
Chris

Re: QoS for iSCSI target?

[Christoph Hellwig]

On Thu, Mar 10, 2016 at 04:24:25PM -0600, Chris Friesen wrote:
> Hi,
> 
> I'm looking for information on whether the iSCSI target in the kernel offers
> any way to do QoS between traffic driven by different initiators.
> 
> I'm trying to make sure that one initiator can't do a denial-of-service
> attack against others.
> 
> Does the kernel target have this sort of thing built-in, or do I need to
> look at network traffic-shaping to achieve this?

It doesn't right now, but it shouldn't be hard to integrate it with
blk cgroups.

Re: QoS for iSCSI target?

[Nicholas A. Bellinger]

On Thu, 2016-03-10 at 23:30 -0800, Christoph Hellwig wrote:
> On Thu, Mar 10, 2016 at 04:24:25PM -0600, Chris Friesen wrote:
> > Hi,
> > 
> > I'm looking for information on whether the iSCSI target in the kernel offers
> > any way to do QoS between traffic driven by different initiators.
> > 
> > I'm trying to make sure that one initiator can't do a denial-of-service
> > attack against others.
> > 
> > Does the kernel target have this sort of thing built-in, or do I need to
> > look at network traffic-shaping to achieve this?
> 
> It doesn't right now, but it shouldn't be hard to integrate it with
> blk cgroups.

For iscsi-target application QoS, the per session command sequence
number window depth (CmdSN) exists to enforce per InitiatorName limits
via TPG default_cmdsn_depth + se_node_acl->queue_depth configfs
attributes.

Note these values can be changed on the fly for iscsi-target using
explicit se_node_acl->acl_group, but currently require a se_session
reinstatement event for updated ExpCmdSN + MaxCmdSN to take effect.

Re: QoS for iSCSI target?

[Chris Friesen]

On 03/11/2016 01:45 AM, Nicholas A. Bellinger wrote:
> On Thu, 2016-03-10 at 23:30 -0800, Christoph Hellwig wrote:
>> On Thu, Mar 10, 2016 at 04:24:25PM -0600, Chris Friesen wrote:
>>> Hi,
>>>
>>> I'm looking for information on whether the iSCSI target in the kernel offers
>>> any way to do QoS between traffic driven by different initiators.
>>>
>>> I'm trying to make sure that one initiator can't do a denial-of-service
>>> attack against others.
>>>
>>> Does the kernel target have this sort of thing built-in, or do I need to
>>> look at network traffic-shaping to achieve this?
>>
>> It doesn't right now, but it shouldn't be hard to integrate it with
>> blk cgroups.
>
> For iscsi-target application QoS, the per session command sequence
> number window depth (CmdSN) exists to enforce per InitiatorName limits
> via TPG default_cmdsn_depth + se_node_acl->queue_depth configfs
> attributes.
>
> Note these values can be changed on the fly for iscsi-target using
> explicit se_node_acl->acl_group, but currently require a se_session
> reinstatement event for updated ExpCmdSN + MaxCmdSN to take effect.

On a slightly different note, is there any way to throttle or limit the overall 
bandwidth consumed by the iSCSI target in the kernel?  I'd like to ensure that 
the iSCSI traffic doesn't completely swamp the host accesses to the same block 
device.

I suppose I could do networking-based traffic shaping, but are there any 
controls in the block IO subsystem?

Chris

Re: QoS for iSCSI target?

[Nicholas A. Bellinger]

On Wed, 2016-03-16 at 10:48 -0600, Chris Friesen wrote:
> On 03/11/2016 01:45 AM, Nicholas A. Bellinger wrote:
> > On Thu, 2016-03-10 at 23:30 -0800, Christoph Hellwig wrote:
> >> On Thu, Mar 10, 2016 at 04:24:25PM -0600, Chris Friesen wrote:
> >>> Hi,
> >>>
> >>> I'm looking for information on whether the iSCSI target in the kernel offers
> >>> any way to do QoS between traffic driven by different initiators.
> >>>
> >>> I'm trying to make sure that one initiator can't do a denial-of-service
> >>> attack against others.
> >>>
> >>> Does the kernel target have this sort of thing built-in, or do I need to
> >>> look at network traffic-shaping to achieve this?
> >>
> >> It doesn't right now, but it shouldn't be hard to integrate it with
> >> blk cgroups.
> >
> > For iscsi-target application QoS, the per session command sequence
> > number window depth (CmdSN) exists to enforce per InitiatorName limits
> > via TPG default_cmdsn_depth + se_node_acl->queue_depth configfs
> > attributes.
> >
> > Note these values can be changed on the fly for iscsi-target using
> > explicit se_node_acl->acl_group, but currently require a se_session
> > reinstatement event for updated ExpCmdSN + MaxCmdSN to take effect.
> 
> On a slightly different note, is there any way to throttle or limit the overall 
> bandwidth consumed by the iSCSI target in the kernel?  I'd like to ensure that 
> the iSCSI traffic doesn't completely swamp the host accesses to the same block 
> device.
> 
> I suppose I could do networking-based traffic shaping, but are there any 
> controls in the block IO subsystem?
> 

On a individual block_device backend basis, block cgroups is the
preferred method for doing this.

Note that any rate limiting imposed by block cgroups is subject to the
current se_node_acl->queue_depth enforced across all LUNs within a given
iscsi session.

Re: QoS for iSCSI target?

[Chris Friesen]

On 03/31/2016 01:05 AM, Nicholas A. Bellinger wrote:
> On Wed, 2016-03-16 at 10:48 -0600, Chris Friesen wrote:
>> On 03/11/2016 01:45 AM, Nicholas A. Bellinger wrote:
>>> On Thu, 2016-03-10 at 23:30 -0800, Christoph Hellwig wrote:
>>>> On Thu, Mar 10, 2016 at 04:24:25PM -0600, Chris Friesen wrote:
>>>>> Hi,
>>>>>
>>>>> I'm looking for information on whether the iSCSI target in the kernel offers
>>>>> any way to do QoS between traffic driven by different initiators.
>>>>>
>>>>> I'm trying to make sure that one initiator can't do a denial-of-service
>>>>> attack against others.
>>>>>
>>>>> Does the kernel target have this sort of thing built-in, or do I need to
>>>>> look at network traffic-shaping to achieve this?
>>>>
>>>> It doesn't right now, but it shouldn't be hard to integrate it with
>>>> blk cgroups.
>>>
>>> For iscsi-target application QoS, the per session command sequence
>>> number window depth (CmdSN) exists to enforce per InitiatorName limits
>>> via TPG default_cmdsn_depth + se_node_acl->queue_depth configfs
>>> attributes.
>>>
>>> Note these values can be changed on the fly for iscsi-target using
>>> explicit se_node_acl->acl_group, but currently require a se_session
>>> reinstatement event for updated ExpCmdSN + MaxCmdSN to take effect.
>>
>> On a slightly different note, is there any way to throttle or limit the overall
>> bandwidth consumed by the iSCSI target in the kernel?  I'd like to ensure that
>> the iSCSI traffic doesn't completely swamp the host accesses to the same block
>> device.
>>
>> I suppose I could do networking-based traffic shaping, but are there any
>> controls in the block IO subsystem?
>>
>
> On a individual block_device backend basis, block cgroups is the
> preferred method for doing this.
>
> Note that any rate limiting imposed by block cgroups is subject to the
> current se_node_acl->queue_depth enforced across all LUNs within a given
> iscsi session.


How would I use cgroups with the kernel iSCSI target?  Is there a set of kernel 
threads dedicated to the iSCSI target that I can assign to a particular group?

Chris

Re: QoS for iSCSI target?

[Nicholas A. Bellinger]

On Fri, 2016-04-01 at 12:35 -0600, Chris Friesen wrote:
> On 03/31/2016 01:05 AM, Nicholas A. Bellinger wrote:
> > On Wed, 2016-03-16 at 10:48 -0600, Chris Friesen wrote:
> >> On 03/11/2016 01:45 AM, Nicholas A. Bellinger wrote:
> >>> On Thu, 2016-03-10 at 23:30 -0800, Christoph Hellwig wrote:
> >>>> On Thu, Mar 10, 2016 at 04:24:25PM -0600, Chris Friesen wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I'm looking for information on whether the iSCSI target in the kernel offers
> >>>>> any way to do QoS between traffic driven by different initiators.
> >>>>>
> >>>>> I'm trying to make sure that one initiator can't do a denial-of-service
> >>>>> attack against others.
> >>>>>
> >>>>> Does the kernel target have this sort of thing built-in, or do I need to
> >>>>> look at network traffic-shaping to achieve this?
> >>>>
> >>>> It doesn't right now, but it shouldn't be hard to integrate it with
> >>>> blk cgroups.
> >>>
> >>> For iscsi-target application QoS, the per session command sequence
> >>> number window depth (CmdSN) exists to enforce per InitiatorName limits
> >>> via TPG default_cmdsn_depth + se_node_acl->queue_depth configfs
> >>> attributes.
> >>>
> >>> Note these values can be changed on the fly for iscsi-target using
> >>> explicit se_node_acl->acl_group, but currently require a se_session
> >>> reinstatement event for updated ExpCmdSN + MaxCmdSN to take effect.
> >>
> >> On a slightly different note, is there any way to throttle or limit the overall
> >> bandwidth consumed by the iSCSI target in the kernel?  I'd like to ensure that
> >> the iSCSI traffic doesn't completely swamp the host accesses to the same block
> >> device.
> >>
> >> I suppose I could do networking-based traffic shaping, but are there any
> >> controls in the block IO subsystem?
> >>
> >
> > On a individual block_device backend basis, block cgroups is the
> > preferred method for doing this.
> >
> > Note that any rate limiting imposed by block cgroups is subject to the
> > current se_node_acl->queue_depth enforced across all LUNs within a given
> > iscsi session.
> 
> 
> How would I use cgroups with the kernel iSCSI target?  Is there a set of kernel 
> threads dedicated to the iSCSI target that I can assign to a particular group?
> 

block cgroups can set I/O throttling (bandwidth + IOPs) limits for any
normal struct block_device.

These values are configured via blkio.throttle.* resource class, and the
limits are imposed independently of the block_device's association with
target_core_mod backend driver export.  Namely:

    blkio.throttle.write_iops_device
    blkio.throttle.read_iops_device
    blkio.throttle.write_bps_device
    blkio.throttle.read_bdp_device

Some examples using these values, plus more blkio.* info is here:
http://events.linuxfoundation.org/sites/events/files/slides/cgroups_0.pdf.

Re: QoS for iSCSI target?

[Chris Friesen]

On 04/02/2016 07:15 PM, Nicholas A. Bellinger wrote:
> On Fri, 2016-04-01 at 12:35 -0600, Chris Friesen wrote:

>>>> On a slightly different note, is there any way to throttle or limit the overall
>>>> bandwidth consumed by the iSCSI target in the kernel?  I'd like to ensure that
>>>> the iSCSI traffic doesn't completely swamp the host accesses to the same block
>>>> device.
>>>>
>>>> I suppose I could do networking-based traffic shaping, but are there any
>>>> controls in the block IO subsystem?
>>>>
>>>
>>> On a individual block_device backend basis, block cgroups is the
>>> preferred method for doing this.
>>>
>>> Note that any rate limiting imposed by block cgroups is subject to the
>>> current se_node_acl->queue_depth enforced across all LUNs within a given
>>> iscsi session.
>>
>>
>> How would I use cgroups with the kernel iSCSI target?  Is there a set of kernel
>> threads dedicated to the iSCSI target that I can assign to a particular group?
>>
>
> block cgroups can set I/O throttling (bandwidth + IOPs) limits for any
> normal struct block_device.
>
> These values are configured via blkio.throttle.* resource class, and the
> limits are imposed independently of the block_device's association with
> target_core_mod backend driver export.  Namely:
>
>      blkio.throttle.write_iops_device
>      blkio.throttle.read_iops_device
>      blkio.throttle.write_bps_device
>      blkio.throttle.read_bdp_device
>
> Some examples using these values, plus more blkio.* info is here:
> http://events.linuxfoundation.org/sites/events/files/slides/cgroups_0.pdf.
>

I understand how to set up the cgroups, but don't I need to ensure that the IO 
operations happen in the context of a particular cgroup?  If so, how do I ensure 
that the in-kernel iSCSI target traffic happens in the context of the specified 
group?  Are there a set of kernel threads dedicated to processing iSCSI requests?

Chris

Re: QoS for iSCSI target?

[Nicholas A. Bellinger]

On Mon, 2016-04-04 at 09:20 -0600, Chris Friesen wrote:
> On 04/02/2016 07:15 PM, Nicholas A. Bellinger wrote:
> > On Fri, 2016-04-01 at 12:35 -0600, Chris Friesen wrote:
> 
> >>>> On a slightly different note, is there any way to throttle or limit the overall
> >>>> bandwidth consumed by the iSCSI target in the kernel?  I'd like to ensure that
> >>>> the iSCSI traffic doesn't completely swamp the host accesses to the same block
> >>>> device.
> >>>>
> >>>> I suppose I could do networking-based traffic shaping, but are there any
> >>>> controls in the block IO subsystem?
> >>>>
> >>>
> >>> On a individual block_device backend basis, block cgroups is the
> >>> preferred method for doing this.
> >>>
> >>> Note that any rate limiting imposed by block cgroups is subject to the
> >>> current se_node_acl->queue_depth enforced across all LUNs within a given
> >>> iscsi session.
> >>
> >>
> >> How would I use cgroups with the kernel iSCSI target?  Is there a set of kernel
> >> threads dedicated to the iSCSI target that I can assign to a particular group?
> >>
> >
> > block cgroups can set I/O throttling (bandwidth + IOPs) limits for any
> > normal struct block_device.
> >
> > These values are configured via blkio.throttle.* resource class, and the
> > limits are imposed independently of the block_device's association with
> > target_core_mod backend driver export.  Namely:
> >
> >      blkio.throttle.write_iops_device
> >      blkio.throttle.read_iops_device
> >      blkio.throttle.write_bps_device
> >      blkio.throttle.read_bdp_device
> >
> > Some examples using these values, plus more blkio.* info is here:
> > http://events.linuxfoundation.org/sites/events/files/slides/cgroups_0.pdf.
> >
> 
> I understand how to set up the cgroups, but don't I need to ensure that the IO 
> operations happen in the context of a particular cgroup?  If so, how do I ensure 
> that the in-kernel iSCSI target traffic happens in the context of the specified 
> group?  Are there a set of kernel threads dedicated to processing iSCSI requests?
> 

block cgroups does I/O ratelimiting at struct block_device level.

Eg: The process cgroup (which AFAICT is what your thinking about) is
separate from block cgroups, and doesn't need to be explicitly enabled
in order for block cgroup to function.

Re: QoS for iSCSI target?

[Chris Friesen]

On 04/04/2016 04:29 PM, Nicholas A. Bellinger wrote:
> On Mon, 2016-04-04 at 09:20 -0600, Chris Friesen wrote:
>> On 04/02/2016 07:15 PM, Nicholas A. Bellinger wrote:
>>> On Fri, 2016-04-01 at 12:35 -0600, Chris Friesen wrote:
>>
>>>>>> On a slightly different note, is there any way to throttle or limit the overall
>>>>>> bandwidth consumed by the iSCSI target in the kernel?  I'd like to ensure that
>>>>>> the iSCSI traffic doesn't completely swamp the host accesses to the same block
>>>>>> device.
>>>>>>
>>>>>> I suppose I could do networking-based traffic shaping, but are there any
>>>>>> controls in the block IO subsystem?
>>>>>>
>>>>>
>>>>> On a individual block_device backend basis, block cgroups is the
>>>>> preferred method for doing this.
>>>>>
>>>>> Note that any rate limiting imposed by block cgroups is subject to the
>>>>> current se_node_acl->queue_depth enforced across all LUNs within a given
>>>>> iscsi session.
>>>>
>>>>
>>>> How would I use cgroups with the kernel iSCSI target?  Is there a set of kernel
>>>> threads dedicated to the iSCSI target that I can assign to a particular group?
>>>>
>>>
>>> block cgroups can set I/O throttling (bandwidth + IOPs) limits for any
>>> normal struct block_device.
>>>
>>> These values are configured via blkio.throttle.* resource class, and the
>>> limits are imposed independently of the block_device's association with
>>> target_core_mod backend driver export.  Namely:
>>>
>>>       blkio.throttle.write_iops_device
>>>       blkio.throttle.read_iops_device
>>>       blkio.throttle.write_bps_device
>>>       blkio.throttle.read_bdp_device
>>>
>>> Some examples using these values, plus more blkio.* info is here:
>>> http://events.linuxfoundation.org/sites/events/files/slides/cgroups_0.pdf.
>>>
>>
>> I understand how to set up the cgroups, but don't I need to ensure that the IO
>> operations happen in the context of a particular cgroup?  If so, how do I ensure
>> that the in-kernel iSCSI target traffic happens in the context of the specified
>> group?  Are there a set of kernel threads dedicated to processing iSCSI requests?
>>
>
> block cgroups does I/O ratelimiting at struct block_device level.
>
> Eg: The process cgroup (which AFAICT is what your thinking about) is
> separate from block cgroups, and doesn't need to be explicitly enabled
> in order for block cgroup to function.
>

I'm still confused.

I'm not trying to globally throttle IO on a particular block device.  I'm trying 
to control how much IO the iSCSI target in the kernel is allowed to drive on a 
particular block device.

The goal is to ensure that the iSCSI target does not consume all of the 
available bandwidth for a particular block device.  I want to ensure that some 
of the bandwidth for that device is available to other processes on the host 
(for management purposes) rather than being consumed by a greedy iSCSI initiator.

In an ideal world I would like a set of rules that say things like:
1) if there is contention, ensure that the host is guaranteed X percent of the 
available /dev/sdb IOPS
2) if there is contention, do not allow the iSCSI target traffic to consume more 
than Y percent of /dev/sdb's write traffic

Chris

Re: QoS for iSCSI target?

[Nicholas A. Bellinger]

On Mon, 2016-04-04 at 17:01 -0600, Chris Friesen wrote:
> On 04/04/2016 04:29 PM, Nicholas A. Bellinger wrote:
> > On Mon, 2016-04-04 at 09:20 -0600, Chris Friesen wrote:
> >> On 04/02/2016 07:15 PM, Nicholas A. Bellinger wrote:
> >>> On Fri, 2016-04-01 at 12:35 -0600, Chris Friesen wrote:
> >>
> >>>>>> On a slightly different note, is there any way to throttle or limit the overall
> >>>>>> bandwidth consumed by the iSCSI target in the kernel?  I'd like to ensure that
> >>>>>> the iSCSI traffic doesn't completely swamp the host accesses to the same block
> >>>>>> device.
> >>>>>>
> >>>>>> I suppose I could do networking-based traffic shaping, but are there any
> >>>>>> controls in the block IO subsystem?
> >>>>>>
> >>>>>
> >>>>> On a individual block_device backend basis, block cgroups is the
> >>>>> preferred method for doing this.
> >>>>>
> >>>>> Note that any rate limiting imposed by block cgroups is subject to the
> >>>>> current se_node_acl->queue_depth enforced across all LUNs within a given
> >>>>> iscsi session.
> >>>>
> >>>>
> >>>> How would I use cgroups with the kernel iSCSI target?  Is there a set of kernel
> >>>> threads dedicated to the iSCSI target that I can assign to a particular group?
> >>>>
> >>>
> >>> block cgroups can set I/O throttling (bandwidth + IOPs) limits for any
> >>> normal struct block_device.
> >>>
> >>> These values are configured via blkio.throttle.* resource class, and the
> >>> limits are imposed independently of the block_device's association with
> >>> target_core_mod backend driver export.  Namely:
> >>>
> >>>       blkio.throttle.write_iops_device
> >>>       blkio.throttle.read_iops_device
> >>>       blkio.throttle.write_bps_device
> >>>       blkio.throttle.read_bdp_device
> >>>
> >>> Some examples using these values, plus more blkio.* info is here:
> >>> http://events.linuxfoundation.org/sites/events/files/slides/cgroups_0.pdf.
> >>>
> >>
> >> I understand how to set up the cgroups, but don't I need to ensure that the IO
> >> operations happen in the context of a particular cgroup?  If so, how do I ensure
> >> that the in-kernel iSCSI target traffic happens in the context of the specified
> >> group?  Are there a set of kernel threads dedicated to processing iSCSI requests?
> >>
> >
> > block cgroups does I/O ratelimiting at struct block_device level.
> >
> > Eg: The process cgroup (which AFAICT is what your thinking about) is
> > separate from block cgroups, and doesn't need to be explicitly enabled
> > in order for block cgroup to function.
> >
> 
> I'm still confused.
> 
> I'm not trying to globally throttle IO on a particular block device.  I'm trying 
> to control how much IO the iSCSI target in the kernel is allowed to drive on a 
> particular block device.
> 
> The goal is to ensure that the iSCSI target does not consume all of the 
> available bandwidth for a particular block device.  I want to ensure that some 
> of the bandwidth for that device is available to other processes on the host 
> (for management purposes) rather than being consumed by a greedy iSCSI initiator.
> 

AFAIK, different traffic classes for a single block device is not
supported by block cgroups.

Also given target_core_iblock claims a given block_device for exclusive
access, you can't actually use the same block device for a mounted
filesystem, LVM, MD, etc, once it's been registered for target usage.

> In an ideal world I would like a set of rules that say things like:
> 1) if there is contention, ensure that the host is guaranteed X percent of the 
> available /dev/sdb IOPS
> 2) if there is contention, do not allow the iSCSI target traffic to consume more 
> than Y percent of /dev/sdb's write traffic
> 

Yeah, that doesn't exist in standlone block groups.

You could probably use network cgroups to limit bandwidth at the socket
level for iscsi_target_mod, but that's going to be across all LUNs in a
session, and not at individual LUN level.

Re: QoS for iSCSI target?

[Chris Friesen]

On 04/04/2016 06:25 PM, Nicholas A. Bellinger wrote:
> On Mon, 2016-04-04 at 17:01 -0600, Chris Friesen wrote:

>> I'm not trying to globally throttle IO on a particular block device.  I'm trying
>> to control how much IO the iSCSI target in the kernel is allowed to drive on a
>> particular block device.
>>
>> The goal is to ensure that the iSCSI target does not consume all of the
>> available bandwidth for a particular block device.  I want to ensure that some
>> of the bandwidth for that device is available to other processes on the host
>> (for management purposes) rather than being consumed by a greedy iSCSI initiator.
>>
>
> AFAIK, different traffic classes for a single block device is not
> supported by block cgroups.
>
> Also given target_core_iblock claims a given block_device for exclusive
> access, you can't actually use the same block device for a mounted
> filesystem, LVM, MD, etc, once it's been registered for target usage.

The issue we're running into is in the context of OpenStack Cinder.  It layers 
LVM on top of the bare block device, then creates an LVM volume per cinder 
volume, and exports the logical volumes via iSCSI.

The problem we're seeing is that if one or more iSCSI initiators drive heavy 
traffic on a slow disk, it can become very slow to access anything at all on the 
bare block device (/dev/sdb or equivalent).  This can cause the server-side 
management code to time out on operations like making a new volume, checking 
free disk space, etc.

>> In an ideal world I would like a set of rules that say things like:
>> 1) if there is contention, ensure that the host is guaranteed X percent of the
>> available /dev/sdb IOPS
>> 2) if there is contention, do not allow the iSCSI target traffic to consume more
>> than Y percent of /dev/sdb's write traffic
>>
>
> Yeah, that doesn't exist in standlone block groups.
>
> You could probably use network cgroups to limit bandwidth at the socket
> level for iscsi_target_mod, but that's going to be across all LUNs in a
> session, and not at individual LUN level.

Are there a set of dedicated kernel threads being used for the iSCSI target?  Or 
is it handled by generic softirq mechanisms?

Chris