Re: Reducing amount of logmessages openiscsi/multipath [with md3000i]

Re: Reducing amount of logmessages openiscsi/multipath [with md3000i]

[Mike Christie]

Kees Hoekzema wrote:
> Hello List,

I am adding the dm-devel list.

I think you want different settings in your multipath conf. For example 
it looks like you are using directio to test paths and you want TUR or 
the md3000i specific one which does not cause IO errors to passive paths.

You also want to make sure that you are using the md3000i hw handler or 
scsh_dh_module if you are not already. The dm-devel guys can help you out.

You may want to post your multipath.conf on the thread.

> 
> Recently we bought a md3000i for our storage needs. This box comes with two
> redundant controllers and both controller has two networkports. We connected
> all four ports to our testing network, and I let some servers mount the
> md3000i. The goal is to have all four connections on the server available
> with a redundant network, so that we can failover if one switch dies or if a
> controller dies.
> 
> However, the controller does not have support an active-active state, which
> means that if you open four connections to the four ports, you will get two
> faulty connections. With multipath this isn't a big problem as it can see
> fairly quick if a controller is offline and it switches to the other
> controller:
> 
> $ multipath -ll
> webdata (36001ec9000d163110000067f484e260c) dm-0 DELL    ,MD3000i
> [size=2.0T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][enabled]
>  \_ 4:0:0:0 sda 8:0   [active][ready]
> \_ round-robin 0 [prio=1000][active]
>  \_ 5:0:0:0 sdb 8:16  [active][ready]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 6:0:0:0 sdc 8:32  [active][faulty]
> \_ round-robin 0 [prio=0][enabled]
>  \_ 7:0:0:0 sdd 8:48  [active][faulty]
> 
> (yes, I know the prio's are a bit strange, but that is to test something
> else).
> 
> This all works quite well, but the problem we have is a massive amount of
> spam in dmesg / kern.log / syslog. In syslog we get this every 3 seconds:
> Aug 19 14:54:12 abaris kernel: [691035.298493] end_request: I/O error, dev
> sdc, sector 0
> Aug 19 14:54:12 abaris kernel: [691035.366443] end_request: I/O error, dev
> sdd, sector 0
> Aug 19 14:54:13 abaris multipathd: sdc: directio checker reports path is
> down
> Aug 19 14:54:13 abaris multipathd: sdd: directio checker reports path is
> down
> 
> This is of course useful information, but to get this 28.800 times per day
> in your log is a waste of space, beside making it impossible to read other
> messages without grepping. Now this can still be reduced by adding some
> syslog filters, but the other problem is dmesg:
> 
> $ dmesg
> [691098.658507] end_request: I/O error, dev sdc, sector 0
> [691098.704036] end_request: I/O error, dev sdd, sector 0
> [691101.678930] end_request: I/O error, dev sdc, sector 0
> [691101.738252] end_request: I/O error, dev sdd, sector 0
> 
> And this goes on for every 3 seconds as long as the connection is logged in.
> 
> So my questions:
> - Is there a way to suppress those messages, either in the iscsi programs or
> in multipath?
> - If that fails, is it possible to tweak multipath in such a way that it
> doesn't check the two failed paths, unless there is no other path available?
> These two paths should become available only if the main controller dies.
> 
> Thanks in advance,
> Kees Hoekzema
> 
> 
> > 

Re: Reducing amount of logmessages openiscsi/multipath [with md3000i]

[Konrad Rzeszutek]

On Tue, Aug 19, 2008 at 12:20:05PM -0500, Mike Christie wrote:
> 
> Kees Hoekzema wrote:
> > Hello List,
> 
> I am adding the dm-devel list.
> 
> I think you want different settings in your multipath conf. For example 
> it looks like you are using directio to test paths and you want TUR or 
> the md3000i specific one which does not cause IO errors to passive paths.

He wants the RDAC hw handler and its path checker (rdac). The MD3000i is an
LSI re-branded box and is the same as the IBM DS3300.

> 
> You also want to make sure that you are using the md3000i hw handler or 
> scsh_dh_module if you are not already. The dm-devel guys can help you out.

Here is the multipath.conf he should be using for MD3000i (or for the DS3300
but will need to modify the vendor/model entry)

        # Note: The same as the IBM DS3300
        #
        device {
                vendor                  "DELL"
                product                 "MD3000i"
                product_blacklist       "Universal Xport"
                features                "1 queue_if_no_path"
                path_grouping_policy    group_by_prio
                hardware_handler        "1 rdac"
                path_checker            rdac
                prio                    "rdac"
                failback                immediate
        }

RE: Reducing amount of logmessages openiscsi/multipath [with md3000i]

[Kees Hoekzema]

> -----Original Message-----
> He wants the RDAC hw handler and its path checker (rdac). The MD3000i
> is an
> LSI re-branded box and is the same as the IBM DS3300.
> 
> >
> > You also want to make sure that you are using the md3000i hw handler
> or
> > scsh_dh_module if you are not already. The dm-devel guys can help you
> out.
> 
> Here is the multipath.conf he should be using for MD3000i (or for the
> DS3300
> but will need to modify the vendor/model entry)
> 
>         # Note: The same as the IBM DS3300
>         #
>         device {
>                 vendor                  "DELL"
>                 product                 "MD3000i"
>                 product_blacklist       "Universal Xport"
>                 features                "1 queue_if_no_path"
>                 path_grouping_policy    group_by_prio
>                 hardware_handler        "1 rdac"
>                 path_checker            rdac
>                 prio                    "rdac"
>                 failback                immediate
>         }
> 
> 

This fixed the problem, although a simple flush was not enough to activate
the changes, so it took a bit longer to get the result I wanted. In the end
I saw that it was still using the direction path_checker and noticed that
you cannot change the 'path_checker' on the fly with just a multipath -F /
-v 2, but that a reload of the modules was needed.

$ multipath -ll
webdata (36001ec9000d163110000067f484e260c) dm-0 DELL    ,MD3000i
[size=2.0T][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=1000][active]
 \_ 0:0:0:0 sda 8:0   [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 1:0:0:0 sdb 8:16  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 2:0:0:0 sdc 8:32  [active][ghost]
 \_ 3:0:0:0 sdd 8:48  [active][ghost]

That is the output of multipath now, and no more spam in my syslog or dmesg.

(The different priorities you see are because the second path (with a prio
of 1) is a longer path through 1 more switch, so I'd rather not give it a
high priority, and if the '1000' path fails, it just uses the '1' path.)

A big thanks to everyone who directed me on the right path ;)

-kees

Re: Reducing amount of logmessages openiscsi/multipath [with md3000i]

[Konrad Rzeszutek]

On Wed, Aug 20, 2008 at 12:57:24PM +0200, Kees Hoekzema wrote:
> 
> 
> 
> > -----Original Message-----
> > He wants the RDAC hw handler and its path checker (rdac). The MD3000i
> > is an
> > LSI re-branded box and is the same as the IBM DS3300.
> > 
> > >
> > > You also want to make sure that you are using the md3000i hw handler
> > or
> > > scsh_dh_module if you are not already. The dm-devel guys can help you
> > out.
> > 
> > Here is the multipath.conf he should be using for MD3000i (or for the
> > DS3300
> > but will need to modify the vendor/model entry)
> > 
> >         # Note: The same as the IBM DS3300
> >         #
> >         device {
> >                 vendor                  "DELL"
> >                 product                 "MD3000i"
> >                 product_blacklist       "Universal Xport"
> >                 features                "1 queue_if_no_path"
> >                 path_grouping_policy    group_by_prio
> >                 hardware_handler        "1 rdac"
> >                 path_checker            rdac
> >                 prio                    "rdac"
> >                 failback                immediate
> >         }
> > 
> > 
> 
> This fixed the problem, although a simple flush was not enough to activate

Glad to hear that.

> the changes, so it took a bit longer to get the result I wanted. In the end
> I saw that it was still using the direction path_checker and noticed that
> you cannot change the 'path_checker' on the fly with just a multipath -F /
> -v 2, but that a reload of the modules was needed.

Ohh, you could also do: multipathd -k'reconfigure'