Status of resizing/growing dm-multipath devices on the fly

Status of resizing/growing dm-multipath devices on the fly

[Pasi Kärkkäinen]

Hello!

Is it possible to resize (grow) dm-multipath devices on the fly nowadays? 

I googled and found some discussions about the subject, but the conclusion
seemed to be it's not possible.. that was a while ago, so I was wondering if
this has been fixed/implemented.. 

Using LVM and adding another new LUN/PV is not an option always.. it's a lot
easier to manage the whole thing if it's possible to resize/grow existing
LUNs on the fly.

Thanks!

-- Pasi

RE: Status of resizing/growing dm-multipath devices on the fly

[Domenico Viggiani]

* Pasi Kärkkäinen wrote:
> 
> Is it possible to resize (grow) dm-multipath devices on the 
> fly nowadays?
> 
> I googled and found some discussions about the subject, but 
> the conclusion seemed to be it's not possible.. that was a 
> while ago, so I was wondering if this has been fixed/implemented..
> 
> Using LVM and adding another new LUN/PV is not an option 
> always.. it's a lot easier to manage the whole thing if it's 
> possible to resize/grow existing LUNs on the fly.

Nowadays all disk-arrays make online LUN extension a breeze but
unfortunately it seems that dm-multipath is still not able to see new size
without downtime:
 http://www.redhat.com/archives/dm-devel/2007-August/msg00205.html
Look also at this recent thread on RHEL5 mailing-list where you can find
also a post from a Red Hat representative:
 http://www.redhat.com/archives/rhelv5-list/2008-July/msg00267.html

Sure, you can just add a new LUN, pvcreate and vgextend but it is not a
viable solution because it causes LUN proliferation.

Sincerely, I'm still subscribed to this mailing-list only to see if someone
solve this SHAME!
It's a feature that Linux really needs.

Hope someone pick this cry of pain!

--
Domenico Viggiani

Re: Status of resizing/growing dm-multipath devices on the fly

[Pasi Kärkkäinen]

On Wed, Aug 06, 2008 at 03:48:12PM +0200, Domenico Viggiani wrote:
> * Pasi Kärkkäinen wrote:
> > 
> > Is it possible to resize (grow) dm-multipath devices on the 
> > fly nowadays?
> > 
> > I googled and found some discussions about the subject, but 
> > the conclusion seemed to be it's not possible.. that was a 
> > while ago, so I was wondering if this has been fixed/implemented..
> > 
> > Using LVM and adding another new LUN/PV is not an option 
> > always.. it's a lot easier to manage the whole thing if it's 
> > possible to resize/grow existing LUNs on the fly.
> 
> Nowadays all disk-arrays make online LUN extension a breeze but
> unfortunately it seems that dm-multipath is still not able to see new size
> without downtime:
>  http://www.redhat.com/archives/dm-devel/2007-August/msg00205.html
> Look also at this recent thread on RHEL5 mailing-list where you can find
> also a post from a Red Hat representative:
>  http://www.redhat.com/archives/rhelv5-list/2008-July/msg00267.html
> 
> Sure, you can just add a new LUN, pvcreate and vgextend but it is not a
> viable solution because it causes LUN proliferation.
> 
> Sincerely, I'm still subscribed to this mailing-list only to see if someone
> solve this SHAME!
> It's a feature that Linux really needs.
> 
> Hope someone pick this cry of pain!
> 

Thanks for the reply.

Does someone know what's the actual problem in this? meaning why it hasn't
been done yet.. 

-- Pasi

Re: Status of resizing/growing dm-multipath devices on the fly

[Mike Anderson]

Pasi K?rkk?inen <pasik@iki.fi> wrote:
> On Wed, Aug 06, 2008 at 03:48:12PM +0200, Domenico Viggiani wrote:
> > * Pasi Kärkkäinen wrote:
> > > 
> > > Is it possible to resize (grow) dm-multipath devices on the 
> > > fly nowadays?
> > > 
> > > I googled and found some discussions about the subject, but 
> > > the conclusion seemed to be it's not possible.. that was a 
> > > while ago, so I was wondering if this has been fixed/implemented..
> > > 
> > > Using LVM and adding another new LUN/PV is not an option 
> > > always.. it's a lot easier to manage the whole thing if it's 
> > > possible to resize/grow existing LUNs on the fly.
> > 
> > Nowadays all disk-arrays make online LUN extension a breeze but
> > unfortunately it seems that dm-multipath is still not able to see new size
> > without downtime:
> >  http://www.redhat.com/archives/dm-devel/2007-August/msg00205.html
> > Look also at this recent thread on RHEL5 mailing-list where you can find
> > also a post from a Red Hat representative:
> >  http://www.redhat.com/archives/rhelv5-list/2008-July/msg00267.html
> > 
> > Sure, you can just add a new LUN, pvcreate and vgextend but it is not a
> > viable solution because it causes LUN proliferation.
> > 
> > Sincerely, I'm still subscribed to this mailing-list only to see if someone
> > solve this SHAME!
> > It's a feature that Linux really needs.
> > 
> > Hope someone pick this cry of pain!
> > 
> 
> Thanks for the reply.
> 
> Does someone know what's the actual problem in this? meaning why it hasn't
> been done yet.. 

It has to do with the block layer not using the new size value while there
are openers.

Andrew submitted a patch series (url below) a while ago to try and address
this issue, but I do not know the status. I added Andrew to the cc.

http://thread.gmane.org/gmane.linux.scsi/41623

-andmike
--
Michael Anderson
andmike@linux.vnet.ibm.com

Re: Status of resizing/growing dm-multipath devices on the fly

[Andrew Patterson]

On Wed, 2008-08-06 at 12:36 -0700, Mike Anderson wrote:
> Pasi K?rkk?inen <pasik@iki.fi> wrote:
> > On Wed, Aug 06, 2008 at 03:48:12PM +0200, Domenico Viggiani wrote:
> > > * Pasi Kärkkäinen wrote:
> > > > 
> > > > Is it possible to resize (grow) dm-multipath devices on the 
> > > > fly nowadays?
> > > > 
> > > > I googled and found some discussions about the subject, but 
> > > > the conclusion seemed to be it's not possible.. that was a 
> > > > while ago, so I was wondering if this has been fixed/implemented..
> > > > 
> > > > Using LVM and adding another new LUN/PV is not an option 
> > > > always.. it's a lot easier to manage the whole thing if it's 
> > > > possible to resize/grow existing LUNs on the fly.
> > > 
> > > Nowadays all disk-arrays make online LUN extension a breeze but
> > > unfortunately it seems that dm-multipath is still not able to see new size
> > > without downtime:
> > >  http://www.redhat.com/archives/dm-devel/2007-August/msg00205.html
> > > Look also at this recent thread on RHEL5 mailing-list where you can find
> > > also a post from a Red Hat representative:
> > >  http://www.redhat.com/archives/rhelv5-list/2008-July/msg00267.html
> > > 
> > > Sure, you can just add a new LUN, pvcreate and vgextend but it is not a
> > > viable solution because it causes LUN proliferation.
> > > 
> > > Sincerely, I'm still subscribed to this mailing-list only to see if someone
> > > solve this SHAME!
> > > It's a feature that Linux really needs.
> > > 
> > > Hope someone pick this cry of pain!
> > > 
> > 
> > Thanks for the reply.
> > 
> > Does someone know what's the actual problem in this? meaning why it hasn't
> > been done yet.. 
> 
> It has to do with the block layer not using the new size value while there
> are openers.
> 
> Andrew submitted a patch series (url below) a while ago to try and address
> this issue, but I do not know the status. I added Andrew to the cc.
> 
> http://thread.gmane.org/gmane.linux.scsi/41623
> 

The above patchset works with SCSI devices. I have tested it with single
SCSI devices under dm/LVM, i.e., a PV is on a single SCSI LUN with only
one path to it. It has not yet been accepted upstream due to not getting
sign-off from Al Viro. I plan on resubmitting soon to add cciss support.
Hopefully Al will getting around to reviewing it sometime this century.
I have heard rumors that Al has some all-encompassing solution for this
problem but hasn't got around to implementing it.  I'll test my new
patches with multi-path while I am at it.

Andrew  

Re: Status of resizing/growing dm-multipath devices on the fly

[Pasi Kärkkäinen]

On Wed, Aug 06, 2008 at 02:58:16PM -0600, Andrew Patterson wrote:
> On Wed, 2008-08-06 at 12:36 -0700, Mike Anderson wrote:
> > Pasi K?rkk?inen <pasik@iki.fi> wrote:
> > > On Wed, Aug 06, 2008 at 03:48:12PM +0200, Domenico Viggiani wrote:
> > > > * Pasi Kärkkäinen wrote:
> > > > > 
> > > > > Is it possible to resize (grow) dm-multipath devices on the 
> > > > > fly nowadays?
> > > > > 
> > > > > I googled and found some discussions about the subject, but 
> > > > > the conclusion seemed to be it's not possible.. that was a 
> > > > > while ago, so I was wondering if this has been fixed/implemented..
> > > > > 
> > > > > Using LVM and adding another new LUN/PV is not an option 
> > > > > always.. it's a lot easier to manage the whole thing if it's 
> > > > > possible to resize/grow existing LUNs on the fly.
> > > > 
> > > > Nowadays all disk-arrays make online LUN extension a breeze but
> > > > unfortunately it seems that dm-multipath is still not able to see new size
> > > > without downtime:
> > > >  http://www.redhat.com/archives/dm-devel/2007-August/msg00205.html
> > > > Look also at this recent thread on RHEL5 mailing-list where you can find
> > > > also a post from a Red Hat representative:
> > > >  http://www.redhat.com/archives/rhelv5-list/2008-July/msg00267.html
> > > > 
> > > > Sure, you can just add a new LUN, pvcreate and vgextend but it is not a
> > > > viable solution because it causes LUN proliferation.
> > > > 
> > > > Sincerely, I'm still subscribed to this mailing-list only to see if someone
> > > > solve this SHAME!
> > > > It's a feature that Linux really needs.
> > > > 
> > > > Hope someone pick this cry of pain!
> > > > 
> > > 
> > > Thanks for the reply.
> > > 
> > > Does someone know what's the actual problem in this? meaning why it hasn't
> > > been done yet.. 
> > 
> > It has to do with the block layer not using the new size value while there
> > are openers.
> > 
> > Andrew submitted a patch series (url below) a while ago to try and address
> > this issue, but I do not know the status. I added Andrew to the cc.
> > 
> > http://thread.gmane.org/gmane.linux.scsi/41623
> > 
> 
> The above patchset works with SCSI devices. I have tested it with single
> SCSI devices under dm/LVM, i.e., a PV is on a single SCSI LUN with only
> one path to it. It has not yet been accepted upstream due to not getting
> sign-off from Al Viro. I plan on resubmitting soon to add cciss support.
> Hopefully Al will getting around to reviewing it sometime this century.
> I have heard rumors that Al has some all-encompassing solution for this
> problem but hasn't got around to implementing it.  I'll test my new
> patches with multi-path while I am at it.
> 

Was there some problems/issues with the patchset? Or just lack of time from
Al.. 

-- Pasi

Re: Status of resizing/growing dm-multipath devices on the fly

[Pasi Kärkkäinen]

On Mon, Aug 11, 2008 at 01:36:19PM +0300, Pasi Kärkkäinen wrote:
> On Wed, Aug 06, 2008 at 02:58:16PM -0600, Andrew Patterson wrote:
> > On Wed, 2008-08-06 at 12:36 -0700, Mike Anderson wrote:
> > > Pasi K?rkk?inen <pasik@iki.fi> wrote:
> > > > On Wed, Aug 06, 2008 at 03:48:12PM +0200, Domenico Viggiani wrote:
> > > > > * Pasi Kärkkäinen wrote:
> > > > > > 
> > > > > > Is it possible to resize (grow) dm-multipath devices on the 
> > > > > > fly nowadays?
> > > > > > 
> > > > > > I googled and found some discussions about the subject, but 
> > > > > > the conclusion seemed to be it's not possible.. that was a 
> > > > > > while ago, so I was wondering if this has been fixed/implemented..
> > > > > > 
> > > > > > Using LVM and adding another new LUN/PV is not an option 
> > > > > > always.. it's a lot easier to manage the whole thing if it's 
> > > > > > possible to resize/grow existing LUNs on the fly.
> > > > > 
> > > > > Nowadays all disk-arrays make online LUN extension a breeze but
> > > > > unfortunately it seems that dm-multipath is still not able to see new size
> > > > > without downtime:
> > > > >  http://www.redhat.com/archives/dm-devel/2007-August/msg00205.html
> > > > > Look also at this recent thread on RHEL5 mailing-list where you can find
> > > > > also a post from a Red Hat representative:
> > > > >  http://www.redhat.com/archives/rhelv5-list/2008-July/msg00267.html
> > > > > 
> > > > > Sure, you can just add a new LUN, pvcreate and vgextend but it is not a
> > > > > viable solution because it causes LUN proliferation.
> > > > > 
> > > > > Sincerely, I'm still subscribed to this mailing-list only to see if someone
> > > > > solve this SHAME!
> > > > > It's a feature that Linux really needs.
> > > > > 
> > > > > Hope someone pick this cry of pain!
> > > > > 
> > > > 
> > > > Thanks for the reply.
> > > > 
> > > > Does someone know what's the actual problem in this? meaning why it hasn't
> > > > been done yet.. 
> > > 
> > > It has to do with the block layer not using the new size value while there
> > > are openers.
> > > 
> > > Andrew submitted a patch series (url below) a while ago to try and address
> > > this issue, but I do not know the status. I added Andrew to the cc.
> > > 
> > > http://thread.gmane.org/gmane.linux.scsi/41623
> > > 
> > 
> > The above patchset works with SCSI devices. I have tested it with single
> > SCSI devices under dm/LVM, i.e., a PV is on a single SCSI LUN with only
> > one path to it. It has not yet been accepted upstream due to not getting
> > sign-off from Al Viro. I plan on resubmitting soon to add cciss support.
> > Hopefully Al will getting around to reviewing it sometime this century.
> > I have heard rumors that Al has some all-encompassing solution for this
> > problem but hasn't got around to implementing it.  I'll test my new
> > patches with multi-path while I am at it.
> > 
> 
> Was there some problems/issues with the patchset? Or just lack of time from
> Al.. 
> 

I noticed these patches were accepted for 2.6.28:
http://www.gossamer-threads.com/lists/linux/kernel/967628

Now how about online resizing dm-multipath devices.. ? We want that aswell! :)

-- Pasi

Re: Status of resizing/growing dm-multipath devices on the fly

[Sharif Nassar]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5889 bytes --]

On Wed, 6 Aug 2008, Pasi Kärkkäinen wrote:

> Hello!
> 
> Is it possible to resize (grow) dm-multipath devices on the fly nowadays? 
> 
> I googled and found some discussions about the subject, but the conclusion
> seemed to be it's not possible.. that was a while ago, so I was wondering if
> this has been fixed/implemented.. 
> 
> Using LVM and adding another new LUN/PV is not an option always.. it's a lot
> easier to manage the whole thing if it's possible to resize/grow existing
> LUNs on the fly.


Sure it is, but it's not pretty.  Don't believe the naysayers.
We seem to have a working method for resizing multipath luns from our 
NetApp filers.  It works on CentOS 4 & 5, for us, and our NetApps.
(2 HBA, times 2 paths on each makes for four SCSI LUN to resize)

The poorly formatted, unpolished, YMMV, no warranty recipe is:

1. Record the state of the dm tables

dmsetup table resize_test | tee orig current new final

2. Split one of the paths

multipath -l
resize_test (360a98000486e5452576f4336486d4c74) dm-2 NETAPP,LUN
[size=1.0G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=0][active]
  \_ 5:0:0:0 sdb 8:16  [active][undef]
  \_ 6:0:0:0 sdc 8:32  [active][undef]
\_ round-robin 0 [prio=0][enabled]
  \_ 6:0:1:0 sdd 8:48  [active][undef]
  \_ 5:0:1:0 sde 8:64  [active][undef]

# Choose a pair: sdb+sdc or sdd+sde. For this, let's choose sdd+sde

# Update current
vi current
0 2097152 multipath 1 queue_if_no_path 0 2 1 round-robin 0 2 1 8:16 4000 
8:32 4000 round-robin 0 2 1 8:48 1000 8:64 1000

# Change this to:
0 2097152 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:16 4000 
8:32 4000

# Note this is what happened:
# Remove the second pair beginning with the second "round-robin". We know 
it's sdd+sde based on the major/minor pairs.
# Reduce the number of pairs after "queue_if_no_path" from 2 (0 2 1) to 1 
(0 1 1)

multipathd -k
> del path sdd
> del path sde

# Reload the dm table
dmsetup suspend resize_test current; dmsetup reload resize_test current; 
dmsetup resume resize_test

# Verify you now just have one set of paths
multipath -l
resize_test (360a98000486e5452576f4336486d4c74) dm-2 NETAPP,LUN
[size=1.0G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=0][active]
  \_ 5:0:0:0 sdb 8:16  [active][undef]
  \_ 6:0:0:0 sdc 8:32  [active][undef]

3.  Grow the LUN on the SAN (although, this can happen before steps 1 & 2 
as well)

(NetApp specific:)
lun resize /vol/resize_test/lun 10g

4. # Update partition table

blockdev --rereadpt /dev/sdd
blockdev --rereadpt /dev/sde
blockdev --getsize /dev/sdd
blockdev --getsize /dev/sde

# The values for getsize in both commands must be the SAME
# Record this value
# The above gave us the value 20971520

# Update the dm table for new
vi new

0 2097152 multipath 1 queue_if_no_path 0 2 1 round-robin 0 2 1 8:16 4000 
8:32 4000 round-robin 0 2 1 8:48 1000 8:64 1000

# Change this to:
0 20971520 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:48 1000 
8:64 1000

# Note this is what happened:
# Update the second parameter (num sectors) from 2097152 to 20971520 
(obtained from blockdev --getsize)
# Remove the second pair beginning with the second "round-robin"
# Reduce the number of pairs after "queue_if_no_path" from 2 (0 2 1) to 1 
(0 1 1)

5.  Switch to the new path

multipathd -k
> add path sdd
> add path sde
> del path sdb
> del path sdc

# Note: We are adding the INACTIVE Paths first (sdd+sde) and then removing
# the ACTIVE paths (sdb+sdc). This is VERY important. I tried reversing it 
by
# deleting ACTIVE first then adding INACTIVE. Fortunately, multipathd kept 
one
# of them, refusing to budge.

dmsetup suspend resize_test; dmsetup reload resize_test new; dmsetup 
resume resize_test

# Verify we've switched to the new path:
multipath -l
resize_test (360a98000486e5452576f4336486d4c74) dm-2 NETAPP,LUN
[size=10G][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=0][enabled]
  \_ 6:0:1:0 sdd 8:48  [active][undef]
  \_ 5:0:1:0 sde 8:64  [active][undef]

# Note: multipath now shows the new size (10G). If it still shows the old 
size, chances are
# that one of the block devices in the pair have not had their partition 
tables updated.
# TURN BACK! AND MAKE SURE TO REREAD THE PARTITION TABLES!


6. Update the block devices on the inactive path

blockdev --rereadpt /dev/sdb
blockdev --rereadpt /dev/sdc
blockdev --getsize /dev/sdb
blockdev --getsize /dev/sdc

7. Re-add the final path

# Update the final dm table

vi final
0 2097152 multipath 1 queue_if_no_path 0 2 1 round-robin 0 2 1 8:16 4000 
8:32 4000 round-robin 0 2 1 8:48 1000 8:64 1000

# Change to:
0 20971520 multipath 1 queue_if_no_path 0 2 1 round-robin 0 2 1 8:16 4000 
8:32 4000 round-robin 0 2 1 8:48 1000 8:64 1000

# Note this is what happened:
# Update the second parameter (num sectors) from 2097152 to 20971520
# (obtained from blockdev --getsize)

multipathd -k
add path sdb
add path sdc

dmsetup suspend resize_test; dmsetup reload resize_test final; dmsetup 
resume resize_test

multipathd -k
reconfigure

# Note: This reconfigure is *ESSENTIAL* as it forces multipath to remember 
the new size. Without this step, failing one of the paths
# can cause multipath to reuse the previous block device size and cause 
filesystem problems.

8.  Resize the FS from userland

ext2online /dev/mapper/resize_test
(or resize2fs in newer )


Pfew! Done!
I think this should have been shared by us (myself, Gino Ledesma, Michael Hsu) a long time ago.
We probably wanted to make a pretty pretty script to do it for us.

There is a slightly more convoluted recipe if you were bold and foolish 
enough to partition the LUN (like we once were).

Cheers,

-sharif

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

RE: Status of resizing/growing dm-multipath devices on the fly

[Domenico Viggiani]

* Sharif Nassar wrote:
>
> We seem to have a working method for resizing multipath luns 
> from our NetApp filers.  It works on CentOS 4 & 5, for us, 
> and our NetApps.
> (2 HBA, times 2 paths on each makes for four SCSI LUN to resize)
> 
> The poorly formatted, unpolished, YMMV, no warranty recipe is:
>
> ...
>
> Pfew! Done!

Incredible! I will study this later, thanks.
Any idea if this could work with an EMC Clariion CX 700 array?

Bye
--
Domenico Viggiani

RE: Status of resizing/growing dm-multipath devices on the fly

[Sharif Nassar]

On Thu, 7 Aug 2008, Domenico Viggiani wrote:

> * Sharif Nassar wrote:
>>
>> We seem to have a working method for resizing multipath luns
>> from our NetApp filers.  It works on CentOS 4 & 5, for us,
>> and our NetApps.
>> (2 HBA, times 2 paths on each makes for four SCSI LUN to resize)
>>
>> The poorly formatted, unpolished, YMMV, no warranty recipe is:
>>
>> ...
>>
>> Pfew! Done!
>
> Incredible! I will study this later, thanks.
> Any idea if this could work with an EMC Clariion CX 700 array?


While we haven't tested it, I don't see any reason why not.  Since the 
core concern is freeing the device so the kernel can see the new size, it 
shouldn't be an issue.

-sharif

Re: Status of resizing/growing dm-multipath devices on the fly

[Pasi Kärkkäinen]

On Wed, Aug 06, 2008 at 02:46:57PM -0700, Sharif Nassar wrote:
> On Wed, 6 Aug 2008, Pasi Kärkkäinen wrote:
> 
> >Hello!
> >
> >Is it possible to resize (grow) dm-multipath devices on the fly nowadays? 
> >
> >I googled and found some discussions about the subject, but the conclusion
> >seemed to be it's not possible.. that was a while ago, so I was wondering 
> >if
> >this has been fixed/implemented.. 
> >
> >Using LVM and adding another new LUN/PV is not an option always.. it's a 
> >lot
> >easier to manage the whole thing if it's possible to resize/grow existing
> >LUNs on the fly.
> 
> 
> Sure it is, but it's not pretty.  Don't believe the naysayers.
> We seem to have a working method for resizing multipath luns from our 
> NetApp filers.  It works on CentOS 4 & 5, for us, and our NetApps.
> (2 HBA, times 2 paths on each makes for four SCSI LUN to resize)
> 
> The poorly formatted, unpolished, YMMV, no warranty recipe is:
> 

Thanks a lot! 

Better to have a workaround than nothing :) 

-- Pasi