From mboxrd@z Thu Jan 1 00:00:00 1970 From: "David Guyon Martin" Subject: device mapper integrated loops - and one more year ! Date: Thu, 5 Oct 2006 16:26:54 +0200 Message-ID: <2c53b31c0610050726o47e6784en417de93b98d15e50@mail.gmail.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0479858206==" Return-path: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: dm-devel@redhat.com List-Id: dm-devel.ids --===============0479858206== Content-Type: multipart/alternative; boundary="----=_Part_11298_30099163.1160058414050" ------=_Part_11298_30099163.1160058414050 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello, Do you think dm could have a loop target instead of seting loops with losetup ? We allready have dm-crypt for the encryption stage of a "losetup -e ", dm replaces losetup when source is a device but I understand that we still need losetup if the source is a regular file. Our problem is that we will need to map a great number of files, as if they were PVs of a VG. Linux kernel can have up to 255 loops but we will reach this limit. The idea is to embed to loop itself in a device mapper add-on like dm-crypt, so we won't use any /dev/loop* . Do you think this is stupid or is there some possility out there ? (we would like to have some advice before starting any project in this direction) Thank you, David G.M. PS: Anniversary of this post, one year ago ! http://www.redhat.com/archives/dm-devel/2005-October/msg00031.html ------=_Part_11298_30099163.1160058414050 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello,

Do you think dm could have a loop target instead of seting loops with losetup ?
We allready have dm-crypt for the encryption stage of a "losetup -e <crypt>", dm replaces losetup when source is a device but I understand that we still need losetup if the source is a regular file.
Our problem is that we will need to map a great number of files, as if they were PVs of a VG. Linux kernel can have up to 255 loops but we will reach this limit.

The idea is to embed to loop itself in a device mapper add-on like dm-crypt, so we won't use any /dev/loop* .

Do you think this is stupid or is there some possility out there ?
(we would like to have some advice before starting any project in this direction)

Thank you,
David G.M.

PS:
Anniversary of this post, one year ago !
http://www.redhat.com/archives/dm-devel/2005-October/msg00031.html
------=_Part_11298_30099163.1160058414050-- --===============0479858206== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============0479858206==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alasdair G Kergon Subject: Re: device mapper integrated loops - and one more year ! Date: Thu, 5 Oct 2006 20:40:05 +0100 Message-ID: <20061005194005.GO17654@agk.surrey.redhat.com> References: <2c53b31c0610050726o47e6784en417de93b98d15e50@mail.gmail.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <2c53b31c0610050726o47e6784en417de93b98d15e50@mail.gmail.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: device-mapper development List-Id: dm-devel.ids On Thu, Oct 05, 2006 at 04:26:54PM +0200, David Guyon Martin wrote: > Do you think dm could have a loop target instead of seting loops with > losetup ? Something like: http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-loop.patch now perhaps? (And there are patches for dmsetup to become a drop-in losetup replacement.) Alasdair -- agk@redhat.com From mboxrd@z Thu Jan 1 00:00:00 1970 From: "David Guyon Martin" Subject: Re: device mapper integrated loops - and one more year ! Date: Fri, 6 Oct 2006 11:38:12 +0200 Message-ID: <2c53b31c0610060238nef13171ma6f4a42ba755c3b0@mail.gmail.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: dm-devel@redhat.com List-Id: dm-devel.ids On Thu, 5 Oct 2006 20:40:05 +0100 Alasdair G Kergon wrote: >Something like: > http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-loop.patch >now perhaps? >(And there are patches for dmsetup to become a drop-in losetup replacement.) Yes! This patch add a new file to the kernel tree, dm-loop.c: "This implements a loopback target for device mapper allowing a regular file to be treated as a block device." Great !!! At last :) It seems this dm-loop is quite new, less than a month... As I understand it I need kernel 2.6.18-rc7 to use it, right ? I really thank you a lot for your quick answer and link. Does anyone has yet a feedback about this dm-loop ? David G.M. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jonathan E Brassow Subject: Re: device mapper integrated loops - and one more year ! Date: Mon, 9 Oct 2006 17:36:40 -0500 Message-ID: <950bd3ad1504b8fb03e73b6236a808bc@redhat.com> References: <2c53b31c0610060238nef13171ma6f4a42ba755c3b0@mail.gmail.com> Reply-To: device-mapper development Mime-Version: 1.0 (Apple Message framework v624) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <2c53b31c0610060238nef13171ma6f4a42ba755c3b0@mail.gmail.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: device-mapper development List-Id: dm-devel.ids On Oct 6, 2006, at 4:38 AM, David Guyon Martin wrote: > On Thu, 5 Oct 2006 20:40:05 +0100 Alasdair G Kergon wrote: > >> Something like: >> >> http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/ >> editing/dm-loop.patch >> now perhaps? > >> (And there are patches for dmsetup to become a drop-in losetup >> replacement.) > > Yes! This patch add a new file to the kernel tree, dm-loop.c: > "This implements a loopback target for device mapper allowing a regular > file to be treated as a block device." > > Great !!! At last :) > It seems this dm-loop is quite new, less than a month... > As I understand it I need kernel 2.6.18-rc7 to use it, right ? > > I really thank you a lot for your quick answer and link. > > Does anyone has yet a feedback about this dm-loop ? > I know Heinz Mauelshagen has been looking at it and already has some performance improvements. Perhaps he'll post the updated patches soon. brassow From mboxrd@z Thu Jan 1 00:00:00 1970 From: Heinz Mauelshagen Subject: Re: device mapper integrated loops - and one more year ! Date: Tue, 10 Oct 2006 20:37:45 +0200 Message-ID: <20061010183745.GA7049@redhat.com> References: <2c53b31c0610060238nef13171ma6f4a42ba755c3b0@mail.gmail.com> <950bd3ad1504b8fb03e73b6236a808bc@redhat.com> Reply-To: mauelshagen@redhat.com, device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <950bd3ad1504b8fb03e73b6236a808bc@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: device-mapper development List-Id: dm-devel.ids On Mon, Oct 09, 2006 at 05:36:40PM -0500, Jonathan E Brassow wrote: > > On Oct 6, 2006, at 4:38 AM, David Guyon Martin wrote: > > >On Thu, 5 Oct 2006 20:40:05 +0100 Alasdair G Kergon wrote: > > > >>Something like: > >> > >>http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/ > >>editing/dm-loop.patch > >>now perhaps? > > > >>(And there are patches for dmsetup to become a drop-in losetup > >>replacement.) > > > >Yes! This patch add a new file to the kernel tree, dm-loop.c: > >"This implements a loopback target for device mapper allowing a regular > >file to be treated as a block device." > > > >Great !!! At last :) > >It seems this dm-loop is quite new, less than a month... > >As I understand it I need kernel 2.6.18-rc7 to use it, right ? > > > >I really thank you a lot for your quick answer and link. > > > >Does anyone has yet a feedback about this dm-loop ? > > > > I know Heinz Mauelshagen has been looking at it and already has some > performance improvements. Perhaps he'll post the updated patches soon. Yes, I'm settling the changes, which optimize the linear search in the file extents table and the initial lookup of the extents in the file to create the table with the dm-loop author Bryn Reeves right now. First benchmarks show promissing results :-) We hope to post something shortly. Regards, Heinz -- The LVM Guy -- > > brassow > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Heinz Mauelshagen Red Hat GmbH Consulting Development Engineer Am Sonnenhang 11 Storage Development 56242 Marienrachdorf Germany Mauelshagen@RedHat.com PHONE +49 171 7803392 FAX +49 2626 924446 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "David Guyon Martin" Subject: Re: device mapper integrated loops - and one more year ! Date: Tue, 10 Oct 2006 22:11:27 +0200 Message-ID: <2c53b31c0610101311x6bd97cd3n45d2d941f79c1d2a@mail.gmail.com> References: <2c53b31c0610060238nef13171ma6f4a42ba755c3b0@mail.gmail.com> <950bd3ad1504b8fb03e73b6236a808bc@redhat.com> <20061010183745.GA7049@redhat.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20061010183745.GA7049@redhat.com> Content-Disposition: inline List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: mauelshagen@redhat.com, device-mapper development List-Id: dm-devel.ids 2006/10/10, Heinz Mauelshagen : > On Mon, Oct 09, 2006 at 05:36:40PM -0500, Jonathan E Brassow wrote: > > > > On Oct 6, 2006, at 4:38 AM, David Guyon Martin wrote: > > > > >On Thu, 5 Oct 2006 20:40:05 +0100 Alasdair G Kergon wrote: > > > > > >>Something like: > > >> > > >>http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/ > > >>editing/dm-loop.patch > > >>now perhaps? > > > > > >>(And there are patches for dmsetup to become a drop-in losetup > > >>replacement.) > > > > > >Yes! This patch add a new file to the kernel tree, dm-loop.c: > > >"This implements a loopback target for device mapper allowing a regular > > >file to be treated as a block device." > > > > > >Great !!! At last :) > > >It seems this dm-loop is quite new, less than a month... > > >As I understand it I need kernel 2.6.18-rc7 to use it, right ? > > > > > >I really thank you a lot for your quick answer and link. > > > > > >Does anyone has yet a feedback about this dm-loop ? > > > > > > > I know Heinz Mauelshagen has been looking at it and already has some > > performance improvements. Perhaps he'll post the updated patches soon. > > Yes, I'm settling the changes, which optimize the linear search > in the file extents table and the initial lookup of the extents > in the file to create the table with the dm-loop author Bryn Reeves > right now. > > First benchmarks show promissing results :-) > > We hope to post something shortly. > > Regards, > Heinz -- The LVM Guy -- > Thanks a lot ! I stay tuned ( I can hardly wait the results ). I think many sysadmins underestimate the power of loops, more loops and greater performances will ease eyes opening. >From dm-loop, comes dm-stego. I didn't found anything similar in the dev tree. Does anyone know if someone is working on a dm-stego filter ? David G.M. PS: Heinz, since you've written dm-delay, you may have a clue to http://www.redhat.com/archives/dm-devel/2006-October/msg00048.html From mboxrd@z Thu Jan 1 00:00:00 1970 From: devzero@web.de Subject: Re: device mapper integrated loops - and one more year ! Date: Sun, 19 Nov 2006 12:30:05 +0100 Message-ID: <1439173022@web.de> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: quoted-printable Return-path: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: dm-devel@redhat.com List-Id: dm-devel.ids Hello ! after desparately seeking for a solution to have more than 256 loop devic= es (for a huge cd-rom server), i have recently come across dm-loop, which= looks very promising (to replace loop.c) can i have more than 256 devices which this? our cd-rom server is going to have more than 256 iso-images soon and i ne= ed to upgrade to a more recent OS, anyway, so this won`t be a problem tha= t dm-loop is so brand new. but - any timeline for dm-loop mainline inclusion ? should i wait for 2.6.19 or 2.6.20 ? is it already useable with less recent kernels, i.e. can i download dm-lo= op patch and recent dmsetup tool and use with older kernel ? i really would like to give it a try - no problem if there are still bugs= inside, because i`d happily test and report if there are bugs left. regards Roland K. systems engineer _______________________________________________________________________ Viren-Scan f=FCr Ihren PC! Jetzt f=FCr jeden. Sofort, online und kostenlo= s. Gleich testen! http://www.pc-sicherheit.web.de/freescan/?mc=3D022222 From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Bryn M. Reeves" Subject: Re: Re: device mapper integrated loops - and one more year ! Date: Tue, 21 Nov 2006 21:11:37 +0000 Message-ID: <45636B89.7080407@redhat.com> References: <1439173022@web.de> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1439173022@web.de> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: device-mapper development List-Id: dm-devel.ids -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 devzero@web.de wrote: > Hello ! > > after desparately seeking for a solution to have more than 256 loop > devices (for a huge cd-rom server), i have recently come across > dm-loop, which looks very promising (to replace loop.c) > > can i have more than 256 devices which this? Hi Roland, Yes, dm-loop will support as many loop devices as device-mapper can allow. > our cd-rom server is going to have more than 256 iso-images soon and > i need to upgrade to a more recent OS, anyway, so this won`t be a > problem that dm-loop is so brand new. > > but - any timeline for dm-loop mainline inclusion ? > > should i wait for 2.6.19 or 2.6.20 ? There isn't a fixed timeline. We are currently working to improve the lookup code - the prototype patch uses a linear search which doesn't scale well for large/fragmented image files. I have some tests running at the moment and we hope to have something soon - the results so far are good, it's just a case of selecting between a couple of different methods. The changes dm-loop requires in device-mapper core are pretty minor and have already been merged in 2.6.19, so it's just the dm-loop patch itself remaining. > is it already useable with less recent kernels, i.e. can i download > dm-loop patch and recent dmsetup tool and use with older kernel ? This should work fine - there are a couple of known issues with the existing patch, but unless you are using large images (>2G), or have a severely fragmented filesystem you shouldn't run into these. I'll see if we can get a revised version of the current patch for wider testing in the next couple of weeks - let me know if you would be interested in trying this out. Thanks, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFFY2uJ6YSQoMYUY94RArTwAJwMmCJbTajxmX6Q/ps545zbgvJJOACg28NP A+F6CtqcHH0fOG9lq2XQw/8= =CzES -----END PGP SIGNATURE----- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roland PJ Subject: Re: Re: device mapper integrated loops - and one more year ! Date: Tue, 21 Nov 2006 23:55:27 +0200 Message-ID: <456375CF.10006@rolandpj.com> References: <1439173022@web.de> <45636B89.7080407@redhat.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <45636B89.7080407@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: device-mapper development List-Id: dm-devel.ids Hi Bryn Just to avoid confusion, this is a different Roland from the original query. Looking at http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/dm-loop.patch + if (!cur_block) + goto bad_bmap; ... +bad_bmap: + DMERR("Loopfile has holes"); + dump_extent_map(map); Does this mean that dm-loop does not support sparse loop-back files? Also, it seems that the approach with dm-loop is that you sniff the (current?) file mapping onto its underlying block device. What happens if the loopback file is modified, or the file system hosting the loopback file does some re-arranging? (I might be way off-tack here - this was a first impression of the code). I'd be interested in comparing dm-loop to vanilla loop which we currently use. Regards Roland #2 > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Bryn M. Reeves" Subject: Re: Re: device mapper integrated loops - and one more year ! Date: Tue, 21 Nov 2006 23:20:36 +0000 Message-ID: <456389C4.9000902@redhat.com> References: <1439173022@web.de> <45636B89.7080407@redhat.com> <456375CF.10006@rolandpj.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <456375CF.10006@rolandpj.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: device-mapper development List-Id: dm-devel.ids -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Roland PJ wrote: > Hi Bryn > > Just to avoid confusion, this is a different Roland from the original > query. Hi Roland, Noted ;) > Does this mean that dm-loop does not support sparse loop-back files? This version doesn't, no. It's not hard to add this though, although it does have some implications if they are used. We've had some discussions over whether this is necessary/desirable - is this something you'd like to see? > Also, it seems that the approach with dm-loop is that you sniff the > (current?) file mapping onto its underlying block device. What happens > if the loopback file is modified, or the file system hosting the > loopback file does some re-arranging? (I might be way off-tack here - > this was a first impression of the code). You'll find this in loop_get_file(): + /* + * We overload the S_SWAPFILE flag for loop targets because + * it provides the same no-truncate semantics we require, and holding + * onto i_sem is no longer an option. + */ + mutex_lock(&inode->i_mutex); + inode->i_flags |= S_SWAPFILE; + mutex_unlock(&inode->i_mutex); The bmap approach is also used for swapfiles (mm/swapfile.c) - the S_SWAPFILE inode flag was added in 2.6.16 when the changeover to mutexes happened. > I'd be interested in comparing dm-loop to vanilla loop which we > currently use. > There are patches to dmsetup that allow it to be called as "losetup" or "dmlosetup", providing the same options as the regular version, so it should be straightforward to compare. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFFY4nE6YSQoMYUY94RAuFyAJ9heEQvTZZ7dEkaIXQ9nxFy7mDx+gCfTozW FusS8of73r2GfGTchFWRvdc= =Eisk -----END PGP SIGNATURE----- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roland Paterson-Jones Subject: Re: Re: device mapper integrated loops - and one more year ! Date: Wed, 22 Nov 2006 10:29:02 +0200 Message-ID: <45640A4E.9090509@rolandpj.com> References: <1439173022@web.de> <45636B89.7080407@redhat.com> <456375CF.10006@rolandpj.com> <456389C4.9000902@redhat.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <456389C4.9000902@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: device-mapper development List-Id: dm-devel.ids Hi Bryn Bryn M. Reeves wrote: >>Does this mean that dm-loop does not support sparse loop-back files? >> >> > >This version doesn't, no. It's not hard to add this though, although it >does have some implications if they are used. We've had some discussions >over whether this is necessary/desirable - is this something you'd like >to see? > > We very much need sparse files for our use-case (fairly dynamically setting up (fake) devices on demand). Sparse files save time to copy, for example. Would you do this by using file ops to lazily fill holes on first write? Is this compatible with S_SWAPFILE? For read purposes, holes can be mapped to zero device, I presume. >The bmap approach is also used for swapfiles (mm/swapfile.c) - the >S_SWAPFILE inode flag was added in 2.6.16 when the changeover to mutexes >happened. > > Ah. I did see that - should have made the connection. So which file systems support (obey?) S_SWAPFILE? I much prefer the approach of mapping to underlying device, cos it sidesteps the file/page cache (loopback causes OOM!). The other annoying thing about loopback is that it re-nices itself to super-low (should that be high?) priority, so we can starve other system daemons by banging on the loopback device. >There are patches to dmsetup that allow it to be called as "losetup" or >"dmlosetup", providing the same options as the regular version, so it >should be straightforward to compare. > > Cool. Where can I get a patch to play with? I presume device-mapper plug-ins can be compiled as modules? Regards Roland #2 From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Bryn M. Reeves" Subject: Re: Re: device mapper integrated loops - and one more year ! Date: Wed, 22 Nov 2006 21:37:43 +0000 Message-ID: <4564C327.1080101@redhat.com> References: <1439173022@web.de> <45636B89.7080407@redhat.com> <456375CF.10006@rolandpj.com> <456389C4.9000902@redhat.com> <45640A4E.9090509@rolandpj.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <45640A4E.9090509@rolandpj.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: device-mapper development List-Id: dm-devel.ids -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Roland Paterson-Jones wrote: > We very much need sparse files for our use-case (fairly dynamically > setting up (fake) devices on demand). Sparse files save time to copy, > for example. > > Would you do this by using file ops to lazily fill holes on first write? > Is this compatible with S_SWAPFILE? For read purposes, holes can be > mapped to zero device, I presume. That's what I figured - lazy allocation seems to be the main reason that sparse files are desirable. The problem with sparse files is that they make us do something that dm-loop is explicitly trying to avoid - calling into the filesystem at I/O time. There are two ways we can address this. The dm-loop target needs a non-bmap based fallback for filesystems that can't use direct mapping anyway (e.g. network filesystems). We can either revert to this mechanism entirely for sparse files (means we get many of the drawbacks of regular /dev/loopN), or try to support mixed extent types. That's had some thought & discussion but we don't have an implementation ready yet. > Ah. I did see that - should have made the connection. So which file > systems support (obey?) S_SWAPFILE? Currently this is handled in the VFS. Currently the only filesystem that explicitly checks for S_SWAPFILE is NFS. Filesystems that support online defragmentation will need to be aware of this flag. > I much prefer the approach of mapping to underlying device, cos it > sidesteps the file/page cache (loopback causes OOM!). The other annoying > thing about loopback is that it re-nices itself to super-low (should > that be high?) priority, so we can starve other system daemons by > banging on the loopback device. This is really the motivation for dm-loop: avoiding unpredictable memory allocation at I/O time and the need for a per-device kernel thread. It also ends up simplifying the code a great deal. Early figures on performance are also suggesting wins in that area. >> There are patches to dmsetup that allow it to be called as "losetup" or >> "dmlosetup", providing the same options as the regular version, so it >> should be straightforward to compare. >> >> > Cool. Where can I get a patch to play with? I presume device-mapper > plug-ins can be compiled as modules? The patch currently on kernel.org uses a table format like this: loop So losetup /dev/loop0 /tmp/file becomes roughly dmsetup create loop0 --table "0 N loop /tmp/file 0" --table option is now in cvs; else use stdin N is the /512 make file a multiple of your page size & fs block size If your dmsetup is recent enough to have the --table option. I was just digging out the dmlosetup patch when I noticed it's already present in recent CVS - if you build dmsetup from there, you should be able to symlink dmsetup as losetup/dmlosetup and call it like the regular version. Regards, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFFZMMn6YSQoMYUY94RAvkPAKCWNrp2VghDpu1qd7nC2J/OgwQF+gCcCjvV 3PbJcCxOqZIpteMsNKht3Js= =B/b2 -----END PGP SIGNATURE----- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Roland Paterson-Jones Subject: Re: Re: device mapper integrated loops - and one more year ! Date: Thu, 23 Nov 2006 13:15:13 +0200 Message-ID: <456582C1.3030309@rolandpj.com> References: <1439173022@web.de> <45636B89.7080407@redhat.com> <456375CF.10006@rolandpj.com> <456389C4.9000902@redhat.com> <45640A4E.9090509@rolandpj.com> <4564C327.1080101@redhat.com> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4564C327.1080101@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: device-mapper development List-Id: dm-devel.ids Bryn M. Reeves wrote: >Roland Paterson-Jones wrote: > > >>... sparse files ... >> >>Would you do this by using file ops to lazily fill holes on first write? >>Is this compatible with S_SWAPFILE? For read purposes, holes can be >>mapped to zero device, I presume. >> >> > >That's what I figured - lazy allocation seems to be the main reason that >sparse files are desirable. > >The problem with sparse files is that they make us do something that >dm-loop is explicitly trying to avoid - calling into the filesystem at >I/O time. > > I can live with that, as long as it's only a first-write phenomenon. I.e. first-write of a hole uses file-ops, then sniffs the underlying device mapping for subsequent ops. Although I guess this could cause the same OOM regime for a lot of writes on an initially sparse file (or is this exacerbated by loopbacks elevated priority?). >There are two ways we can address this. The dm-loop target needs a >non-bmap based fallback for filesystems that can't use direct mapping >anyway (e.g. network filesystems). > >We can either revert to this mechanism entirely for sparse files (means >we get many of the drawbacks of regular /dev/loopN), or try to support >mixed extent types. That's had some thought & discussion but we don't >have an implementation ready yet. > > I'd be very much in favour of a lazy mixed approach that converges towards device-only access with substantial use. Otherwise I think I'm back where I started. > >The patch currently on kernel.org uses a table format like this: > loop > > Thanks. I'll play ;) Roland From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Bryn M. Reeves" Subject: Re: device mapper integrated loops - and one more year ! Date: Mon, 22 Jan 2007 09:43:24 +0000 Message-ID: <45B4873C.8030504@redhat.com> References: <1572893983@web.de> Reply-To: device-mapper development Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1572893983@web.de> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: devzero@web.de Cc: device-mapper development List-Id: dm-devel.ids -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 devzero@web.de wrote: > Hello Bryn, > > >> I'll see if we can get a revised version of the current patch for wider >> testing in the next couple of weeks - let me know if you would be >> interested in trying this out. > > not sure if i sent a reply to this or have missed this - but, yes - i`m still interested very much in using dm-loop ! Hi Roland, No, you've not missed anything! I've got an updated version of the patch with some enhancements Heinz (lvmguy) has been working on to improve the driver's lookup performance. This is still being worked on at the moment. > is there anything new with this? > do i have to expect issues with files >2gb ? (i need to loopback mount dvd iso-images) > > is there a download url for the latest dm-loop patch so i can do some tests and report feedback ? This patch is a little out of date now - I'll get a later version (bugfixes mainly - the new lookup code is still being tested) made available here. Thanks, Bryn. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFFtIcl6YSQoMYUY94RAsRqAJ4iKrXKW5IYe0S9NlUofBp1QpzoiQCgxxba gfoC4jE1+anDhZwz/+MFtUs= =IBSf -----END PGP SIGNATURE-----