Any device mapper target that stores data in files?

Any device mapper target that stores data in files?

[Thiago Padilha]

[-- Attachment #1.1: Type: text/plain, Size: 1962 bytes --]

Hi

First of all, I'm very new on the subject(kernel programming/device
mappers/block devices), so forgive if I say anything stupid.

I need a device mapper target that stores data in files of fixed sizes,
probably defined when the virtual device is first created. For the sake of
explanation, lets call this target as "dirdm" and assume there's an
userspace tool of the same name that can be used to manage such virtual
devices. Now let's say I have an empty directory "/dirdm" and want to
create a virtual device with 4k "block size" on top of it:

    dirdm create --size 10G --block-size 4k /dirdm

After this command is executed, there's a new file with 4k size at
/dirdm/0. This file will be used to store the first block of the device.
Here are the filenames that represent some blocks of this virtual device:

- block 0: 0
- block 1: 4096
- block 2 8192
- block 3: 12288
- block 4: 16384

That is, each block is stored in a file named after the first byte on the
block. Querying a block that has no data written will simply return 0(block
files are also initialized with 0), so it behaves like a sparse block
device.

Is there any device mapper implementation that manages data in a similar
manner? Note that these details about filenames or directory structure are
not important, all I care about is that the dm target splits data in
relatively small files on a directory.

The reason for these requirements is to be able to efficiently store
arbitrary block devices in a cloud storage service such a google
drive/onedrive without syncing a big loop device on every modification(Only
files representing affected blocks are modified). One useful application
would be to create a big device(such as 100g size) and add luks/ext4 on top
of it, which would be a great way to store private data.

If there are no existing dm targets matching these requirements, what would
be a recommended reading/documentation to get started on implementing this
on my own?

[-- Attachment #1.2: Type: text/html, Size: 2316 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Any device mapper target that stores data in files?

[Doug Dumitru]

[-- Attachment #1.1: Type: text/plain, Size: 3323 bytes --]

Mr. Padilha,

In general, what you are trying to do is discouraged.  It is possible to
open and read/write files from kernel space, but the architecture is not
very portable.  I actually do this all the time, but only for debugging
traces.

I would recommend another implementation method if you want to implement a
block device backed by files.  The "Network Block Device" is a project that
includes a kernel module (nbd.ko) and a user-space server.  The kernel
module is usable without the user space module and lets you implement a
decent, multi-threaded, block device completely from user space code.

nbd.ko is reached by opening a master device (which is a character device)
and then reading/writing commands into the single file handle.  Operations
are "async" so they multi-thread easily and you can easily implement all of
the block device functions including new stuff like discard.

Normally, the user-space nature of this would be a performance hit, but if
you are using the "cloud", the extra latency might not matter much.

Doug Dumitru
EasyCo LLC


On Sun, Aug 16, 2015 at 9:55 PM, Thiago Padilha <tpadilha84@gmail.com>
wrote:

> Hi
>
> First of all, I'm very new on the subject(kernel programming/device
> mappers/block devices), so forgive if I say anything stupid.
>
> I need a device mapper target that stores data in files of fixed sizes,
> probably defined when the virtual device is first created. For the sake of
> explanation, lets call this target as "dirdm" and assume there's an
> userspace tool of the same name that can be used to manage such virtual
> devices. Now let's say I have an empty directory "/dirdm" and want to
> create a virtual device with 4k "block size" on top of it:
>
>     dirdm create --size 10G --block-size 4k /dirdm
>
> After this command is executed, there's a new file with 4k size at
> /dirdm/0. This file will be used to store the first block of the device.
> Here are the filenames that represent some blocks of this virtual device:
>
> - block 0: 0
> - block 1: 4096
> - block 2 8192
> - block 3: 12288
> - block 4: 16384
>
> That is, each block is stored in a file named after the first byte on the
> block. Querying a block that has no data written will simply return 0(block
> files are also initialized with 0), so it behaves like a sparse block
> device.
>
> Is there any device mapper implementation that manages data in a similar
> manner? Note that these details about filenames or directory structure are
> not important, all I care about is that the dm target splits data in
> relatively small files on a directory.
>
> The reason for these requirements is to be able to efficiently store
> arbitrary block devices in a cloud storage service such a google
> drive/onedrive without syncing a big loop device on every modification(Only
> files representing affected blocks are modified). One useful application
> would be to create a big device(such as 100g size) and add luks/ext4 on top
> of it, which would be a great way to store private data.
>
> If there are no existing dm targets matching these requirements, what
> would be a recommended reading/documentation to get started on implementing
> this on my own?
>
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>



-- 
Doug Dumitru
EasyCo LLC

[-- Attachment #1.2: Type: text/html, Size: 4944 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Any device mapper target that stores data in files?

[Thiago Padilha]

[-- Attachment #1.1: Type: text/plain, Size: 3716 bytes --]

The network block device seems like a good fit for what I'm trying to do.

Thanks Doug, I will investigate it.

On Mon, Aug 17, 2015 at 10:51 PM Doug Dumitru <doug@easyco.com> wrote:

> Mr. Padilha,
>
> In general, what you are trying to do is discouraged.  It is possible to
> open and read/write files from kernel space, but the architecture is not
> very portable.  I actually do this all the time, but only for debugging
> traces.
>
> I would recommend another implementation method if you want to implement a
> block device backed by files.  The "Network Block Device" is a project that
> includes a kernel module (nbd.ko) and a user-space server.  The kernel
> module is usable without the user space module and lets you implement a
> decent, multi-threaded, block device completely from user space code.
>
> nbd.ko is reached by opening a master device (which is a character device)
> and then reading/writing commands into the single file handle.  Operations
> are "async" so they multi-thread easily and you can easily implement all of
> the block device functions including new stuff like discard.
>
> Normally, the user-space nature of this would be a performance hit, but if
> you are using the "cloud", the extra latency might not matter much.
>
> Doug Dumitru
> EasyCo LLC
>
>
> On Sun, Aug 16, 2015 at 9:55 PM, Thiago Padilha <tpadilha84@gmail.com>
> wrote:
>
>> Hi
>>
>> First of all, I'm very new on the subject(kernel programming/device
>> mappers/block devices), so forgive if I say anything stupid.
>>
>> I need a device mapper target that stores data in files of fixed sizes,
>> probably defined when the virtual device is first created. For the sake of
>> explanation, lets call this target as "dirdm" and assume there's an
>> userspace tool of the same name that can be used to manage such virtual
>> devices. Now let's say I have an empty directory "/dirdm" and want to
>> create a virtual device with 4k "block size" on top of it:
>>
>>     dirdm create --size 10G --block-size 4k /dirdm
>>
>> After this command is executed, there's a new file with 4k size at
>> /dirdm/0. This file will be used to store the first block of the device.
>> Here are the filenames that represent some blocks of this virtual device:
>>
>> - block 0: 0
>> - block 1: 4096
>> - block 2 8192
>> - block 3: 12288
>> - block 4: 16384
>>
>> That is, each block is stored in a file named after the first byte on the
>> block. Querying a block that has no data written will simply return 0(block
>> files are also initialized with 0), so it behaves like a sparse block
>> device.
>>
>> Is there any device mapper implementation that manages data in a similar
>> manner? Note that these details about filenames or directory structure are
>> not important, all I care about is that the dm target splits data in
>> relatively small files on a directory.
>>
>> The reason for these requirements is to be able to efficiently store
>> arbitrary block devices in a cloud storage service such a google
>> drive/onedrive without syncing a big loop device on every modification(Only
>> files representing affected blocks are modified). One useful application
>> would be to create a big device(such as 100g size) and add luks/ext4 on top
>> of it, which would be a great way to store private data.
>>
>> If there are no existing dm targets matching these requirements, what
>> would be a recommended reading/documentation to get started on implementing
>> this on my own?
>>
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>
>
>
> --
> Doug Dumitru
> EasyCo LLC
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

[-- Attachment #1.2: Type: text/html, Size: 5933 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: Any device mapper target that stores data in files?

[Zdenek Kabelac]

Dne 18.8.2015 v 08:45 Thiago Padilha napsal(a):
>
>         I need a device mapper target that stores data in files of fixed
>         sizes, probably defined when the virtual device is first created. For
>         the sake of explanation, lets call this target as "dirdm" and assume
>         there's an userspace tool of the same name that can be used to manage
>         such virtual devices. Now let's say I have an empty directory "/dirdm"
>         and want to create a virtual device with 4k "block size" on top of it:
>
>              dirdm create --size 10G --block-size 4k /dirdm
>
>         After this command is executed, there's a new file with 4k size at
>         /dirdm/0. This file will be used to store the first block of the
>         device. Here are the filenames that represent some blocks of this
>         virtual device:

Hi


I don't quite follow what you try to invent here - it looks like you play with 
the idea of doing 'btrfs' without btrfs.

DM is about devices - as soon as you start to mix 'filesystem-layer' with 
'block-layer' you get into troubles you cannot resolve - simply because of the 
interacting locking inside kernel is not supported for this use-case.

i.e.  for initial playing you could use  any file  via  'loop' device.
(There is even  dm-loop support which however is not upstream since there
is already 'loop' target.)

At the end however you will get into cases you cannot resolve once your 
filesystem on DM device will get blocked by underlying filesystem used to hold 
backend file for loop device.

For DM - you should stay with pure block devices. Loop over filesystem is good 
for 'testing' but it should be avoided as advised way for using any DM device.

For b-tree layered storage please check thin-pool.

Regards

Zdenek