Re: Any device mapper target that stores data in files?

[Thiago Padilha <tpadilha84@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 3716 bytes --]

The network block device seems like a good fit for what I'm trying to do.

Thanks Doug, I will investigate it.

On Mon, Aug 17, 2015 at 10:51 PM Doug Dumitru <doug@easyco.com> wrote:

> Mr. Padilha,
>
> In general, what you are trying to do is discouraged.  It is possible to
> open and read/write files from kernel space, but the architecture is not
> very portable.  I actually do this all the time, but only for debugging
> traces.
>
> I would recommend another implementation method if you want to implement a
> block device backed by files.  The "Network Block Device" is a project that
> includes a kernel module (nbd.ko) and a user-space server.  The kernel
> module is usable without the user space module and lets you implement a
> decent, multi-threaded, block device completely from user space code.
>
> nbd.ko is reached by opening a master device (which is a character device)
> and then reading/writing commands into the single file handle.  Operations
> are "async" so they multi-thread easily and you can easily implement all of
> the block device functions including new stuff like discard.
>
> Normally, the user-space nature of this would be a performance hit, but if
> you are using the "cloud", the extra latency might not matter much.
>
> Doug Dumitru
> EasyCo LLC
>
>
> On Sun, Aug 16, 2015 at 9:55 PM, Thiago Padilha <tpadilha84@gmail.com>
> wrote:
>
>> Hi
>>
>> First of all, I'm very new on the subject(kernel programming/device
>> mappers/block devices), so forgive if I say anything stupid.
>>
>> I need a device mapper target that stores data in files of fixed sizes,
>> probably defined when the virtual device is first created. For the sake of
>> explanation, lets call this target as "dirdm" and assume there's an
>> userspace tool of the same name that can be used to manage such virtual
>> devices. Now let's say I have an empty directory "/dirdm" and want to
>> create a virtual device with 4k "block size" on top of it:
>>
>>     dirdm create --size 10G --block-size 4k /dirdm
>>
>> After this command is executed, there's a new file with 4k size at
>> /dirdm/0. This file will be used to store the first block of the device.
>> Here are the filenames that represent some blocks of this virtual device:
>>
>> - block 0: 0
>> - block 1: 4096
>> - block 2 8192
>> - block 3: 12288
>> - block 4: 16384
>>
>> That is, each block is stored in a file named after the first byte on the
>> block. Querying a block that has no data written will simply return 0(block
>> files are also initialized with 0), so it behaves like a sparse block
>> device.
>>
>> Is there any device mapper implementation that manages data in a similar
>> manner? Note that these details about filenames or directory structure are
>> not important, all I care about is that the dm target splits data in
>> relatively small files on a directory.
>>
>> The reason for these requirements is to be able to efficiently store
>> arbitrary block devices in a cloud storage service such a google
>> drive/onedrive without syncing a big loop device on every modification(Only
>> files representing affected blocks are modified). One useful application
>> would be to create a big device(such as 100g size) and add luks/ext4 on top
>> of it, which would be a great way to store private data.
>>
>> If there are no existing dm targets matching these requirements, what
>> would be a recommended reading/documentation to get started on implementing
>> this on my own?
>>
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>
>
>
> --
> Doug Dumitru
> EasyCo LLC
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

[-- Attachment #1.2: Type: text/html, Size: 5933 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]