UBIFS: Is it possible to get the compressed size of a file?

UBIFS: Is it possible to get the compressed size of a file?

[markus.heininger]

Hello,
 
when using compression along with UBIfs, fstat reports the uncompressed file size
(as it should do). Is there a simple way to get the compressed file size, thus the real space
a single file needs on the flash memory?
 
Many thanks,
Markus

Re: UBIFS: Is it possible to get the compressed size of a file?

[hujianyang]

On 2014/12/12 16:10, markus.heininger@online.de wrote:
> Hello,
>  
> when using compression along with UBIfs, fstat reports the uncompressed file size
> (as it should do). Is there a simple way to get the compressed file size, thus the real space
> a single file needs on the flash memory?

Why you need this?

I know *df* shows the space of a partition. Run *sync* after
a file created or appended, the space change of this partition
maybe could show you the compressed size of this file.

Is that enough for you?

>  
> Many thanks,
> Markus
> 

Aw: Re: UBIFS: Is it possible to get the compressed size of a file?

[markus.heininger]

Hello,

in our system we´re using ring buffers where it is necessary to
remove old files from certain directories when the physical usage
of each directory is above a certain level which is different for each 
folder.

Evaluating the output of "df" after write access might be difficult
since there are several concurrent writing processes.

But many thanks for your answer, it seems that there is no easy
way to get the information needed and we must investigate further on
our own.

With best regards,
Markus

Gesendet: Mittwoch, 17. Dezember 2014 um 07:22 Uhr
Von: hujianyang <hujianyang@huawei.com>
An: "markus.heininger@online.de" <markus.heininger@online.de>
Cc: linux-mtd@lists.infradead.org, "Artem Bityutskiy" <dedekind1@gmail.com>
Betreff: Re: UBIFS: Is it possible to get the compressed size of a file?
On 2014/12/12 16:10, markus.heininger@online.de wrote:
> Hello,
>
> when using compression along with UBIfs, fstat reports the uncompressed file size
> (as it should do). Is there a simple way to get the compressed file size, thus the real space
> a single file needs on the flash memory?

Why you need this?

I know *df* shows the space of a partition. Run *sync* after
a file created or appended, the space change of this partition
maybe could show you the compressed size of this file.

Is that enough for you?

>
> Many thanks,
> Markus
>

 

Re: UBIFS: Is it possible to get the compressed size of a file?

[Artem Bityutskiy]

Hi Markus,

On Thu, 2014-12-18 at 12:08 +0100, markus.heininger@online.de wrote:
> in our system we´re using ring buffers where it is necessary to
> remove old files from certain directories when the physical usage
> of each directory is above a certain level which is different for each 
> folder.
> 
> Evaluating the output of "df" after write access might be difficult
> since there are several concurrent writing processes.
> 
> But many thanks for your answer, it seems that there is no easy
> way to get the information needed and we must investigate further on
> our own.

Yes, no easy way, but I think implementing what you need is possible. I
do not have plans and time to work on this, but I can help by giving
advises and review.

The question has 2 major parts.

1. The interface
2. The implementation


For the former, one need to carefully investigate if there is something
like this already implemented for other file-systems. I think btrfs may
have it. If it is, then UBIFS should use similar interface, probably.

And whatever is the interface choice, it should be discussed in the
linux-fsdevel@vger.kernel.org mailing list, which I am CCing.


For the latter, I'd suggest to try this.

a. 'struct ubifs_ino_node' has unused space, use it to add the
compressed size field.
b. maintain this field
c. this field will only be correct for the part of the file which are on
the media. The dirty data in the page cache has not yet been compressed,
so we do not know its compressed size yet.
e. when user asks for the compressed size, you have to sync the inode
first, in order to make sure the compressed size is correct.

And the implementation should be backward-compatible. That is, if new
driver mounts the old media, you return something predicatable. I guess
uncompressed size could be it.


Hope this helps,
Artem.

Re: UBIFS: Is it possible to get the compressed size of a file?

[Andreas Dilger]

On Feb 2, 2015, at 2:33 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> Hi Markus,
> 
> On Thu, 2014-12-18 at 12:08 +0100, markus.heininger@online.de wrote:
>> in our system we´re using ring buffers where it is necessary to
>> remove old files from certain directories when the physical usage
>> of each directory is above a certain level which is different for each 
>> folder.
>> 
>> Evaluating the output of "df" after write access might be difficult
>> since there are several concurrent writing processes.
>> 
>> But many thanks for your answer, it seems that there is no easy
>> way to get the information needed and we must investigate further on
>> our own.
> 
> Yes, no easy way, but I think implementing what you need is possible. I
> do not have plans and time to work on this, but I can help by giving
> advises and review.
> 
> The question has 2 major parts.
> 
> 1. The interface
> 2. The implementation
> 
> For the former, one need to carefully investigate if there is something
> like this already implemented for other file-systems. I think btrfs may
> have it. If it is, then UBIFS should use similar interface, probably.
> 
> And whatever is the interface choice, it should be discussed in the
> linux-fsdevel@vger.kernel.org mailing list, which I am CCing.

One option that was discussed for btrfs was to use the first fe_reserved
field for the FIEMAP ioctl struct fiemap_extent to fe_phys_length to
hold the compressed size of each extent in the file.

http://lwn.net/Articles/607552/
http://thread.gmane.org/gmane.comp.file-systems.btrfs/37312

I'm not sure what happened to that patch series - I was looking forward
to it landing, and it was in very good shape I think.

I think FIEMAP would be a good userspace API to get this kind of
low-level allocation information about the compressed size of data.
For most files it would probably only need a single fiemap_extent
to return the allocation data.

Cheers, Andreas

> For the latter, I'd suggest to try this.
> 
> a. 'struct ubifs_ino_node' has unused space, use it to add the
> compressed size field.
> b. maintain this field
> c. this field will only be correct for the part of the file which are on
> the media. The dirty data in the page cache has not yet been compressed,
> so we do not know its compressed size yet.
> e. when user asks for the compressed size, you have to sync the inode
> first, in order to make sure the compressed size is correct.
> 
> And the implementation should be backward-compatible. That is, if new
> driver mounts the old media, you return something predicatable. I guess
> uncompressed size could be it.
> 
> 
> Hope this helps,
> Artem.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Cheers, Andreas

Re: UBIFS: Is it possible to get the compressed size of a file?

[Artem Bityutskiy]

On Mon, 2015-02-02 at 22:45 -0700, Andreas Dilger wrote:
> I think FIEMAP would be a good userspace API to get this kind of
> low-level allocation information about the compressed size of data.
> For most files it would probably only need a single fiemap_extent
> to return the allocation data.

Thanks Andreas,

indeed FIEMAP sounds like a good answer to the problem.

Artem.

Re: UBIFS: Is it possible to get the compressed size of a file?

[hujianyang]

Hi Artem and Andreas,

I had worked on this feature for a few days. Now I meet some problems,
I want to discuss them with you and wish you could give me some advises.

On 2015/2/3 13:45, Andreas Dilger wrote:
> On Feb 2, 2015, at 2:33 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
>>
>> Yes, no easy way, but I think implementing what you need is possible. I
>> do not have plans and time to work on this, but I can help by giving
>> advises and review.
>>
>> The question has 2 major parts.
>>
>> 1. The interface
>> 2. The implementation
>>
>> For the former, one need to carefully investigate if there is something
>> like this already implemented for other file-systems. I think btrfs may
>> have it. If it is, then UBIFS should use similar interface, probably.
>>
>> And whatever is the interface choice, it should be discussed in the
>> linux-fsdevel@vger.kernel.org mailing list, which I am CCing.
>

First, talking about the interface.

> One option that was discussed for btrfs was to use the first fe_reserved
> field for the FIEMAP ioctl struct fiemap_extent to fe_phys_length to
> hold the compressed size of each extent in the file.
>

I don't think fiemap is a good interface for UBIFS and for compressed
size reporting feature.

The contents of files in UBIFS are sorted into *ubifs_data_node* which
contains as much as UBIFS_BLOCK_SIZE(4096) bytes of data. A single file
may contain lots of data nodes and these data nodes may locate on flash
in order or in disorder because of out of place update.

An fiemap ioctl from userspace need lots of memory to store discontinuous
data mapping and copy these fiemap_extent may cost a lot of time.

> http://lwn.net/Articles/607552/
> http://thread.gmane.org/gmane.comp.file-systems.btrfs/37312
> 
> I'm not sure what happened to that patch series - I was looking forward
> to it landing, and it was in very good shape I think.
> 

Since the *fe_phys_length* of fiemap_extent is not import to mainline,
current fiemap can only report logical data length of an extent. Regret
to say, it's no use for getting the compressed size of a file in UBIFS.


Then, looking at the implement.

>>
>> a. 'struct ubifs_ino_node' has unused space, use it to add the
>> compressed size field.
>> b. maintain this field
>> c. this field will only be correct for the part of the file which are on
>> the media. The dirty data in the page cache has not yet been compressed,
>> so we do not know its compressed size yet.
>> e. when user asks for the compressed size, you have to sync the inode
>> first, in order to make sure the compressed size is correct.
>>
>> And the implementation should be backward-compatible. That is, if new
>> driver mounts the old media, you return something predicatable. I guess
>> uncompressed size could be it.
>>

I'm worry about power cut recovery of compressed size if we introduce it
into 'struct ubifs_ino_node'. We can't write a new metadata node after each
changing of data node. Dirty data may not change the logical size of a file,
but it must change the compressed size. How to keep the consistency between
real compressed size(amount of each data nodes) and the record in metadata
node?

In logic size case, we could solve this problem by block number, because the
size of each blocks are UBIFS_BLOCK_SIZE, each exist data node could tell
the logic size of a file. Actually we use this method to fix the logic size
of a file in recovery path. Since the physic size of each data node are not
equal, we couldn't get the physic size of a file by a single data node in
journal. And we couldn't record the total compressed size of a file in data
node because it doesn't have enough reserve space, we couldn't use the same
functionality for compressed size.

Since metadata node(ubifs_ino_node) and data node(ubifs_data_node) are stored
in different journals, I didn't find a easy way to keep consistency when a
power cut happen. Seems a rebuilding whole scan can not be avoid in this case.


So could we use a simply method? just a private ioctl which scan the tnc tree
and report the compressed size of a UBIFS file? I found an old patch for Btrfs,
but it is not import to mainline.

https://patchwork.kernel.org/patch/117782/

Since the files in UBIFS are not too large, maybe we could test if the cost of
time is acceptable for ordinary use case.

Further, for both fiemap or this private ioctl method, current tnc tree lookup
mechanism seems always copying the whole ubifs node, but only the header of a
node is used in this case. Do we have a way to only read part of a node from
tnc tree?

Thanks,
Hu

Re: UBIFS: Is it possible to get the compressed size of a file?

[David Sterba]

On Sat, Feb 28, 2015 at 03:31:14PM +0800, hujianyang wrote:
> > http://lwn.net/Articles/607552/
> > http://thread.gmane.org/gmane.comp.file-systems.btrfs/37312
> > 
> > I'm not sure what happened to that patch series - I was looking forward
> > to it landing, and it was in very good shape I think.

> Since the *fe_phys_length* of fiemap_extent is not import to mainline,
> current fiemap can only report logical data length of an extent. Regret
> to say, it's no use for getting the compressed size of a file in UBIFS.

Yeah, not the best example how a patchset should be pushed. I'll try to
get back soon to it given the almost-ready state and another potential
user of the interface.

> So could we use a simply method? just a private ioctl which scan the tnc tree
> and report the compressed size of a UBIFS file? I found an old patch for Btrfs,
> but it is not import to mainline.
> 
> https://patchwork.kernel.org/patch/117782/

This was at the beginning of the fiemap extensions. I've picked that
patch, updated and sent

http://www.mail-archive.com/linux-btrfs%40vger.kernel.org/msg14371.html

but it was not merged. We've discussed that and came to conclusion that
a special ioctl was not the right approach when we could extend the
existing interfaces. The ioctl was iterating over all extents and just
summing some numbers, effectively repeating the core code of fiemap.

Though you're (relatively) free to introduce private ioctls for your
filesystem, I'd rather not recommend that. Once an ioctl is introduced,
you have to support it forever, the deprecation periods span several
years.

> Further, for both fiemap or this private ioctl method, current tnc tree lookup
> mechanism seems always copying the whole ubifs node, but only the header of a
> node is used in this case. Do we have a way to only read part of a node from
> tnc tree?

I don't understand the details, but this seems that even the private
ioctl would help too much even if you'd be more free to do some
performance optimizations that would not fit fiemap.