Actual usage of files in ubifs

Actual usage of files in ubifs

[Ricard Wanderlof]

Is there a (simple) way to find out exactly how much space each specific 
file uses in an ubifs fils system?

I looked into running mkfs.ubifs with some form of option to give me the 
info, the closest I got was -g 3 which output a lot of sizes but it was 
not obvious exactly what they meant.

I suppose it would be fully possible to add code to mkfs.ubifs to output 
this information, as it surely must be there at some point, since it 
selects the compression algorithm on a file per file basis, debending on 
the amount of compression acheived for each algorithm.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Actual usage of files in ubifs

[Richard Weinberger]

On Mon, Sep 11, 2017 at 11:47 AM, Ricard Wanderlof
<ricard.wanderlof@axis.com> wrote:
>
> Is there a (simple) way to find out exactly how much space each specific
> file uses in an ubifs fils system?
>
> I looked into running mkfs.ubifs with some form of option to give me the
> info, the closest I got was -g 3 which output a lot of sizes but it was
> not obvious exactly what they meant.
>
> I suppose it would be fully possible to add code to mkfs.ubifs to output
> this information, as it surely must be there at some point, since it
> selects the compression algorithm on a file per file basis, debending on
> the amount of compression acheived for each algorithm.

Well, how exactly should it be?
I guess you want more than the compressed size on flash.
Do you also want to know how much bytes a file occupies in the index
tree and the inode itself?
This is non-trivial.

-- 
Thanks,
//richard

Re: Actual usage of files in ubifs

[Ricard Wanderlof]

On Mon, 11 Sep 2017, Richard Weinberger wrote:

> On Mon, Sep 11, 2017 at 11:47 AM, Ricard Wanderlof
> <ricard.wanderlof@axis.com> wrote:
> >
> > Is there a (simple) way to find out exactly how much space each specific
> > file uses in an ubifs fils system?
> >
> > I looked into running mkfs.ubifs with some form of option to give me the
> > info, the closest I got was -g 3 which output a lot of sizes but it was
> > not obvious exactly what they meant.
> >
> > I suppose it would be fully possible to add code to mkfs.ubifs to output
> > this information, as it surely must be there at some point, since it
> > selects the compression algorithm on a file per file basis, debending on
> > the amount of compression acheived for each algorithm.
> 
> Well, how exactly should it be?
> I guess you want more than the compressed size on flash.
> Do you also want to know how much bytes a file occupies in the index
> tree and the inode itself?
> This is non-trivial.

Good point. I was solely thinking along the lines of how much space the 
actual file occupied, not considering metadata. That would be a good 
starting point. I'm guessing that for moderate file sizes the metadata 
would be relatively small compared to the file itself?

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Actual usage of files in ubifs

[Artem Bityutskiy]

On Mon, 2017-09-11 at 17:18 +0200, Ricard Wanderlof wrote:
> On Mon, 11 Sep 2017, Richard Weinberger wrote:
> 
> > On Mon, Sep 11, 2017 at 11:47 AM, Ricard Wanderlof
> > <ricard.wanderlof@axis.com> wrote:
> > > 
> > > Is there a (simple) way to find out exactly how much space each
> > > specific
> > > file uses in an ubifs fils system?
> > > 
> > > I looked into running mkfs.ubifs with some form of option to give
> > > me the
> > > info, the closest I got was -g 3 which output a lot of sizes but
> > > it was
> > > not obvious exactly what they meant.
> > > 
> > > I suppose it would be fully possible to add code to mkfs.ubifs to
> > > output
> > > this information, as it surely must be there at some point, since
> > > it
> > > selects the compression algorithm on a file per file basis,
> > > debending on
> > > the amount of compression acheived for each algorithm.
> > 
> > Well, how exactly should it be?
> > I guess you want more than the compressed size on flash.
> > Do you also want to know how much bytes a file occupies in the
> > index
> > tree and the inode itself?
> > This is non-trivial.
> 
> Good point. I was solely thinking along the lines of how much space
> the 
> actual file occupied, not considering metadata. That would be a good 
> starting point. I'm guessing that for moderate file sizes the
> metadata 
> would be relatively small compared to the file itself?

I would think a "slow" version of this would not be that hard to
implement - walk the index and sum up node sizes. Subtract header sizes
if you do not want metadata.

I am not sure what would be the API? Do other FSes implement something
like this?

Re: Actual usage of files in ubifs

[Richard Weinberger]

Am Montag, 11. September 2017, 17:44:09 CEST schrieb Artem Bityutskiy:
> > Good point. I was solely thinking along the lines of how much space
> > the
> > actual file occupied, not considering metadata. That would be a good
> > starting point. I'm guessing that for moderate file sizes the
> > metadata
> > would be relatively small compared to the file itself?
> 
> I would think a "slow" version of this would not be that hard to
> implement - walk the index and sum up node sizes. Subtract header sizes
> if you do not want metadata.
> 
> I am not sure what would be the API? Do other FSes implement something
> like this?

I think a "show MTD usage by inode" should be implementable via debugfs.
Maybe, after a discussion on linux-fsdevel a per-file ioctl().

But first I'd like to know more about the use-case and where to draw the 
border.
e.g. If a file as xattrs, do you also account them? UBIFS modules xattrs
via inodes. So, they have a rather huge space overhead.

Thanks,
//richard

-- 
sigma star gmbh - Eduard-Bodem-Gasse 6 - 6020 Innsbruck - Austria
ATU66964118 - FN 374287y

Re: Actual usage of files in ubifs

[Ricard Wanderlof]

On Mon, 11 Sep 2017, Richard Weinberger wrote:

> Am Montag, 11. September 2017, 17:44:09 CEST schrieb Artem Bityutskiy:
> > > Good point. I was solely thinking along the lines of how much space
> > > the
> > > actual file occupied, not considering metadata. That would be a good
> > > starting point. I'm guessing that for moderate file sizes the
> > > metadata
> > > would be relatively small compared to the file itself?
> > 
> > I would think a "slow" version of this would not be that hard to
> > implement - walk the index and sum up node sizes. Subtract header sizes
> > if you do not want metadata.
> > 
> > I am not sure what would be the API? Do other FSes implement something
> > like this?
> 
> I think a "show MTD usage by inode" should be implementable via debugfs.
> Maybe, after a discussion on linux-fsdevel a per-file ioctl().
> 
> But first I'd like to know more about the use-case and where to draw the 
> border. e.g. If a file as xattrs, do you also account them? UBIFS 
> modules xattrs via inodes. So, they have a rather huge space overhead.

In our specific situation the use case is basically that given a specific 
set files (in the root file system), how much flash is actually needed, 
also considering that different types of files have different compression 
ratios depending on the content. I.e. a large text file might not hurt as 
much as a large binary since the former compresses better.

So primarily it is the total amount needed for the whole file system, but 
it quickly comes down how much individual files consume. In our case it's 
not necessary to do the operation on a running file system, but in the 
more general case that might be more useful.

I would think that this is something that is known somewhere inside 
mkfs.ubifs, or is the information too convoluted to be easily extractable?

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Actual usage of files in ubifs

[Richard Weinberger]

Ricard,

Am Dienstag, 12. September 2017, 11:15:25 CEST schrieb Ricard Wanderlof:
> On Mon, 11 Sep 2017, Richard Weinberger wrote:
> > Am Montag, 11. September 2017, 17:44:09 CEST schrieb Artem Bityutskiy:
> > > > Good point. I was solely thinking along the lines of how much space
> > > > the
> > > > actual file occupied, not considering metadata. That would be a good
> > > > starting point. I'm guessing that for moderate file sizes the
> > > > metadata
> > > > would be relatively small compared to the file itself?
> > > 
> > > I would think a "slow" version of this would not be that hard to
> > > implement - walk the index and sum up node sizes. Subtract header sizes
> > > if you do not want metadata.
> > > 
> > > I am not sure what would be the API? Do other FSes implement something
> > > like this?
> > 
> > I think a "show MTD usage by inode" should be implementable via debugfs.
> > Maybe, after a discussion on linux-fsdevel a per-file ioctl().
> > 
> > But first I'd like to know more about the use-case and where to draw the
> > border. e.g. If a file as xattrs, do you also account them? UBIFS
> > modules xattrs via inodes. So, they have a rather huge space overhead.
> 
> In our specific situation the use case is basically that given a specific
> set files (in the root file system), how much flash is actually needed,
> also considering that different types of files have different compression
> ratios depending on the content. I.e. a large text file might not hurt as
> much as a large binary since the former compresses better.
> 
> So primarily it is the total amount needed for the whole file system, but
> it quickly comes down how much individual files consume. In our case it's
> not necessary to do the operation on a running file system, but in the
> more general case that might be more useful.
> 
> I would think that this is something that is known somewhere inside
> mkfs.ubifs, or is the information too convoluted to be easily extractable?

Adding this to mkfs.ubifs shouldn't be a big deal.
And I agree your use case makes sense.

Do you want to send me a patch? :)

Thanks,
//richard

-- 
sigma star gmbh - Eduard-Bodem-Gasse 6 - 6020 Innsbruck - Austria
ATU66964118 - FN 374287y

Re: Actual usage of files in ubifs

[Ricard Wanderlof]

On Tue, 12 Sep 2017, Richard Weinberger wrote:

> > > But first I'd like to know more about the use-case and where to draw the
> > > border. e.g. If a file as xattrs, do you also account them? UBIFS
> > > modules xattrs via inodes. So, they have a rather huge space overhead.
> > 
> > In our specific situation the use case is basically that given a specific
> > set files (in the root file system), how much flash is actually needed,
> > also considering that different types of files have different compression
> > ratios depending on the content. I.e. a large text file might not hurt as
> > much as a large binary since the former compresses better.
> > 
> > So primarily it is the total amount needed for the whole file system, but
> > it quickly comes down how much individual files consume. In our case it's
> > not necessary to do the operation on a running file system, but in the
> > more general case that might be more useful.
> > 
> > I would think that this is something that is known somewhere inside
> > mkfs.ubifs, or is the information too convoluted to be easily extractable?
> 
> Adding this to mkfs.ubifs shouldn't be a big deal.
> And I agree your use case makes sense.
> 
> Do you want to send me a patch? :)

Yes, that would be very much appreciated!

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Actual usage of files in ubifs

[Richard Weinberger]

Am Samstag, 16. September 2017, 00:08:53 CEST schrieb Ricard Wanderlof:
> > Do you want to send me a patch? :)
> 
> Yes, that would be very much appreciated!

Uhh, sorry. Mad bad English tricked you.
I meant whether you want to create a patch and send it to me. :-)

Thanks,
//richard

-- 
sigma star gmbh - Eduard-Bodem-Gasse 6 - 6020 Innsbruck - Austria
ATU66964118 - FN 374287y

Re: Actual usage of files in ubifs

[Ricard Wanderlof]

On Sat, 16 Sep 2017, Richard Weinberger wrote:

> Am Samstag, 16. September 2017, 00:08:53 CEST schrieb Ricard Wanderlof:
> > > Do you want to send me a patch? :)
> > 
> > Yes, that would be very much appreciated!
> 
> Uhh, sorry. Mad bad English tricked you.
> I meant whether you want to create a patch and send it to me. :-)

Sorry, my mistake, it's not your English at all, it's completely fine, 
it's rather that I shouldn't be responding to mtd list posts on a Saturday 
morning when I'm not really awake and not reading clearly. :-)

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30