Re: [Qemu-devel] [QEMU : VVFAT] vvfat.c - help required for understanding/modification

Re: [Qemu-devel] [QEMU : VVFAT] vvfat.c - help required for understanding/modification

[Pintu Kumar]

Hello Mr. Johannes, Kevin

I already did some work for scanning only top level directory in vvfat.
Using the following logic in read_directory()

if(parent_index >= 0 & (!dot & !dotdot))
{
       free(buffer);
       break;
}

Hope this is correct logic for skipping sub-directories content by
just scanning only dot and dotdot entries inside it.

But as per the recent analysis it was observed that there is some
problem in skipping sub-directory scanning itself.
Because when I issue "df" command on VVFAT partition I get
read_cluster errors during these sector location.

Mr. Johannes, can you clarify on this first.

Then we will look into the dynamic mapping part later.


Thank you very much.


Thanks, Regards,
Pintu


On Thu, Oct 20, 2011 at 1:43 PM, Kevin Wolf <kwolf@redhat.com> wrote:
> Am 19.10.2011 20:34, schrieb Pintu Kumar:
>> Dear Mr. Johannes,
>>
>> Thank you very much for your reply.
>>
>> So according to you implementing dynamic mapping logic after mounting
>> in VVFAT is possible.
>>
>> But from your following mail, I could not understand how to do that.
>>
>> Can you explain me with reference from the vvfat.c code.
>>
>> What should I do to scan only top level files/folders in read_directory?
>>
>> And what should I need to do in vvfatd_process_req (case : READ) to
>> create dynamic mapping on the fly before displaying the content on
>> guest OS.
>>
>> Right now I very this on my Linux PC it using "ls" command.
>>
>> Example:
>> If my vvfat mount path is /mnt/movifs and it contains one
>> sub-directory say "test1"
>> Then I use "ls -l /mnt/movifs/test1" in which case vvfatd_process_req
>> is invoked and then vvfat_read is invoked.
>>
>>
>> Please let me know.
>
> You asked in a second mail that you sent privately that I explain to you
> how to implement this, as you didn't fully understand what Johannes
> wrote. I'm going to add my reply here so that we don't have several
> separate mail threads. Actually, adding the mailing list wouldn't hurt
> either, but that's your decision.
>
> There's really one main thing that I would recommend you to finally do:
> Take a look at how FAT works. Completely independent of vvfat. And then
> think about how to implement something like this. And only then go back
> to vvfat and do it there. I won't do that for you, even if you double
> the length of the email thread.
>
>>> ---------- Forwarded message ----------
>>> From: Johannes Schindelin <Johannes.Schindelin@gmx.de>
>>> Date: Wed, Oct 19, 2011 at 1:37 AM
>>> Subject: Re: [QEMU : VVFAT] vvfat.c - help required for
>>> understanding/modification
>>> To: Pintu Kumar <pintu.ping@gmail.com>
>>>
>>>
>>> Hi,
>>>
>>> On Sun, 16 Oct 2011, Pintu Kumar wrote:
>>>
>>>> In short, we wanted to implement partial scanning in the beginning and
>>>> dynamic mappings for VVFAT during runtime and after VVFAT is mounted.
>>>> That is, to scan only top level files and directories (skip
>>>> sub-directories content) during vvfat_prepare() and mount it.
>>>>
>>>> Later we a particular sub-directory is accessed by the guest OS, we
>>>> wanted to create its mapping on the fly.
>>>
>>> That sounds doable, since you can initialize the respective blocks in the
>>> FAT lazily, when they are accessed.
>
> I think it's possible in a very limited way. The nasty thing is that you
> don't really have the notion of directories when the guest accesses the
> file system. You have accesses to sectors that contain the FAT.
>
> Once the guest has read in a sector of the FAT, you must consider this
> sector final. You can't later allocate new clusters in this sector of
> the FAT because in general the guest OS will have cached the sector and
> won't reread it.
>
> So the real problem is to manage which clusters already have been read;
> which clusters you already referenced, but are still open for additions;
> and which files/subdirectories still need to be integrated somewhere.
> The guest can read the sectors of the FAT in any order, and it could
> read sectors that you haven't referenced yet at all (because, who reads
> single sectors?), which should add a bit to the fun.
>
>>>> I did some work on it but facing some problem. The sub-directory
>>>> contents are not visible after dynamic mapping and updating the FAT
>>>> content.
>>>>
>>>> I wanted to know your opinion if such kind of logic is possible to
>>>> implement in VVFAT.
>>>
>>> I think the basic problem is to know when to update the FAT content. As I
>>> alluded to above, I would do it thusly: Initialize the top-level directory
>>> and remember the blocks you want to use for the subdirectories. Whenever
>>> the corresponding blocks' contents are requested, initialize them, again
>>> ear-marking blocks for the subdirectories.
>>>
>>> But maybe I misunderstood?
>
> It really boils down to this question: Do we have enough space in the
> FAT for leaving most of it sparse? It looks like we're talking about a
> rather big directory tree, otherwise reading it in on startup wouldn't
> be a problem.
>
> And of course you're wasting precious disk space: If I understand the
> code right we only have 504 MB for the whole file system (should still
> be possible to increase a bit, but at some point you'll need FAT32). The
> available space will be reduced by same percentage as FAT entries stay
> unused.
>
> Kevin
>

Re: [Qemu-devel] [QEMU : VVFAT] vvfat.c - help required for understanding/modification

[Johannes Schindelin]

Hi Pintu,

On Sat, 22 Oct 2011, Pintu Kumar wrote:

> I already did some work for scanning only top level directory in vvfat. 
> Using the following logic in read_directory()
> 
> if(parent_index >= 0 & (!dot & !dotdot))
> {
>        free(buffer);
>        break;
> }

Sorry, this is way too deep in the code (and I'd have to guess because you 
did not grace me with exact locations), I cannot afford to dive into it 
that far. But what I can say is that it is not all that easy; you'll need 
to refactor the code.

You _will_ need to reserve sectors for future directory parsing. You 
_will_ need to add code that intercepts reads to those sectors. You _will_ 
have to read the directories in question when those sectors are accessed 
and thereby commit the sectors to a final state.

BTW both Kevin and I said that before.

> Hope this is correct logic for skipping sub-directories content by just 
> scanning only dot and dotdot entries inside it.

It would be, if all you want are empty directories.

> But as per the recent analysis it was observed that there is some 
> problem in skipping sub-directory scanning itself. Because when I issue 
> "df" command on VVFAT partition I get read_cluster errors during these 
> sector location.

When you issue "df", you need to commit to which sectors are reserved. By 
that, you take almost all the advantage of lazy directory parsing away.

> Mr. Johannes, can you clarify on this first.
> 
> Then we will look into the dynamic mapping part later.

I am very sorry, but I do get the impression that the intent is to lure me 
into doing all the work, including the investigation what it would take to 
turn the code into a version that reads just the top-level first and only 
later the subdirectories.

This will not happen. I have enough projects that I want to tackle myself, 
and I am extremely unwilling to do the work for somebody else when that 
somebody else is paid to do the work. For the things I _want_ to do I 
already have too little time, given that my day job is quite demanding and 
asks software design and architecture of me already.

So: you _will_ have to acquaint yourself with the FAT and then the VFAT 
specification. You _will_ have to make a proper plan how the client may be 
fooled into believing that there is a fixed filesystem when there is 
actually just directories of files. And you _will_ have to write that code 
yourself.

I hope this clarified it?

Ciao,
Johannes

Re: [Qemu-devel] [QEMU : VVFAT] vvfat.c - help required for understanding/modification

[Pintu Kumar]

Dear Mr. Johannes,

I am sorry but I think you took me wrong. I never asked you to do things for me.
I just wanted few clarifications to proceed further as I was stuck
after performing few experiments as below:

For scanning only top-level directory I used the following approach.
You can refer to this link for the vvfat original code:
http://www.netmite.com/android/mydroid/external/qemu/block-vvfat.c

read_directory(.....)
{
         ---
        while((entry=readdir(dir)))
        {
               ---
               direntry=create_short_and_long_name(s, i,
entry->d_name, is_dot || is_dotdot);

               /* skip sub-directory here */
               if(!is_dot && !is_dotdot && (S_ISDIR(st.st_mode)))
               {
                       free(buffer);
                       s->current_mapping = NULL;
                       break;
               }
               -----
        }
        ----
        -----
}

If I use the above logic I could able to scan only top-level files
with empty sub-directories.
As follows:
# ls -la /mnt/vvfatfs
.
..
File1.txt
File2.txt
test1

But there is one problem here if I use the above logic.
When I issue "ls" command "vvfat_read" is not getting triggered.
So I think there is some problem and I could figure out where to
implement the dynamic scanning of sub-directory later.

But in the above logic if I scan at least the "dot" and "dotdot"
entries for the sub-directories I could get the trigger point in
vvfat_read during "ls" command.
That is if the output of "ls" is as follows:
# ls -la /mnt/vvfatfs
.  <dot for vvfatfs>
.. <dotdot fot vvfatfs>
File1.txt
File2.txt
test1
   .   <dot for test1>
   ..  <dotdot for test1>

I am not sure why this is happening. So I need your help in understanding.

Similarly I implemented dynamic creation of mapping for
sub-directories in "vvfat_read" by remembering the previous mappings
for the sub-directories.
And after that I could able to see to get the modified FAT content.
But the content itself is not visible (except the dot and dotdot
entries) for the sub-directory when we use "ls" command to see them.
I will talk about in detail later once you could confirm the
top-directory only scanning method.

Other comments are in-lined in your mail below.


Thanks,
Pintu


On Sun, Oct 23, 2011 at 12:28 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi Pintu,
>
> On Sat, 22 Oct 2011, Pintu Kumar wrote:
>
>> I already did some work for scanning only top level directory in vvfat.
>> Using the following logic in read_directory()
>>
>> if(parent_index >= 0 & (!dot & !dotdot))
>> {
>>        free(buffer);
>>        break;
>> }
>
> Sorry, this is way too deep in the code (and I'd have to guess because you
> did not grace me with exact locations), I cannot afford to dive into it
> that far. But what I can say is that it is not all that easy; you'll need
> to refactor the code.
>
> You _will_ need to reserve sectors for future directory parsing. You
> _will_ need to add code that intercepts reads to those sectors. You _will_
> have to read the directories in question when those sectors are accessed
> and thereby commit the sectors to a final state.
>
> BTW both Kevin and I said that before.
>
>> Hope this is correct logic for skipping sub-directories content by just
>> scanning only dot and dotdot entries inside it.
>
> It would be, if all you want are empty directories.
>
>> But as per the recent analysis it was observed that there is some
>> problem in skipping sub-directory scanning itself. Because when I issue
>> "df" command on VVFAT partition I get read_cluster errors during these
>> sector location.
>
> When you issue "df", you need to commit to which sectors are reserved. By
> that, you take almost all the advantage of lazy directory parsing away.
>


Actually, after using "df" command on the original code itself, it
give read cluster error.
So, there is something messy here with the original code itself.



>> Mr. Johannes, can you clarify on this first.
>>
>> Then we will look into the dynamic mapping part later.
>
> I am very sorry, but I do get the impression that the intent is to lure me
> into doing all the work, including the investigation what it would take to
> turn the code into a version that reads just the top-level first and only
> later the subdirectories.
>
> This will not happen. I have enough projects that I want to tackle myself,
> and I am extremely unwilling to do the work for somebody else when that
> somebody else is paid to do the work. For the things I _want_ to do I
> already have too little time, given that my day job is quite demanding and
> asks software design and architecture of me already.
>


I think you took me wrong. I am sorry but I never asked you to do it for me.
I just wanted to get some clarification quickly since you are the
author and who else can best explain to me except you.
Right now I myself is not fully convinced if some thing like this is
completely possible and feasible.
I need to perform various experiments and dont want to leave even a
single chance before reaching the conclusion that it is not possible.



> So: you _will_ have to acquaint yourself with the FAT and then the VFAT
> specification. You _will_ have to make a proper plan how the client may be
> fooled into believing that there is a fixed filesystem when there is
> actually just directories of files. And you _will_ have to write that code
> yourself.
>



Yes, I have already tried implementing it myself and I got some amount
of success.
But the moment I am stuck, I came to the mailing list for some
clarifications and help.



> I hope this clarified it?
>
> Ciao,
> Johannes
>

Re: [Qemu-devel] [QEMU : VVFAT] vvfat.c - help required for understanding/modification

[Kevin Wolf]

Am 26.10.2011 21:17, schrieb Pintu Kumar:
> Dear Mr. Johannes,
> 
> I am sorry but I think you took me wrong. I never asked you to do things for me.
> I just wanted few clarifications to proceed further as I was stuck
> after performing few experiments as below:

You are stuck because you still didn't think about the theory before
jumping to the code. Please do this before asking more questions. Both
Johannes and I have told you that it's not as easy as you seem to think.

When you have a design to solve the problem (and I believe it might be
better to start that from scratch rather than extending vvfat as it
would end up being a rewrite anyway), we can discuss that design. But it
doesn't make any sense to discuss detailed changes in vvfat when you
don't even seem to understand the problem.

> But there is one problem here if I use the above logic.
> When I issue "ls" command "vvfat_read" is not getting triggered.
> So I think there is some problem and I could figure out where to
> implement the dynamic scanning of sub-directory later.

Why do you expect that vvfat_read is called? If the guest OS has the
directory entries already cached, there's no reason for it to read them
from disk.

Kevin

Re: [Qemu-devel] [QEMU : VVFAT] vvfat.c - help required for understanding/modification

[Johannes Schindelin]

Hi Kevin,

On Thu, 27 Oct 2011, Kevin Wolf wrote:

> When you have a design to solve the problem (and I believe it might be
> better to start that from scratch rather than extending vvfat as it
> would end up being a rewrite anyway), we can discuss that design.

I fully agree on all three accounts. VVFAT never was intended as anything 
more than a hack, and it was never intended for large directories/tree 
structures. And one fundamental problem cannot be overcome: when the host 
changes the underlying files.

Ciao,
Johannes