Re: Problem Reading from XenStore in DomU

[Dagaen Golomb <dgolomb@seas.upenn.edu>]

>> Hi All,
>>
>> I'm having an interesting issue. I am working on a project that
>> requires me to share memory between dom0 and domUs. I have this
>> successfully working using the grant table and the XenStore to
>> communicate grefs.
>>
>> My issue is this. I have one domU running Ubuntu 12.04 with a default
>> 3.8.x kernel that has no issue reading or writing from the XenStore.
>> My work also requires some kernel modifications, and we have made
>> these changes in the 4.1.0 kernel. In particular, we've only added a
>> simple hypercall. This modified kernel is what dom0 is running, on top
>> of Xen 4.7 rc1.
>>
>> The guest I mentioned before with the default kernel can still read
>> and write the XenStore just fine when on Xen 4.7 rc1 and with dom0
>> running our kernel.
>>
>> The issue I'm having is with another newly created guest (i.e., it is
>> not a copy of the working one, this is because I needed more space and
>> couldn't expand the original disk image). It is also running Ubuntu
>> 12.04 but has been upgraded to our modified kernel. On this guest I
>> can write to the XenStore, and see that the writes were indeed
>> successful using xenstore-ls in dom0. However, when this same guest
>> attempts to read from the XenStore, it doesn't return an error code
>> but instead just blocks indefinitely. I've waiting many minutes to
>> make sure its not just blocking for a while, it appears like it will
>> block forever. The block is happening when I start the transaction.
>> I've also tried not using a transaction, in which case it blocks on
>> the read itself.
>>
>> I have an inkling this may be something as simple as a configuration
>> issue, but I can't seem to find anything. Also, the fact that writes
>> work fine but reads do not is perplexing me.
>>
>> Any help would be appreciated!
>
> Nothing should block like this.  Without seeing your patch, I can't
> comment as to whether you have accidentally broken things.

I don't see any way the patch could be causing this. It simply adds
another function and case clause to an already-existing hypercall, and
when you call the hypercall with that option it returns the current
budget of a passed-in vcpu. It doesn't even come close to touching
grant mechanics, and doesn't modify any state - it simply returns a
value that previously was hidden in the kernel.

> Other avenues of investigation are to look at what the xenstored process
> is doing in dom0 (is it idle? or is it spinning?), and to look in the
> xenstored log file to see if anything suspicious occurs.

I tried booting into older, stock kernels. They all work with the
read. However, I do not see why the kernel modification would be the
issue as described above. I also have the dom0 running this kernel and
it reads and writes the XenStore just dandy. Are there any kernel
config issues that could do this? This is very confusing. I have one
instance of it working with reads so it can be inherent to the kernel
build itself...

Xenstore (or any grep-ed "xen" process for that matter) are acting
fine. No changes in memory or CPU when blocking vs before blocking.
xenstored-access.log is this:

[20160515T19:45:42.207Z]  A992         newconn
[20160515T19:45:42.210Z]  A992         endconn
[20160515T19:45:42.223Z]  A993         newconn
[20160515T19:45:42.223Z]  A993         write     /local/domain/15/mbroe \017\001
[20160515T19:45:42.223Z]  A993         endconn
[20160515T19:45:42.232Z]  A994         newconn
[20160515T19:45:42.232Z]  A994.1       setperms  /local/domain/15/mbroe n0 b15
[20160515T19:45:42.233Z]  A994.1       setperms
/local/domain/15/mbroe/gref n0 b15
[20160515T19:45:42.233Z]  A994.1       setperms
/local/domain/15/mbroe/min_budget n0 b15
[20160515T19:45:42.234Z]  A994.1       commit
[20160515T19:45:42.234Z]  A994         endconn
[20160515T19:46:25.215Z]  A995         newconn
[20160515T19:46:25.215Z]  A995         endconn
[20160515T19:46:29.540Z]  A996         newconn
[20160515T19:46:29.541Z]  A996         watch
/local/domain/15/mbroe/max_block max_block
[20160515T19:46:29.541Z]  A996         w event
/local/domain/15/mbroe/max_block max_block
[20160515T19:46:29.542Z]  A996.1       commit
[20160515T19:46:29.543Z]  A996         watch
/local/domain/15/mbroe/num_blocks num_blocks
[20160515T19:46:29.543Z]  A996         w event
/local/domain/15/mbroe/num_blocks num_blocks
[20160515T19:46:29.544Z]  A996.2       commit
[20160515T19:46:29.545Z]  A996.3       write     /local/domain/15/mbroe/gref 8
[20160515T19:46:29.546Z]  A996.3       commit
[20160515T19:46:29.546Z]  A996.4       write
/local/domain/15/mbroe/min_budget 10
[20160515T19:46:29.547Z]  A996.4       commit
[20160515T19:46:29.548Z]  A996         endconn

** Above is dom0, running our kernel, doing both read and write just
fine, and setting permission for dom15. **

[20160515T19:47:44.562Z]  D15          watch
/local/domain/15/mbroe/gref FFFF88007B2F3390
[20160515T19:47:44.562Z]  D15          w event
/local/domain/15/mbroe/gref FFFF88007B2F3390

** Above is shown while dom15 is trying to read, but blocking **

[20160515T19:48:39.392Z]  D15          unwatch
/local/domain/15/mbroe/gref FFFF88007B2F3390

** This line is after terminating dom15 program while its still waiting.

The only issue I could see is the FFFF88007B2F3390. This is supposed
to be the key name a presume (such as gref). Maybe its an issue with
the compiled binary, and it ends up watching on a key that doesn't
exist (nor ever will). I will look into this as it looks promising!
Thanks!

I'm don't see any other logs for xenstore, if there are more I please
point them to me. xenstored.log in same directory is recognized as
binary fine, and when I open anyways all I see is "Xen Storage Daemon,
version 1.0" repeatedly.

Regards,
Dagaen Golomb
Ph.D. Student, University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel