Need help figuring out why my windows pv drivers wont work with a 32 bit dom0...

Need help figuring out why my windows pv drivers wont work with a 32 bit dom0...

[James Harper]

All my development up until now on the Windows PV drivers has been on an
AMD64 system with a 64 bit hypervisor, a 64 bit Dom0, and 32 or 64 bit
Windows DomU's, and that has all worked fine for me.

I've been getting reports of problems from other people, and initially
assumed (incorrectly I think) that it was a problem with Intel, but as
it turns out it appears to be a problem when Dom0 is 32 bit. When I test
with a 64 bit hypervisor and a 32 bit Dom0, the block device drivers
don't work - the first request gives a failure, and a few requests later
I get a crash. The crash is most likely because I may not be handling
failures correctly.

Anyway, I can't seem to figure out why things are breaking under vbd
when Dom0 is 32 bit. Vif works perfectly. Before some very recent
testing, I wasn't even setting the 'protocol' frontend xenstore
parameter, but I have since tried setting it to XEN_IO_PROTO_ABI_NATIVE,
XEN_IO_PROTO_ABI_X86_64 and XEN_IO_PROTO_ABI_X86_32, without any
success.

At around the time of the error, Dom0 logs "(XEN) grant_table.c:264:d0
Bad flags (0) or dom (0). (expected dom 0)", which I assume is probably
related.

Can anyone suggest what is different about the backend that I might not
be taking into account? My thoughts so far are:
. Windows is padding structures differently under 32 and 64 bits.
Strange that everything else works fine though...
. The pfn I get from MmGetMdlPfnArray (windows kernel api) isn't valid
to a 32 bit Dom0 (this function is used identically in the vif driver
though)

Any and all suggestions will be greatly appreciated!

Thanks

James

RE: Need help figuring out why my windows pv drivers wontwork with a 32 bit dom0...

[James Harper]

> . Windows is padding structures differently under 32 and 64 bits.
> Strange that everything else works fine though...

I think I may be onto something with this...

32 bit Linux (using gcc and a quick .c file):
sizeof(uint8_t) = 1
sizeof(uint16_t) = 2
sizeof(uint32_t) = 4
sizeof(uint64_t) = 8
sizeof(struct blkif_request) = 108
sizeof(struct blkif_request_segment) = 8
sizeof(struct blkif_response) = 12

64 bit Linux (using gcc and a quick .c file):
sizeof(uint8_t) = 1
sizeof(uint16_t) = 2
sizeof(uint32_t) = 4
sizeof(uint64_t) = 8
sizeof(struct blkif_request) = 112
sizeof(struct blkif_request_segment) = 8
sizeof(struct blkif_response) = 16

32 bit Windows (using Microsoft DDK Compiler and debug statements in the
driver)
sizeof(uint8_t) = 1
sizeof(uint16_t) = 2
sizeof(uint32_t) = 4
sizeof(uint64_t) = 8
sizeof(struct blkif_request) = 112
sizeof(struct blkif_request_segment) = 8
sizeof(struct blkif_response) = 16

Haven't tried this test on 64 bit windows yet, but if 32 bit windows and
32 bit Linux align their structures differently, I can imagine that
problems are going to arise...

James

RE: Need help figuring out why my windows pv driverswontwork with a 32 bit dom0...

[James Harper]

> > . Windows is padding structures differently under 32 and 64 bits.
> > Strange that everything else works fine though...
> 
> I think I may be onto something with this...
> 
> 32 bit Linux (using gcc and a quick .c file):
> sizeof(uint8_t) = 1
> sizeof(uint16_t) = 2
> sizeof(uint32_t) = 4
> sizeof(uint64_t) = 8
> sizeof(struct blkif_request) = 108
> sizeof(struct blkif_request_segment) = 8
> sizeof(struct blkif_response) = 12
> 
> 64 bit Linux (using gcc and a quick .c file):
> sizeof(uint8_t) = 1
> sizeof(uint16_t) = 2
> sizeof(uint32_t) = 4
> sizeof(uint64_t) = 8
> sizeof(struct blkif_request) = 112
> sizeof(struct blkif_request_segment) = 8
> sizeof(struct blkif_response) = 16
> 
> 32 bit Windows (using Microsoft DDK Compiler and debug statements in
the
> driver)
> sizeof(uint8_t) = 1
> sizeof(uint16_t) = 2
> sizeof(uint32_t) = 4
> sizeof(uint64_t) = 8
> sizeof(struct blkif_request) = 112
> sizeof(struct blkif_request_segment) = 8
> sizeof(struct blkif_response) = 16
> 
> Haven't tried this test on 64 bit windows yet, but if 32 bit windows
and
> 32 bit Linux align their structures differently, I can imagine that
> problems are going to arise...
> 

"#pragma pack(4)" before the blkif definitions and "#pragma pack()"
afterwards fixed it up.

All appears to be working now. I've pushed the changes to hg
(http://xenbits.xensource.com/ext/win-pvdrivers.hg). I haven't yet
tested under amd64 in case I've broken that...

James

Re: Need help figuring out why my windows pv drivers wontwork with a 32 bit dom0...

[Samuel Thibault]

James Harper, le Fri 01 Feb 2008 15:52:55 +1100, a écrit :
> > . Windows is padding structures differently under 32 and 64 bits.
> > Strange that everything else works fine though...
> 
> I think I may be onto something with this...
> 
> 32 bit Linux (using gcc and a quick .c file):
> sizeof(uint8_t) = 1
> sizeof(uint16_t) = 2
> sizeof(uint32_t) = 4
> sizeof(uint64_t) = 8
> sizeof(struct blkif_request) = 108
> sizeof(struct blkif_request_segment) = 8
> sizeof(struct blkif_response) = 12
> 
> 64 bit Linux (using gcc and a quick .c file):
> sizeof(uint8_t) = 1
> sizeof(uint16_t) = 2
> sizeof(uint32_t) = 4
> sizeof(uint64_t) = 8
> sizeof(struct blkif_request) = 112
> sizeof(struct blkif_request_segment) = 8
> sizeof(struct blkif_response) = 16
> 
> 32 bit Windows (using Microsoft DDK Compiler and debug statements in the
> driver)
> sizeof(uint8_t) = 1
> sizeof(uint16_t) = 2
> sizeof(uint32_t) = 4
> sizeof(uint64_t) = 8
> sizeof(struct blkif_request) = 112
> sizeof(struct blkif_request_segment) = 8
> sizeof(struct blkif_response) = 16
> 
> Haven't tried this test on 64 bit windows yet, but if 32 bit windows and
> 32 bit Linux align their structures differently, I can imagine that
> problems are going to arise...

Mmm, then we probably need to define windows-specific ABI types...

Samuel

RE: Need help figuring out why my windows pv drivers wontwork with a 32 bit dom0...

[James Harper]

> > Haven't tried this test on 64 bit windows yet, but if 32 bit windows
and
> > 32 bit Linux align their structures differently, I can imagine that
> > problems are going to arise...
> 
> Mmm, then we probably need to define windows-specific ABI types...

I have now gotten it to the point where:
Dom0=32, DomU=32 works (didn't work previously)
Dom0=64, DomU=64 works (worked previously)
Dom0=64, DomU=32 doesn't work (worked previously)

I'm completely baffled... the Dom0=64 + DomU=32 case is the one I've
been testing with all along, and it's only stopped working since I've
fixed up the structure alignment and told the backend that the frontend
is using the 32 bit abi.

James

RE: Need help figuring out why my windows pv driverswontwork with a 32 bit dom0...

[Daniel Stodden]

On Fri, 2008-02-01 at 16:48 +1100, James Harper wrote:

> "#pragma pack(4)" before the blkif definitions and "#pragma pack()"
> afterwards fixed it up.

yes, packing should fix it, but is it the right approach?
the fields are not naturally (i.e. multiple of the word size) aligned,
and that is the problem. different compilers will chose alignments
different from the declared ones for good reason. The 64bit values are
hanging across a dword boundary, and expect the memory interface to hate
that.

could you add explicit, sane pad values to the structures and see
whether that negatively affects the message sizes which fit on the
sring? I don't think that this would be the case. Pad, _then_ pack them.
Should be packed for gcc as well, as there are no guarantees that chosen
alignments are maintained across past and future revisions:
__attribute__((packed))

btw: there's an offsetof() macro (C99?) which should give you more
insight which variables are actually affected in your test code:
#define offsetof(type,memb) ((unsigned long) &((type*)0)->memb)

regards,
daniel

-- 
Daniel Stodden
LRR     -      Lehrstuhl für Rechnertechnik und Rechnerorganisation
Institut für Informatik der TU München             D-85748 Garching
http://www.lrr.in.tum.de/~stodden         mailto:stodden@cs.tum.edu
PGP Fingerprint: F5A4 1575 4C56 E26A 0B33  3D80 457E 82AE B0D8 735B

RE: Need help figuring out why my windows pv drivers wontwork with a 32 bit dom0...

[James Harper]

> 
> Mmm, then we probably need to define windows-specific ABI types...
> 

Actually... I just tried a Linux -i386 DomU kernel while running a
-amd64 Dom0 kernel, and it crashes in a way that looks remarkably
similar to the way that Windows crashes.

Do you know if there was such a bug that has been fixed? Maybe I just
need to upgrade something?

James

RE: Need help figuring out why my windows pvdriverswontwork with a 32 bit dom0...

[James Harper]

> On Fri, 2008-02-01 at 16:48 +1100, James Harper wrote:
> 
> > "#pragma pack(4)" before the blkif definitions and "#pragma pack()"
> > afterwards fixed it up.
> 
> yes, packing should fix it, but is it the right approach?

I think it is. I'm just matching the packing to the way Linux does it
for that one structure.

See my previous post, the problem I'm having (64 bit Dom0 with 32 bit
DomU) isn't just limited to windows PV domains - Linux DomU's are
crashing in exactly the same way! I'm just trying to figure out if there
is already a fix for this and I just need to apply it somewhere...

James