Re: KVM and variable-endianness guest CPUs

Victor Kamensky
2014-01-24T00:09:30Z

On 23 January 2014 08:25, Victor Kamensky <victor.kamensky@linaro.org> wrote:
> On 23 January 2014 07:33, Peter Maydell <peter.maydell@linaro.org> wrote:
>> On 23 January 2014 15:06, Victor Kamensky <victor.kamensky@linaro.org> wrote:
>>> In [1] I wrote
>>>
>>> "I don't see why you so attached to desire to describe
>>> data part of memory transaction as just one of int
>>> types. If we are talking about bunch of hypothetical
>>> cases imagine such bus that allow transaction with
>>> size of 6 bytes. How do you describe such data in
>>> your ints speak? What endianity you can assign to
>>> sequence of 6 bytes? While note that description of
>>> such transaction as set of 6 byte values at address
>>> $whatever makes perfect sense."
>>>
>>> But notice that in your next reply [2] you just dropped it
>>
>> Yes. This is because it was one of the places where
>> I would have just had to repeat "no, I'm afraid you're wrong
>> about how hardware works". I think in general it's going
>> to be better if I don't try to reply point by point to this
>> email; I think you should go back and reread the emails I've
>> sent. Key points:
>> (1) hardware is not doing anything involving arrays
>> of bytes
>
> Array of bytes or integers is just a way to describe data lines
> on the bus. Did you look at this document?
>
> http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0290g/ch06s05s01.html
>
> A0, A1, ,,, A7 byte values are the same for both LE and BE-8
> case (first two columns in the table) and they unambiguously
> describe data bus signals
>
>> (2) the API between kernel and userspace needs to define
>> the semantics of mmio.data, ie how to map between
>> "x byte wide transaction with value v" and the array,
>> and that is primarily what this conversation is about
>> (3) the only choice which is both (a) sensible and (b)
>> not breaking existing usage is to say "the array is
>> in host-kernel-byte-order"
>> (4) PPC CPUs in BE mode and ARM CPUs in BE mode are not
>> the same, because in the ARM case it is doing an
>> internal-to-CPU byteswap, and in the PPC case it is not
>
> That is one of the key disconnects. I'll go find real examples
> in ARM LE, ARM BE, and PPC BE Linux kernel. Just for
> everybody sake's here is summary of the disconnect:
>
> If we have the same h/w connected to memory bus in ARM
> and PPC systems and we have the following three pieces
> of code that work with r0 having same device same
> register address:
>
> 1. ARM LE word write of 0x04030201:
> setend le
> mov r1, #0x04030201
> str r1, [r0]
>
> 2. ARM BE word write of 0x01020304:
> setend be
> mov r1, #0x01020304
> str r1, [r0]
>
> 3. PPC BE word write of 0x01020304:
> lis r1,0x102
> ori r1,r1,0x304
> stw r1,0(r0)
>
> I claim that h/w will see the same data on bus lines in all
> three cases, and h/w would acts the same in all three
> cases. Peter says that ARM BE and PPC BE case h/w
> would act differently.
>
> If anyone else can offer opinion on that while I am looking
> for real examples that would be great.

Here is my example:

Let's look at isp1760 USB host controller (effectively
one used by TC2). Source code is at
drivers/usb/host/isp1760-hcd.c file. The driver could
be enabled in kernel by adding CONFIG_USB_ISP1760_HCD=y
config I enabled it in ppc image build, arm TC2 already
have it

isp1760 USB host controller driver registers are
in LE format. isp1760 USB host controller driver uses
reg_write32 function to write memory mapped controler
registers. That in turns calls writel, which is LE
device word write:

void reg_write32(void __iomem *base, u32 reg, u32 val)
{
writel(val, base + reg);
}

In C terms writel is LE word write function. It is
effective memory barrier, cpu_to_le32 and write.
cpu_to_le32 will do byteswap only in BE case.

LE ARM
------

00002e04 <reg_write32>:
2e04: e92d4070 push {r4, r5, r6, lr}
2e08: e1a04000 mov r4, r0
2e0c: e1a05001 mov r5, r1
2e10: e1a06002 mov r6, r2
2e14: f57ff04e dsb st
2e18: e59f3018 ldr r3, [pc, #24] ; 2e38
<reg_write32+0x34>
2e1c: e5933018 ldr r3, [r3, #24]
2e20: e3530000 cmp r3, #0
2e24: 0a000000 beq 2e2c <reg_write32+0x28>
2e28: e12fff33 blx r3
2e2c: e0841005 add r1, r4, r5
2e30: e5816000 str r6, [r1] @ <-------
2e34: e8bd8070 pop {r4, r5, r6, pc}
2e38: 00000000 .word 0x00000000

operates in LE, just write value to device register mem location

BE ARM
------

00000590 <reg_write32>:
590: e92d4070 push {r4, r5, r6, lr}
594: e1a04000 mov r4, r0
598: e1a05001 mov r5, r1
59c: e1a06002 mov r6, r2
5a0: f57ff04e dsb st
5a4: e59f301c ldr r3, [pc, #28] ; 5c8 <reg_write32+0x38>
5a8: e5933018 ldr r3, [r3, #24]
5ac: e3530000 cmp r3, #0
5b0: 0a000000 beq 5b8 <reg_write32+0x28>
5b4: e12fff33 blx r3
5b8: e6bf2f36 rev r2, r6 @ <-------
5bc: e0841005 add r1, r4, r5
5c0: e5812000 str r2, [r1] @ <-------
5c4: e8bd8070 pop {r4, r5, r6, pc}
5c8: 00000000 .word 0x00000000

operates in BE, byteswap first, then write value to device mem location

BE PPC
------

00003070 <reg_write32>:
3070: 7c 08 02 a6 mflr r0
3074: 90 01 00 04 stw r0,4(r1)
3078: 48 00 00 01 bl 3078 <reg_write32+0x8>
307c: 94 21 ff f0 stwu r1,-16(r1)
3080: 7c 08 02 a6 mflr r0
3084: 90 01 00 14 stw r0,20(r1)
3088: 7c 83 22 14 add r4,r3,r4
308c: 7c 00 04 ac sync
3090: 7c a0 25 2c stwbrx r5,0,r4 @ <-------
3094: 80 01 00 14 lwz r0,20(r1)
3098: 38 21 00 10 addi r1,r1,16
309c: 7c 08 03 a6 mtlr r0
30a0: 4e 80 00 20 blr

stwbrx - is 'Store Word Byte-Reverse Indexed' instruction
bascially byteswap word and store it (r3 is base, r4
is reg, r5 is val).

operates in BE, byteswap and store value but now it is
done in one instruction. But it is exactly what ARM BE
code does.

So if one will call reg_write32(0x1000, 0, 0x04030201) it
would work the same way as earlier cited sequences - ARM
LE case would use 0x04030201 and write it out. BE ARM and
BE PPC would get byteswap value of (0x04030201)=0x01020304
and write it out.

Device see the same thing for all three cases, that is how
BE-8 memory bus works.

Thanks,
Victor

> Thanks,
> Victor
>
>> thanks
>> -- PMM