[Qemu-devel] [TestDays] s390x emulation error

[Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Hello,

I get the following error:

$ s390x-softmmu/qemu-system-s390x
qemu-system-s390x: Guest moved used index from 0 to 47802

Same if specifying SLES 11 SP2 Beta8 DVD .iso via -cdrom.

First time trying it, am I missing anything?
The error message is not really telling either way and should be fixed.

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 09.11.2011 23:53, schrieb Andreas Färber:
> I get the following error:
> 
> $ s390x-softmmu/qemu-system-s390x
> qemu-system-s390x: Guest moved used index from 0 to 47802

This is coming from hw/virtio.c:

static int virtqueue_num_heads(VirtQueue *vq, unsigned int idx)
{
    uint16_t num_heads = vring_avail_idx(vq) - idx;

    /* Check it isn't doing very strange things with descriptor numbers. */
    if (num_heads > vq->vring.num) {
        error_report("Guest moved used index from %u to %u",
                     idx, vring_avail_idx(vq));
        exit(1);
    }

    return num_heads;
}

Any idea? Couldn't find a related s390-specific patch on the list.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

Am 10.11.2011 um 00:17 schrieb Andreas Färber <afaerber@suse.de>:

> Am 09.11.2011 23:53, schrieb Andreas Färber:
>> I get the following error:
>> 
>> $ s390x-softmmu/qemu-system-s390x
>> qemu-system-s390x: Guest moved used index from 0 to 47802
> 
> This is coming from hw/virtio.c:
> 
> static int virtqueue_num_heads(VirtQueue *vq, unsigned int idx)
> {
>    uint16_t num_heads = vring_avail_idx(vq) - idx;
> 
>    /* Check it isn't doing very strange things with descriptor numbers. */
>    if (num_heads > vq->vring.num) {
>        error_report("Guest moved used index from %u to %u",
>                     idx, vring_avail_idx(vq));
>        exit(1);
>    }
> 
>    return num_heads;
> }
> 
> Any idea? Couldn't find a related s390-specific patch on the list.

There is a good chance my simple in-zipl virtio implementation does something wrong :(. IIRC I always keep the outstanding requests on 1 (something outstanding) or 0 (all is done) so I can keep the logic a lot easier.

Alex

> 
> Andreas
> 
> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

Am 09.11.2011 um 23:53 schrieb Andreas Färber <afaerber@suse.de>:

> Hello,
> 
> I get the following error:
> 
> $ s390x-softmmu/qemu-system-s390x
> qemu-system-s390x: Guest moved used index from 0 to 47802
> 
> Same if specifying SLES 11 SP2 Beta8 DVD .iso via -cdrom.
> 
> First time trying it, am I missing anything?
> The error message is not really telling either way and should be fixed.

S390 doesn't have a standardized way of booting off CDs, so you have to manually run the image from the CD by passing it to -kernel. It's done the same way on real hardware :(.

Alex

> 
> Regards,
> Andreas
> 
> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 10.11.2011 00:18, schrieb Alexander Graf:
> Am 09.11.2011 um 23:53 schrieb Andreas Färber <afaerber@suse.de>:
>> I get the following error:
>>
>> $ s390x-softmmu/qemu-system-s390x
>> qemu-system-s390x: Guest moved used index from 0 to 47802
>>
>> Same if specifying SLES 11 SP2 Beta8 DVD .iso via -cdrom.
>>
>> First time trying it, am I missing anything?
>> The error message is not really telling either way and should be fixed.
> 
> S390 doesn't have a standardized way of booting off CDs, so you have to manually run the image from the CD by passing it to -kernel. It's done the same way on real hardware :(.

Fine with me. Just please have the emulator say so! :)

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

Am 10.11.2011 um 00:20 schrieb Andreas Färber <afaerber@suse.de>:

> Am 10.11.2011 00:18, schrieb Alexander Graf:
>> Am 09.11.2011 um 23:53 schrieb Andreas Färber <afaerber@suse.de>:
>>> I get the following error:
>>> 
>>> $ s390x-softmmu/qemu-system-s390x
>>> qemu-system-s390x: Guest moved used index from 0 to 47802
>>> 
>>> Same if specifying SLES 11 SP2 Beta8 DVD .iso via -cdrom.
>>> 
>>> First time trying it, am I missing anything?
>>> The error message is not really telling either way and should be fixed.
>> 
>> S390 doesn't have a standardized way of booting off CDs, so you have to manually run the image from the CD by passing it to -kernel. It's done the same way on real hardware :(.
> 
> Fine with me. Just please have the emulator say so! :)

Hm. That is zipl business really. It should print a warning when it can't find a boot record I suppose :).

Alex

> 
> Andreas
> 
> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 10.11.2011 00:21, schrieb Alexander Graf:
> 
> 
> Am 10.11.2011 um 00:20 schrieb Andreas Färber <afaerber@suse.de>:
> 
>> Am 10.11.2011 00:18, schrieb Alexander Graf:
>>> Am 09.11.2011 um 23:53 schrieb Andreas Färber <afaerber@suse.de>:
>>>> I get the following error:
>>>>
>>>> $ s390x-softmmu/qemu-system-s390x
>>>> qemu-system-s390x: Guest moved used index from 0 to 47802
>>>>
>>>> Same if specifying SLES 11 SP2 Beta8 DVD .iso via -cdrom.
>>>>
>>>> First time trying it, am I missing anything?
>>>> The error message is not really telling either way and should be fixed.
>>>
>>> S390 doesn't have a standardized way of booting off CDs, so you have to manually run the image from the CD by passing it to -kernel. It's done the same way on real hardware :(.
>>
>> Fine with me. Just please have the emulator say so! :)
> 
> Hm. That is zipl business really. It should print a warning when it can't find a boot record I suppose :).

Or the machine, knowing what QEMU-specific firmware it uses, could error
if it doesn't get -kernel arguments. :) Me, I don't know zipl.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

[-- Attachment #1: Type: text/plain, Size: 1386 bytes --]


On 10.11.2011, at 00:23, Andreas Färber wrote:

> Am 10.11.2011 00:21, schrieb Alexander Graf:
>> 
>> 
>> Am 10.11.2011 um 00:20 schrieb Andreas Färber <afaerber@suse.de>:
>> 
>>> Am 10.11.2011 00:18, schrieb Alexander Graf:
>>>> Am 09.11.2011 um 23:53 schrieb Andreas Färber <afaerber@suse.de>:
>>>>> I get the following error:
>>>>> 
>>>>> $ s390x-softmmu/qemu-system-s390x
>>>>> qemu-system-s390x: Guest moved used index from 0 to 47802
>>>>> 
>>>>> Same if specifying SLES 11 SP2 Beta8 DVD .iso via -cdrom.
>>>>> 
>>>>> First time trying it, am I missing anything?
>>>>> The error message is not really telling either way and should be fixed.
>>>> 
>>>> S390 doesn't have a standardized way of booting off CDs, so you have to manually run the image from the CD by passing it to -kernel. It's done the same way on real hardware :(.
>>> 
>>> Fine with me. Just please have the emulator say so! :)
>> 
>> Hm. That is zipl business really. It should print a warning when it can't find a boot record I suppose :).
> 
> Or the machine, knowing what QEMU-specific firmware it uses, could error
> if it doesn't get -kernel arguments. :) Me, I don't know zipl.

There is a very simple zipl-derived bootloader in QEMU that can boot off disk images. It just can't boot off CDs:

  http://repo.or.cz/w/s390-tools.git/shortlog/refs/heads/virtio-zipl


Alex


[-- Attachment #2: Type: text/html, Size: 3940 bytes --]

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 09.11.2011 23:53, schrieb Andreas Färber:
> I get the following error:
> 
> $ s390x-softmmu/qemu-system-s390x
> qemu-system-s390x: Guest moved used index from 0 to 47802

Now with -kernel and -drive if=virtio on s390-next I got the following
fatal error during installation of SLES 11 SP2 Beta8:

qemu: fatal: EXECUTE on instruction prefix 0xdc00 not implemented


������������������������������������������������������������������������Ŀ

R00=0000000000010000 R01=000000000011071a R02=0000000000714bb0
R03=000000003cf47dcf�Total│    │        │00:00
                     �
R04=00000000ffffff07 R05=0000000000525f08 R06=000000003cb55600
R07=000000000000000c�
                     �
R08=000003ffff8fccf8 R09=000000003f59fc98 R10=000000003cfc0000
R11=000000003c786a50�─������������������������������������������������������������������──�─�

R12=000000003cf47dcf R13=0000000000513540 R14=000000003cf47d70
R15=000000003cf47d28│~kB)
                     │
F00=4130000000000000 F01=3f80000000000000 F02=0000000000000000
F03=0000000000000000│Installing gdm-2.24.0-24.69.21.s390x.rpm (installed
size 2.37 MB)       │
F04=4130000000000000 F05=40a0000000000000 F06=0000000000000000
F07=0000000000000000│grep: sbin/conf.d/SuSEconfig.gdm: No such file or
directory             ─
F08=0000000000000000 F09=0000000000000000 F10=0000000000000000
F11=0000000000000000Installing gdm-2.24.0-24.69.21.s390x.rpm (installed
size 2.37 MB)
F12=0000000000000000 F13=0000000000000000 F14=0000000000000000
F15=0000000000000000

   Installing Packages... (Remaining: 2.37 MB / 00:05)

C00=0000000014354e12 C01=00000000007a1007 C02=0000000000011140
C03=0000000000000000

C04=0000000000000a5a C05=0000000000011140 C06=0000000010000000
C07=000000003d6cc1c7
C08=0000000000000000 C09=0000000000000000 C10=0000000000000000
C11=0000000000000000
C12=0000000000000000 C13=000000003d6cc1c7 C14=00000000db000000
C15=0000000000000000

PSW=mask 0704000180000000 addr 0000000000110732 cc 01

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Is there a known issue with running multiple instances of
qemu-system-s390x? I got a hang on openSUSE 12.1 RC2 x86_64 host:

0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f0de7f651c5 in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f0de7f6501a in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000048b4a9 in qemu_mutex_lock (mutex=<optimized out>)
    at /home/andreas/QEMU/qemu/qemu-thread-posix.c:54
#4  0x00000000004cd3df in qemu_mutex_lock_iothread ()
    at /home/andreas/QEMU/qemu/cpus.c:843
#5  0x0000000000467580 in main_loop_wait (nonblocking=<optimized out>)
    at /home/andreas/QEMU/qemu/main-loop.c:459
#6  0x0000000000408984 in main_loop () at /home/andreas/QEMU/qemu/vl.c:1481
#7  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
    at /home/andreas/QEMU/qemu/vl.c:3474

Key presses didn't work, SDL window doesn't close, at 99.9% CPU.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

On 10.11.2011, at 10:53, Andreas Färber <afaerber@suse.de> wrote:

> Is there a known issue with running multiple instances of
> qemu-system-s390x? I got a hang on openSUSE 12.1 RC2 x86_64 host:
> 
> 0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
> (gdb) bt
> #0  0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00007f0de7f651c5 in _L_lock_883 () from /lib64/libpthread.so.0
> #2  0x00007f0de7f6501a in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x000000000048b4a9 in qemu_mutex_lock (mutex=<optimized out>)
>    at /home/andreas/QEMU/qemu/qemu-thread-posix.c:54
> #4  0x00000000004cd3df in qemu_mutex_lock_iothread ()
>    at /home/andreas/QEMU/qemu/cpus.c:843
> #5  0x0000000000467580 in main_loop_wait (nonblocking=<optimized out>)
>    at /home/andreas/QEMU/qemu/main-loop.c:459
> #6  0x0000000000408984 in main_loop () at /home/andreas/QEMU/qemu/vl.c:1481
> #7  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
>    at /home/andreas/QEMU/qemu/vl.c:3474
> 
> Key presses didn't work, SDL window doesn't close, at 99.9% CPU.

Huh? This is all generic code O_o. And no, it's not a known issue.

Alex

> 
> Andreas
> 
> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 10.11.2011 11:32, schrieb Alexander Graf:
> 
> On 10.11.2011, at 10:53, Andreas Färber <afaerber@suse.de> wrote:
> 
>> Is there a known issue with running multiple instances of
>> qemu-system-s390x? I got a hang on openSUSE 12.1 RC2 x86_64 host:
>>
>> 0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>> (gdb) bt
>> #0  0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>> #1  0x00007f0de7f651c5 in _L_lock_883 () from /lib64/libpthread.so.0
>> #2  0x00007f0de7f6501a in pthread_mutex_lock () from /lib64/libpthread.so.0
>> #3  0x000000000048b4a9 in qemu_mutex_lock (mutex=<optimized out>)
>>    at /home/andreas/QEMU/qemu/qemu-thread-posix.c:54
>> #4  0x00000000004cd3df in qemu_mutex_lock_iothread ()
>>    at /home/andreas/QEMU/qemu/cpus.c:843
>> #5  0x0000000000467580 in main_loop_wait (nonblocking=<optimized out>)
>>    at /home/andreas/QEMU/qemu/main-loop.c:459
>> #6  0x0000000000408984 in main_loop () at /home/andreas/QEMU/qemu/vl.c:1481
>> #7  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
>>    at /home/andreas/QEMU/qemu/vl.c:3474
>>
>> Key presses didn't work, SDL window doesn't close, at 99.9% CPU.
> 
> Huh? This is all generic code O_o. And no, it's not a known issue.

Hm, reproducable by running

$ s390x-softmmu/qemu-system-s390x

(without arguments) on s390-next branch.

I get compat_monitor0 console, but monitor or switching to real console
don't work and neither does closing. Same backtrace.

Doesn't happen with ppc-softmmu/qemu-system-ppc or x86_64. Maybe an
issue with the monitor, only apparent when then monitor is shown at
startup rather than later user-invoked? Any suggestions?

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

On 10.11.2011, at 12:29, Andreas Färber <afaerber@suse.de> wrote:

> Am 10.11.2011 11:32, schrieb Alexander Graf:
>> 
>> On 10.11.2011, at 10:53, Andreas Färber <afaerber@suse.de> wrote:
>> 
>>> Is there a known issue with running multiple instances of
>>> qemu-system-s390x? I got a hang on openSUSE 12.1 RC2 x86_64 host:
>>> 
>>> 0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>> (gdb) bt
>>> #0  0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>> #1  0x00007f0de7f651c5 in _L_lock_883 () from /lib64/libpthread.so.0
>>> #2  0x00007f0de7f6501a in pthread_mutex_lock () from /lib64/libpthread.so.0
>>> #3  0x000000000048b4a9 in qemu_mutex_lock (mutex=<optimized out>)
>>>   at /home/andreas/QEMU/qemu/qemu-thread-posix.c:54
>>> #4  0x00000000004cd3df in qemu_mutex_lock_iothread ()
>>>   at /home/andreas/QEMU/qemu/cpus.c:843
>>> #5  0x0000000000467580 in main_loop_wait (nonblocking=<optimized out>)
>>>   at /home/andreas/QEMU/qemu/main-loop.c:459
>>> #6  0x0000000000408984 in main_loop () at /home/andreas/QEMU/qemu/vl.c:1481
>>> #7  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
>>>   at /home/andreas/QEMU/qemu/vl.c:3474
>>> 
>>> Key presses didn't work, SDL window doesn't close, at 99.9% CPU.
>> 
>> Huh? This is all generic code O_o. And no, it's not a known issue.
> 
> Hm, reproducable by running
> 
> $ s390x-softmmu/qemu-system-s390x
> 
> (without arguments) on s390-next branch.
> 
> I get compat_monitor0 console, but monitor or switching to real console
> don't work and neither does closing. Same backtrace.
> 
> Doesn't happen with ppc-softmmu/qemu-system-ppc or x86_64. Maybe an
> issue with the monitor, only apparent when then monitor is shown at
> startup rather than later user-invoked? Any suggestions?

Maybe the guest is executing an endless loop? Could you try to compile this asm blob for ppc and execute it with -kernel?

Test.S:

_start:
  b .


Compile with gcc test.S -nostdlib


Alex

> 

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 10.11.2011 13:36, schrieb Alexander Graf:
> 
> On 10.11.2011, at 12:29, Andreas Färber <afaerber@suse.de> wrote:
> 
>> Am 10.11.2011 11:32, schrieb Alexander Graf:
>>>
>>> On 10.11.2011, at 10:53, Andreas Färber <afaerber@suse.de> wrote:
>>>
>>>> Is there a known issue with running multiple instances of
>>>> qemu-system-s390x? I got a hang on openSUSE 12.1 RC2 x86_64 host:
>>>>
>>>> 0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>> (gdb) bt
>>>> #0  0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>> #1  0x00007f0de7f651c5 in _L_lock_883 () from /lib64/libpthread.so.0
>>>> #2  0x00007f0de7f6501a in pthread_mutex_lock () from /lib64/libpthread.so.0
>>>> #3  0x000000000048b4a9 in qemu_mutex_lock (mutex=<optimized out>)
>>>>   at /home/andreas/QEMU/qemu/qemu-thread-posix.c:54
>>>> #4  0x00000000004cd3df in qemu_mutex_lock_iothread ()
>>>>   at /home/andreas/QEMU/qemu/cpus.c:843
>>>> #5  0x0000000000467580 in main_loop_wait (nonblocking=<optimized out>)
>>>>   at /home/andreas/QEMU/qemu/main-loop.c:459
>>>> #6  0x0000000000408984 in main_loop () at /home/andreas/QEMU/qemu/vl.c:1481
>>>> #7  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
>>>>   at /home/andreas/QEMU/qemu/vl.c:3474
>>>>
>>>> Key presses didn't work, SDL window doesn't close, at 99.9% CPU.
>>>
>>> Huh? This is all generic code O_o. And no, it's not a known issue.
>>
>> Hm, reproducable by running
>>
>> $ s390x-softmmu/qemu-system-s390x
>>
>> (without arguments) on s390-next branch.
>>
>> I get compat_monitor0 console, but monitor or switching to real console
>> don't work and neither does closing. Same backtrace.
>>
>> Doesn't happen with ppc-softmmu/qemu-system-ppc or x86_64. Maybe an
>> issue with the monitor, only apparent when then monitor is shown at
>> startup rather than later user-invoked? Any suggestions?
> 
> Maybe the guest is executing an endless loop? Could you try to compile this asm blob for ppc and execute it with -kernel?
> 
> Test.S:
> 
> _start:
>   b .
> 
> 
> Compile with gcc test.S -nostdlib

Nope, doesn't hang.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

On 11/14/2011 06:29 PM, Andreas Färber wrote:
> Am 10.11.2011 13:36, schrieb Alexander Graf:
>> On 10.11.2011, at 12:29, Andreas Färber<afaerber@suse.de>  wrote:
>>
>>> Am 10.11.2011 11:32, schrieb Alexander Graf:
>>>> On 10.11.2011, at 10:53, Andreas Färber<afaerber@suse.de>  wrote:
>>>>
>>>>> Is there a known issue with running multiple instances of
>>>>> qemu-system-s390x? I got a hang on openSUSE 12.1 RC2 x86_64 host:
>>>>>
>>>>> 0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>>> (gdb) bt
>>>>> #0  0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>>> #1  0x00007f0de7f651c5 in _L_lock_883 () from /lib64/libpthread.so.0
>>>>> #2  0x00007f0de7f6501a in pthread_mutex_lock () from /lib64/libpthread.so.0
>>>>> #3  0x000000000048b4a9 in qemu_mutex_lock (mutex=<optimized out>)
>>>>>    at /home/andreas/QEMU/qemu/qemu-thread-posix.c:54
>>>>> #4  0x00000000004cd3df in qemu_mutex_lock_iothread ()
>>>>>    at /home/andreas/QEMU/qemu/cpus.c:843
>>>>> #5  0x0000000000467580 in main_loop_wait (nonblocking=<optimized out>)
>>>>>    at /home/andreas/QEMU/qemu/main-loop.c:459
>>>>> #6  0x0000000000408984 in main_loop () at /home/andreas/QEMU/qemu/vl.c:1481
>>>>> #7  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
>>>>>    at /home/andreas/QEMU/qemu/vl.c:3474
>>>>>
>>>>> Key presses didn't work, SDL window doesn't close, at 99.9% CPU.
>>>> Huh? This is all generic code O_o. And no, it's not a known issue.
>>> Hm, reproducable by running
>>>
>>> $ s390x-softmmu/qemu-system-s390x
>>>
>>> (without arguments) on s390-next branch.
>>>
>>> I get compat_monitor0 console, but monitor or switching to real console
>>> don't work and neither does closing. Same backtrace.
>>>
>>> Doesn't happen with ppc-softmmu/qemu-system-ppc or x86_64. Maybe an
>>> issue with the monitor, only apparent when then monitor is shown at
>>> startup rather than later user-invoked? Any suggestions?
>> Maybe the guest is executing an endless loop? Could you try to compile this asm blob for ppc and execute it with -kernel?
>>
>> Test.S:
>>
>> _start:
>>    b .
>>
>>
>> Compile with gcc test.S -nostdlib
> Nope, doesn't hang.

Hrm. Paolo, any idea?


Alex

Re: [Qemu-devel] [TestDays] s390x emulation error

[Paolo Bonzini]

On 11/14/2011 06:36 PM, Alexander Graf wrote:
> Hrm. Paolo, any idea?

Sorry for asking the obvious, but can it be bisected?

Paolo

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 10.11.2011 12:29, schrieb Andreas Färber:
> Am 10.11.2011 11:32, schrieb Alexander Graf:
>>
>> On 10.11.2011, at 10:53, Andreas Färber <afaerber@suse.de> wrote:
>>
>>> Is there a known issue with running multiple instances of
>>> qemu-system-s390x? I got a hang on openSUSE 12.1 RC2 x86_64 host:
>>>
>>> 0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>> (gdb) bt
>>> #0  0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>> #1  0x00007f0de7f651c5 in _L_lock_883 () from /lib64/libpthread.so.0
>>> #2  0x00007f0de7f6501a in pthread_mutex_lock () from /lib64/libpthread.so.0
>>> #3  0x000000000048b4a9 in qemu_mutex_lock (mutex=<optimized out>)
>>>    at /home/andreas/QEMU/qemu/qemu-thread-posix.c:54
>>> #4  0x00000000004cd3df in qemu_mutex_lock_iothread ()
>>>    at /home/andreas/QEMU/qemu/cpus.c:843
>>> #5  0x0000000000467580 in main_loop_wait (nonblocking=<optimized out>)
>>>    at /home/andreas/QEMU/qemu/main-loop.c:459
>>> #6  0x0000000000408984 in main_loop () at /home/andreas/QEMU/qemu/vl.c:1481
>>> #7  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
>>>    at /home/andreas/QEMU/qemu/vl.c:3474
>>>
>>> Key presses didn't work, SDL window doesn't close, at 99.9% CPU.
>>
>> Huh? This is all generic code O_o. And no, it's not a known issue.
> 
> Hm, reproducable by running
> 
> $ s390x-softmmu/qemu-system-s390x
> 
> (without arguments) on s390-next branch.
> 
> I get compat_monitor0 console, but monitor or switching to real console
> don't work and neither does closing. Same backtrace.

Same in my WIP qemu-system-rl78.

I found that the following main-loop change works around it for s390x
and rl78 but breaks x86_64 SeaBIOS boot. Paolo, any ideas?

diff --git a/main-loop.c b/main-loop.c
index 60e9748..2ab5023 100644
--- a/main-loop.c
+++ b/main-loop.c
@@ -460,7 +460,7 @@ int main_loop_wait(int nonblocking)
     }

     glib_select_poll(&rfds, &wfds, &xfds, (ret < 0));
-    qemu_iohandler_poll(&rfds, &wfds, &xfds, ret);
+    qemu_iohandler_poll(&rfds, &wfds, &xfds, (ret < 0));
 #ifdef CONFIG_SLIRP
     slirp_select_poll(&rfds, &wfds, &xfds, (ret < 0));
 #endif

A deadlock between iothread and main?

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Stefan Weil]

Am 12.11.2011 11:08, schrieb Andreas Färber:
> Am 10.11.2011 12:29, schrieb Andreas Färber:
> I found that the following main-loop change works around it for s390x
> and rl78 but breaks x86_64 SeaBIOS boot. Paolo, any ideas?
>
> diff --git a/main-loop.c b/main-loop.c
> index 60e9748..2ab5023 100644
> --- a/main-loop.c
> +++ b/main-loop.c
> @@ -460,7 +460,7 @@ int main_loop_wait(int nonblocking)
> }
>
> glib_select_poll(&rfds, &wfds, &xfds, (ret < 0));
> - qemu_iohandler_poll(&rfds, &wfds, &xfds, ret);
> + qemu_iohandler_poll(&rfds, &wfds, &xfds, (ret < 0));
> #ifdef CONFIG_SLIRP
> slirp_select_poll(&rfds, &wfds, &xfds, (ret < 0));
> #endif
>
> A deadlock between iothread and main?
>
> Andreas

I just tried s390x on a 386 host (32 bit!) and got a different crash
(modulo operation / division with 0.0).

Are 32 bit hosts supported?

Stefan

(gdb) r
Starting program: 
/home/stefan/src/qemu/qemu.org/qemu/bin/debug/386/s390x-softmmu/qemu-system-s390x 

[Thread debugging using libthread_db enabled]
[New Thread 0xae9d0b70 (LWP 6841)]

Program received signal SIGFPE, Arithmetic exception.
[Switching to Thread 0xae9d0b70 (LWP 6841)]
0x08199f6b in __umoddi3 ()
(gdb) i s
#0  0x08199f6b in __umoddi3 ()
#1  0x08168a48 in helper_dlg (r1=2, v2=0) at 
/home/stefan/src/qemu/qemu.org/qemu/target-s390x/op_helper.c:369
#2  0x00eb5a88 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) up
#1  0x08168a48 in helper_dlg (r1=2, v2=0) at 
/home/stefan/src/qemu/qemu.org/qemu/target-s390x/op_helper.c:369
369            env->regs[r1] = env->regs[r1+1] % divisor;
(gdb) l
364    {
365        uint64_t divisor = v2;
366
367        if (!env->regs[r1]) {
368            /* 64 -> 64/64 case */
369            env->regs[r1] = env->regs[r1+1] % divisor;
370            env->regs[r1+1] = env->regs[r1+1] / divisor;
371            return;
372        } else {
373
(gdb) p divisor
$1 = 0
(gdb) p v2
$2 = 0

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

On 12.11.2011, at 11:40, Stefan Weil <sw@weilnetz.de> wrote:

> Am 12.11.2011 11:08, schrieb Andreas Färber:
>> Am 10.11.2011 12:29, schrieb Andreas Färber:
>> I found that the following main-loop change works around it for s390x
>> and rl78 but breaks x86_64 SeaBIOS boot. Paolo, any ideas?
>> 
>> diff --git a/main-loop.c b/main-loop.c
>> index 60e9748..2ab5023 100644
>> --- a/main-loop.c
>> +++ b/main-loop.c
>> @@ -460,7 +460,7 @@ int main_loop_wait(int nonblocking)
>> }
>> 
>> glib_select_poll(&rfds, &wfds, &xfds, (ret < 0));
>> - qemu_iohandler_poll(&rfds, &wfds, &xfds, ret);
>> + qemu_iohandler_poll(&rfds, &wfds, &xfds, (ret < 0));
>> #ifdef CONFIG_SLIRP
>> slirp_select_poll(&rfds, &wfds, &xfds, (ret < 0));
>> #endif
>> 
>> A deadlock between iothread and main?
>> 
>> Andreas
> 
> I just tried s390x on a 386 host (32 bit!) and got a different crash
> (modulo operation / division with 0.0).
> 
> Are 32 bit hosts supported?
> 
> Stefan
> 
> (gdb) r
> Starting program: /home/stefan/src/qemu/qemu.org/qemu/bin/debug/386/s390x-softmmu/qemu-system-s390x 
> [Thread debugging using libthread_db enabled]
> [New Thread 0xae9d0b70 (LWP 6841)]
> 
> Program received signal SIGFPE, Arithmetic exception.
> [Switching to Thread 0xae9d0b70 (LWP 6841)]
> 0x08199f6b in __umoddi3 ()
> (gdb) i s
> #0  0x08199f6b in __umoddi3 ()
> #1  0x08168a48 in helper_dlg (r1=2, v2=0) at /home/stefan/src/qemu/qemu.org/qemu/target-s390x/op_helper.c:369
> #2  0x00eb5a88 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> (gdb) up
> #1  0x08168a48 in helper_dlg (r1=2, v2=0) at /home/stefan/src/qemu/qemu.org/qemu/target-s390x/op_helper.c:369
> 369            env->regs[r1] = env->regs[r1+1] % divisor;
> (gdb) l
> 364    {
> 365        uint64_t divisor = v2;
> 366
> 367        if (!env->regs[r1]) {
> 368            /* 64 -> 64/64 case */
> 369            env->regs[r1] = env->regs[r1+1] % divisor;
> 370            env->regs[r1+1] = env->regs[r1+1] / divisor;
> 371            return;
> 372        } else {
> 373
> (gdb) p divisor
> $1 = 0
> (gdb) p v2
> $2 = 0
> 

No, that's the expected result. I don't special-case division by 0 and my small virtio zipl boot rom triggers div by 0 when no hd is attached :)

Alex

Re: [Qemu-devel] [TestDays] s390x emulation error

[Paolo Bonzini]

On 11/12/2011 11:08 AM, Andreas Färber wrote:
>
> diff --git a/main-loop.c b/main-loop.c
> index 60e9748..2ab5023 100644
> --- a/main-loop.c
> +++ b/main-loop.c
> @@ -460,7 +460,7 @@ int main_loop_wait(int nonblocking)
>       }
>
>       glib_select_poll(&rfds,&wfds,&xfds, (ret<  0));
> -    qemu_iohandler_poll(&rfds,&wfds,&xfds, ret);
> +    qemu_iohandler_poll(&rfds,&wfds,&xfds, (ret<  0));
>   #ifdef CONFIG_SLIRP
>       slirp_select_poll(&rfds,&wfds,&xfds, (ret<  0));
>   #endif

No, this is definitely wrong. :)  It will break all iohandlers.

Paolo

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 13.11.2011 09:48, schrieb Paolo Bonzini:
> On 11/12/2011 11:08 AM, Andreas Färber wrote:
>> I found that the following main-loop change works around it for s390x
>> and rl78 but breaks x86_64 SeaBIOS boot. Paolo, any ideas?
>>
>> diff --git a/main-loop.c b/main-loop.c
>> index 60e9748..2ab5023 100644
>> --- a/main-loop.c
>> +++ b/main-loop.c
>> @@ -460,7 +460,7 @@ int main_loop_wait(int nonblocking)
>>       }
>>
>>       glib_select_poll(&rfds,&wfds,&xfds, (ret<  0));
>> -    qemu_iohandler_poll(&rfds,&wfds,&xfds, ret);
>> +    qemu_iohandler_poll(&rfds,&wfds,&xfds, (ret<  0));
>>   #ifdef CONFIG_SLIRP
>>       slirp_select_poll(&rfds,&wfds,&xfds, (ret<  0));
>>   #endif
> 
> No, this is definitely wrong. :)  It will break all iohandlers.

Yeah, I noticed that myself in the part you had snipped above.
Question is, why are some use cases broken "iff" used right. :)
Any suggestions where/how to debug this?

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 12.11.2011 11:08, schrieb Andreas Färber:
> Am 10.11.2011 12:29, schrieb Andreas Färber:
>> Am 10.11.2011 11:32, schrieb Alexander Graf:
>>>
>>> On 10.11.2011, at 10:53, Andreas Färber <afaerber@suse.de> wrote:
>>>
>>>> Is there a known issue with running multiple instances of
>>>> qemu-system-s390x? I got a hang on openSUSE 12.1 RC2 x86_64 host:
>>>>
>>>> 0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>> (gdb) bt
>>>> #0  0x00007f0de7f698d4 in __lll_lock_wait () from /lib64/libpthread.so.0
>>>> #1  0x00007f0de7f651c5 in _L_lock_883 () from /lib64/libpthread.so.0
>>>> #2  0x00007f0de7f6501a in pthread_mutex_lock () from /lib64/libpthread.so.0
>>>> #3  0x000000000048b4a9 in qemu_mutex_lock (mutex=<optimized out>)
>>>>    at /home/andreas/QEMU/qemu/qemu-thread-posix.c:54
>>>> #4  0x00000000004cd3df in qemu_mutex_lock_iothread ()
>>>>    at /home/andreas/QEMU/qemu/cpus.c:843
>>>> #5  0x0000000000467580 in main_loop_wait (nonblocking=<optimized out>)
>>>>    at /home/andreas/QEMU/qemu/main-loop.c:459
>>>> #6  0x0000000000408984 in main_loop () at /home/andreas/QEMU/qemu/vl.c:1481
>>>> #7  main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>)
>>>>    at /home/andreas/QEMU/qemu/vl.c:3474
>>>>
>>>> Key presses didn't work, SDL window doesn't close, at 99.9% CPU.
>>>
>>> Huh? This is all generic code O_o. And no, it's not a known issue.
>>
>> Hm, reproducable by running
>>
>> $ s390x-softmmu/qemu-system-s390x
>>
>> (without arguments) on s390-next branch.
>>
>> I get compat_monitor0 console, but monitor or switching to real console
>> don't work and neither does closing. Same backtrace.
> 
> Same in my WIP qemu-system-rl78.
> 
> I found that the following main-loop change works around it for s390x
> and rl78 but breaks x86_64 SeaBIOS boot. [...]
> 
> diff --git a/main-loop.c b/main-loop.c
> index 60e9748..2ab5023 100644
> --- a/main-loop.c
> +++ b/main-loop.c
> @@ -460,7 +460,7 @@ int main_loop_wait(int nonblocking)
>      }
> 
>      glib_select_poll(&rfds, &wfds, &xfds, (ret < 0));
> -    qemu_iohandler_poll(&rfds, &wfds, &xfds, ret);
> +    qemu_iohandler_poll(&rfds, &wfds, &xfds, (ret < 0));
>  #ifdef CONFIG_SLIRP
>      slirp_select_poll(&rfds, &wfds, &xfds, (ret < 0));
>  #endif

For the record: While s390x and rl78 showed an identical backtrace in
gdb, the actual causes of the lockups turned out to be different:

For rl78, tlb_fill() was not yet fully/correctly implemented, with no TB
being generated for execution. We should probably error out rather than
going into an infinite loop then.

For s390x, the hang was workaroundable by memset()'ing not just the
virtio memory region but the whole of RAM. Still investigating.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

On 10.11.2011, at 10:32, Andreas Färber <afaerber@suse.de> wrote:

> Am 09.11.2011 23:53, schrieb Andreas Färber:
>> I get the following error:
>> 
>> $ s390x-softmmu/qemu-system-s390x
>> qemu-system-s390x: Guest moved used index from 0 to 47802
> 
> Now with -kernel and -drive if=virtio on s390-next I got the following
> fatal error during installation of SLES 11 SP2 Beta8:
> 
> qemu: fatal: EXECUTE on instruction prefix 0xdc00 not implemented

Ouch - so gcc does generate new execute indirects now. I guess we need to add it then :)

Alex

> 
> 
> ������������������������������������������������������������������������Ŀ
> 
> R00=0000000000010000 R01=000000000011071a R02=0000000000714bb0
> R03=000000003cf47dcf�Total│    │        │00:00
>                     �
> R04=00000000ffffff07 R05=0000000000525f08 R06=000000003cb55600
> R07=000000000000000c�
>                     �
> R08=000003ffff8fccf8 R09=000000003f59fc98 R10=000000003cfc0000
> R11=000000003c786a50�─������������������������������������������������������������������──�─�
> 
> R12=000000003cf47dcf R13=0000000000513540 R14=000000003cf47d70
> R15=000000003cf47d28│~kB)
>                     │
> F00=4130000000000000 F01=3f80000000000000 F02=0000000000000000
> F03=0000000000000000│Installing gdm-2.24.0-24.69.21.s390x.rpm (installed
> size 2.37 MB)       │
> F04=4130000000000000 F05=40a0000000000000 F06=0000000000000000
> F07=0000000000000000│grep: sbin/conf.d/SuSEconfig.gdm: No such file or
> directory             ─
> F08=0000000000000000 F09=0000000000000000 F10=0000000000000000
> F11=0000000000000000Installing gdm-2.24.0-24.69.21.s390x.rpm (installed
> size 2.37 MB)
> F12=0000000000000000 F13=0000000000000000 F14=0000000000000000
> F15=0000000000000000
> 
>   Installing Packages... (Remaining: 2.37 MB / 00:05)
> 
> C00=0000000014354e12 C01=00000000007a1007 C02=0000000000011140
> C03=0000000000000000
> 
> C04=0000000000000a5a C05=0000000000011140 C06=0000000010000000
> C07=000000003d6cc1c7
> C08=0000000000000000 C09=0000000000000000 C10=0000000000000000
> C11=0000000000000000
> C12=0000000000000000 C13=000000003d6cc1c7 C14=00000000db000000
> C15=0000000000000000
> 
> PSW=mask 0704000180000000 addr 0000000000110732 cc 01
> 
> Andreas
> 
> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [TestDays] s390x emulation error

[Alexander Graf]

On 10.11.2011, at 10:32, Andreas Färber wrote:

> Am 09.11.2011 23:53, schrieb Andreas Färber:
>> I get the following error:
>> 
>> $ s390x-softmmu/qemu-system-s390x
>> qemu-system-s390x: Guest moved used index from 0 to 47802
> 
> Now with -kernel and -drive if=virtio on s390-next I got the following
> fatal error during installation of SLES 11 SP2 Beta8:
> 
> qemu: fatal: EXECUTE on instruction prefix 0xdc00 not implemented

Could you please try out the following patch and see if that fixes the issue for you?


diff --git a/target-s390x/op_helper.c b/target-s390x/op_helper.c
index 137bae7..5ddc7b9 100644
--- a/target-s390x/op_helper.c
+++ b/target-s390x/op_helper.c
@@ -636,6 +636,9 @@ uint32_t HELPER(ex)(uint32_t cc, uint64_t v1, uint64_t addr, uint64_t ret)
         case 0x700:
             cc = helper_xc(l, get_address(0, b1, d1), get_address(0, b2, d2));
             break;
+        case 0xc00:
+            helper_tr(l, get_address(0, b1, d1), get_address(0, b2, d2));
+            break;
         default:
             goto abort;
             break;


Alex

Re: [Qemu-devel] [TestDays] s390x emulation error

[Andreas Färber]

Am 18.11.2011 16:33, schrieb Alexander Graf:
> 
> On 10.11.2011, at 10:32, Andreas Färber wrote:
> 
>> Am 09.11.2011 23:53, schrieb Andreas Färber:
>>> I get the following error:
>>>
>>> $ s390x-softmmu/qemu-system-s390x
>>> qemu-system-s390x: Guest moved used index from 0 to 47802
>>
>> Now with -kernel and -drive if=virtio on s390-next I got the following
>> fatal error during installation of SLES 11 SP2 Beta8:
>>
>> qemu: fatal: EXECUTE on instruction prefix 0xdc00 not implemented
> 
> Could you please try out the following patch and see if that fixes the issue for you?
> 
> 
> diff --git a/target-s390x/op_helper.c b/target-s390x/op_helper.c
> index 137bae7..5ddc7b9 100644
> --- a/target-s390x/op_helper.c
> +++ b/target-s390x/op_helper.c
> @@ -636,6 +636,9 @@ uint32_t HELPER(ex)(uint32_t cc, uint64_t v1, uint64_t addr, uint64_t ret)
>          case 0x700:
>              cc = helper_xc(l, get_address(0, b1, d1), get_address(0, b2, d2));
>              break;
> +        case 0xc00:
> +            helper_tr(l, get_address(0, b1, d1), get_address(0, b2, d2));
> +            break;
>          default:
>              goto abort;
>              break;

Tested-by: Andreas Färber <afaerber@suse.de>

Compiles and gets me further through the install.


Now for SLES 11 SP2 Beta9 I get:

 YaST2 - installation @ 10.0.2.15



 Switching to the installed system has failed

 More information can be found near the end of the
'/var/log/YaST2/y2log' file.
  ┌Checking the Installed
System...──────────────────────────────────────────┐
  │Checking whether RPM package gzip-1.3.12-69.19.1 is installed...
Passed   │
┬ │Checking whether RPM package initviocons-0.5-55.21 is installed...
Passed │
│ │Checking whether RPM package pciutils-3.0.1-2.26.10 is installed...
Passed┴
  │
     │
┬ │=== Running complex check on package yast2-packager... ===
     │
┬ │Checking whether RPM package yast2-packager is installed... Passed
     │
┬ │Checking what requires RPM package yast2-packager... Passed
     │
┬ │Checking whether RPM package coreutils-8.12-6.15.13 is installed...
Passed│
│ │Checking whether RPM package perl-base-5.10.0-64.55.1 is installed...
Pass┴
  │~ed
     │
  │
     │
┬ │=== Checking YaST log file /mnt//var/log/YaST2//y2log... ===
     │
│ │Opening file... Failed
     ┴
  │
     │
  │This is worth reporting a bug at http://bugzilla.novell.com/.
     ┬
  │Please, attach also all YaST logs stored in the '/var/log/YaST2/'
director┴

└──────────────────────────────────────────────────────────────────────────┘



 9 0 OK                              all


And after pressing F3, I got a screen saying zipl could not be installed.

Will try to re-check with Beta8 or to compare Beta9 on z/VM.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg