[syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    de10553fce40 Merge tag 'x86-apic-2023-04-24' of git://git...
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=14bdae68280000
kernel config:  https://syzkaller.appspot.com/x/.config?x=975b8311f6b96bca
dashboard link: https://syzkaller.appspot.com/bug?extid=d692037148a8169fc9dd
compiler:       arm-linux-gnueabi-gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+d692037148a8169fc9dd@syzkaller.appspotmail.com

8<--- cut here ---
Unable to handle kernel NULL pointer dereference at virtual address 000005fc when read
[000005fc] *pgd=80000080004003, *pmd=00000000
Internal error: Oops: 206 [#1] PREEMPT SMP ARM
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-syzkaller #0
Hardware name: ARM-Versatile Express
Insufficient stack space to handle exception!
Task stack:     [0xdf85c000..0xdf85e000]
IRQ stack:      [0xdf804000..0xdf806000]
Overflow stack: [0x828ae000..0x828af000]
Internal error: kernel stack overflow: 0 [#2] PREEMPT SMP ARM
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-syzkaller #0
Hardware name: ARM-Versatile Express
PC is at __dabt_svc+0x14/0x60 arch/arm/kernel/entry-armv.S:210
LR is at vsnprintf+0x378/0x408 lib/vsprintf.c:2862
pc : [<80200a74>]    lr : [<817ad5d8>]    psr: 00000193
sp : df804028  ip : df805868  fp : df805864
r10: 00000060  r9 : ffffffff  r8 : 00000010
r7 : 00000020  r6 : 00000004  r5 : ffffffff  r4 : df805960
r3 : ffffffff  r2 : 00000040  r1 : ffffffff  r0 : 8264d250
Flags: nzcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 30c5387d  Table: 80003000  DAC: 00000000
Register r0 information:
8<--- cut here ---
Unable to handle kernel NULL pointer dereference at virtual address 000001ff when read
[000001ff] *pgd=80000080004003, *pmd=00000000
Internal error: Oops: 206 [#3] PREEMPT SMP ARM
Modules linked in:
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-syzkaller #0
Hardware name: ARM-Versatile Express
PC is at __find_vmap_area mm/vmalloc.c:841 [inline]
PC is at find_vmap_area mm/vmalloc.c:1862 [inline]
PC is at find_vm_area mm/vmalloc.c:2571 [inline]
PC is at vmalloc_dump_obj+0x38/0xb4 mm/vmalloc.c:4108
LR is at __raw_spin_lock include/linux/spinlock_api_smp.h:132 [inline]
LR is at _raw_spin_lock+0x18/0x58 kernel/locking/spinlock.c:154
pc : [<8046b2f0>]    lr : [<817db0f4>]    psr: 20000193
sp : 828aeef8  ip : 828aeee0  fp : 828aef0c
r10: 828f3980  r9 : 8241c964  r8 : 8264d41c
r7 : 60000193  r6 : 00000001  r5 : 8264e000  r4 : 00000207
r3 : 80216bd4  r2 : 00001d4c  r1 : 00000000  r0 : 00000001
Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 30c5387d  Table: 80003000  DAC: 00000000


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Miguel Ojeda]

Hi syzbot engineers,

On Tue, Apr 25, 2023 at 10:06 PM syzbot
<syzbot+d692037148a8169fc9dd@syzkaller.appspotmail.com> wrote:
>
> HEAD commit:    de10553fce40 Merge tag 'x86-apic-2023-04-24' of git://git...
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=14bdae68280000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=975b8311f6b96bca
> dashboard link: https://syzkaller.appspot.com/bug?extid=d692037148a8169fc9dd
> compiler:       arm-linux-gnueabi-gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> userspace arch: arm

I am not sure what triggered the bot to consider Rust here -- the
config does not enable it.

What am I missing?

Cheers,
Miguel

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Dmitry Vyukov]

On Tue, 25 Apr 2023 at 23:36, Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> Hi syzbot engineers,
>
> On Tue, Apr 25, 2023 at 10:06 PM syzbot
> <syzbot+d692037148a8169fc9dd@syzkaller.appspotmail.com> wrote:
> >
> > HEAD commit:    de10553fce40 Merge tag 'x86-apic-2023-04-24' of git://git...
> > git tree:       upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=14bdae68280000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=975b8311f6b96bca
> > dashboard link: https://syzkaller.appspot.com/bug?extid=d692037148a8169fc9dd
> > compiler:       arm-linux-gnueabi-gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> > userspace arch: arm
>
> I am not sure what triggered the bot to consider Rust here -- the
> config does not enable it.
>
> What am I missing?

Hi Miguel,

The crash is in lib/vsprintf.c and:

$ scripts/get_maintainer.pl -f lib/vsprintf.c
...
rust-for-linux@vger.kernel.org (open list:RUST)
...

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Miguel Ojeda]

Hi Dmitry,

On Wed, Apr 26, 2023 at 8:34 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> The crash is in lib/vsprintf.c and:
>
> $ scripts/get_maintainer.pl -f lib/vsprintf.c
> ...
> rust-for-linux@vger.kernel.org (open list:RUST)
> ...

Ah, yes, thanks!

For the moment it is fine, since there are not many reports nor
keyword instances, but perhaps in the future we could consider
filtering out `RUST` on the bot side if `CONFIG_RUST=n` and the match
was based on `K:` (via diff with `--no-keywords`?).

Cheers,
Miguel

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Dmitry Vyukov]

On Wed, 26 Apr 2023 at 12:09, Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> Hi Dmitry,
>
> On Wed, Apr 26, 2023 at 8:34 AM Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> > The crash is in lib/vsprintf.c and:
> >
> > $ scripts/get_maintainer.pl -f lib/vsprintf.c
> > ...
> > rust-for-linux@vger.kernel.org (open list:RUST)
> > ...
>
> Ah, yes, thanks!
>
> For the moment it is fine, since there are not many reports nor
> keyword instances, but perhaps in the future we could consider

In which of the dozens of kernel testing systems? ;)
And also in heads of thousands of kernel developers and users?
All of them use get_maintainer.pl.


> filtering out `RUST` on the bot side if `CONFIG_RUST=n` and the match
> was based on `K:` (via diff with `--no-keywords`?).
>
> Cheers,
> Miguel

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Miguel Ojeda]

On Wed, Apr 26, 2023 at 12:12 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> In which of the dozens of kernel testing systems? ;)
> And also in heads of thousands of kernel developers and users?
> All of them use get_maintainer.pl.

I am aware, but `get_maintainer.pl` is fine as it is -- we still want
to know about things that touch things that mention Rust in general,
so that we can possibly be helpful to others, especially early on.

However, if a bot is testing the kernel with Rust actually disabled at
runtime, what I am saying is that the chance that it has something to
do with Rust is quite low, especially if matched via `K:` rather than
`F:`. Thus my request.

Now, it could be nice to have some logic like that in
`get_maintainer.pl` encoded for all bots to filter things out based on
the kernel config and the type of match; but otherwise, yes, the bots
would need to add the logic.

Cc'ing Joe in case this is already possible in `get_maintainer.pl` or
whether there could be a better approach.

Cheers,
Miguel

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Dmitry Vyukov]

On Wed, 26 Apr 2023 at 12:30, Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
> > In which of the dozens of kernel testing systems? ;)
> > And also in heads of thousands of kernel developers and users?
> > All of them use get_maintainer.pl.
>
> I am aware, but `get_maintainer.pl` is fine as it is -- we still want
> to know about things that touch things that mention Rust in general,
> so that we can possibly be helpful to others, especially early on.
>
> However, if a bot is testing the kernel with Rust actually disabled at
> runtime, what I am saying is that the chance that it has something to
> do with Rust is quite low, especially if matched via `K:` rather than
> `F:`. Thus my request.
>
> Now, it could be nice to have some logic like that in
> `get_maintainer.pl` encoded for all bots to filter things out based on
> the kernel config and the type of match; but otherwise, yes, the bots
> would need to add the logic.
>
> Cc'ing Joe in case this is already possible in `get_maintainer.pl` or
> whether there could be a better approach.

I understand your intentions and they make sense.
But adding this logic to syzbot won't help thousands of users of
get_maintainer.pl and dozens of other testing systems. There also will
be a bit of get_maintainer.pl inside of syzbot code, so now all kernel
developers will need to be aware of it and also submit changes to
syzbot when they want to change maintainers logic.

I think this also equally applies to all other users of K:.
And a number of them had similar complaints re how K; works.

I am thinking if K: should actually apply just to patches and be
ignored for source files?
If there are files that belong to "rust" (or "bpf" or any other user
of K:), then I think these should be just listed explicitly in the
subsystem (that should be a limited set of files that can be
enumerated with wildcards).
It's also reasonable to apply K: to patches.
But if a random source file happened to mention "rust" somewhere once,
I am not sure you want to be CCed on all issues in that file.
Does it sound reasonable?

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Miguel Ojeda]

On Wed, Apr 26, 2023 at 12:43 PM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> I understand your intentions and they make sense.

Thanks! I am glad you agree it may have some value -- please see below
for details.

> But adding this logic to syzbot won't help thousands of users of
> get_maintainer.pl and dozens of other testing systems. There also will

I haven't said otherwise -- as I said, I think it would be nice to
have this be part of the kernel itself. :)

> be a bit of get_maintainer.pl inside of syzbot code, so now all kernel
> developers will need to be aware of it and also submit changes to
> syzbot when they want to change maintainers logic.
>
> I think this also equally applies to all other users of K:.
> And a number of them had similar complaints re how K; works.

Yeah, I would imagine so.

> I am thinking if K: should actually apply just to patches and be
> ignored for source files?

I considered that -- for things like Rust, it could make sense, but
perhaps somebody is already using `K:` to match files they do care
about, rather than `F:`. So we would need to ask others, but I think
it is fine.

> If there are files that belong to "rust" (or "bpf" or any other user
> of K:), then I think these should be just listed explicitly in the
> subsystem (that should be a limited set of files that can be
> enumerated with wildcards).

Yes, at least for Rust, modulo omissions, we match files explicitly
with `F:`. We have a couple unimportant omissions, e.g.
`.rustfmt.toml`, but I can send a patch.

Personally, I have always seen `F:` files (and `N:`-matched ones) as
having more weight than `K:`-matched ones, i.e. I saw `K:` as more of
a "it depends on what it matches -- discretion needed".

From a quick look, most `K:`-using subsystems seem to list `F:` and
`N:` as I would expect.

> It's also reasonable to apply K: to patches.

Yes, definitely, for Rust, that is our main use case, i.e. it is
mainly why we wanted to have the `K:` entry: to catch changes to
things that are tagged with "Rust" in C files (early on, at least).

It is particularly important for us, since we are also considering
having more of these annotations in the future.

> But if a random source file happened to mention "rust" somewhere once,
> I am not sure you want to be CCed on all issues in that file.
> Does it sound reasonable?

For Rust, yes, that would probably work for us. Not sure for all
subsystems using `K:`, though.

Having said that, I suggested including the kernel config too in this
decision (i.e. not for the patches case, but for testers finding
runtime issues), because it adds information: it leaves reports out
when something is not even enabled but matched via `K:`, but still
allows a Cc when matched via `K:` and enabled. It is, of course, still
potentially a false positive, but some subsystems may want to hear
about those.

For instance, for Rust, this would be fine early on, since we don't
expect many to have `RUST=y` to begin with, and thus the odd false
positive report via `K:` is fine. Later on, this heuristic may change,
and we may not change those matches anymore (especially since, by
then, the goal is that subsystems would be taking care of their own
Rust bits).

This is what I was suggesting to then put in `get_maintainer.pl`, e.g.
a `--bot` option (or `--runtime`, or `--config-based-filtering`, or
similar) option. Then the bots can add that option on their side.

Thanks again for considering this!

Cheers,
Miguel

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Dmitry Vyukov]

On Wed, 26 Apr 2023 at 13:37, Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
> > I understand your intentions and they make sense.
>
> Thanks! I am glad you agree it may have some value -- please see below
> for details.
>
> > But adding this logic to syzbot won't help thousands of users of
> > get_maintainer.pl and dozens of other testing systems. There also will
>
> I haven't said otherwise -- as I said, I think it would be nice to
> have this be part of the kernel itself. :)
>
> > be a bit of get_maintainer.pl inside of syzbot code, so now all kernel
> > developers will need to be aware of it and also submit changes to
> > syzbot when they want to change maintainers logic.
> >
> > I think this also equally applies to all other users of K:.
> > And a number of them had similar complaints re how K; works.
>
> Yeah, I would imagine so.
>
> > I am thinking if K: should actually apply just to patches and be
> > ignored for source files?
>
> I considered that -- for things like Rust, it could make sense, but
> perhaps somebody is already using `K:` to match files they do care
> about, rather than `F:`. So we would need to ask others, but I think
> it is fine.
>
> > If there are files that belong to "rust" (or "bpf" or any other user
> > of K:), then I think these should be just listed explicitly in the
> > subsystem (that should be a limited set of files that can be
> > enumerated with wildcards).
>
> Yes, at least for Rust, modulo omissions, we match files explicitly
> with `F:`. We have a couple unimportant omissions, e.g.
> `.rustfmt.toml`, but I can send a patch.
>
> Personally, I have always seen `F:` files (and `N:`-matched ones) as
> having more weight than `K:`-matched ones, i.e. I saw `K:` as more of
> a "it depends on what it matches -- discretion needed".
>
> From a quick look, most `K:`-using subsystems seem to list `F:` and
> `N:` as I would expect.
>
> > It's also reasonable to apply K: to patches.
>
> Yes, definitely, for Rust, that is our main use case, i.e. it is
> mainly why we wanted to have the `K:` entry: to catch changes to
> things that are tagged with "Rust" in C files (early on, at least).
>
> It is particularly important for us, since we are also considering
> having more of these annotations in the future.
>
> > But if a random source file happened to mention "rust" somewhere once,
> > I am not sure you want to be CCed on all issues in that file.
> > Does it sound reasonable?
>
> For Rust, yes, that would probably work for us. Not sure for all
> subsystems using `K:`, though.
>
> Having said that, I suggested including the kernel config too in this
> decision (i.e. not for the patches case, but for testers finding
> runtime issues), because it adds information: it leaves reports out
> when something is not even enabled but matched via `K:`, but still
> allows a Cc when matched via `K:` and enabled. It is, of course, still
> potentially a false positive, but some subsystems may want to hear
> about those.
>
> For instance, for Rust, this would be fine early on, since we don't
> expect many to have `RUST=y` to begin with, and thus the odd false
> positive report via `K:` is fine. Later on, this heuristic may change,
> and we may not change those matches anymore (especially since, by
> then, the goal is that subsystems would be taking care of their own
> Rust bits).
>
> This is what I was suggesting to then put in `get_maintainer.pl`, e.g.
> a `--bot` option (or `--runtime`, or `--config-based-filtering`, or
> similar) option. Then the bots can add that option on their side.
>
> Thanks again for considering this!

Was looking at what's the status of this, and if we need to file a
feature request for syzbot.
Turns out Joe Perches fixed this a bit ago by adding
--keywords-in-file flag (off by default):
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=71ca5ee18708c1f9f086e20ac0a657009bcfe43a

I think that's the right thing to do. syzbot won't be confused by
widely matched K: patterns with sending reports (since it runs
get_maintainers.pl on files).

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Sergey Senozhatsky]

On (23/04/25 13:06), syzbot wrote:
> 8<--- cut here ---
> Unable to handle kernel NULL pointer dereference at virtual address 000005fc when read
> [000005fc] *pgd=80000080004003, *pmd=00000000
> Internal error: Oops: 206 [#1] PREEMPT SMP ARM
> Modules linked in:
> CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-syzkaller #0
> Hardware name: ARM-Versatile Express

> Insufficient stack space to handle exception!

So much stuff is going on there.

> Task stack:     [0xdf85c000..0xdf85e000]
> IRQ stack:      [0xdf804000..0xdf806000]
> Overflow stack: [0x828ae000..0x828af000]
> Internal error: kernel stack overflow: 0 [#2] PREEMPT SMP ARM
> Modules linked in:
> CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-syzkaller #0
> Hardware name: ARM-Versatile Express
> PC is at __dabt_svc+0x14/0x60 arch/arm/kernel/entry-armv.S:210
> LR is at vsnprintf+0x378/0x408 lib/vsprintf.c:2862
> pc : [<80200a74>]    lr : [<817ad5d8>]    psr: 00000193
> sp : df804028  ip : df805868  fp : df805864
> r10: 00000060  r9 : ffffffff  r8 : 00000010
> r7 : 00000020  r6 : 00000004  r5 : ffffffff  r4 : df805960
> r3 : ffffffff  r2 : 00000040  r1 : ffffffff  r0 : 8264d250
> Flags: nzcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
> Control: 30c5387d  Table: 80003000  DAC: 00000000
> Register r0 information:
> 8<--- cut here ---
> Unable to handle kernel NULL pointer dereference at virtual address 000001ff when read
> [000001ff] *pgd=80000080004003, *pmd=00000000
> Internal error: Oops: 206 [#3] PREEMPT SMP ARM
> Modules linked in:
> CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-syzkaller #0
> Hardware name: ARM-Versatile Express
> PC is at __find_vmap_area mm/vmalloc.c:841 [inline]
> PC is at find_vmap_area mm/vmalloc.c:1862 [inline]
> PC is at find_vm_area mm/vmalloc.c:2571 [inline]
> PC is at vmalloc_dump_obj+0x38/0xb4 mm/vmalloc.c:4108
> LR is at __raw_spin_lock include/linux/spinlock_api_smp.h:132 [inline]
> LR is at _raw_spin_lock+0x18/0x58 kernel/locking/spinlock.c:154

Not sure if I can make sense out of this.

Re: [syzbot] upstream boot error: BUG: unable to handle kernel NULL pointer dereference in __dabt_svc

[Dmitry Vyukov]

On Wed, 26 Apr 2023 at 06:49, Sergey Senozhatsky
<senozhatsky@chromium.org> wrote:
>
> On (23/04/25 13:06), syzbot wrote:
> > 8<--- cut here ---
> > Unable to handle kernel NULL pointer dereference at virtual address 000005fc when read
> > [000005fc] *pgd=80000080004003, *pmd=00000000
> > Internal error: Oops: 206 [#1] PREEMPT SMP ARM
> > Modules linked in:
> > CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-syzkaller #0
> > Hardware name: ARM-Versatile Express
>
> > Insufficient stack space to handle exception!
>
> So much stuff is going on there.
>
> > Task stack:     [0xdf85c000..0xdf85e000]
> > IRQ stack:      [0xdf804000..0xdf806000]
> > Overflow stack: [0x828ae000..0x828af000]
> > Internal error: kernel stack overflow: 0 [#2] PREEMPT SMP ARM
> > Modules linked in:
> > CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-syzkaller #0
> > Hardware name: ARM-Versatile Express
> > PC is at __dabt_svc+0x14/0x60 arch/arm/kernel/entry-armv.S:210
> > LR is at vsnprintf+0x378/0x408 lib/vsprintf.c:2862
> > pc : [<80200a74>]    lr : [<817ad5d8>]    psr: 00000193
> > sp : df804028  ip : df805868  fp : df805864
> > r10: 00000060  r9 : ffffffff  r8 : 00000010
> > r7 : 00000020  r6 : 00000004  r5 : ffffffff  r4 : df805960
> > r3 : ffffffff  r2 : 00000040  r1 : ffffffff  r0 : 8264d250
> > Flags: nzcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
> > Control: 30c5387d  Table: 80003000  DAC: 00000000
> > Register r0 information:
> > 8<--- cut here ---
> > Unable to handle kernel NULL pointer dereference at virtual address 000001ff when read
> > [000001ff] *pgd=80000080004003, *pmd=00000000
> > Internal error: Oops: 206 [#3] PREEMPT SMP ARM
> > Modules linked in:
> > CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-syzkaller #0
> > Hardware name: ARM-Versatile Express
> > PC is at __find_vmap_area mm/vmalloc.c:841 [inline]
> > PC is at find_vmap_area mm/vmalloc.c:1862 [inline]
> > PC is at find_vm_area mm/vmalloc.c:2571 [inline]
> > PC is at vmalloc_dump_obj+0x38/0xb4 mm/vmalloc.c:4108
> > LR is at __raw_spin_lock include/linux/spinlock_api_smp.h:132 [inline]
> > LR is at _raw_spin_lock+0x18/0x58 kernel/locking/spinlock.c:154
>
> Not sure if I can make sense out of this.

+linux-arm-kernel@

I suspect this is some recent arch/arm related corruption.
There are also these similar boot crashes that started happening at
roughly the same time:
https://syzkaller.appspot.com/bug?id=4d697346183db2f86ba2f76acb7d66e7731f88df
https://syzkaller.appspot.com/bug?id=dcd98d67539fe4d0d28d2e655e510569eda6f4de