All of lore.kernel.org
 help / color / mirror / Atom feed
* [Bug 95031] New: [NVE4] Random GPU lockups
@ 2016-04-20  7:44 bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
  0 siblings, 1 reply; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-20  7:44 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 1499 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

            Bug ID: 95031
           Summary: [NVE4] Random GPU lockups
           Product: xorg
           Version: unspecified
          Hardware: Other
                OS: All
            Status: NEW
          Severity: normal
          Priority: medium
         Component: Driver/nouveau
          Assignee: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
          Reporter: lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
        QA Contact: xorg-team-go0+a7rfsptAfugRpC6u6w@public.gmane.org

Created attachment 123085
  --> https://bugs.freedesktop.org/attachment.cgi?id=123085&action=edit
nouveau bugs

Having random lockups on a GTX 660 Ti (NVE4), since kernel 4.1 I guess, using
DRI2.
[    0.267666] nouveau 0000:02:00.0: NVIDIA GK104 (0e4030a2)
[    0.378583] nouveau 0000:02:00.0: bios: version 80.04.4b.00.1a
[    0.379302] nouveau 0000:02:00.0: fb: 2048 MiB GDDR5

Now on gentoo ~amd64 using:

sys-kernel/gentoo-sources-4.5.1
x11-base/xorg-server-1.18.3
x11-drivers/xf86-video-nouveau-1.0.12

Also tried Karol Herbst reclocking branch v4
(https://github.com/karolherbst/nouveau/tree/stable_reclocking_kepler_v4),
reclocked to pstate 07 and tried all 3 boost states. All hang sooner or later.

Will continue testing other pstates and boost configurations.

Sometimes the kernel log becomes corrupted, but I managed to get a working log
(attached).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 2893 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
@ 2016-04-20  7:52   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-04-20 13:06   ` [Bug 95031] [NVE4] 660 Ti " bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
                     ` (8 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-20  7:52 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 402 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

--- Comment #1 from Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> ---
Forgot to add: did not try earlier kernels, so this behaviour might exist since
the card was supported. On Windows it works well.

It hangs on normal browsing or opening a video on VLC.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1089 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] 660 Ti Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
  2016-04-20  7:52   ` [Bug 95031] " bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
@ 2016-04-20 13:06   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-04-20 20:54   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
                     ` (7 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-20 13:06 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 465 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|[NVE4] Random GPU lockups   |[NVE4] 660 Ti Random GPU
                   |                            |lockups

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1063 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] 660 Ti Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
  2016-04-20  7:52   ` [Bug 95031] " bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-04-20 13:06   ` [Bug 95031] [NVE4] 660 Ti " bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
@ 2016-04-20 20:54   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-04-20 20:54   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
                     ` (6 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-20 20:54 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 308 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

--- Comment #2 from Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> ---
Different kernel log with some nouveau info.

So far tried pstate 07 with boost 0, 1 and 2.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1009 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] 660 Ti Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
                     ` (2 preceding siblings ...)
  2016-04-20 20:54   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
@ 2016-04-20 20:54   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-04-21  3:12   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
                     ` (5 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-20 20:54 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 346 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

--- Comment #3 from Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> ---
Created attachment 123098
  --> https://bugs.freedesktop.org/attachment.cgi?id=123098&action=edit
another log with different info

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1202 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] 660 Ti Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
                     ` (3 preceding siblings ...)
  2016-04-20 20:54   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
@ 2016-04-21  3:12   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-04-21  4:38   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
                     ` (4 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-21  3:12 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 715 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

--- Comment #4 from Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> ---
OK, some interesting findings.

07 pstate on Linux has:
core 324 MHz memory 648 MHz AC DC
GPU core:     +0.99 V
Increasing GPU core voltage to 1.09V has wielded a stable system so far.

On Windows, the idle state has (checked with gpu-z):
core 324 MHz memory 162MHz
GPU core voltage: 0.99V
It has never managed to hang.

So maybe 07 pstate on Linux has a memory clock too high for its voltage. Also,
as it is the lowest pstate, maybe memory clock could be reduced further to
162MHz (as in Windows).

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1416 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] 660 Ti Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
                     ` (4 preceding siblings ...)
  2016-04-21  3:12   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
@ 2016-04-21  4:38   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-04-21  6:23   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
                     ` (3 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-21  4:38 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 481 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

--- Comment #5 from Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> ---
Created attachment 123105
  --> https://bugs.freedesktop.org/attachment.cgi?id=123105&action=edit
vbios

Yea, finally managed to hang the system at 07 pstate with 1.09V (+0.1V), with a
corrupted kernel log as well. I dunno what else to do. I'm attaching the vbios.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1285 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] 660 Ti Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
                     ` (5 preceding siblings ...)
  2016-04-21  4:38   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
@ 2016-04-21  6:23   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-04-21  8:19   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
                     ` (2 subsequent siblings)
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-21  6:23 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 460 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

--- Comment #6 from Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> ---
Created attachment 123106
  --> https://bugs.freedesktop.org/attachment.cgi?id=123106&action=edit
kernel log 3

This time running pstate 0f with +0.1V, total 1.15V. Hangs, corrupts kernel log
and then starts flooding it with a different error.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1278 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] 660 Ti Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
                     ` (6 preceding siblings ...)
  2016-04-21  6:23   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
@ 2016-04-21  8:19   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-04-21 17:15   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-05-26 23:43   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-21  8:19 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 900 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

--- Comment #7 from Karol Herbst <freedesktop-lIBOoy2+GI7scQ4cX5LuPg@public.gmane.org> ---
(In reply to Lucas Ribeiro from comment #4)
> 
> So maybe 07 pstate on Linux has a memory clock too high for its voltage.
> Also, as it is the lowest pstate, maybe memory clock could be reduced
> further to 162MHz (as in Windows).

And maybe not. There are other issues which aren't exactly voltage related. If
such a high votlage won't help, then it is usually something else, we just have
to figure out what it is.

Also regarding the lower clocks: yeah I know that sometimes nvidia clocks
further down, but there is no real value in doing so if there is no voltage
information for those low clocks and it doesn't make any difference regarding
power consumption as far as I know.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1692 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] 660 Ti Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
                     ` (7 preceding siblings ...)
  2016-04-21  8:19   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
@ 2016-04-21 17:15   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  2016-05-26 23:43   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-04-21 17:15 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 1048 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

--- Comment #8 from Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> ---
(In reply to Karol Herbst from comment #7)
> (In reply to Lucas Ribeiro from comment #4)
> > 
> > So maybe 07 pstate on Linux has a memory clock too high for its voltage.
> > Also, as it is the lowest pstate, maybe memory clock could be reduced
> > further to 162MHz (as in Windows).
> 
> And maybe not. There are other issues which aren't exactly voltage related.
> If such a high votlage won't help, then it is usually something else, we
> just have to figure out what it is.
> 
> Also regarding the lower clocks: yeah I know that sometimes nvidia clocks
> further down, but there is no real value in doing so if there is no voltage
> information for those low clocks and it doesn't make any difference
> regarding power consumption as far as I know.

Thanks for clearing that up.

I'm out of ideas, should I capture a mmiotrace?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 1909 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Bug 95031] [NVE4] 660 Ti Random GPU lockups
       [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
                     ` (8 preceding siblings ...)
  2016-04-21 17:15   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
@ 2016-05-26 23:43   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
  9 siblings, 0 replies; 11+ messages in thread
From: bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ @ 2016-05-26 23:43 UTC (permalink / raw)
  To: nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW


[-- Attachment #1.1: Type: text/plain, Size: 755 bytes --]

https://bugs.freedesktop.org/show_bug.cgi?id=95031

Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #9 from Lucas Ribeiro <lucasout-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> ---
Running kernel 4.6 has improved the driver. I don't know what changed, but I
have yet to see lockups on this card. No out of tree patches applied.

Will post again if I experience a freeze.

Thanks!

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #1.2: Type: text/html, Size: 2140 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2016-05-26 23:43 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-04-20  7:44 [Bug 95031] New: [NVE4] Random GPU lockups bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
     [not found] ` <bug-95031-8800-V0hAGp6uBxMKqLRl/0Ahz6D7qz1kEfGD2LY78lusg7I@public.gmane.org/>
2016-04-20  7:52   ` [Bug 95031] " bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
2016-04-20 13:06   ` [Bug 95031] [NVE4] 660 Ti " bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
2016-04-20 20:54   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
2016-04-20 20:54   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
2016-04-21  3:12   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
2016-04-21  4:38   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
2016-04-21  6:23   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
2016-04-21  8:19   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
2016-04-21 17:15   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ
2016-05-26 23:43   ` bugzilla-daemon-CC+yJ3UmIYqDUpFQwHEjaQ

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.