Re: [Fdutils] DMA cache consistency bug introduced in 2.6.28

[Mark Hounschell <dmarkh@cfl.rr.com>]

On 01/11/2010 07:19 PM, Pallipadi, Venkatesh wrote:
> On Fri, 2010-01-08 at 09:42 -0800, Mark Hounschell wrote:
>> On 12/23/2009 03:30 PM, Pallipadi, Venkatesh wrote:
>>
>>>>> Can you try this one line patch either on .28 or .32 (with /proc/interrupts
>>>>> output).
>>>>> This disables hpet2 and lapic timer should then be used on CPU 0. If things
>>>>> work with this test patch, we will know that the failure is somehow related
>>>>> to HPET usage in MSI mode.
>>>>>
>>>>> Thanks,
>>>>> Venki
>>>>>
>>>>> Reduce the rating of percpu hpet timer
>>>>>
>>>>> Signed-off-by: Venkatesh Pallipadi <venkatesh.pallipadi@intel.com>
>>>>> ---
>>>>>  arch/x86/kernel/hpet.c |    2 +-
>>>>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>>>>
>>>>> diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c
>>>>> index cafb1c6..f89d17a 100644
>>>>> --- a/arch/x86/kernel/hpet.c
>>>>> +++ b/arch/x86/kernel/hpet.c
>>>>> @@ -480,7 +480,7 @@ static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu)
>>>>>  	hpet_setup_irq(hdev);
>>>>>  	evt->irq = hdev->irq;
>>>>>  
>>>>> -	evt->rating = 110;
>>>>> +	evt->rating = 40;
>>>>>  	evt->features = CLOCK_EVT_FEAT_ONESHOT;
>>>>>  	if (hdev->flags & HPET_DEV_PERI_CAP)
>>>>>  		evt->features |= CLOCK_EVT_FEAT_PERIODIC;
>>>>
>>>> That made it work. Used 2.6.32.2
>>>>
>>>> cat /proc/interrupts
>>>>            CPU0       CPU1       CPU2       CPU3
>>>>   0:         82          0          0          1   IO-APIC-edge      timer
>>>>   1:          0          0          0         67   IO-APIC-edge      i8042
>>>>   3:          0          0          0          6   IO-APIC-edge
>>>>   4:          0          0          0          4   IO-APIC-edge
>>>>   6:          0          0          0          4   IO-APIC-edge      floppy
>>>>   8:          0          0          0          8   IO-APIC-edge      rtc0
>>>>   9:          0          0          0          0   IO-APIC-fasteoi   acpi
>>>>  12:          0          0         10       1519   IO-APIC-edge      i8042
>>>>  14:          0          0         39      10995   IO-APIC-edge
>>>> pata_atiixp
>>>>  15:          0          0          3        391   IO-APIC-edge
>>>> pata_atiixp
>>>>  16:          0          0          2        606   IO-APIC-fasteoi
>>>> aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel, Digi DBX2, ni-pci-gpib
>>>>  17:          0          0          0          3   IO-APIC-fasteoi
>>>> ehci_hcd:usb1, parport0, ni-pci-gpib
>>>>  18:          0          0         10       2168   IO-APIC-fasteoi
>>>> ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, Digi DBX2, nvidia
>>>>  19:          0          0          0        130   IO-APIC-fasteoi
>>>> aic7xxx, ehci_hcd:usb2, ttySLG0, eth1
>>>>  22:          0          0          8       1151   IO-APIC-fasteoi   ahci
>>>>  24:          0          0          0          0  HPET_MSI-edge      hpet2
>>>>  29:          0          0          0         48   PCI-MSI-edge
>>>> sky2@pci:0000:04:00.0
>>>> NMI:          0          0          0          0   Non-maskable interrupts
>>>> LOC:      34842      30177      29672      29632   Local timer interrupts
>>>> SPU:          0          0          0          0   Spurious interrupts
>>>> PMI:          0          0          0          0   Performance monitoring
>>>> interrupts
>>>> PND:          0          0          0          0   Performance pending work
>>>> RES:      17501      20449      16670      11224   Rescheduling interrupts
>>>> CAL:      10554       2336       1102       1071   Function call interrupts
>>>> TLB:        364        562        753        468   TLB shootdowns
>>>> ERR:          0
>>>> MIS:          0
>>>>
>>>>
>>>> # fdformat /dev/fd0u1440
>>>> Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB.
>>>> Formatting ... done
>>>> Verifying ... done
>>>
>>> Hmmm.. Thats very interesting indeed.
>>>
>>> That clearly says that HPET MSI interrupts somehow is causing some
>>> caching side effect in the chipset that results in this floppy dma
>>> failure.
>>>
>>> Here's is what we have until now.
>>> IRQ 0 is based on HPET legacy interrupt and HPET device is also capable
>>> of MSI on this platform. So we also have a percpu hpet (hpet2 tied to
>>> CPU0). percpu hpet was added to avoid the usage of IRQ0+LAPIC broadcast
>>> in cases where LAPIC timer will stop working in deep C-state. As we have
>>> only one HPET channel free for percpu HPET, we only have hpet2 tied to
>>> CPU 0 and other CPUs still have to go through IRQ0+LAPIC broadcast with
>>> deep C-state.
>>>
>>> One problem here is that percpu hpet should only get used when LAPIC
>>> cannot be used (that is when CPU enters deep C-state). Using hpet2 in
>>> place of LAPIC timer even when deep C-state is not supported is not
>>> right in terms of performance. We need some changes here to fix that
>>> [Problem 1].
>>>
>>> But, that still does not explain why we are seeing this problem in the
>>> first place. I mean, using hpet2 is not optimal, but should not have
>>> functionality issues like this. Even fixing [Problem 1] above, we may
>>> see this problem on some other platform that supports deep C-state and
>>> so has hpet2 enabled for a valid reason.
>>>
>>> Also, I am not sure whether the problem also happens if legacy HPET
>>> interrupts are used during run time in place of LAPIC timer (May be
>>> worth to try this with a simple test patch, let me think about it). In
>>> this case, legacy HPET interrupt rightly goes quiet after boot, giving
>>> priority to LAPIC timer.
>>>
>>> With hpet MSI interrupts, we do a write followed by read of HPET
>>> memmapped register to set a HPET channel timeout + read of global HPET
>>> timer. This happens on every timer interrupt on CPU 0. And we also have
>>> MSI interrupt being delivered to CPU 0. I cannot think of any reason why
>>> this can break dma. We can probably try adding some dummy HPET read
>>> after dma write, to see if that flushes things properly.
>>>
>>
>> Haven't seen any activity on this thread in a while. Just curious, are we
>> still working this?
>> Is there anything else I can do to help?
> 
> Sorry for not following up on this. We have narrowed this down to HPET
> MSI and floppy DMA. I still don't know how HPET MSI interrupts are
> breaking floppy DMA.
> 
> You are seeing the problem on two different systems. Correct? Do you
> have any system where this works with HPET MSI enabled?
> 

I see the problem on every system in which the HPET2 shows up in
/proc/interrupts. The machines that work with HPET enabled don't show HPET
at all in /proc/interrupts.  I have some of each. All the boxes that fail
here use the (AMD) 790x series chip sets.

> Couple of options on how we can go about this one:
> 1) Change the HPET-MSI change to not get activated when there are no
> C-states with LAPIC stoppage involved. This will resolve the problem on
> the systems you reported as there are no deep C-states. But, I fear that
> with the actual problem unresolved, we may hit it in future with this or
> some other platform having same issue with CPUs that support deep
> C-state.
> 2) Try this testcase on few other platforms that support HPET-MSI and
> deep C-states and check how widespread the problem is and then add a
> whitelist-blacklist for HPET MSI usage.
> 
> I think, for 2.6.33 option 1 is better. Will work on that and send in
> patches for you test.
> 

OK, thanks
Mark