Re: [Fdutils] DMA cache consistency bug introduced in 2.6.28

[Mark Hounschell <markh@compro.net>]

On 12/23/2009 03:30 PM, Pallipadi, Venkatesh wrote:

>>> Can you try this one line patch either on .28 or .32 (with /proc/interrupts
>>> output).
>>> This disables hpet2 and lapic timer should then be used on CPU 0. If things
>>> work with this test patch, we will know that the failure is somehow related
>>> to HPET usage in MSI mode.
>>>
>>> Thanks,
>>> Venki
>>>
>>> Reduce the rating of percpu hpet timer
>>>
>>> Signed-off-by: Venkatesh Pallipadi <venkatesh.pallipadi@intel.com>
>>> ---
>>>  arch/x86/kernel/hpet.c |    2 +-
>>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>>
>>> diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c
>>> index cafb1c6..f89d17a 100644
>>> --- a/arch/x86/kernel/hpet.c
>>> +++ b/arch/x86/kernel/hpet.c
>>> @@ -480,7 +480,7 @@ static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu)
>>>  	hpet_setup_irq(hdev);
>>>  	evt->irq = hdev->irq;
>>>  
>>> -	evt->rating = 110;
>>> +	evt->rating = 40;
>>>  	evt->features = CLOCK_EVT_FEAT_ONESHOT;
>>>  	if (hdev->flags & HPET_DEV_PERI_CAP)
>>>  		evt->features |= CLOCK_EVT_FEAT_PERIODIC;
>>
>> That made it work. Used 2.6.32.2
>>
>> cat /proc/interrupts
>>            CPU0       CPU1       CPU2       CPU3
>>   0:         82          0          0          1   IO-APIC-edge      timer
>>   1:          0          0          0         67   IO-APIC-edge      i8042
>>   3:          0          0          0          6   IO-APIC-edge
>>   4:          0          0          0          4   IO-APIC-edge
>>   6:          0          0          0          4   IO-APIC-edge      floppy
>>   8:          0          0          0          8   IO-APIC-edge      rtc0
>>   9:          0          0          0          0   IO-APIC-fasteoi   acpi
>>  12:          0          0         10       1519   IO-APIC-edge      i8042
>>  14:          0          0         39      10995   IO-APIC-edge
>> pata_atiixp
>>  15:          0          0          3        391   IO-APIC-edge
>> pata_atiixp
>>  16:          0          0          2        606   IO-APIC-fasteoi
>> aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel, Digi DBX2, ni-pci-gpib
>>  17:          0          0          0          3   IO-APIC-fasteoi
>> ehci_hcd:usb1, parport0, ni-pci-gpib
>>  18:          0          0         10       2168   IO-APIC-fasteoi
>> ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, Digi DBX2, nvidia
>>  19:          0          0          0        130   IO-APIC-fasteoi
>> aic7xxx, ehci_hcd:usb2, ttySLG0, eth1
>>  22:          0          0          8       1151   IO-APIC-fasteoi   ahci
>>  24:          0          0          0          0  HPET_MSI-edge      hpet2
>>  29:          0          0          0         48   PCI-MSI-edge
>> sky2@pci:0000:04:00.0
>> NMI:          0          0          0          0   Non-maskable interrupts
>> LOC:      34842      30177      29672      29632   Local timer interrupts
>> SPU:          0          0          0          0   Spurious interrupts
>> PMI:          0          0          0          0   Performance monitoring
>> interrupts
>> PND:          0          0          0          0   Performance pending work
>> RES:      17501      20449      16670      11224   Rescheduling interrupts
>> CAL:      10554       2336       1102       1071   Function call interrupts
>> TLB:        364        562        753        468   TLB shootdowns
>> ERR:          0
>> MIS:          0
>>
>>
>> # fdformat /dev/fd0u1440
>> Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB.
>> Formatting ... done
>> Verifying ... done
> 
> Hmmm.. Thats very interesting indeed.
> 
> That clearly says that HPET MSI interrupts somehow is causing some
> caching side effect in the chipset that results in this floppy dma
> failure.
> 
> Here's is what we have until now.
> IRQ 0 is based on HPET legacy interrupt and HPET device is also capable
> of MSI on this platform. So we also have a percpu hpet (hpet2 tied to
> CPU0). percpu hpet was added to avoid the usage of IRQ0+LAPIC broadcast
> in cases where LAPIC timer will stop working in deep C-state. As we have
> only one HPET channel free for percpu HPET, we only have hpet2 tied to
> CPU 0 and other CPUs still have to go through IRQ0+LAPIC broadcast with
> deep C-state.
> 
> One problem here is that percpu hpet should only get used when LAPIC
> cannot be used (that is when CPU enters deep C-state). Using hpet2 in
> place of LAPIC timer even when deep C-state is not supported is not
> right in terms of performance. We need some changes here to fix that
> [Problem 1].
> 
> But, that still does not explain why we are seeing this problem in the
> first place. I mean, using hpet2 is not optimal, but should not have
> functionality issues like this. Even fixing [Problem 1] above, we may
> see this problem on some other platform that supports deep C-state and
> so has hpet2 enabled for a valid reason.
> 
> Also, I am not sure whether the problem also happens if legacy HPET
> interrupts are used during run time in place of LAPIC timer (May be
> worth to try this with a simple test patch, let me think about it). In
> this case, legacy HPET interrupt rightly goes quiet after boot, giving
> priority to LAPIC timer.
> 
> With hpet MSI interrupts, we do a write followed by read of HPET
> memmapped register to set a HPET channel timeout + read of global HPET
> timer. This happens on every timer interrupt on CPU 0. And we also have
> MSI interrupt being delivered to CPU 0. I cannot think of any reason why
> this can break dma. We can probably try adding some dummy HPET read
> after dma write, to see if that flushes things properly.
> 

Haven't seen any activity on this thread in a while. Just curious, are we
still working this?
Is there anything else I can do to help?

Thanks
Mark