Re: smp.c && barriers (Was: [PATCH 1/4] generic-smp: remove single ipi fallback for smp_call_function_many())

Ingo Molnar
2009-02-25T12:48:39Z

* Nick Piggin <npiggin@suse.de> wrote:

> On Fri, Feb 20, 2009 at 10:56:54AM -0800, Suresh B wrote:
> > On Fri, 2009-02-20 at 02:56 -0800, Ingo Molnar wrote:
> > > * Suresh Siddha <suresh.b.siddha@intel.com> wrote:
> > >
> > > > On Thu, 2009-02-19 at 04:20 -0800, Ingo Molnar wrote:
> > > > > Could you please refresh this patch to latest tip:master? The
> > > > > APIC drivers moved to arch/x86/kernel/apic/.
> > > >
> > > > Appended the refreshed patch. Thanks.
> > >
> > > thanks. Two details i noticed:
> > >
> > > Firstly:
> > >
> > > > +++ b/arch/x86/kernel/apic/x2apic_cluster.c
> > > > +++ b/arch/x86/kernel/apic/x2apic_phys.c
> > >
> > > how about x2apic_uv.c? It uses uv_write_global_mmr64() in its
> > > IPI sending method, which uses:
> > >
> > > static inline void uv_write_global_mmr64(int pnode, unsigned long offset,
> > > unsigned long val)
> > > {
> > > *uv_global_mmr64_address(pnode, offset) = val;
> > > }
> > >
> > > which uses ->mmr_base, which is mapped via:
> > >
> > > init_extra_mapping_uc(UV_LOCAL_MMR_BASE, UV_LOCAL_MMR_SIZE);
> > >
> > > so it should be fine because uncached - but at minimum we should
> > > put a comment into x2apic_uv.c that the generic IPI code relies
> > > on the lowlevel code serializing - i.e. relies on the UC PAT
> > > attribute.
> >
> > Oops. I forgot to mention that and copy Jack to confirm our
> > understanding. I can send a followup patch adding the comments for UV.
> > Jack, are you ok?
> >
> > >
> > > Secondly, you added smp_mb(), which will translate to an MFENCE.
> > >
> > > But in theory it should be enough to have a wmb() here. [Note,
> > > not an smp_wmb() that i suggested before.] That will translate
> > > to an SFENCE - which will serialize writes but still allows
> > > reads/prefetches to pass.
> > >
> > > So the question is, is an SFENCE there enough to serialize the
> > > WRMSR with previous memory-writes? It's not specified in the
> > > x2apic docs as far as i could see.
> >
> > No. sfence is not enough (wrmsr to x2apic regs was still passing ahead).
> > We have done a small experiment to demonstrate the issue and adding
> > mfence fixes the issue but not sfence. We need a serializing instruction
> > or mfence. I will try to get the SDM updated.
>
> Just if I may add something -- I would probably slightly
> prefer if you explicitly used an sfence or other serializing
> instruction rather than smp_mb(). Maybe call it wrmsr_fence()
> or something. Apart from being self documenting, and less
> confusing (wrmsr is not part of normal ordering), I assume you
> technically also need it on UP systems?

Hm, UP systems shouldnt worry too much about sending IPIs to
other CPUs, right? We optimize out same-CPU IPIs already so that
bit should not matter.

Agreed on the self-documentation aspect.

Ingo