[x86] fs, gs purpose & multicore prog

[x86] fs, gs purpose & multicore prog

[Eric Lacombe]

Hello,

I've some questions about IA-32e in Linux.

- What is the FS and GS segments role inside the kernel ?
(I was thinking about thread local storage)

- When I do a "mov %fs ..." instruction (in a module), it seems that %fs is 
equal to 0 (idem for %gs). Are these registers not always filled ?

- What is the purpose of MSR_FS_BASE and MSR_GS_BASE ? 
(I thought they were filled with "gdt[fs_entry].base")

- My last question is about the kernel programation of multi-core (or 
multiprocessor)
architecture. I don't see a lot of documentation about that on Internet.
Do you have some docs/urls about this topic.
Maybe someone can briefly explain how the execution flow are given to the 
different cores.

Thanks in advance.

        Eric

Re: [x86] fs, gs purpose & multicore prog

[Jeremy Fitzhardinge]

Eric Lacombe wrote:
> Hello,
>
> I've some questions about IA-32e in Linux.
>
> - What is the FS and GS segments role inside the kernel ?
> (I was thinking about thread local storage)
>   

In a 32-bit kernel %fs is the base of the per-cpu data area.  In a
64-bit kernel %gs points to the pda (processor data area).  The pda is a
single structure, whereas per-cpu data is a section that per-cpu
variables get put into.

> - When I do a "mov %fs ..." instruction (in a module), it seems that %fs is 
> equal to 0 (idem for %gs). Are these registers not always filled ?
>   

On 32-bit they will always have a value, or you'll get a GPF.  On 64-bit
the value of the selector doesn't matter because the MSRs are the real
content.

> - What is the purpose of MSR_FS_BASE and MSR_GS_BASE ? 
> (I thought they were filled with "gdt[fs_entry].base")
>   
On 64-bit, the GDT isn't large enough to hold a 64-bit offset, so it
only stores the low 32-bits.  When you load a segment register with a
selector, it picks up from the gdt.  If you want a full 64-bit offset,
you need to write it to the msr.

> - My last question is about the kernel programation of multi-core (or 
> multiprocessor)
> architecture. I don't see a lot of documentation about that on Internet.
> Do you have some docs/urls about this topic.
> Maybe someone can briefly explain how the execution flow are given to the 
> different cores.
>   

To the kernel they're all just cpus, and it runs tasks on them as
usual.  There are a few tweaks in the scheduler to pay attention to the
shared caches and so on.

    J

Re: [x86] fs, gs purpose & multicore prog

[Eric Lacombe]

Hi,

Thanks for your answers. I've some new questions now ;)

On Thursday 04 September 2008 18:49:42 Jeremy Fitzhardinge wrote:
> Eric Lacombe wrote:

> > - What is the FS and GS segments role inside the kernel ?
> > (I was thinking about thread local storage)
>
> In a 32-bit kernel %fs is the base of the per-cpu data area.  In a
> 64-bit kernel %gs points to the pda (processor data area).  The pda is a
> single structure, whereas per-cpu data is a section that per-cpu
> variables get put into.

So, %gs is not used in 32-bit kernel and %fs is not used in 64-bit kernel. Is 
it right ? Why there were different choices of design ?

>
> > - When I do a "mov %fs ..." instruction (in a module), it seems that %fs
> > is equal to 0 (idem for %gs). Are these registers not always filled ?
>
> On 32-bit they will always have a value, or you'll get a GPF.  On 64-bit
> the value of the selector doesn't matter because the MSRs are the real
> content.

ok, but what about the limits and access types?

>
> > - What is the purpose of MSR_FS_BASE and MSR_GS_BASE ?
> > (I thought they were filled with "gdt[fs_entry].base")
>
> On 64-bit, the GDT isn't large enough to hold a 64-bit offset, so it
> only stores the low 32-bits.  When you load a segment register with a
> selector, it picks up from the gdt.  If you want a full 64-bit offset,
> you need to write it to the msr.

Ok, I just saw that a 64-bit base in segment descriptor is only available for 
the system descriptor.

>
> > - My last question is about the kernel programation of multi-core (or
> > multiprocessor)
> > architecture. I don't see a lot of documentation about that on Internet.
> > Do you have some docs/urls about this topic.
> > Maybe someone can briefly explain how the execution flow are given to the
> > different cores.
>
> To the kernel they're all just cpus, and it runs tasks on them as
> usual.  There are a few tweaks in the scheduler to pay attention to the
> shared caches and so on.

Ok, but how does the kernel technically run tasks on different processor (or 
core)? My question was ambiguous, I was not assuming that I knew how 
multiprocessor works.

Thanks again.

	Eric

>
>     J

Re: [x86] fs, gs purpose & multicore prog

[Jeremy Fitzhardinge]

Eric Lacombe wrote:
> So, %gs is not used in 32-bit kernel and %fs is not used in 64-bit kernel. Is 
> it right ? Why there were different choices of design ?
>   

They both use the opposite from what their respective usermodes use for
thread local storage, since there may be a slight performance advantage
to doing so (at least on 32-bit).  32-bit's use of %gs for usermode TLS
is old and arbitrary, probably predating x86-64 (at least its wide
availability).  The use of %gs for kernel per-cpu data is architectural
on 64-bit, because of the "swapgs" instruction; there's no "swapfs"
instruction.

>>> - When I do a "mov %fs ..." instruction (in a module), it seems that %fs
>>> is equal to 0 (idem for %gs). Are these registers not always filled ?
>>>       
>> On 32-bit they will always have a value, or you'll get a GPF.  On 64-bit
>> the value of the selector doesn't matter because the MSRs are the real
>> content.
>>     
>
> ok, but what about the limits and access types?
>   

They don't exist in 64-bit.  The GDT contains them, but they're not
enforced.

>>> - What is the purpose of MSR_FS_BASE and MSR_GS_BASE ?
>>> (I thought they were filled with "gdt[fs_entry].base")
>>>       
>> On 64-bit, the GDT isn't large enough to hold a 64-bit offset, so it
>> only stores the low 32-bits.  When you load a segment register with a
>> selector, it picks up from the gdt.  If you want a full 64-bit offset,
>> you need to write it to the msr.
>>     
>
> Ok, I just saw that a 64-bit base in segment descriptor is only available for 
> the system descriptor.
>   

Yes, the IDT has double-wide entries to fit 64-bit values, but they
didn't extend that to the GDT.  Or something - I last looked at this a
couple of months ago, and it never sticks in my brain for long.

> Ok, but how does the kernel technically run tasks on different processor (or 
> core)? My question was ambiguous, I was not assuming that I knew how 
> multiprocessor works.
>   

That's a very broad question.  A good proportion of the core kernel code
is dedicated to doing just that.  Very roughly, at boot time it brings
up all the cpus, and they more or less run independently each looking
for work to do in the form of processes waiting to run on the run
queue.  They collectively run the scheduler algorithms to work out who
runs what when; almost everything run in the kernel is a task - both
usermode processes and kernel threads.  Except for the stuff which isn't.

    J

Re: [x86] fs, gs purpose & multicore prog

[Eric Lacombe]

Thanks again ;)

On Vendredi 05 Septembre 2008 16:51:39 Jeremy Fitzhardinge wrote:
> Eric Lacombe wrote:
[...]
> >>> - What is the purpose of MSR_FS_BASE and MSR_GS_BASE ?
> >>> (I thought they were filled with "gdt[fs_entry].base")
> >>
> >> On 64-bit, the GDT isn't large enough to hold a 64-bit offset, so it
> >> only stores the low 32-bits.  When you load a segment register with a
> >> selector, it picks up from the gdt.  If you want a full 64-bit offset,
> >> you need to write it to the msr.
> >
> > Ok, I just saw that a 64-bit base in segment descriptor is only available
> > for the system descriptor.
>
> Yes, the IDT has double-wide entries to fit 64-bit values, but they
> didn't extend that to the GDT.  Or something - I last looked at this a
> couple of months ago, and it never sticks in my brain for long.

It seems, there also are the TSS and LDT.

>
> > Ok, but how does the kernel technically run tasks on different processor
> > (or core)? My question was ambiguous, I was not assuming that I knew how
> > multiprocessor works.
>
> That's a very broad question.  A good proportion of the core kernel code
> is dedicated to doing just that.  Very roughly, at boot time it brings
> up all the cpus, and they more or less run independently each looking
> for work to do in the form of processes waiting to run on the run
> queue.  They collectively run the scheduler algorithms to work out who
> runs what when; almost everything run in the kernel is a task - both
> usermode processes and kernel threads.  Except for the stuff which isn't.

I know these things ;) but what I wanted to know is the "x86 architectural 
details". In fact, I saw that during the machine init the BIOS select a cpu on 
the bus to be the BSP (bootstrap proc). The others are then the APs (Appli 
proc). Then the kernel runs on the BSP. What I wonder is how the kernel gives 
execution flow to the APs.

Thanks.

	Eric

>
>     J

Re: [x86] fs, gs purpose & multicore prog

[Jeremy Fitzhardinge]

Eric Lacombe wrote:
> I know these things ;) but what I wanted to know is the "x86 architectural 
> details". In fact, I saw that during the machine init the BIOS select a cpu on 
> the bus to be the BSP (bootstrap proc). The others are then the APs (Appli 
> proc). Then the kernel runs on the BSP. What I wonder is how the kernel gives 
> execution flow to the APs.
>   

See arch/x86/kernel/smpboot.c (esp do_boot_cpu()), and chapter 7 of
"Intel® 64 and IA-32 Architectures Software Developer’s Manual", volume 3a.

    J