Arches that don't support PREEMPT

Arches that don't support PREEMPT

[Matthew Wilcox]

On Tue, Sep 19, 2023 at 02:30:59PM +0200, Thomas Gleixner wrote:
> Though it just occured to me that there are dragons lurking:
> 
> arch/alpha/Kconfig:     select ARCH_NO_PREEMPT
> arch/hexagon/Kconfig:   select ARCH_NO_PREEMPT
> arch/m68k/Kconfig:      select ARCH_NO_PREEMPT if !COLDFIRE
> arch/um/Kconfig:        select ARCH_NO_PREEMPT

Sounds like three-and-a-half architectures which could be queued up for
removal right behind ia64 ...

I suspect none of these architecture maintainers have any idea there's a
problem.  Look at commit 87a4c375995e and the discussion in
https://lore.kernel.org/lkml/20180724175646.3621-1-hch@lst.de/

Let's cc those maintainers so they can remove this and fix whatever
breaks.

Re: Arches that don't support PREEMPT

[Geert Uytterhoeven]

Hi Willy,

On Tue, Sep 19, 2023 at 3:01 PM Matthew Wilcox <willy@infradead.org> wrote:
> On Tue, Sep 19, 2023 at 02:30:59PM +0200, Thomas Gleixner wrote:
> > Though it just occured to me that there are dragons lurking:
> >
> > arch/alpha/Kconfig:     select ARCH_NO_PREEMPT
> > arch/hexagon/Kconfig:   select ARCH_NO_PREEMPT
> > arch/m68k/Kconfig:      select ARCH_NO_PREEMPT if !COLDFIRE
> > arch/um/Kconfig:        select ARCH_NO_PREEMPT
>
> Sounds like three-and-a-half architectures which could be queued up for
> removal right behind ia64 ...
>
> I suspect none of these architecture maintainers have any idea there's a
> problem.  Look at commit 87a4c375995e and the discussion in
> https://lore.kernel.org/lkml/20180724175646.3621-1-hch@lst.de/

These links don't really point out there is a grave problem?

> Let's cc those maintainers so they can remove this and fix whatever
> breaks.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Arches that don't support PREEMPT

[John Paul Adrian Glaubitz]

On Tue, 2023-09-19 at 14:00 +0100, Matthew Wilcox wrote:
> On Tue, Sep 19, 2023 at 02:30:59PM +0200, Thomas Gleixner wrote:
> > Though it just occured to me that there are dragons lurking:
> > 
> > arch/alpha/Kconfig:     select ARCH_NO_PREEMPT
> > arch/hexagon/Kconfig:   select ARCH_NO_PREEMPT
> > arch/m68k/Kconfig:      select ARCH_NO_PREEMPT if !COLDFIRE
> > arch/um/Kconfig:        select ARCH_NO_PREEMPT
> 
> Sounds like three-and-a-half architectures which could be queued up for
> removal right behind ia64 ...

The agreement to kill off ia64 wasn't an invitation to kill off other stuff
that people are still working on! Can we please not do this?

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Arches that don't support PREEMPT

[Peter Zijlstra]

On Tue, Sep 19, 2023 at 03:37:24PM +0200, John Paul Adrian Glaubitz wrote:
> On Tue, 2023-09-19 at 14:00 +0100, Matthew Wilcox wrote:
> > On Tue, Sep 19, 2023 at 02:30:59PM +0200, Thomas Gleixner wrote:
> > > Though it just occured to me that there are dragons lurking:
> > > 
> > > arch/alpha/Kconfig:     select ARCH_NO_PREEMPT
> > > arch/hexagon/Kconfig:   select ARCH_NO_PREEMPT
> > > arch/m68k/Kconfig:      select ARCH_NO_PREEMPT if !COLDFIRE
> > > arch/um/Kconfig:        select ARCH_NO_PREEMPT
> > 
> > Sounds like three-and-a-half architectures which could be queued up for
> > removal right behind ia64 ...
> 
> The agreement to kill off ia64 wasn't an invitation to kill off other stuff
> that people are still working on! Can we please not do this?

If you're working on one of them, then surely it's a simple matter of
working on adding CONFIG_PREEMPT support :-)

Re: Arches that don't support PREEMPT

[John Paul Adrian Glaubitz]

On Tue, 2023-09-19 at 15:42 +0200, Peter Zijlstra wrote:
> > The agreement to kill off ia64 wasn't an invitation to kill off other stuff
> > that people are still working on! Can we please not do this?
> 
> If you're working on one of them, then surely it's a simple matter of
> working on adding CONFIG_PREEMPT support :-)

As Geert poined out, I'm not seeing anything particular problematic with the
architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
something about organizing KConfig files.

I find it a bit unfair that maintainers of architectures that have huge companies
behind them use their manpower to urge less popular architectures for removal just
because they don't have 150 people working on the port so they can keep up with
design changes quickly.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Arches that don't support PREEMPT

[Peter Zijlstra]

On Tue, Sep 19, 2023 at 03:48:09PM +0200, John Paul Adrian Glaubitz wrote:
> On Tue, 2023-09-19 at 15:42 +0200, Peter Zijlstra wrote:
> > > The agreement to kill off ia64 wasn't an invitation to kill off other stuff
> > > that people are still working on! Can we please not do this?
> > 
> > If you're working on one of them, then surely it's a simple matter of
> > working on adding CONFIG_PREEMPT support :-)
> 
> As Geert poined out, I'm not seeing anything particular problematic with the
> architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
> something about organizing KConfig files.

The plan in the parent thread is to remove PREEMPT_NONE and
PREEMPT_VOLUNTARY and only keep PREEMPT_FULL.

> I find it a bit unfair that maintainers of architectures that have huge companies
> behind them use their manpower to urge less popular architectures for removal just
> because they don't have 150 people working on the port so they can keep up with
> design changes quickly.

PREEMPT isn't something new. Also, I don't think the arch part for
actually supporting it is particularly hard, mostly it is sticking the
preempt_schedule_irq() call in return from interrupt code path.

If you convert the arch to generic-entry (a much larger undertaking)
then you get this for free.

Re: Arches that don't support PREEMPT

[Thomas Gleixner]

On Tue, Sep 19 2023 at 15:48, John Paul Adrian Glaubitz wrote:
> On Tue, 2023-09-19 at 15:42 +0200, Peter Zijlstra wrote:
>> > The agreement to kill off ia64 wasn't an invitation to kill off other stuff
>> > that people are still working on! Can we please not do this?
>> 
>> If you're working on one of them, then surely it's a simple matter of
>> working on adding CONFIG_PREEMPT support :-)
>
> As Geert poined out, I'm not seeing anything particular problematic with the
> architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
> something about organizing KConfig files.
>
> I find it a bit unfair that maintainers of architectures that have huge companies
> behind them use their manpower to urge less popular architectures for removal just
> because they don't have 150 people working on the port so they can keep up with
> design changes quickly.

I don't urge for removal. I just noticed that these four architectures
lack PREEMPT support. The only thing which is missing is the actual
preemption point in the return to kernel code path.

But otherwise it should just work, which I obviously can't confirm :)

Even without that preemption point it should build and boot. There might
be some minor latency issues when that preemption point is not there,
but adding it is not rocket science either. It's probably about 10 lines
of ASM code, if at all.

Though not adding that might cause a blocking issue for the rework of
the whole preemption logic in order to remove the sprinkled around
cond_resched() muck or force us to maintain some nasty workaround just
for the benefit of a few stranglers.

So I can make the same argument the other way around, that it's
unjustified that some architectures which are just supported for
nostalgia throw roadblocks into kernel developemnt.

If my ALPHA foo wouldn't be very close to zero, I'd write that ASM hack
myself, but that's going to cost more of my and your time than it's
worth the trouble,

Hmm. I could delegate that to Linus, he might still remember :)

Thanks,

        tglx

Re: Arches that don't support PREEMPT

[Anton Ivanov]

On 19/09/2023 14:42, Peter Zijlstra wrote:
> On Tue, Sep 19, 2023 at 03:37:24PM +0200, John Paul Adrian Glaubitz wrote:
>> On Tue, 2023-09-19 at 14:00 +0100, Matthew Wilcox wrote:
>>> On Tue, Sep 19, 2023 at 02:30:59PM +0200, Thomas Gleixner wrote:
>>>> Though it just occured to me that there are dragons lurking:
>>>>
>>>> arch/alpha/Kconfig:     select ARCH_NO_PREEMPT
>>>> arch/hexagon/Kconfig:   select ARCH_NO_PREEMPT
>>>> arch/m68k/Kconfig:      select ARCH_NO_PREEMPT if !COLDFIRE
>>>> arch/um/Kconfig:        select ARCH_NO_PREEMPT
>>>
>>> Sounds like three-and-a-half architectures which could be queued up for
>>> removal right behind ia64 ...
>>
>> The agreement to kill off ia64 wasn't an invitation to kill off other stuff
>> that people are still working on! Can we please not do this?
> 
> If you're working on one of them, then surely it's a simple matter of
> working on adding CONFIG_PREEMPT support :-)

In the case of UML adding preempt will be quite difficult. I looked at this a few years back.

At the same time it is used for kernel test and other stuff. It is not exactly abandonware on a CPU found in archaeological artifacts of past civilizations like ia64.

> 
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um
> 

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

Re: Arches that don't support PREEMPT

[John Paul Adrian Glaubitz]

On Tue, 2023-09-19 at 16:16 +0200, Peter Zijlstra wrote:
> > I find it a bit unfair that maintainers of architectures that have huge companies
> > behind them use their manpower to urge less popular architectures for removal just
> > because they don't have 150 people working on the port so they can keep up with
> > design changes quickly.
> 
> PREEMPT isn't something new. Also, I don't think the arch part for
> actually supporting it is particularly hard, mostly it is sticking the
> preempt_schedule_irq() call in return from interrupt code path.
> 
> If you convert the arch to generic-entry (a much larger undertaking)
> then you get this for free.

If the conversion isn't hard, why is the first reflex the urge to remove an architecture
instead of offering advise how to get the conversion done?

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Arches that don't support PREEMPT

[Matthew Wilcox]

On Tue, Sep 19, 2023 at 04:24:48PM +0200, John Paul Adrian Glaubitz wrote:
> If the conversion isn't hard, why is the first reflex the urge to remove an architecture
> instead of offering advise how to get the conversion done?

Because PREEMPT has been around since before 2005 (cc19ca86a023 created
Kconfig.preempt and I don't need to go back further than that to make my
point), and you haven't done the work yet.  Clearly it takes the threat
of removal to get some kind of motion.

Re: Arches that don't support PREEMPT

[H. Peter Anvin]

On September 19, 2023 7:17:04 AM PDT, Thomas Gleixner <tglx@linutronix.de> wrote:
>On Tue, Sep 19 2023 at 15:48, John Paul Adrian Glaubitz wrote:
>> On Tue, 2023-09-19 at 15:42 +0200, Peter Zijlstra wrote:
>>> > The agreement to kill off ia64 wasn't an invitation to kill off other stuff
>>> > that people are still working on! Can we please not do this?
>>> 
>>> If you're working on one of them, then surely it's a simple matter of
>>> working on adding CONFIG_PREEMPT support :-)
>>
>> As Geert poined out, I'm not seeing anything particular problematic with the
>> architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
>> something about organizing KConfig files.
>>
>> I find it a bit unfair that maintainers of architectures that have huge companies
>> behind them use their manpower to urge less popular architectures for removal just
>> because they don't have 150 people working on the port so they can keep up with
>> design changes quickly.
>
>I don't urge for removal. I just noticed that these four architectures
>lack PREEMPT support. The only thing which is missing is the actual
>preemption point in the return to kernel code path.
>
>But otherwise it should just work, which I obviously can't confirm :)
>
>Even without that preemption point it should build and boot. There might
>be some minor latency issues when that preemption point is not there,
>but adding it is not rocket science either. It's probably about 10 lines
>of ASM code, if at all.
>
>Though not adding that might cause a blocking issue for the rework of
>the whole preemption logic in order to remove the sprinkled around
>cond_resched() muck or force us to maintain some nasty workaround just
>for the benefit of a few stranglers.
>
>So I can make the same argument the other way around, that it's
>unjustified that some architectures which are just supported for
>nostalgia throw roadblocks into kernel developemnt.
>
>If my ALPHA foo wouldn't be very close to zero, I'd write that ASM hack
>myself, but that's going to cost more of my and your time than it's
>worth the trouble,
>
>Hmm. I could delegate that to Linus, he might still remember :)
>
>Thanks,
>
>        tglx

Does *anyone* actually run Alpha at this point?

Re: Arches that don't support PREEMPT

[Matt Turner]

On Tue, Sep 19, 2023 at 10:51 AM H. Peter Anvin <hpa@zytor.com> wrote:
>
> On September 19, 2023 7:17:04 AM PDT, Thomas Gleixner <tglx@linutronix.de> wrote:
> >On Tue, Sep 19 2023 at 15:48, John Paul Adrian Glaubitz wrote:
> >> On Tue, 2023-09-19 at 15:42 +0200, Peter Zijlstra wrote:
> >>> > The agreement to kill off ia64 wasn't an invitation to kill off other stuff
> >>> > that people are still working on! Can we please not do this?
> >>>
> >>> If you're working on one of them, then surely it's a simple matter of
> >>> working on adding CONFIG_PREEMPT support :-)
> >>
> >> As Geert poined out, I'm not seeing anything particular problematic with the
> >> architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
> >> something about organizing KConfig files.
> >>
> >> I find it a bit unfair that maintainers of architectures that have huge companies
> >> behind them use their manpower to urge less popular architectures for removal just
> >> because they don't have 150 people working on the port so they can keep up with
> >> design changes quickly.
> >
> >I don't urge for removal. I just noticed that these four architectures
> >lack PREEMPT support. The only thing which is missing is the actual
> >preemption point in the return to kernel code path.
> >
> >But otherwise it should just work, which I obviously can't confirm :)
> >
> >Even without that preemption point it should build and boot. There might
> >be some minor latency issues when that preemption point is not there,
> >but adding it is not rocket science either. It's probably about 10 lines
> >of ASM code, if at all.
> >
> >Though not adding that might cause a blocking issue for the rework of
> >the whole preemption logic in order to remove the sprinkled around
> >cond_resched() muck or force us to maintain some nasty workaround just
> >for the benefit of a few stranglers.
> >
> >So I can make the same argument the other way around, that it's
> >unjustified that some architectures which are just supported for
> >nostalgia throw roadblocks into kernel developemnt.
> >
> >If my ALPHA foo wouldn't be very close to zero, I'd write that ASM hack
> >myself, but that's going to cost more of my and your time than it's
> >worth the trouble,
> >
> >Hmm. I could delegate that to Linus, he might still remember :)
> >
> >Thanks,
> >
> >        tglx
>
> Does *anyone* actually run Alpha at this point?

I do, as part of maintaining the Gentoo distribution for Alpha.

I'm listed in MAINTAINERS, but really only so I can collect patches
send them to Linus after testing. I don't have copious amounts of free
time to be proactive in kernel development and it's also not really my
area of expertise so I'm nowhere near effective at it.

I would be happy to test any patches sent my way (but I acknowledge
that writing these patches wouldn't be high on anyone's priority list,
etc)

(A video my friend Ian and I made about a particularly large
AlphaServer I have in my basement, in case anyone is interested:
https://www.youtube.com/watch?v=z658a8Js5qg)

Re: Arches that don't support PREEMPT

[Thomas Gleixner]

On Tue, Sep 19 2023 at 15:21, Anton Ivanov wrote:
> On 19/09/2023 14:42, Peter Zijlstra wrote:
>> If you're working on one of them, then surely it's a simple matter of
>> working on adding CONFIG_PREEMPT support :-)
>
> In the case of UML adding preempt will be quite difficult. I looked at
> this a few years back.

What's so difficult about it?

Thanks,

        tglx

Re: Arches that don't support PREEMPT

[Anton Ivanov]

On 19/09/2023 16:17, Thomas Gleixner wrote:
> On Tue, Sep 19 2023 at 15:21, Anton Ivanov wrote:
>> On 19/09/2023 14:42, Peter Zijlstra wrote:
>>> If you're working on one of them, then surely it's a simple matter of
>>> working on adding CONFIG_PREEMPT support :-)
>> In the case of UML adding preempt will be quite difficult. I looked at
>> this a few years back.
> What's so difficult about it?

It's been a while. I remember that I dropped it at the time, but do not remember the full details.

There was some stuff related to FP state and a few other issues I ran into while rewriting the interrupt controller. Some of it may be resolved by now as we are using host cpu flags, etc.

I can give it another go :)

>
> Thanks,
>
>          tglx
>
-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

Re: Arches that don't support PREEMPT

[Steven Rostedt]

On Tue, 19 Sep 2023 15:32:05 +0100
Matthew Wilcox <willy@infradead.org> wrote:

> On Tue, Sep 19, 2023 at 04:24:48PM +0200, John Paul Adrian Glaubitz wrote:
> > If the conversion isn't hard, why is the first reflex the urge to remove an architecture
> > instead of offering advise how to get the conversion done?  
> 
> Because PREEMPT has been around since before 2005 (cc19ca86a023 created
> Kconfig.preempt and I don't need to go back further than that to make my
> point), and you haven't done the work yet.  Clearly it takes the threat
> of removal to get some kind of motion.

Or the use case of a preempt kernel on said arch has never been a request.
Just because it was available doesn't necessarily mean it's required.

Please, let's not jump to threats of removal just to get a feature in.
Simply ask first. I didn't see anyone reaching out to the maintainers
asking for this as it will be needed for a new feature that will likely
make maintaining said arch easier.

Everything is still in brainstorming mode.

-- Steve

Re: Arches that don't support PREEMPT

[Richard Weinberger]

----- Ursprüngliche Mail -----
> Von: "anton ivanov" <anton.ivanov@cambridgegreys.com>
> It's been a while. I remember that I dropped it at the time, but do not remember
> the full details.
> 
> There was some stuff related to FP state and a few other issues I ran into while
> rewriting the interrupt controller. Some of it may be resolved by now as we are
> using host cpu flags, etc.

I remember also having a hacky but working version almost 10 years ago.
It was horrible slow because of the extra scheduler rounds.
But yes, if PREEMPT will be a must-have feature we'll have to try again.

Thanks,
//richard

Re: Arches that don't support PREEMPT

[Anton Ivanov]

On 19/09/2023 17:22, Richard Weinberger wrote:
> ----- Ursprüngliche Mail -----
>> Von: "anton ivanov" <anton.ivanov@cambridgegreys.com>
>> It's been a while. I remember that I dropped it at the time, but do not remember
>> the full details.
>>
>> There was some stuff related to FP state and a few other issues I ran into while
>> rewriting the interrupt controller. Some of it may be resolved by now as we are
>> using host cpu flags, etc.
> 
> I remember also having a hacky but working version almost 10 years ago.
> It was horrible slow because of the extra scheduler rounds.
> But yes, if PREEMPT will be a must-have feature we'll have to try again.

We will need proper fpu primitives for starters that's for sure. fpu_star/end in UML are presently NOOP.

Some of the default spinlocks and other stuff which we pick up from generic may need to change as well.

This is off the top of my head and something which we can fix straight away. I will send some patches to the mailing list tomorrow or on Thu.

A.

> 
> Thanks,
> //richard
> 
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

Re: Arches that don't support PREEMPT

[Ulrich Teichert]

Hi,

[del]
> Does *anyone* actually run Alpha at this point?

Yes, at least I'm still trying to keep my boxes running from time to time,

CU,
Uli
-- 
Dipl. Inf. Ulrich Teichert|e-mail: Ulrich.Teichert@gmx.de | Listening to:
Stormweg 24               |The Baboon Show: Rolling, The Chats: 6L GTR,Plosivs:
24539 Neumuenster, Germany|Hit the Breaks, Nightwatchers: On a Mission

Re: Arches that don't support PREEMPT

[Linus Torvalds]

On Tue, 19 Sept 2023 at 06:48, John Paul Adrian Glaubitz
<glaubitz@physik.fu-berlin.de> wrote:
>
> As Geert poined out, I'm not seeing anything particular problematic with the
> architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
> something about organizing KConfig files.

It can definitely be problematic.

Not the Kconfig file part, and not the preempt count part itself.

But the fact that it has never been used and tested means that there
might be tons of "this architecture code knows it's not preemptible,
because this architecture doesn't support preemption".

So you may have basic architecture code that simply doesn't have the
"preempt_disable()/enable()" pairs that it needs.

PeterZ mentioned the generic entry code, which does this for the entry
path. But it actually goes much deeper: just do a

    git grep preempt_disable arch/x86/kernel

and then do the same for some other architectures.

Looking at alpha, for example, there *are* hits for it, so at least
some of the code there clearly *tries* to do it. But does it cover all
the required parts? If it's never been tested, I'd be surprised if
it's all just ready to go.

I do think we'd need to basically continue to support ARCH_NO_PREEMPT
- and such architectures migth end up with the worst-cast latencies of
only scheduling at return to user space.

               Linus

Re: Arches that don't support PREEMPT

[Thomas Gleixner]

On Tue, Sep 19 2023 at 17:41, Anton Ivanov wrote:
> On 19/09/2023 17:22, Richard Weinberger wrote:
>> ----- Ursprüngliche Mail -----
>>> Von: "anton ivanov" <anton.ivanov@cambridgegreys.com>
>>> It's been a while. I remember that I dropped it at the time, but do not remember
>>> the full details.
>>>
>>> There was some stuff related to FP state and a few other issues I ran into while
>>> rewriting the interrupt controller. Some of it may be resolved by now as we are
>>> using host cpu flags, etc.
>> 
>> I remember also having a hacky but working version almost 10 years ago.
>> It was horrible slow because of the extra scheduler rounds.

Which can be completely avoided as the proposed change will have the
preemption points, but they are only utilized when preempt FULL is
enabled (at boot or runtime). So the behaviour can still be like preempt
NONE, but with a twist to get rid of the cond_resched()/might_resched()
and other heuristic approaches to prevent starvation by long running
functions. That twist needs the preemption points.

See https://lore.kernel.org/lkml/87cyyfxd4k.ffs@tglx

>> But yes, if PREEMPT will be a must-have feature we'll have to try again.
>
> We will need proper fpu primitives for starters that's for
> sure. fpu_star/end in UML are presently NOOP.
>
> Some of the default spinlocks and other stuff which we pick up from
> generic may need to change as well.
>
> This is off the top of my head and something which we can fix straight
> away. I will send some patches to the mailing list tomorrow or on Thu.

I think it does not have to be perfect. UM is far from perfect in
mimicing a real kernel. The main point is that it provides the preempt
counter in the first place and some minimal amount of preemption points
aside of those which come with the preempt_enable() machinery for free.

Thanks,

        tglx

Re: Arches that don't support PREEMPT

[John Paul Adrian Glaubitz]

Hi Linus!

On Tue, 2023-09-19 at 10:25 -0700, Linus Torvalds wrote:
> On Tue, 19 Sept 2023 at 06:48, John Paul Adrian Glaubitz
> <glaubitz@physik.fu-berlin.de> wrote:
> > 
> > As Geert poined out, I'm not seeing anything particular problematic with the
> > architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
> > something about organizing KConfig files.
> 
> It can definitely be problematic.
> 
> Not the Kconfig file part, and not the preempt count part itself.
> 
> But the fact that it has never been used and tested means that there
> might be tons of "this architecture code knows it's not preemptible,
> because this architecture doesn't support preemption".
> 
> So you may have basic architecture code that simply doesn't have the
> "preempt_disable()/enable()" pairs that it needs.
> 
> PeterZ mentioned the generic entry code, which does this for the entry
> path. But it actually goes much deeper: just do a
> 
>     git grep preempt_disable arch/x86/kernel
> 
> and then do the same for some other architectures.
> 
> Looking at alpha, for example, there *are* hits for it, so at least
> some of the code there clearly *tries* to do it. But does it cover all
> the required parts? If it's never been tested, I'd be surprised if
> it's all just ready to go.

Thanks for the detailed explanation.

> I do think we'd need to basically continue to support ARCH_NO_PREEMPT
> - and such architectures migth end up with the worst-cast latencies of
> only scheduling at return to user space.

Great to hear, thank you.

And, yes, eventually I would be happy to help get alpha and m68k converted.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Arches that don't support PREEMPT

[Thomas Gleixner]

On Tue, Sep 19 2023 at 10:25, Linus Torvalds wrote:
> On Tue, 19 Sept 2023 at 06:48, John Paul Adrian Glaubitz
> <glaubitz@physik.fu-berlin.de> wrote:
>>
>> As Geert poined out, I'm not seeing anything particular problematic with the
>> architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
>> something about organizing KConfig files.
>
> It can definitely be problematic.
>
> Not the Kconfig file part, and not the preempt count part itself.
>
> But the fact that it has never been used and tested means that there
> might be tons of "this architecture code knows it's not preemptible,
> because this architecture doesn't support preemption".
>
> So you may have basic architecture code that simply doesn't have the
> "preempt_disable()/enable()" pairs that it needs.
>
> PeterZ mentioned the generic entry code, which does this for the entry
> path. But it actually goes much deeper: just do a
>
>     git grep preempt_disable arch/x86/kernel
>
> and then do the same for some other architectures.
>
> Looking at alpha, for example, there *are* hits for it, so at least
> some of the code there clearly *tries* to do it. But does it cover all
> the required parts? If it's never been tested, I'd be surprised if
> it's all just ready to go.
>
> I do think we'd need to basically continue to support ARCH_NO_PREEMPT
> - and such architectures migth end up with the worst-cast latencies of
> only scheduling at return to user space.

The only thing these architectures should gain is the preempt counter
itself, but yes the extra preemption points are not mandatory to
have, i.e. we simply do not enable them for the nostalgia club.

The removal of cond_resched() might cause latencies, but then I doubt
that these museus pieces are used for real work :)

Thanks,

        tglx

Re: Arches that don't support PREEMPT

[Steven Rostedt]

On Tue, 19 Sep 2023 20:31:50 +0200
Thomas Gleixner <tglx@linutronix.de> wrote:

> The removal of cond_resched() might cause latencies, but then I doubt
> that these museus pieces are used for real work :)

We could simply leave the cond_resched() around but defined as nops for
everything but the "nostalgia club" to keep them from having any regressions.

-- Steve

Re: Arches that don't support PREEMPT

[Linus Torvalds]

On Tue, 19 Sept 2023 at 11:37, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> We could simply leave the cond_resched() around but defined as nops for
> everything but the "nostalgia club" to keep them from having any regressions.

I doubt the nostalgia club cares about some latencies (that are
usually only noticeable under extreme loads anyway).

And if they do, maybe that would make somebody sit down and look into
doing it right.

So I think keeping it around would actually be both useless and
counter-productive.

              Linus

Re: Arches that don't support PREEMPT

[Thomas Gleixner]

On Tue, Sep 19 2023 at 11:52, Linus Torvalds wrote:
> On Tue, 19 Sept 2023 at 11:37, Steven Rostedt <rostedt@goodmis.org> wrote:
>>
>> We could simply leave the cond_resched() around but defined as nops for
>> everything but the "nostalgia club" to keep them from having any regressions.
>
> I doubt the nostalgia club cares about some latencies (that are
> usually only noticeable under extreme loads anyway).
>
> And if they do, maybe that would make somebody sit down and look into
> doing it right.
>
> So I think keeping it around would actually be both useless and
> counter-productive.

Amen to that.

Re: Arches that don't support PREEMPT

[Ingo Molnar]

* Thomas Gleixner <tglx@linutronix.de> wrote:

> On Tue, Sep 19 2023 at 10:25, Linus Torvalds wrote:
> > On Tue, 19 Sept 2023 at 06:48, John Paul Adrian Glaubitz
> > <glaubitz@physik.fu-berlin.de> wrote:
> >>
> >> As Geert poined out, I'm not seeing anything particular problematic with the
> >> architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
> >> something about organizing KConfig files.
> >
> > It can definitely be problematic.
> >
> > Not the Kconfig file part, and not the preempt count part itself.
> >
> > But the fact that it has never been used and tested means that there
> > might be tons of "this architecture code knows it's not preemptible,
> > because this architecture doesn't support preemption".
> >
> > So you may have basic architecture code that simply doesn't have the
> > "preempt_disable()/enable()" pairs that it needs.
> >
> > PeterZ mentioned the generic entry code, which does this for the entry
> > path. But it actually goes much deeper: just do a
> >
> >     git grep preempt_disable arch/x86/kernel
> >
> > and then do the same for some other architectures.
> >
> > Looking at alpha, for example, there *are* hits for it, so at least
> > some of the code there clearly *tries* to do it. But does it cover all
> > the required parts? If it's never been tested, I'd be surprised if
> > it's all just ready to go.
> >
> > I do think we'd need to basically continue to support ARCH_NO_PREEMPT
> > - and such architectures migth end up with the worst-cast latencies of
> > only scheduling at return to user space.
> 
> The only thing these architectures should gain is the preempt counter 
> itself, [...]

And if any of these machines are still used, there's the small benefit of 
preempt_count increasing debuggability of scheduling in supposedly 
preempt-off sections that were ignored silently previously, as most of 
these architectures do not even enable CONFIG_DEBUG_ATOMIC_SLEEP=y in their 
defconfigs:

  $ for ARCH in alpha hexagon m68k um; do git grep DEBUG_ATOMIC_SLEEP arch/$ARCH; done
  $

Plus the efficiency of CONFIG_DEBUG_ATOMIC_SLEEP=y is much reduced on 
non-PREEMPT kernels to begin with: it will basically only detect scheduling 
in hardirqs-off critical sections.

So IMHO there's a distinct debuggability & robustness plus in enabling the 
preemption count on all architectures, even if they don't or cannot use the 
rescheduling points.

> [...] but yes the extra preemption points are not mandatory to have, i.e. 
> we simply do not enable them for the nostalgia club.
> 
> The removal of cond_resched() might cause latencies, but then I doubt 
> that these museus pieces are used for real work :)

I'm not sure we should initially remove *explicit* legacy cond_resched() 
points, except from high-freq paths where they hurt - and of course remove 
them from might_sleep().

Thanks,

	Ingo

Re: Arches that don't support PREEMPT

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> On Tue, 19 Sep 2023 20:31:50 +0200
> Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> > The removal of cond_resched() might cause latencies, but then I doubt
> > that these museus pieces are used for real work :)
> 
> We could simply leave the cond_resched() around but defined as nops for
> everything but the "nostalgia club" to keep them from having any regressions.

That's not a good idea IMO, it's an invitation for accelerated rate bitrot 
turning cond_resched() meaningless very quickly.

We should remove cond_resched() - but probably not as the first step. They 
are conceptually independent of NEED_RESCHED_LAZY and we don't *have to* 
remove them straight away.

By removing cond_resched() separately there's an easily bisectable point to 
blame for any longer latencies on legacy platforms, should any of them 
still be used with recent kernels.

Thanks,

	Ingo

Re: Arches that don't support PREEMPT

[Thomas Gleixner]

On Tue, Sep 19 2023 at 10:25, Linus Torvalds wrote:
> PeterZ mentioned the generic entry code, which does this for the entry
> path. But it actually goes much deeper: just do a
>
>     git grep preempt_disable arch/x86/kernel
>
> and then do the same for some other architectures.
>
> Looking at alpha, for example, there *are* hits for it, so at least
> some of the code there clearly *tries* to do it. But does it cover all
> the required parts? If it's never been tested, I'd be surprised if
> it's all just ready to go.

Interestingly enough m68k has zero instances, but it supports PREEMPT on
the COLDFIRE subarchitecture...

RE: Arches that don't support PREEMPT

[David Laight]

From: Linus Torvalds
> Sent: 19 September 2023 18:25
> 
> On Tue, 19 Sept 2023 at 06:48, John Paul Adrian Glaubitz
> <glaubitz@physik.fu-berlin.de> wrote:
> >
> > As Geert poined out, I'm not seeing anything particular problematic with the
> > architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
> > something about organizing KConfig files.
> 
> It can definitely be problematic.
> 
> Not the Kconfig file part, and not the preempt count part itself.
> 
> But the fact that it has never been used and tested means that there
> might be tons of "this architecture code knows it's not preemptible,
> because this architecture doesn't support preemption".

Do distos even build x86 kernels with PREEMPT_FULL?
I know I've had issues with massive latencies caused graphics driver
forcing write-backs of all the framebuffer memory.
(I think it is a failed attempt to fix a temporary display corruption.)

OTOH SMP support and CONFIG_RT will test most of the code.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Arches that don't support PREEMPT

[Anton Ivanov]

On 19/09/2023 15:16, Peter Zijlstra wrote:
> On Tue, Sep 19, 2023 at 03:48:09PM +0200, John Paul Adrian Glaubitz wrote:
>> On Tue, 2023-09-19 at 15:42 +0200, Peter Zijlstra wrote:
>>>> The agreement to kill off ia64 wasn't an invitation to kill off other stuff
>>>> that people are still working on! Can we please not do this?
>>>
>>> If you're working on one of them, then surely it's a simple matter of
>>> working on adding CONFIG_PREEMPT support :-)
>>
>> As Geert poined out, I'm not seeing anything particular problematic with the
>> architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
>> something about organizing KConfig files.
> 
> The plan in the parent thread is to remove PREEMPT_NONE and
> PREEMPT_VOLUNTARY and only keep PREEMPT_FULL.
> 
>> I find it a bit unfair that maintainers of architectures that have huge companies
>> behind them use their manpower to urge less popular architectures for removal just
>> because they don't have 150 people working on the port so they can keep up with
>> design changes quickly.
> 
> PREEMPT isn't something new. Also, I don't think the arch part for
> actually supporting it is particularly hard, mostly it is sticking the
> preempt_schedule_irq() call in return from interrupt code path.

That calls local_irq_enable() which does various signal related/irq pending work on UML. That in turn does no like being invoked again (as you may have already been invoked out of that) in the IRQ return path.

So it is likely to end up being slightly more difficult than that for UML - it will need to be wrapped so it can be invoked from the "host" side signal code as well as invoked with some additional checks to avoid making a hash out of the IRQ handling.

It may be necessary to modify some of the existing reentrancy prevention logic in the signal handlers as well and change it to make use of the preempt count instead of its own flags/counters.

> 
> If you convert the arch to generic-entry (a much larger undertaking)
> then you get this for free.
> 
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um
> 

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

Re: Arches that don't support PREEMPT

[Arnd Bergmann]

On Tue, Sep 19, 2023, at 10:16, Peter Zijlstra wrote:
> On Tue, Sep 19, 2023 at 03:48:09PM +0200, John Paul Adrian Glaubitz wrote:
>> On Tue, 2023-09-19 at 15:42 +0200, Peter Zijlstra wrote:
>> > > The agreement to kill off ia64 wasn't an invitation to kill off other stuff
>> > > that people are still working on! Can we please not do this?
>> > 
>> > If you're working on one of them, then surely it's a simple matter of
>> > working on adding CONFIG_PREEMPT support :-)
>> 
>> As Geert poined out, I'm not seeing anything particular problematic with the
>> architectures lacking CONFIG_PREEMPT at the moment. This seems to be more
>> something about organizing KConfig files.
>
> The plan in the parent thread is to remove PREEMPT_NONE and
> PREEMPT_VOLUNTARY and only keep PREEMPT_FULL.
...
>
> PREEMPT isn't something new. Also, I don't think the arch part for
> actually supporting it is particularly hard, mostly it is sticking the
> preempt_schedule_irq() call in return from interrupt code path.
>
> If you convert the arch to generic-entry (a much larger undertaking)
> then you get this for free.

I checked the default configurations for both in-kernel targets and
general-purpose distros and was surprised to learn that very few
actually turn on full preemption by default:

- All distros I looked at (rhel, debian, opensuse) use PREEMPT_VOLUNTARY
  by default, though they usually also set PREEMPT_DYNAMIC to let users
  override it at boot time.

- The majority (220) of all defconfig files in the kernel don't select
  any preemption options, and just get PREEMPT_NONE automatically.
  This includes the generic configs for armv7, s390 and mips.
  
- A small number (24) set PREEMPT_VOLUNTARY, but this notably
  includes x86 and ppc64. x86 is the only one of those that sets
  PREEMPT_DYNAMIC

- CONFIG_PREEMPT=y (full preemption) is used on 89 defconfigs,
  including arm64 and a lot of the older arm32, arc and
  mips platforms.

If we want to have a chance of removing both PREEMPT_NONE and
PREEMPT_VOLUNTARY, I think we should start with changing the
defaults first, so defconfigs that don't specify anything else
get PREEMPT=y, and distros that use PREEMPT_VOLUNTARY use it
use it in the absence of a command line argument. If that
doesn't cause too many regressions, the next step might be
to hide the choice under CONFIG_EXPERT until all m68k and
alpha no longer require PREEMPT_NONE.

     Arnd

Re: Arches that don't support PREEMPT

[Geert Uytterhoeven]

Hi Willy,

On Tue, Sep 19, 2023 at 3:01 PM Matthew Wilcox <willy@infradead.org> wrote:
> On Tue, Sep 19, 2023 at 02:30:59PM +0200, Thomas Gleixner wrote:
> > Though it just occured to me that there are dragons lurking:
> >
> > arch/alpha/Kconfig:     select ARCH_NO_PREEMPT
> > arch/hexagon/Kconfig:   select ARCH_NO_PREEMPT
> > arch/m68k/Kconfig:      select ARCH_NO_PREEMPT if !COLDFIRE
> > arch/um/Kconfig:        select ARCH_NO_PREEMPT
>
> Sounds like three-and-a-half architectures which could be queued up for
> removal right behind ia64 ...
>
> I suspect none of these architecture maintainers have any idea there's a
> problem.  Look at commit 87a4c375995e and the discussion in
> https://lore.kernel.org/lkml/20180724175646.3621-1-hch@lst.de/
>
> Let's cc those maintainers so they can remove this and fix whatever
> breaks.

Looks like your scare tactics are working ;-)
[PATCH/RFC] m68k: Add full preempt support
https://lore.kernel.org/all/7858a184cda66e0991fd295c711dfed7e4d1248c.1696603287.git.geert@linux-m68k.org

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds