[PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible

[PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible

[nagalakshmi.nandigama]

When CONFIG_DEBUG_PREEMPT is enabled, bug is observed in the smp_processor_id().
This is because smp_processor_id() is not called in preempt safe condition.

To fix this issue, use raw_smp_processor_id instead of smp_processor_id.

Signed-off-by: Nagalakshmi Nandigama <nagalakshmi.nandigama@lsi.com>
CC: stable@vger.kernel.org
---

diff --git a/drivers/scsi/mpt2sas/mpt2sas_base.c b/drivers/scsi/mpt2sas/mpt2sas_base.c
index 272fab7..b010be0 100644
--- a/drivers/scsi/mpt2sas/mpt2sas_base.c
+++ b/drivers/scsi/mpt2sas/mpt2sas_base.c
@@ -1792,7 +1792,7 @@ static inline void _base_writeq(__u64 b, volatile void __iomem *addr,
 static inline u8
 _base_get_msix_index(struct MPT2SAS_ADAPTER *ioc)
 {
-	return ioc->cpu_msix_table[smp_processor_id()];
+	return ioc->cpu_msix_table[raw_smp_processor_id()];
 }
 
 /**

Re: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible

[Matthew Wilcox]

On Tue, Apr 17, 2012 at 11:25:04AM +0530, nagalakshmi.nandigama@lsi.com wrote:
> When CONFIG_DEBUG_PREEMPT is enabled, bug is observed in the smp_processor_id().
> This is because smp_processor_id() is not called in preempt safe condition.
> 
> To fix this issue, use raw_smp_processor_id instead of smp_processor_id.

Just so we're clear, "fix this issue" really means "paper over the warning".

 * NOTE: raw_smp_processor_id() is for internal use only
 * (smp_processor_id() is the preferred variant), but in rare
 * instances it might also be used to turn off false positives
 * (i.e. smp_processor_id() use that the debugging code reports but
 * which use for some reason is legal). Don't use this to hack around
 * the warning message, as your code might not work under PREEMPT.

As far as I can tell, it's called under a lock already so should be
non-preemptable ... can you share the backtrace you get?

> Signed-off-by: Nagalakshmi Nandigama <nagalakshmi.nandigama@lsi.com>
> CC: stable@vger.kernel.org
> ---
> 
> diff --git a/drivers/scsi/mpt2sas/mpt2sas_base.c b/drivers/scsi/mpt2sas/mpt2sas_base.c
> index 272fab7..b010be0 100644
> --- a/drivers/scsi/mpt2sas/mpt2sas_base.c
> +++ b/drivers/scsi/mpt2sas/mpt2sas_base.c
> @@ -1792,7 +1792,7 @@ static inline void _base_writeq(__u64 b, volatile void __iomem *addr,
>  static inline u8
>  _base_get_msix_index(struct MPT2SAS_ADAPTER *ioc)
>  {
> -	return ioc->cpu_msix_table[smp_processor_id()];
> +	return ioc->cpu_msix_table[raw_smp_processor_id()];
>  }
>  
>  /**
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

RE: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible

[Moore, Eric]

 Jan Schmidt suggested using raw_smp_processor_id()  back in February, see this: http://marc.info/?t=132974687100003&r=1&w=2

Alex Shi recently suggested using preempt_disable() and preempt_enable() to solve the same issue, see this: http://marc.info/?t=133274303900003&r=1&w=2

I believe the stack traces are there in both email discussion, they occur when CONFIG_DEBUG_PREEMPT is enabled.

I would rather go with the solution giving the best performance.   James Bottomley is there on the discussion with  Jan Schmidt, he suggested using get_cpu() and put_cpu().

Eric
________________________________________
From: Matthew Wilcox [matthew@wil.cx]
Sent: Tuesday, April 17, 2012 9:14 PM
To: Nandigama, Nagalakshmi
Cc: stable@vger.kernel.org; linux-scsi@vger.kernel.org; Prakash, Sathya; Moore, Eric; jejb@kernel.org
Subject: Re: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id()       in preemptible

On Tue, Apr 17, 2012 at 11:25:04AM +0530, nagalakshmi.nandigama@lsi.com wrote:
> When CONFIG_DEBUG_PREEMPT is enabled, bug is observed in the smp_processor_id().
> This is because smp_processor_id() is not called in preempt safe condition.
>
> To fix this issue, use raw_smp_processor_id instead of smp_processor_id.

Just so we're clear, "fix this issue" really means "paper over the warning".

 * NOTE: raw_smp_processor_id() is for internal use only
 * (smp_processor_id() is the preferred variant), but in rare
 * instances it might also be used to turn off false positives
 * (i.e. smp_processor_id() use that the debugging code reports but
 * which use for some reason is legal). Don't use this to hack around
 * the warning message, as your code might not work under PREEMPT.

As far as I can tell, it's called under a lock already so should be
non-preemptable ... can you share the backtrace you get?

> Signed-off-by: Nagalakshmi Nandigama <nagalakshmi.nandigama@lsi.com>
> CC: stable@vger.kernel.org
> ---
>
> diff --git a/drivers/scsi/mpt2sas/mpt2sas_base.c b/drivers/scsi/mpt2sas/mpt2sas_base.c
> index 272fab7..b010be0 100644
> --- a/drivers/scsi/mpt2sas/mpt2sas_base.c
> +++ b/drivers/scsi/mpt2sas/mpt2sas_base.c
> @@ -1792,7 +1792,7 @@ static inline void _base_writeq(__u64 b, volatile void __iomem *addr,
>  static inline u8
>  _base_get_msix_index(struct MPT2SAS_ADAPTER *ioc)
>  {
> -     return ioc->cpu_msix_table[smp_processor_id()];
> +     return ioc->cpu_msix_table[raw_smp_processor_id()];
>  }
>
>  /**
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Matthew Wilcox                          Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible

[Matthew Wilcox]

On Tue, Apr 17, 2012 at 09:35:49PM -0600, Moore, Eric wrote:
>  Jan Schmidt suggested using raw_smp_processor_id()  back in February, see this: http://marc.info/?t=132974687100003&r=1&w=2
> 
> Alex Shi recently suggested using preempt_disable() and preempt_enable() to solve the same issue, see this: http://marc.info/?t=133274303900003&r=1&w=2
> 
> I believe the stack traces are there in both email discussion, they occur when CONFIG_DEBUG_PREEMPT is enabled.
> 
> I would rather go with the solution giving the best performance.   James Bottomley is there on the discussion with  Jan Schmidt, he suggested using get_cpu() and put_cpu().

I use get_cpu() / put_cpu() in the NVMe driver in similar circumstances.
It's not noticable in the profiles :-)

Where my usage differs from the patch for mpt2sas is that I hold a
reference to the CPU over the submission.  This works out well for
me because I have per-CPU state.  Might be worth considering for your
driver ...

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

RE: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() inpreemptible

[Nandigama, Nagalakshmi]

[-- Attachment #1: Type: text/plain, Size: 1636 bytes --]

Accepted. 
Resent the patch.
Please see attached mail.

Regards,
Nagalakshmi


-----Original Message-----
From: Matthew Wilcox [mailto:matthew@wil.cx] 
Sent: Wednesday, April 18, 2012 9:44 AM
To: Moore, Eric
Cc: Nandigama, Nagalakshmi; stable@vger.kernel.org; linux-scsi@vger.kernel.org; Prakash, Sathya; jejb@kernel.org
Subject: Re: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible

On Tue, Apr 17, 2012 at 09:35:49PM -0600, Moore, Eric wrote:
>  Jan Schmidt suggested using raw_smp_processor_id()  back in February, see this: http://marc.info/?t=132974687100003&r=1&w=2
> 
> Alex Shi recently suggested using preempt_disable() and preempt_enable() to solve the same issue, see this: http://marc.info/?t=133274303900003&r=1&w=2
> 
> I believe the stack traces are there in both email discussion, they occur when CONFIG_DEBUG_PREEMPT is enabled.
> 
> I would rather go with the solution giving the best performance.   James Bottomley is there on the discussion with  Jan Schmidt, he suggested using get_cpu() and put_cpu().

I use get_cpu() / put_cpu() in the NVMe driver in similar circumstances.
It's not noticable in the profiles :-)

Where my usage differs from the patch for mpt2sas is that I hold a
reference to the CPU over the submission.  This works out well for
me because I have per-CPU state.  Might be worth considering for your
driver ...

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

[-- Attachment #2: Type: message/rfc822, Size: 2247 bytes --]

From: "Nandigama, Nagalakshmi" <Nagalakshmi.Nandigama@lsi.com>
To: "stable@vger.kernel.org" <stable@vger.kernel.org>, "Nandigama, Nagalakshmi" <Nagalakshmi.Nandigama@lsi.com>, "Moore, Eric" <Eric.Moore@lsi.com>
Cc: "jejb@kernel.org" <jejb@kernel.org>
Subject: [RESEND] [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible
Date: Fri, 20 Apr 2012 16:28:39 +0530
Message-ID: <201204201058.q3KAwPlo019992@milmhbs0.lsil.com>

When CONFIG_DEBUG_PREEMPT is enabled, bug is observed in the smp_processor_id().
This is because smp_processor_id() is not called in preempt safe condition.

To fix this issue, use get_cpu() and put_cpu() instead of smp_processor_id.

Signed-off-by: Nagalakshmi Nandigama <nagalakshmi.nandigama@lsi.com>
CC: stable@vger.kernel.org
---

diff --git a/drivers/scsi/mpt2sas/mpt2sas_base.c b/drivers/scsi/mpt2sas/mpt2sas_base.c
index 272fab7..0f47f56 100644
--- a/drivers/scsi/mpt2sas/mpt2sas_base.c
+++ b/drivers/scsi/mpt2sas/mpt2sas_base.c
@@ -1792,7 +1792,10 @@ static inline void _base_writeq(__u64 b, volatile void __iomem *addr,
 static inline u8
 _base_get_msix_index(struct MPT2SAS_ADAPTER *ioc)
 {
-       return ioc->cpu_msix_table[smp_processor_id()];
+       int cpu = get_cpu();
+       put_cpu();
+       return ioc->cpu_msix_table[cpu];
+
 }

 /**

Re: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible

[James Bottomley]

On Tue, 2012-04-17 at 22:13 -0600, Matthew Wilcox wrote:
> On Tue, Apr 17, 2012 at 09:35:49PM -0600, Moore, Eric wrote:
> >  Jan Schmidt suggested using raw_smp_processor_id()  back in February, see this: http://marc.info/?t=132974687100003&r=1&w=2
> > 
> > Alex Shi recently suggested using preempt_disable() and preempt_enable() to solve the same issue, see this: http://marc.info/?t=133274303900003&r=1&w=2
> > 
> > I believe the stack traces are there in both email discussion, they occur when CONFIG_DEBUG_PREEMPT is enabled.
> > 
> > I would rather go with the solution giving the best performance.   James Bottomley is there on the discussion with  Jan Schmidt, he suggested using get_cpu() and put_cpu().
> 
> I use get_cpu() / put_cpu() in the NVMe driver in similar circumstances.
> It's not noticable in the profiles :-)
> 
> Where my usage differs from the patch for mpt2sas is that I hold a
> reference to the CPU over the submission.  This works out well for
> me because I have per-CPU state.  Might be worth considering for your
> driver ...

Right, so in this case mpt2sas doesn't care.  It literally only needs a
notion of the current CPU for cache hotness of MSI queue return.  It
doesn't need pinning or anything else.  I think the use of
raw_smp_processor_id() in this instance is ideal, because it actually is
a "need to think about how to do this better" flag, which may mean
actually doing it in a more per-CPU state type way.

Mind you, something like

cpu = get_cpu(); put_cpu();

Is also an instant smack in the eyeballs too because it looks so daft,
so I'm not incredibly bothered.  I lean towards raw_smp_processor_id()
because it's what's actually wanted, but not strongly.

James

Re: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() inpreemptible

[Alex Shi]

On 04/20/2012 07:00 PM, Nandigama, Nagalakshmi wrote:

> diff --git a/drivers/scsi/mpt2sas/mpt2sas_base.c b/drivers/scsi/mpt2sas/mpt2sas_base.c
> index 272fab7..0f47f56 100644
> --- a/drivers/scsi/mpt2sas/mpt2sas_base.c
> +++ b/drivers/scsi/mpt2sas/mpt2sas_base.c
> @@ -1792,7 +1792,10 @@ static inline void _base_writeq(__u64 b, volatile void __iomem *addr,
>  static inline u8
>  _base_get_msix_index(struct MPT2SAS_ADAPTER *ioc)
>  {
> -       return ioc->cpu_msix_table[smp_processor_id()];
> +       int cpu = get_cpu();
> +       put_cpu();
> +       return ioc->cpu_msix_table[cpu];
> +
>  }


If we don't care the correctness interrupt CPU. Do
preempt_enable/preempt_disable is just waste of time.
A patch like following is enough, :)

 _base_get_msix_index(struct MPT2SAS_ADAPTER *ioc)
 {
-       return ioc->cpu_msix_table[smp_processor_id()];
+       return ioc->cpu_msix_table[raw_smp_processor_id()];
 }


BTW, I didn't subscribe linux-scsi mailing list, so cc to me is
appreciated.

RE: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible

[Nandigama, Nagalakshmi]

James, this patch is not picked up for 3.5. Request you to push this patch for next release.

Regards,
Nagalakshmi


-----Original Message-----
From: nagalakshmi.nandigama@lsi.com [mailto:nagalakshmi.nandigama@lsi.com] 
Sent: Tuesday, April 17, 2012 11:25 AM
To: stable@vger.kernel.org; linux-scsi@vger.kernel.org; Nandigama, Nagalakshmi; Prakash, Sathya; Moore, Eric
Cc: jejb@kernel.org
Subject: [PATCH] [SCSI] mpt2sas : Fix unsafe using smp_processor_id() in preemptible

When CONFIG_DEBUG_PREEMPT is enabled, bug is observed in the smp_processor_id().
This is because smp_processor_id() is not called in preempt safe condition.

To fix this issue, use raw_smp_processor_id instead of smp_processor_id.

Signed-off-by: Nagalakshmi Nandigama <nagalakshmi.nandigama@lsi.com>
CC: stable@vger.kernel.org
---

diff --git a/drivers/scsi/mpt2sas/mpt2sas_base.c b/drivers/scsi/mpt2sas/mpt2sas_base.c
index 272fab7..b010be0 100644
--- a/drivers/scsi/mpt2sas/mpt2sas_base.c
+++ b/drivers/scsi/mpt2sas/mpt2sas_base.c
@@ -1792,7 +1792,7 @@ static inline void _base_writeq(__u64 b, volatile void __iomem *addr,
 static inline u8
 _base_get_msix_index(struct MPT2SAS_ADAPTER *ioc)
 {
-	return ioc->cpu_msix_table[smp_processor_id()];
+	return ioc->cpu_msix_table[raw_smp_processor_id()];
 }
 
 /**