[RESEND 2][PATCH v4] hexdump: fix for non-aligned buffers

[RESEND 2][PATCH v4] hexdump: fix for non-aligned buffers

[Horacio Mijail Antón Quiles]

An hexdump with a buf not aligned to the groupsize causes
non-naturally-aligned memory accesses. This was causing a kernel panic on
the processor BlackFin BF527, when such an unaligned buffer was fed by the
function ubifs_scanned_corruption in fs/ubifs/scan.c .

To fix this, if the buffer is not aligned to groupsize in a platform which
does not define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, then change the
groupsize to 1, so alignment is no longer a problem.
This behavior is coherent with the way the function currently deals with
inappropriate parameter combinations, which is to fall back to safe
"defaults", even if that means changing the output format and the implicit
access patterns that could have been expected.

Signed-off-by: Horacio Mijail Antón Quiles <hmijail@gmail.com>
---
Resent on 8 Jul because of no answers.

Resent on 29 Jun because of no answers.

Changes in v4:
 - Added space before "*/" (per Joe Perches' indication)

Changes in v3:
 - Removed trailing whitespace (per Joe Perches' indication)
 - Changed mail headers to avoid encoding issues (per Joe Perches' hint)
  
Changes in v2:
  - Changed from ad-hoc calculation to IS_ALIGNED() for readability (per 
      Joe Perches' indication)
  - Made the explanation clearer (aligned vs naturally aligned)

---
 lib/hexdump.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/hexdump.c b/lib/hexdump.c
index 7ea0969..0601a35 100644
--- a/lib/hexdump.c
+++ b/lib/hexdump.c
@@ -124,6 +124,11 @@ int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize,
 	if ((len % groupsize) != 0)	/* no mixed size output */
 		groupsize = 1;
 
+	/* fall back to 1-byte groups if buf is not aligned to groupsize */
+	if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
+	    !IS_ALIGNED((uintptr_t)buf, groupsize))
+		groupsize = 1;
+
 	ngroups = len / groupsize;
 	ascii_column = rowsize * 2 + rowsize / groupsize + 1;
 
-- 
2.1.4

Re: [RESEND 2][PATCH v4] hexdump: fix for non-aligned buffers

[Joe Perches]

On Thu, 2015-07-09 at 01:44 +0200, Horacio Mijail Antón Quiles wrote:
> An hexdump with a buf not aligned to the groupsize causes
> non-naturally-aligned memory accesses. This was causing a kernel panic on
> the processor BlackFin BF527, when such an unaligned buffer was fed by the
> function ubifs_scanned_corruption in fs/ubifs/scan.c .

Seems sensible enough to me.

> ---
> diff --git a/lib/hexdump.c b/lib/hexdump.c
[]
> @@ -124,6 +124,11 @@ int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize,
>  	if ((len % groupsize) != 0)	/* no mixed size output */
>  		groupsize = 1;
>  
> +	/* fall back to 1-byte groups if buf is not aligned to groupsize */
> +	if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
> +	    !IS_ALIGNED((uintptr_t)buf, groupsize))
> +		groupsize = 1;
> +
>  	ngroups = len / groupsize;
>  	ascii_column = rowsize * 2 + rowsize / groupsize + 1;
>  

Re: [RESEND 2][PATCH v4] hexdump: fix for non-aligned buffers

[Andrew Morton]

On Thu, 9 Jul 2015 01:44:18 +0200 Horacio Mijail Ant__n Quiles <hmijail@gmail.com> wrote:

> An hexdump with a buf not aligned to the groupsize causes
> non-naturally-aligned memory accesses. This was causing a kernel panic on
> the processor BlackFin BF527, when such an unaligned buffer was fed by the
> function ubifs_scanned_corruption in fs/ubifs/scan.c .
> 
> To fix this, if the buffer is not aligned to groupsize in a platform which
> does not define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, then change the
> groupsize to 1, so alignment is no longer a problem.
> This behavior is coherent with the way the function currently deals with
> inappropriate parameter combinations, which is to fall back to safe
> "defaults", even if that means changing the output format and the implicit
> access patterns that could have been expected.

CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS seems inappropriate for this. 
Having this unset means "can do unaligned accesses, but they're
inefficient".  It doesn't mean "unaligned accesses go oops".

But I can't the appropriate config knob.  There's a
CONFIG_CPU_HAS_NO_UNALIGNED, but that's an m68k-private thing.

> Resent on 8 Jul because of no answers.
> 
> Resent on 29 Jun because of no answers.

During the merge window.  So I have everything sitting there in my
patches-to-process pile.  The backlog is excessive this time (700+
emails) so I'm thinking I'll change things so I'll henceforth be
processing patches-for-the-next-cycle during this-cycle, while keeping
patches-for-next-cycle out of linux-next.

> --- a/lib/hexdump.c
> +++ b/lib/hexdump.c
> @@ -124,6 +124,11 @@ int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize,
>  	if ((len % groupsize) != 0)	/* no mixed size output */
>  		groupsize = 1;
>  
> +	/* fall back to 1-byte groups if buf is not aligned to groupsize */

That isn't a great comment.  It tells us what the code does (which is
quite obvious just from reading it) but it doesn't tell us *why* it
does it.  Something like "if buf is not aligned to groupsize then fall
back to 1-byte groups to avoid unaligned memory accesses on
architectures which do not support them"?

But as mentioned above, that's different from "architectures which do
not support them efficently"!  Maybe we need a new config variable.

Or maybe blackfin should be handling the unaligned access trap and
transparently handling it, like sparc?

> +	if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
> +	    !IS_ALIGNED((uintptr_t)buf, groupsize))
> +		groupsize = 1;
> +
>
> ...
>

Re: [RESEND 2][PATCH v4] hexdump: fix for non-aligned buffers

[H. Mijail]

> On 09 Jul 2015, at 02:03, Andrew Morton <akpm@linux-foundation.org> wrote:
> 
> On Thu, 9 Jul 2015 01:44:18 +0200 Horacio Mijail Ant__n Quiles <hmijail@gmail.com> wrote:
> 
>> An hexdump with a buf not aligned to the groupsize causes
>> non-naturally-aligned memory accesses. This was causing a kernel panic on
>> the processor BlackFin BF527, when such an unaligned buffer was fed by the
>> function ubifs_scanned_corruption in fs/ubifs/scan.c .
>> 
>> To fix this, if the buffer is not aligned to groupsize in a platform which
>> does not define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, then change the
>> groupsize to 1, so alignment is no longer a problem.
>> This behavior is coherent with the way the function currently deals with
>> inappropriate parameter combinations, which is to fall back to safe
>> "defaults", even if that means changing the output format and the implicit
>> access patterns that could have been expected.
> 
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS seems inappropriate for this. 
> Having this unset means "can do unaligned accesses, but they're
> inefficient".  It doesn't mean "unaligned accesses go oops".
> 
> But I can't the appropriate config knob.  There's a
> CONFIG_CPU_HAS_NO_UNALIGNED, but that’s an m68k-private thing.

I’m only a newbie, but I will argue that the lesser evil is checking
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS - until a new configure variable
is defined.

In Documentation/unaligned-memory-access.txt, an undefined
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is taken as if to mean “the 
hardware isn’t able to access memory on arbitrary boundaries”.

And I certainly can’t see any more appropriate CONFIG_* variable.

The other alternative in Documentation/unaligned-memory-access.txt is the
macro get_unaligned() from asm/unaligned.h. However, using get_unaligned()
would mean a much more intrusive patch, since each case of the groupsize 
would be changed, and anyway we would still need to check 
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS to avoid penalising everyone.

Lastly, as noted on the patch’s description, hex_dump_to_buffer() as it is 
has no qualms about sanitizing the received parameter values down to 
conservative values, and so any current callers are already used to having
the expected output and the implicit memory access patterns changed. So 
I’d say that changing the groupsize is not only harmless, but expected.

> 
>> Resent on 8 Jul because of no answers.
>> 
>> Resent on 29 Jun because of no answers.
> 
> During the merge window.  So I have everything sitting there in my
> patches-to-process pile.  The backlog is excessive this time (700+
> emails) so I'm thinking I'll change things so I'll henceforth be
> processing patches-for-the-next-cycle during this-cycle, while keeping
> patches-for-next-cycle out of linux-next.

No problem for me - if I should squelch the next version of this patch
for some time, please let me know.

> 
>> --- a/lib/hexdump.c
>> +++ b/lib/hexdump.c
>> @@ -124,6 +124,11 @@ int hex_dump_to_buffer(const void *buf, size_t len, int rowsize, int groupsize,
>> 	if ((len % groupsize) != 0)	/* no mixed size output */
>> 		groupsize = 1;
>> 
>> +	/* fall back to 1-byte groups if buf is not aligned to groupsize */
> 
> That isn't a great comment.  It tells us what the code does (which is
> quite obvious just from reading it) but it doesn't tell us *why* it
> does it.  Something like "if buf is not aligned to groupsize then fall
> back to 1-byte groups to avoid unaligned memory accesses on
> architectures which do not support them”?

Thanks, note taken.

> 
> But as mentioned above, that's different from "architectures which do
> not support them efficently"!  Maybe we need a new config variable.
> 
> Or maybe blackfin should be handling the unaligned access trap and
> transparently handling it, like sparc?
> 

I’ll wait for anyone else to weight in…



>> +	if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) &&
>> +	    !IS_ALIGNED((uintptr_t)buf, groupsize))
>> +		groupsize = 1;
>> +
>> 
>> ...
>> 

Re: [RESEND 2][PATCH v4] hexdump: fix for non-aligned buffers

[Andrew Morton]

On Thu, 9 Jul 2015 03:36:02 +0200 "H. Mijail" <hmijail@gmail.com> wrote:

> 
> > On 09 Jul 2015, at 02:03, Andrew Morton <akpm@linux-foundation.org> wrote:
> > 
> > On Thu, 9 Jul 2015 01:44:18 +0200 Horacio Mijail Ant__n Quiles <hmijail@gmail.com> wrote:
> > 
> >> An hexdump with a buf not aligned to the groupsize causes
> >> non-naturally-aligned memory accesses. This was causing a kernel panic on
> >> the processor BlackFin BF527, when such an unaligned buffer was fed by the
> >> function ubifs_scanned_corruption in fs/ubifs/scan.c .
> >> 
> >> To fix this, if the buffer is not aligned to groupsize in a platform which
> >> does not define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, then change the
> >> groupsize to 1, so alignment is no longer a problem.
> >> This behavior is coherent with the way the function currently deals with
> >> inappropriate parameter combinations, which is to fall back to safe
> >> "defaults", even if that means changing the output format and the implicit
> >> access patterns that could have been expected.
> > 
> > CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS seems inappropriate for this. 
> > Having this unset means "can do unaligned accesses, but they're
> > inefficient".  It doesn't mean "unaligned accesses go oops".
> > 
> > But I can't the appropriate config knob.  There's a
> > CONFIG_CPU_HAS_NO_UNALIGNED, but that's an m68k-private thing.
> 
> I'm only a newbie, but I will argue that the lesser evil is checking
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS - until a new configure variable
> is defined.
> 
> In Documentation/unaligned-memory-access.txt, an undefined
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is taken as if to mean 'the 
> hardware isn't able to access memory on arbitrary boundaries'.

hm, yes, OK, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is a poor name.

> The other alternative in Documentation/unaligned-memory-access.txt is the
> macro get_unaligned() from asm/unaligned.h. However, using get_unaligned()
> would mean a much more intrusive patch, since each case of the groupsize 
> would be changed, and anyway we would still need to check 
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS to avoid penalising everyone.

Actually, I think using get_unaligned() would be a better solution. 
For architectures which have CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y,
get_unaligned() should be fast - just one instruction.

This way we avoid having different-appearing output on different
architectures.  

> >> Resent on 8 Jul because of no answers.
> >> 
> >> Resent on 29 Jun because of no answers.
> > 
> > During the merge window.  So I have everything sitting there in my
> > patches-to-process pile.  The backlog is excessive this time (700+
> > emails) so I'm thinking I'll change things so I'll henceforth be
> > processing patches-for-the-next-cycle during this-cycle, while keeping
> > patches-for-next-cycle out of linux-next.
> 
> No problem for me - if I should squelch the next version of this patch
> for some time, please let me know.

The merge window has ended ;)

> > 
> > But as mentioned above, that's different from "architectures which do
> > not support them efficently"!  Maybe we need a new config variable.
> > 
> > Or maybe blackfin should be handling the unaligned access trap and
> > transparently handling it, like sparc?
> > 
> 
> I'll wait for anyone else to weight in'

Possibly blackfin *could* emulate unaligned accesses.  But according to
the documentation, hex_dump_to_buffer() needs to be altered anyway
because we cannot rely on the architecture handling such accesses.

Re: [RESEND 2][PATCH v4] hexdump: fix for non-aligned buffers

[Geert Uytterhoeven]

On Thu, Jul 9, 2015 at 4:07 AM, Andrew Morton <akpm@linux-foundation.org> wrote:
>> The other alternative in Documentation/unaligned-memory-access.txt is the
>> macro get_unaligned() from asm/unaligned.h. However, using get_unaligned()
>> would mean a much more intrusive patch, since each case of the groupsize
>> would be changed, and anyway we would still need to check
>> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS to avoid penalising everyone.
>
> Actually, I think using get_unaligned() would be a better solution.
> For architectures which have CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y,
> get_unaligned() should be fast - just one instruction.
>
> This way we avoid having different-appearing output on different
> architectures.

Definitely.

A less optimal get_unaligned() will just be noise in the snprintf() processing
time.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [RESEND 2][PATCH v4] hexdump: fix for non-aligned buffers

[H. Mijail]

> On 09 Jul 2015, at 09:16, Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> 
> On Thu, Jul 9, 2015 at 4:07 AM, Andrew Morton <akpm@linux-foundation.org> wrote:
>>> The other alternative in Documentation/unaligned-memory-access.txt is the
>>> macro get_unaligned() from asm/unaligned.h. However, using get_unaligned()
>>> would mean a much more intrusive patch, since each case of the groupsize
>>> would be changed, and anyway we would still need to check
>>> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS to avoid penalising everyone.
>> 
>> Actually, I think using get_unaligned() would be a better solution.
>> For architectures which have CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y,
>> get_unaligned() should be fast - just one instruction.

D’oh! You’re right, of course.

>> 
>> This way we avoid having different-appearing output on different
>> architectures.
> 
> Definitely.
> 
> A less optimal get_unaligned() will just be noise in the snprintf() processing
> time.
> 

OK, so thanks Andrew and Geert for your comments. I’ll reimplement using get_unaligned.