* [PATCH] fixdep: handle short reads in read_file @ 2024-09-06 15:29 Sam James 2024-09-07 2:02 ` Masahiro Yamada 0 siblings, 1 reply; 6+ messages in thread From: Sam James @ 2024-09-06 15:29 UTC (permalink / raw) To: Masahiro Yamada, Nathan Chancellor, Nicolas Schier Cc: Sam James, Masahiro Yamada, linux-kbuild, linux-kernel 50% or so of kernel builds within our package manager fail for me with 'fixdep: read: success' because read(), for some reason - possibly ptrace, only read a short amount, not the full size. Unfortunately, this didn't trigger a -Wunused-result warning because we _are_ checking the return value, but with a bad comparison (it's completely fine for read() to not read the whole file in one gulp). Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e Signed-off-by: Sam James <sam@gentoo.org> --- scripts/basic/fixdep.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c index 84b6efa849f4d..04d7742c99ac2 100644 --- a/scripts/basic/fixdep.c +++ b/scripts/basic/fixdep.c @@ -233,9 +233,15 @@ static void *read_file(const char *filename) perror("fixdep: malloc"); exit(2); } - if (read(fd, buf, st.st_size) != st.st_size) { - perror("fixdep: read"); - exit(2); + ssize_t bytes = 0; + while (bytes < st.st_size) { + ssize_t cur = read(fd, buf + bytes, st.st_size - bytes); + if (cur == -1) { + perror("fixdep: read"); + exit(2); + } else { + bytes += cur; + } } buf[st.st_size] = '\0'; close(fd); -- 2.46.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] fixdep: handle short reads in read_file 2024-09-06 15:29 [PATCH] fixdep: handle short reads in read_file Sam James @ 2024-09-07 2:02 ` Masahiro Yamada 2024-09-07 10:14 ` Sam James 0 siblings, 1 reply; 6+ messages in thread From: Masahiro Yamada @ 2024-09-07 2:02 UTC (permalink / raw) To: Sam James; +Cc: Nathan Chancellor, Nicolas Schier, linux-kbuild, linux-kernel [-- Attachment #1: Type: text/plain, Size: 1858 bytes --] On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: > > 50% or so of kernel builds within our package manager fail for me with > 'fixdep: read: success' because read(), for some reason - possibly ptrace, > only read a short amount, not the full size. > > Unfortunately, this didn't trigger a -Wunused-result warning because > we _are_ checking the return value, but with a bad comparison (it's completely > fine for read() to not read the whole file in one gulp). > > Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file to buffer") I guess, another approach would be to use fread() instead of read(). Does the attached diff fix the issue too? > Signed-off-by: Sam James <sam@gentoo.org> > --- > scripts/basic/fixdep.c | 12 +++++++++--- > 1 file changed, 9 insertions(+), 3 deletions(-) > > diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c > index 84b6efa849f4d..04d7742c99ac2 100644 > --- a/scripts/basic/fixdep.c > +++ b/scripts/basic/fixdep.c > @@ -233,9 +233,15 @@ static void *read_file(const char *filename) > perror("fixdep: malloc"); > exit(2); > } > - if (read(fd, buf, st.st_size) != st.st_size) { > - perror("fixdep: read"); > - exit(2); > + ssize_t bytes = 0; > + while (bytes < st.st_size) { > + ssize_t cur = read(fd, buf + bytes, st.st_size - bytes); > + if (cur == -1) { > + perror("fixdep: read"); > + exit(2); > + } else { > + bytes += cur; > + } > } > buf[st.st_size] = '\0'; > close(fd); > -- > 2.46.0 > -- Best Regards Masahiro Yamada [-- Attachment #2: use_fread.diff --] [-- Type: text/x-patch, Size: 1213 bytes --] diff --git a/scripts/basic/fixdep.c b/scripts/basic/fixdep.c index cdd5da7e009b..7a0ca9f0fbbe 100644 --- a/scripts/basic/fixdep.c +++ b/scripts/basic/fixdep.c @@ -89,10 +89,7 @@ * but I don't think the added complexity is worth it) */ -#include <sys/types.h> #include <sys/stat.h> -#include <unistd.h> -#include <fcntl.h> #include <string.h> #include <stdbool.h> #include <stdlib.h> @@ -213,27 +210,28 @@ static void parse_config_file(const char *p) static void *read_file(const char *filename) { struct stat st; - int fd; + FILE *file; char *buf; - fd = open(filename, O_RDONLY); - if (fd < 0) { + file = fopen(filename, "rb"); + if (!file) { fprintf(stderr, "fixdep: error opening file: "); perror(filename); exit(2); } - if (fstat(fd, &st) < 0) { - fprintf(stderr, "fixdep: error fstat'ing file: "); + if (stat(filename, &st) < 0) { + fprintf(stderr, "fixdep: error stat'ing file: "); perror(filename); exit(2); } + buf = xmalloc(st.st_size + 1); - if (read(fd, buf, st.st_size) != st.st_size) { + if (fread(buf, 1, st.st_size, file) != st.st_size) { perror("fixdep: read"); exit(2); } buf[st.st_size] = '\0'; - close(fd); + fclose(file); return buf; } ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] fixdep: handle short reads in read_file 2024-09-07 2:02 ` Masahiro Yamada @ 2024-09-07 10:14 ` Sam James 2024-09-07 13:26 ` Sam James 0 siblings, 1 reply; 6+ messages in thread From: Sam James @ 2024-09-07 10:14 UTC (permalink / raw) To: Masahiro Yamada Cc: Nathan Chancellor, Nicolas Schier, linux-kbuild, linux-kernel Masahiro Yamada <masahiroy@kernel.org> writes: > On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: Hi Masahiro, >> >> 50% or so of kernel builds within our package manager fail for me with >> 'fixdep: read: success' because read(), for some reason - possibly ptrace, >> only read a short amount, not the full size. >> >> Unfortunately, this didn't trigger a -Wunused-result warning because >> we _are_ checking the return value, but with a bad comparison (it's completely >> fine for read() to not read the whole file in one gulp). >> >> Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e > > > Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file > to buffer") > Ah, thanks. I'll fix that and send v2 depending on how we decide to move forward wrt below. > > I guess, another approach would be to use fread() instead of read(). > > Does the attached diff fix the issue too? > > Unfortunately no. It failed for me in the same way as before :( The man page mentions: > On success, fread() and fwrite() return the number of items read or > written. This number equals the number of bytes transferred only when size is 1. so I guess it suffers from the same pitfall. I checked POSIX & ISO C as well which says: > If a partial element is read, its value is unspecified. and > The fread() function shall return the number of elements successfully > read, which shall be less than nitems only if an error or end-of-file > is encountered, or size is zero. The error reference is kind of mysterious there though. It kind of looks like fread *should* work. I'll send this mail and then think about it a bit later and ask around to see if I'm missing something obvious? > [...] thanks, sam ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fixdep: handle short reads in read_file 2024-09-07 10:14 ` Sam James @ 2024-09-07 13:26 ` Sam James 2024-09-07 23:48 ` Masahiro Yamada 0 siblings, 1 reply; 6+ messages in thread From: Sam James @ 2024-09-07 13:26 UTC (permalink / raw) To: Masahiro Yamada Cc: Nathan Chancellor, Nicolas Schier, linux-kbuild, linux-kernel Sam James <sam@gentoo.org> writes: > Masahiro Yamada <masahiroy@kernel.org> writes: > >> On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: > > Hi Masahiro, > >>> >>> 50% or so of kernel builds within our package manager fail for me with >>> 'fixdep: read: success' because read(), for some reason - possibly ptrace, >>> only read a short amount, not the full size. >>> >>> Unfortunately, this didn't trigger a -Wunused-result warning because >>> we _are_ checking the return value, but with a bad comparison (it's completely >>> fine for read() to not read the whole file in one gulp). >>> >>> Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e >> >> >> Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file >> to buffer") >> > > Ah, thanks. I'll fix that and send v2 depending on how we decide to move > forward wrt below. > >> >> I guess, another approach would be to use fread() instead of read(). >> >> Does the attached diff fix the issue too? >> >> > > Unfortunately no. It failed for me in the same way as before :( > > The man page mentions: >> On success, fread() and fwrite() return the number of items read or >> written. This number equals the number of bytes transferred only when size is 1. > > so I guess it suffers from the same pitfall. I checked POSIX & ISO C as well > which says: >> If a partial element is read, its value is unspecified. > and >> The fread() function shall return the number of elements successfully >> read, which shall be less than nitems only if an error or end-of-file >> is encountered, or size is zero. > > The error reference is kind of mysterious there though. > > It kind of looks like fread *should* work. I'll send this mail and then > think about it a bit later and ask around to see if I'm missing > something obvious? OK, others disagree with my reading of fread and think it is ambiguous. With your patch, I was able to get failures albeit possibly less frequently. I'm trying my patch again in a loop now. > >> [...] > > thanks, > sam ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fixdep: handle short reads in read_file 2024-09-07 13:26 ` Sam James @ 2024-09-07 23:48 ` Masahiro Yamada 2024-09-08 8:20 ` Sam James 0 siblings, 1 reply; 6+ messages in thread From: Masahiro Yamada @ 2024-09-07 23:48 UTC (permalink / raw) To: Sam James; +Cc: Nathan Chancellor, Nicolas Schier, linux-kbuild, linux-kernel On Sat, Sep 7, 2024 at 10:26 PM Sam James <sam@gentoo.org> wrote: > > Sam James <sam@gentoo.org> writes: > > > Masahiro Yamada <masahiroy@kernel.org> writes: > > > >> On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: > > > > Hi Masahiro, > > > >>> > >>> 50% or so of kernel builds within our package manager fail for me with > >>> 'fixdep: read: success' because read(), for some reason - possibly ptrace, > >>> only read a short amount, not the full size. > >>> > >>> Unfortunately, this didn't trigger a -Wunused-result warning because > >>> we _are_ checking the return value, but with a bad comparison (it's completely > >>> fine for read() to not read the whole file in one gulp). > >>> > >>> Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e > >> > >> > >> Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file > >> to buffer") > >> > > > > Ah, thanks. I'll fix that and send v2 depending on how we decide to move > > forward wrt below. > > > >> > >> I guess, another approach would be to use fread() instead of read(). > >> > >> Does the attached diff fix the issue too? > >> > >> > > > > Unfortunately no. It failed for me in the same way as before :( > > > > The man page mentions: > >> On success, fread() and fwrite() return the number of items read or > >> written. This number equals the number of bytes transferred only when size is 1. > > > > so I guess it suffers from the same pitfall. I checked POSIX & ISO C as well > > which says: > >> If a partial element is read, its value is unspecified. > > and > >> The fread() function shall return the number of elements successfully > >> read, which shall be less than nitems only if an error or end-of-file > >> is encountered, or size is zero. > > > > The error reference is kind of mysterious there though. > > > > It kind of looks like fread *should* work. I'll send this mail and then > > think about it a bit later and ask around to see if I'm missing > > something obvious? > > OK, others disagree with my reading of fread and think it is ambiguous. > > With your patch, I was able to get failures albeit possibly less > frequently. I'm trying my patch again in a loop now. > > > > >> [...] > > > > thanks, > > sam > Your quotation of the POSIX fread() spec: (https://pubs.opengroup.org/onlinepubs/000095399/functions/fread.html) > If a partial element is read, its value is unspecified. and > The fread() function shall return the number of elements successfully > read, which shall be less than nitems only if an error or end-of-file > is encountered, or size is zero. I think this is clear enough. The end-of-file should not be encountered, as we check the file size in advance. So, some error might be happening. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] fixdep: handle short reads in read_file 2024-09-07 23:48 ` Masahiro Yamada @ 2024-09-08 8:20 ` Sam James 0 siblings, 0 replies; 6+ messages in thread From: Sam James @ 2024-09-08 8:20 UTC (permalink / raw) To: Masahiro Yamada Cc: Nathan Chancellor, Nicolas Schier, linux-kbuild, linux-kernel Masahiro Yamada <masahiroy@kernel.org> writes: > On Sat, Sep 7, 2024 at 10:26 PM Sam James <sam@gentoo.org> wrote: >> >> Sam James <sam@gentoo.org> writes: >> >> > Masahiro Yamada <masahiroy@kernel.org> writes: >> > >> >> On Sat, Sep 7, 2024 at 12:29 AM Sam James <sam@gentoo.org> wrote: >> > >> > Hi Masahiro, >> > >> >>> >> >>> 50% or so of kernel builds within our package manager fail for me with >> >>> 'fixdep: read: success' because read(), for some reason - possibly ptrace, >> >>> only read a short amount, not the full size. >> >>> >> >>> Unfortunately, this didn't trigger a -Wunused-result warning because >> >>> we _are_ checking the return value, but with a bad comparison (it's completely >> >>> fine for read() to not read the whole file in one gulp). >> >>> >> >>> Fixes: 01b5cbe7012fb1eeffc5c143865569835bcd405e >> >> >> >> >> >> Fixes: 01b5cbe7012f ("fixdep: use malloc() and read() to load dep_file >> >> to buffer") >> >> >> > >> > Ah, thanks. I'll fix that and send v2 depending on how we decide to move >> > forward wrt below. >> > >> >> >> >> I guess, another approach would be to use fread() instead of read(). >> >> >> >> Does the attached diff fix the issue too? >> >> >> >> >> > >> > Unfortunately no. It failed for me in the same way as before :( >> > >> > The man page mentions: >> >> On success, fread() and fwrite() return the number of items read or >> >> written. This number equals the number of bytes transferred only when size is 1. >> > >> > so I guess it suffers from the same pitfall. I checked POSIX & ISO C as well >> > which says: >> >> If a partial element is read, its value is unspecified. >> > and >> >> The fread() function shall return the number of elements successfully >> >> read, which shall be less than nitems only if an error or end-of-file >> >> is encountered, or size is zero. >> > >> > The error reference is kind of mysterious there though. >> > >> > It kind of looks like fread *should* work. I'll send this mail and then >> > think about it a bit later and ask around to see if I'm missing >> > something obvious? >> >> OK, others disagree with my reading of fread and think it is ambiguous. >> >> With your patch, I was able to get failures albeit possibly less >> frequently. I'm trying my patch again in a loop now. >> >> > >> >> [...] >> > >> > thanks, >> > sam >> > > > > > > > Your quotation of the POSIX fread() spec: > (https://pubs.opengroup.org/onlinepubs/000095399/functions/fread.html) > >> If a partial element is read, its value is unspecified. > and >> The fread() function shall return the number of elements successfully >> read, which shall be less than nitems only if an error or end-of-file >> is encountered, or size is zero. > > > I think this is clear enough. > > > The end-of-file should not be encountered, as we check the file > size in advance. I believe that it's referring to the number of records, not if you read *1* record of size N. I looked at the musl and glibc sources and neither seem to retry partial reads in that case. I don't see any error indicator set. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-09-08 8:21 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-09-06 15:29 [PATCH] fixdep: handle short reads in read_file Sam James 2024-09-07 2:02 ` Masahiro Yamada 2024-09-07 10:14 ` Sam James 2024-09-07 13:26 ` Sam James 2024-09-07 23:48 ` Masahiro Yamada 2024-09-08 8:20 ` Sam James
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox