Filename too long

Filename too long

[Mark Hatle]

I found a problem today working through a problem report.

We have a situation where we automatically add certain project directories as
premirrors within the environment.

During the fetch2 work, the system takes this long pathname, and then embeds it
into ud.lockfile parameter, we end up with something like:

The path to the build/download dir is about 250 characters... (abbreviated below)

ud.lockfile =
/my/really/log/path/to/my/project/build/download/git2/file....my.really.log.path.to.my.project.layers.wr-kernel.recipes-kernel.linux.......git.kernel-4.8.x.git.lock

This triggers an exception, IOError with the errno '36', and error string of
'File name too long'.  (Note, this is the filename, NOT the path being too long!)

In the bitbake/lib/bb/utils.py: def lockfile(name, shared=False, retry=True,
block=False)

The system tries to:

while True:
    try:
        lf = open(name, 'a+')
        ... more work here ...
    except Exception:
        ... close the file, if opened ...
        pass
    if not retry:
        return None

This triggers an endless loop if this happens.

So at a minimum we probably need to extend the exception handling to look for
file (or path) too long errors and return/stop.

I'm wondering if, something like the following should be added:

 exception IOError as e:
     errno, strerror = e.args
     logger.fatal("IOError on lock: [%s] %s" % (errno, strerror))
     try:
         lf.close()
     except Exception:
         pass
     retry = False
     pass

(Is stopping on all IOErrors too broad of a check?)


Second, I'm not sure where the lockfile name is determined, but should there be
a way in there to shorten the lockfile to say the 'basename' instead of a full
path (if specified)?

Any suggestions would be appreciated.

--Mark

Re: Filename too long

[Robert Yang]

Hi Mark,

We have a local patch to fix the problem, I had sent it to bitbake-devel
before, but not merged:

https://patchwork.openembedded.org/patch/61321/

Here is the updated patch in our bitbake:

commit ebb1190c01a15fc6e91c9810c46313c3b3fb305c
Author: Robert Yang <liezhi.yang@windriver.com>
Date:   Fri Nov 8 23:32:11 2013 +0800

     fetch2/git.py: fix the filename too long error

     When we use the PREMIRROR/MIRROR from the local disk, and if it is in a
     deep directory, for example, when len(path) > 255 (The NAME_MAX), we
     will get the "filename too long error".

     This is becuase we use ud.path.replace('/', '.') to change the path to
     the filename. We seldom see such a long url, but it is easy to produce
     it on the local machine.

     Another problem is that:

     file:///local/path

     will be converted into .local.path which is a hidden file by default.

     Don't convert the name when ud.proto == "file" will fix the problem.

     Signed-off-by: Robert Yang <liezhi.yang@windriver.com>

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index f7a0c01..cc17e5a 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -195,7 +195,10 @@ class Git(FetchMethod):
                      ud.unresolvedrev[name] = ud.revisions[name]
                  ud.revisions[name] = self.latest_revision(ud, d, name)

-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', 
'.').replace('*', '.'))
+        if ud.proto == "file":
+            gitsrcname = '%s%s' % ("file...", os.path.basename(ud.path))
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), 
ud.path.replace('/', '.').replace('*', '.'))
          if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]


// Robert

On 01/19/2017 03:02 AM, Mark Hatle wrote:
> I found a problem today working through a problem report.
>
> We have a situation where we automatically add certain project directories as
> premirrors within the environment.
>
> During the fetch2 work, the system takes this long pathname, and then embeds it
> into ud.lockfile parameter, we end up with something like:
>
> The path to the build/download dir is about 250 characters... (abbreviated below)
>
> ud.lockfile =
> /my/really/log/path/to/my/project/build/download/git2/file....my.really.log.path.to.my.project.layers.wr-kernel.recipes-kernel.linux.......git.kernel-4.8.x.git.lock
>
> This triggers an exception, IOError with the errno '36', and error string of
> 'File name too long'.  (Note, this is the filename, NOT the path being too long!)
>
> In the bitbake/lib/bb/utils.py: def lockfile(name, shared=False, retry=True,
> block=False)
>
> The system tries to:
>
> while True:
>     try:
>         lf = open(name, 'a+')
>         ... more work here ...
>     except Exception:
>         ... close the file, if opened ...
>         pass
>     if not retry:
>         return None
>
> This triggers an endless loop if this happens.
>
> So at a minimum we probably need to extend the exception handling to look for
> file (or path) too long errors and return/stop.
>
> I'm wondering if, something like the following should be added:
>
>  exception IOError as e:
>      errno, strerror = e.args
>      logger.fatal("IOError on lock: [%s] %s" % (errno, strerror))
>      try:
>          lf.close()
>      except Exception:
>          pass
>      retry = False
>      pass
>
> (Is stopping on all IOErrors too broad of a check?)
>
>
> Second, I'm not sure where the lockfile name is determined, but should there be
> a way in there to shorten the lockfile to say the 'basename' instead of a full
> path (if specified)?
>
> Any suggestions would be appreciated.
>
> --Mark
>

Re: Filename too long

[Richard Purdie]

On Thu, 2017-01-19 at 14:13 +0800, Robert Yang wrote:
> Hi Mark,
> 
> We have a local patch to fix the problem, I had sent it to bitbake-
> devel
> before, but not merged:
> 
> https://patchwork.openembedded.org/patch/61321/
> 
> Here is the updated patch in our bitbake:
> 
> commit ebb1190c01a15fc6e91c9810c46313c3b3fb305c
> Author: Robert Yang <liezhi.yang@windriver.com>
> Date:   Fri Nov 8 23:32:11 2013 +0800
> 
>      fetch2/git.py: fix the filename too long error
> 
>      When we use the PREMIRROR/MIRROR from the local disk, and if it
> is in a
>      deep directory, for example, when len(path) > 255 (The
> NAME_MAX), we
>      will get the "filename too long error".
> 
>      This is becuase we use ud.path.replace('/', '.') to change the
> path to
>      the filename. We seldom see such a long url, but it is easy to
> produce
>      it on the local machine.
> 
>      Another problem is that:
> 
>      file:///local/path
> 
>      will be converted into .local.path which is a hidden file by
> default.

Looking at the code in your patch, this second piece clearly isn't true
due to this code:

          if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]

I also don't believe this patch is correct since we're meant to be
creating a filename, not a path and if you leave the '/' characters in,
it isn't a filename. I think we likely just need to truncate the path
to the last say 225 chars if its over length. The unique piece should
be towards the right hand end of the name.

Cheers,

Richard

Re: Filename too long

[Mark Hatle]

On 1/19/17 4:21 AM, Richard Purdie wrote:
> On Thu, 2017-01-19 at 14:13 +0800, Robert Yang wrote:
>> Hi Mark,
>>
>> We have a local patch to fix the problem, I had sent it to bitbake-
>> devel
>> before, but not merged:
>>
>> https://patchwork.openembedded.org/patch/61321/
>>
>> Here is the updated patch in our bitbake:
>>
>> commit ebb1190c01a15fc6e91c9810c46313c3b3fb305c
>> Author: Robert Yang <liezhi.yang@windriver.com>
>> Date:   Fri Nov 8 23:32:11 2013 +0800
>>
>>      fetch2/git.py: fix the filename too long error
>>
>>      When we use the PREMIRROR/MIRROR from the local disk, and if it
>> is in a
>>      deep directory, for example, when len(path) > 255 (The
>> NAME_MAX), we
>>      will get the "filename too long error".
>>
>>      This is becuase we use ud.path.replace('/', '.') to change the
>> path to
>>      the filename. We seldom see such a long url, but it is easy to
>> produce
>>      it on the local machine.
>>
>>      Another problem is that:
>>
>>      file:///local/path
>>
>>      will be converted into .local.path which is a hidden file by
>> default.
> 
> Looking at the code in your patch, this second piece clearly isn't true
> due to this code:
> 
>           if gitsrcname.startswith('.'):
>               gitsrcname = gitsrcname[1:]
> 
> I also don't believe this patch is correct since we're meant to be
> creating a filename, not a path and if you leave the '/' characters in,
> it isn't a filename. I think we likely just need to truncate the path
> to the last say 225 chars if its over length. The unique piece should
> be towards the right hand end of the name.

Ok.. I can certainly implement that.  Should the truncation be in the lockfile
code or in the code that generatd the name itself?

BTW I did some further looking, the issue is that we have a recipe that includes
a local git repository.  So essentially the recipe does:

recipe-foo/foo/foo_1.0.bb:

SRC_URI = "git://${THISDIR}/foo.git;type=file"

When I was looking through everything I was mistaking the path and premirror
work.  It's actually the SRC_URI that is very long, and changes dynamically
based on the path to the recipe itself.... leading to the problem.

--Mark

> Cheers,
> 
> Richard
> 

Re: Filename too long

[Richard Purdie]

On Thu, 2017-01-19 at 09:07 -0600, Mark Hatle wrote:
> On 1/19/17 4:21 AM, Richard Purdie wrote:
> > 
> > On Thu, 2017-01-19 at 14:13 +0800, Robert Yang wrote:
> > > 
> > Looking at the code in your patch, this second piece clearly isn't
> > true
> > due to this code:
> > 
> >           if gitsrcname.startswith('.'):
> >               gitsrcname = gitsrcname[1:]
> > 
> > I also don't believe this patch is correct since we're meant to be
> > creating a filename, not a path and if you leave the '/' characters
> > in,
> > it isn't a filename. I think we likely just need to truncate the
> > path
> > to the last say 225 chars if its over length. The unique piece
> > should
> > be towards the right hand end of the name.
> Ok.. I can certainly implement that.  Should the truncation be in the
> lockfile code or in the code that generatd the name itself?

Gut feeling says I think the generated name is safest.

Cheers,

Richard