[RFC 0/2] Sanity check network connectivity

[RFC 0/2] Sanity check network connectivity

[Joshua Lock]

v3 of this change set based on extra feedback from Khem Raj.
Changes since v2:
* Remove references to Yocto, don't offer default uri's to check
* Make the error message configurable through a variable, CONNECTIVITY_CHECK_MSG,
whilst providing a reasonable default.

"In response to a Yocto Bugzilla request[1] I've written a sanity test to
check whether BitBake is able to fecth from http, https and git sources. The
idea being that if the user is behing a proxy and this test fails we can more
easily help them diagnose and fix their problem.

I've built on the existing infrastructure for less frequent sanity tests so
whilst this test is reasonably heavy it will only run when TMPDIR changes
(usually first run?). Further I added a variable to disable just this sanity
check. People shipping offline installs to customers should just be able to
set the variable in their shipped configuration and not worry about this
sanity check irritating people."

The following changes since commit ff014d9634638457622f6019b163e75bafcefada:

  task-base: add 3G into DISTRO_FEATURE (2011-06-29 14:46:46 +0100)

are available in the git repository at:
  git://git.openembedded.org/openembedded-core-contrib josh/connection-test
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=josh/connection-test

Joshua Lock (2):
  sanity.bbclass: pass the data object to the less frequent test
    harnesses
  sanity: implement network connectivity test

 meta/classes/sanity.bbclass |   53 ++++++++++++++++++++++++++++++++++++------
 1 files changed, 45 insertions(+), 8 deletions(-)

-- 
1.7.5.4

[RFC 1/2] sanity.bbclass: pass the data object to the less frequent test harnesses

[Joshua Lock]

By passing the data object to the less frequently run test harnesses
(check_sanity_tmpdir_change(), check_sanity_sstate_dir_change() and
check_sanity_version_change()) we can run tests against BitBake data here
too.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 meta/classes/sanity.bbclass |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index d296c86..720777a 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -21,7 +21,7 @@ def check_conf_exists(fn, data):
             return True
     return False
 
-def check_sanity_sstate_dir_change(sstate_dir):
+def check_sanity_sstate_dir_change(sstate_dir, data):
     # Sanity checks to be done when the value of SSTATE_DIR changes
 
     # Check that SSTATE_DIR isn't on a filesystem with limited filename length (eg. eCryptFS)
@@ -30,14 +30,14 @@ def check_sanity_sstate_dir_change(sstate_dir):
         testmsg = check_create_long_filename(sstate_dir, "SSTATE_DIR")
     return testmsg
 
-def check_sanity_tmpdir_change(tmpdir):
+def check_sanity_tmpdir_change(tmpdir, data):
     # Sanity checks to be done when the value of TMPDIR changes
 
     # Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS)
     testmsg = check_create_long_filename(tmpdir, "TMPDIR")
     return testmsg
         
-def check_sanity_version_change():
+def check_sanity_version_change(data):
     # Sanity checks to be done when SANITY_VERSION changes
     return ""
 
@@ -266,14 +266,14 @@ def check_sanity(e):
     
     sanity_version = int(data.getVar('SANITY_VERSION', e.data, True) or 1)
     if last_sanity_version < sanity_version: 
-        messages = messages + check_sanity_version_change()
-        messages = messages + check_sanity_tmpdir_change(tmpdir)
-        messages = messages + check_sanity_sstate_dir_change(sstate_dir)
+        messages = messages + check_sanity_version_change(e.data)
+        messages = messages + check_sanity_tmpdir_change(tmpdir, e.data)
+        messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data)
     else: 
         if last_tmpdir != tmpdir:
-            messages = messages + check_sanity_tmpdir_change(tmpdir)
+            messages = messages + check_sanity_tmpdir_change(tmpdir, e.data)
         if last_sstate_dir != sstate_dir:
-            messages = messages + check_sanity_sstate_dir_change(sstate_dir)
+            messages = messages + check_sanity_sstate_dir_change(sstate_dir, e.data)
 
     if os.path.exists("conf"):
         f = file(sanityverfile, 'w')
-- 
1.7.5.4

[RFC 2/2] sanity: implement network connectivity test

[Joshua Lock]

Sanity test to verify files can be fetched from the network using git, http
and https fetchers point users at a page to help get set up in the case of a
failure.

Requires a variable CONNECTIVITY_CHECK_URIS to be set, using the same pattern
as SRC_URI, of URI's to test against.
The variable CONNECTIVITY_CHECK_MSG can be set to provide a custom error
message, such as a pointer to some help, when this check fails.

Addresses [YOCTO #933]

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 meta/classes/sanity.bbclass |   37 +++++++++++++++++++++++++++++++++++++
 1 files changed, 37 insertions(+), 0 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 720777a..c9d37c9 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -35,6 +35,8 @@ def check_sanity_tmpdir_change(tmpdir, data):
 
     # Check that TMPDIR isn't on a filesystem with limited filename length (eg. eCryptFS)
     testmsg = check_create_long_filename(tmpdir, "TMPDIR")
+    # Check that we can fetch from various network transports
+    testmsg = testmsg + check_connectivity(data)
     return testmsg
         
 def check_sanity_version_change(data):
@@ -79,6 +81,41 @@ def check_create_long_filename(filepath, pathname):
             return "Failed to create a file in %s: %s" % (pathname, strerror)
     return ""
 
+def check_connectivity(d):
+    # URI's to check can be set in the CONNECTIVITY_CHECK_URIS variable
+    # using the same syntax as for SRC_URI. If the variable is not set
+    # the check is skipped
+    test_uris = (bb.data.getVar('CONNECTIVITY_CHECK_URIS', d, True) or "").split()
+    retval = ""
+
+    # Only check connectivity if network enabled and the
+    # CONNECTIVITY_CHECK_URIS are set
+    network_enabled = not bb.data.getVar('BB_NO_NETWORK', d, True)
+    check_enabled = len(test_uris)
+    if check_enabled and network_enabled:
+        data = bb.data.createCopy(d)
+        bookmark = os.getcwd()
+        dldir = bb.data.expand('${TMPDIR}/sanity', data)
+        bb.data.setVar('DL_DIR', dldir, data)
+
+        try:
+            fetcher = bb.fetch2.Fetch(test_uris, data)
+            fetcher.download()
+            fetcher.clean(test_uris)
+        except Exception:
+            # Allow the message to be configured so that users can be
+            # pointed to a support mechanism.
+            msg = bb.data.getVar('CONNECTIVITY_CHECK_MSG', d, True) or ""
+            if len(msg) == 0:
+                msg = "Failed to fetch test data from the network. Please ensure your network is configured correctly.\n"
+            retval = msg
+        finally:
+            # Make sure we tidy up the cruft
+            oe.path.remove(dldir)
+            os.chdir(bookmark)
+
+    return retval
+
 def check_sanity(e):
     from bb import note, error, data, __version__
 
-- 
1.7.5.4

Re: [RFC 0/2] Sanity check network connectivity

[Koen Kooi]

Op 29 jun 2011, om 23:55 heeft Joshua Lock het volgende geschreven:

> v3 of this change set based on extra feedback from Khem Raj.
> Changes since v2:
> * Remove references to Yocto, don't offer default uri's to check

Can't you offer an oe.org url to check instead?

Re: [RFC 0/2] Sanity check network connectivity

[Joshua Lock]

On Thu, 2011-06-30 at 00:00 +0200, Koen Kooi wrote:
> Op 29 jun 2011, om 23:55 heeft Joshua Lock het volgende geschreven:
> 
> > v3 of this change set based on extra feedback from Khem Raj.
> > Changes since v2:
> > * Remove references to Yocto, don't offer default uri's to check
> 
> Can't you offer an oe.org url to check instead?

Can we? Sure. Should we? I don't know...

If this is deemed desirable perhaps we'll need some small uri's to fetch
from and test at least http, https and git fetches. Any suggestions?

Could this be a follow on patch?

Joshua
-- 
Joshua Lock
        Yocto Project "Johannes factotum"
        Intel Open Source Technology Centre

Re: [RFC 2/2] sanity: implement network connectivity test

[Richard Purdie]

On Wed, 2011-06-29 at 14:55 -0700, Joshua Lock wrote:
> Sanity test to verify files can be fetched from the network using git, http
> and https fetchers point users at a page to help get set up in the case of a
> failure.
> 
> Requires a variable CONNECTIVITY_CHECK_URIS to be set, using the same pattern
> as SRC_URI, of URI's to test against.
> The variable CONNECTIVITY_CHECK_MSG can be set to provide a custom error
> message, such as a pointer to some help, when this check fails.
> 
> Addresses [YOCTO #933]
> 
> Signed-off-by: Joshua Lock <josh@linux.intel.com>

Merged to master, thanks.

Richard