[PATCH] add net-lib, nfs-lib, url-lib, and img-lib

[PATCH] add net-lib, nfs-lib, url-lib, and img-lib

[Will Woods]

Hi all,

This set of patches adds a bunch of "libraries" to dracut:
- net-lib adds some utility functions to module 40network
- nfs-lib provides the special nfs handling that was in 95nfs/nfsroot
- url-lib is a new module for handling http{s}/ftp URLs with curl
- img-lib is a new module for dealing with archives & fs images.

They might need tweaking for style/GPL headers/etc - just let me know and
I'll fix up the patches.

Only patch 3 and 5 change existing code - the rest should be very low-risk.

Patch 3 ports the existing nfsroot to nfs-lib. This is kind of a
proof-of-concept (look how short it is! whee!) but it *has not been tested*.
Feel free to take or leave that patch - but if someone could apply it and
help test it, I would be *very* thankful!

Patch 5 moves 90livenet to url-lib, and I *have* tested this to confirm that
it works.

If these are accepted, some next steps might be:
- move things out of dracut-lib.sh to net-lib or nfs-lib
  (wait_for_if_up, nfs_root_to_var, ip_to_var, etc.)
- add rpm/deb support to img-lib
  (is there a way to do optional dependencies, specified in dracut.conf?)
- add support for live "updates" images to livenet

Thanks for looking these over,

-w

[PATCH 1/9] 40network: add net-lib.sh

[Will Woods]

net-lib.sh is a library of useful functions for network stuff.

More things may get added/moved here in the future.

Signed-off-by: Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 modules.d/40network/module-setup.sh |    1 +
 modules.d/40network/net-lib.sh      |   33 +++++++++++++++++++++++++++++++++
 2 files changed, 34 insertions(+), 0 deletions(-)
 create mode 100644 modules.d/40network/net-lib.sh

diff --git a/modules.d/40network/module-setup.sh b/modules.d/40network/module-setup.sh
index ef7818a..2f4d93d 100755
--- a/modules.d/40network/module-setup.sh
+++ b/modules.d/40network/module-setup.sh
@@ -76,6 +76,7 @@ install() {
     inst "$moddir/ifup" "/sbin/ifup"
     inst "$moddir/netroot" "/sbin/netroot"
     inst "$moddir/dhclient-script" "/sbin/dhclient-script"
+    inst "$moddir/net-lib.sh" "/lib/net-lib.sh"
     inst_simple "$moddir/dhclient.conf" "/etc/dhclient.conf"
     inst_hook pre-udev 50 "$moddir/ifname-genrules.sh"
     inst_hook pre-udev 60 "$moddir/net-genrules.sh"
diff --git a/modules.d/40network/net-lib.sh b/modules.d/40network/net-lib.sh
new file mode 100644
index 0000000..0ed80b8
--- /dev/null
+++ b/modules.d/40network/net-lib.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+
+get_ip() {
+    local iface="$1" ip=""
+    ip=$(ip -o -f inet addr show $iface)
+    ip=${ip%%/*}
+    ip=${ip##* }
+}
+
+iface_for_remote_addr() {
+    set -- $(ip -o route get to $1)
+    echo $5
+}
+
+iface_for_mac() {
+    local interface="" mac="$(echo $1 | tr '[:upper:]' '[:lower:]')"
+    for interface in /sys/class/net/*; do
+        if [ $(cat $interface/address) = "$mac" ]; then
+            echo ${interface##*/}
+        fi
+    done
+}
+
+iface_has_link() {
+    local interface="$1" flags=""
+    [ -n "$interface" ] || return 2
+    interface="/sys/class/net/$interface"
+    [ -d "$interface" ] || return 2
+    flags=$(cat $interface/flags)
+    echo $(($flags|0x41)) > $interface/flags # 0x41: IFF_UP|IFF_RUNNING
+    [ "$(cat $interface/carrier)" = 1 ] || return 1
+    # XXX Do we need to reset the flags here? anaconda never bothered..
+}
-- 
1.7.7.6

[PATCH 2/9] 95nfs: add nfs-lib.sh

[Will Woods]

nfs-lib.sh contains a bunch of functions used to parse NFS "url"s of
various types, pull nfs information out of dhcp info, and actually
perform nfs mounts sanely.

Signed-off-by: Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 modules.d/95nfs/module-setup.sh |    1 +
 modules.d/95nfs/nfs-lib.sh      |  139 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 140 insertions(+), 0 deletions(-)
 create mode 100755 modules.d/95nfs/nfs-lib.sh

diff --git a/modules.d/95nfs/module-setup.sh b/modules.d/95nfs/module-setup.sh
index 0e921ff..19d618c 100755
--- a/modules.d/95nfs/module-setup.sh
+++ b/modules.d/95nfs/module-setup.sh
@@ -60,6 +60,7 @@ install() {
     inst_hook pre-udev 99 "$moddir/nfs-start-rpc.sh"
     inst_hook pre-pivot 99 "$moddir/nfsroot-cleanup.sh"
     inst "$moddir/nfsroot" "/sbin/nfsroot"
+    inst "$moddir/nfs-lib.sh" "/lib/nfs-lib.sh"
     mkdir -m 0755 -p "$initdir/var/lib/nfs/rpc_pipefs"
     mkdir -m 0755 -p "$initdir/var/lib/rpcbind"
     mkdir -m 0755 -p "$initdir/var/lib/nfs/statd/sm"
diff --git a/modules.d/95nfs/nfs-lib.sh b/modules.d/95nfs/nfs-lib.sh
new file mode 100755
index 0000000..4f3d184
--- /dev/null
+++ b/modules.d/95nfs/nfs-lib.sh
@@ -0,0 +1,139 @@
+#!/bin/sh
+# -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*-
+# ex: ts=8 sw=4 sts=4 et filetype=sh
+
+type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
+. /lib/net-lib.sh
+
+# TODO: make these things not pollute the calling namespace
+
+# nfs_to_var NFSROOT [NETIF]
+# use NFSROOT to set $nfs, $server, $path, and $options.
+# NFSROOT is something like: nfs[4]:<server>:/<path>[:<options>|,<options>]
+# NETIF is used to get information from DHCP options, if needed.
+nfs_to_var() {
+    # Unfortunately, there's multiple styles of nfs "URL" in use, so we need
+    # extra functions to parse them into $nfs, $server, $path, and $options.
+    # FIXME: local netif=${2:-$netif}?
+    case "$1" in
+        nfs://*) rfc2224_nfs_to_var "$1" ;;
+        nfs:*:*:/*) anaconda_nfs_to_var "$1" ;;
+        *) nfsroot_to_var "$1" ;;
+    esac
+    # if anything's missing, try to fill it in from DHCP options
+    if [ -z "$server" ] || [ -z "$path" ]; then nfsroot_from_dhcp $2; fi
+    # if there's a "%s" in the path, replace it with the hostname/IP
+    if strstr "$path" "%s"; then
+        local node=""
+        read node < /proc/sys/kernel/hostname
+        [ "$node" = "(none)" ] && node=$(get_ip $2)
+        path=${path%%%s*}$node${path#*%s} # replace only the first %s
+    fi
+}
+
+# root=nfs:[<server-ip>:]<root-dir>[:<nfs-options>]
+# root=nfs4:[<server-ip>:]<root-dir>[:<nfs-options>]
+nfsroot_to_var() {
+    # strip nfs[4]:
+    local arg="$@:"
+    nfs="${arg%%:*}"
+    arg="${arg##$nfs:}"
+
+    # check if we have a server
+    if strstr "$arg" ':/*' ; then
+        server="${arg%%:/*}"
+        arg="/${arg##*:/}"
+    fi
+
+    path="${arg%%:*}"
+
+    # rest are options
+    options="${arg##$path}"
+    # strip leading ":"
+    options="${options##:}"
+    # strip  ":"
+    options="${options%%:}"
+
+    # Does it really start with '/'?
+    [ -n "${path%%/*}" ] && path="error";
+
+    #Fix kernel legacy style separating path and options with ','
+    if [ "$path" != "${path#*,}" ] ; then
+        options=${path#*,}
+        path=${path%%,*}
+    fi
+}
+
+# RFC2224: nfs://<server>[:<port>]/<path>
+rfc2224_nfs_to_var() {
+    nfs="nfs"
+    server="${1#nfs://}"
+    path="/${server#*/}"
+    server="${server%%/*}"
+    server="${server%%:}" # anaconda compat (nfs://<server>:/<path>)
+    local port="${server##*:}"
+    [ "$port" != "$server" ] && options="port=$port"
+}
+
+# Anaconda-style path with options: nfs:<options>:<server>:/<path>
+# (without mount options, anaconda is the same as dracut)
+anaconda_nfs_to_var() {
+    nfs="nfs"
+    options="${1#nfs:}"
+    server="${options#*:}"
+    server="${server%:/*}"
+    options="${options%%:*}"
+    path="/${1##*:/}"
+}
+
+# nfsroot_from_dhcp NETIF
+# fill in missing server/path from DHCP options.
+nfsroot_from_dhcp() {
+    local f
+    for f in /tmp/net.$1.override /tmp/dhclient.$1.dhcpopts; do
+        [ -f $f ] && . $f || return
+    done
+    [ -n "$new_root_path" ] && nfsroot_to_var "$nfs:$new_root_path"
+    [ -z "$path" ] && [ "$(getarg root=)" == "/dev/nfs" ] && path=/tftpboot/%s
+    [ -z "$server" ] && server=$new_dhcp_server_identifier
+    [ -z "$server" ] && server=$new_dhcp_next_server
+    [ -z "$server" ] && server=${new_root_path%%:*}
+}
+
+# Look through $options, fix "rw"/"ro", move "lock"/"nolock" to $nfslock
+munge_nfs_options() {
+    local f="" flags="" nfsrw="ro" OLDIFS="$IFS"
+    IFS=,
+    for f in $options; do
+        case $f in
+            ro|rw) nfsrw=$f ;;
+            lock|nolock) nfslock=$f ;;
+            *) flags=${flags:+$flags,}$f ;;
+        esac
+    done
+    IFS="$OLDIFS"
+
+    # Override rw/ro if set on cmdline
+    getarg ro >/dev/null && nfsrw=ro
+    getarg rw >/dev/null && nfsrw=rw
+
+    options=$nfsrw${flags:+,$flags}
+}
+
+# mount_nfs NFSROOT MNTDIR [NETIF]
+mount_nfs() {
+    local nfsroot="$1" mntdir="$2" netif="$3"
+    local nfs="" server="" path="" options=""
+    nfs_to_var $nfsroot $netif
+    munge_nfs_options
+    if [ "$nfs" = "nfs4" ]; then
+        options=$options${nfslock+,$nfslock}
+    else
+        # NFSv{2,3} doesn't support using locks as it requires a helper to
+        # transfer the rpcbind state to the new root
+        [ "$nfslock" = "lock" ] \
+            && warn "Locks unsupported on NFSv{2,3}, using nolock" 1>&2
+        options=$options,nolock
+    fi
+    mount -t $nfs -o$options $server:$path $mntdir
+}
-- 
1.7.7.6

[PATCH 3/9] port nfsroot to nfs-lib

[Will Woods]

Rewrite nfsroot to use nfs-lib. The functionality should be unchanged.

Signed-off-by: Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 modules.d/95nfs/nfsroot |   91 ++--------------------------------------------
 1 files changed, 4 insertions(+), 87 deletions(-)

diff --git a/modules.d/95nfs/nfsroot b/modules.d/95nfs/nfsroot
index 764971b..589d4ed 100755
--- a/modules.d/95nfs/nfsroot
+++ b/modules.d/95nfs/nfsroot
@@ -3,17 +3,9 @@
 # ex: ts=8 sw=4 sts=4 et filetype=sh
 
 type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
+. /lib/nfs-lib.sh
 
-PATH=/usr/sbin:/usr/bin:/sbin:/bin
-
-# Huh? Empty $1?
-[ -z "$1" ] && exit 1
-
-# Huh? Empty $2?
-[ -z "$2" ] && exit 1
-
-# Huh? Empty $3?
-[ -z "$3" ] && exit 1
+[ "$#" = 3 ] || exit 1
 
 # root is in the form root=nfs[4]:[server:]path[:options], either from
 # cmdline or dhcp root-path
@@ -21,85 +13,10 @@ netif="$1"
 root="$2"
 NEWROOT="$3"
 
-# Continue if nfs prefix
-case "${root%%:*}" in
-    nfs|nfs4);;
-    *) return;;
-esac
-
-nfsroot_to_var $root
-
-#Load other data that might provide info
-[ -f /tmp/net.$netif.override ] && . /tmp/net.$netif.override
-[ -f /tmp/dhclient.$netif.dhcpopts ] && . /tmp/dhclient.$netif.dhcpopts
-
-#Empty path means try dhcp root-path, this is ok here since parse-nfsroot.sh
-#already takes care of nfs:... formatted root-path
-[ -z "$path" ] && nfsroot_to_var $nfs:$new_root_path
-
-#Empty path defaults to "/tftpboot/%s" only in nfsroot.txt legacy mode
-[ -z "$path" ] && [ "$(getarg root=)" = "/dev/nfs" ] && path="/tftpboot/%s"
-
-if [ -z "$server" ] ; then
-    # XXX new_dhcp_next_server is unconfirmed this is an assumption
-    for var in $srv $new_dhcp_server_identifier $new_dhcp_next_server $new_root_path '' ; do
-        [ -n "$var" ] && server=$var && break;
-    done
-
-    # XXX This blindly assumes that if new_root_path has to used that
-    # XXX it really can be used as server
-    server=${server%%:*}
-fi
-
+nfs_to_var $root $netif
 [ -z "$server" ] && die "Required parameter 'server' is missing"
 
-# Kernel replaces first %s with host name, and falls back to the ip address
-# if it isn't set. Only the first %s is substituted.
-if [ "${path#*%s}" != "$path" ]; then
-    ip=$(ip -o -f inet addr show $netif)
-    ip=${ip%%/*}
-    ip=${ip##* }
-    read node < /proc/sys/kernel/hostname
-    [ "$node" = "(none)" ] && node=$ip
-    path=${path%%%s*}$node${path#*%s}
-fi
-
-# Look through the options and remove rw/locking options
-OLDIFS="$IFS"
-IFS=,
-for f in $options ; do
-    [ "$f" = "ro" -o "$f" = "rw" ] && nfsrw=$f && continue
-    [ "$f" = "lock" -o "$f" = "nolock" ] && nfslock=$f && continue
-    flags=${flags:+$flags,}$f
-done
-IFS="$OLDIFS"
-options=$flags
-
-# Override rw/ro if set on cmdline
-getarg ro && nfsrw=ro
-getarg rw && nfsrw=rw
-
-# Default to ro if unset
-[ -z "$nfsrw" ] && nfsrw=ro
-
-options=${options:+$options,}$nfsrw
-
-if [ "$nfs" = "nfs4" ]; then
-    # XXX Should we loop here?
-    mount -t nfs4 -o$options${nfslock+,$nfslock} \
-        $server:$path $NEWROOT \
-        && { [ -e /dev/root ] || >/dev/root ; }
-else
-    # NFSv{2,3} doesn't support using locks as it requires a helper to transfer
-    # the rpcbind state to the new root
-    [ "$nfslock" = "lock" ] && \
-        warn "Locks unsupported on NFSv{2,3}, using nolock" 1>&2
-
-    # XXX Should we loop here?
-    { mount -t nfs -o$options${options:+,}nolock,nfsvers=3 $server:$path $NEWROOT || \
-        mount -t nfs -o$options${options:+,}nolock,nfsvers=2 $server:$path $NEWROOT ; } \
-        && { [ -e /dev/root ] || >/dev/root ; }
-fi
+mount_nfs $root $NEWROOT $netif && { [ -e /dev/root ] || >/dev/root ; }
 
 # inject new exit_if_exists
 echo 'settle_exit_if_exists="--exit-if-exists=/dev/root"; rm "$job"' > $hookdir/initqueue/nfs.sh
-- 
1.7.7.6

[PATCH 4/9] add module 45url-lib

[Will Woods]

url-lib adds some functions for dealing with URLs (mostly for fetching
files, for the moment).

It uses curl to handle http/https/ftp URLs, but it can be extended by other
modules at runtime by using the "add_url_handler" function.

Signed-off-by: Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 modules.d/45url-lib/module-setup.sh |   24 +++++++++++++
 modules.d/45url-lib/url-lib.sh      |   66 +++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+), 0 deletions(-)
 create mode 100755 modules.d/45url-lib/module-setup.sh
 create mode 100755 modules.d/45url-lib/url-lib.sh

diff --git a/modules.d/45url-lib/module-setup.sh b/modules.d/45url-lib/module-setup.sh
new file mode 100755
index 0000000..eab9f25
--- /dev/null
+++ b/modules.d/45url-lib/module-setup.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+# module-setup for url-lib
+
+check() {
+    command -v curl >/dev/null || return 1
+    return 255
+}
+
+depends() {
+    echo network
+    return 0
+}
+
+install() {
+    inst "$moddir/url-lib.sh" "/lib/url-lib.sh"
+    dracut_install curl
+    mkdir -m 0755 -p "$initdir/etc/ssl/certs"
+    if ! inst_any -t /etc/ssl/certs/ca-bundle.crt \
+            /etc/ssl/certs/ca-bundle.crt \
+            /etc/ssl/certs/ca-certificates.crt; then
+        dwarn "Couldn't find SSL CA cert bundle; HTTPS won't work."
+    fi
+}
+
diff --git a/modules.d/45url-lib/url-lib.sh b/modules.d/45url-lib/url-lib.sh
new file mode 100755
index 0000000..51340d5
--- /dev/null
+++ b/modules.d/45url-lib/url-lib.sh
@@ -0,0 +1,66 @@
+#!/bin/bash
+# url-lib.sh - functions for handling URLs (file fetching etc.)
+#
+# Authors:
+#   Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
+
+type mkuniqdir >/dev/null 2>&1 || . /lib/dracut-lib.sh
+
+# fetch_url URL [OUTFILE]
+#   fetch the given URL to a locally-visible location.
+#   if OUTFILE is given, the URL will be fetched to that filename,
+#   overwriting it if present.
+#   the return values are as follows:
+#   0: success
+#   253: unknown error (file missing)
+#   254: unhandled URL scheme / protocol
+#   255: bad arguments / unparseable URLs
+#   other: fetch command failure (whatever curl/mount/etc return)
+fetch_url() {
+    local url="$1" outloc="$2"
+    local handler="$(get_url_handler $url)"
+    [ -n "$handler" ] || return 254
+    [ -n "$url" ] || return 255
+    $handler "$url" "$outloc"
+}
+
+# get_url_handler URL
+#   returns the first HANDLERNAME corresponding to the URL's scheme
+get_url_handler() {
+    local scheme="${1%%:*}" item=""
+    for item in $url_handler_map; do
+        [ "$scheme" = "${item%%:*}" ] && echo "${item#*:}" && return 0
+    done
+    return 1
+}
+
+# add_url_handler HANDLERNAME SCHEME [SCHEME...]
+#   associate the named handler with the named scheme(s).
+add_url_handler() {
+    local handler="$1"; shift
+    local schemes="$@" scheme=""
+    set --
+    for scheme in $schemes; do
+        set -- "$@" "$scheme:$handler"
+    done
+    set -- $@ $url_handler_map # add new items to *front* of list
+    url_handler_map="$@"
+}
+
+curl_args="--location --retry 3 --fail --show-error"
+curl_fetch_url() {
+    local url="$1" outloc="$2"
+    if [ -n "$outloc" ]; then
+        curl $curl_args --output "$outloc" "$url" || return $?
+    else
+        local outdir="$(mkuniqdir /tmp curl_fetch_url)"
+        local cwd="$(pwd)"
+        cd "$outdir"
+        curl $curl_args --remote-name "$url" || return $?
+        cd "$cwd"
+        outloc="$(echo $outdir/*)"
+    fi
+    [ -f "$outloc" ] || return 253
+    echo "$outloc"
+}
+add_url_handler curl_fetch_url http https ftp
-- 
1.7.7.6

[PATCH 5/9] 90livenet: port to url-lib

[Will Woods]

This makes the livenetroot module use url-lib for fetching its root
image/filesystem. There's also some minor tweaks for POSIX compliance.

Signed-off-by: Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 modules.d/90livenet/livenetroot      |   37 ++++++++++-----------------------
 modules.d/90livenet/module-setup.sh  |   13 +-----------
 modules.d/90livenet/parse-livenet.sh |   19 ++++++++++-------
 3 files changed, 23 insertions(+), 46 deletions(-)

diff --git a/modules.d/90livenet/livenetroot b/modules.d/90livenet/livenetroot
index 1e8e34e..bc62760 100755
--- a/modules.d/90livenet/livenetroot
+++ b/modules.d/90livenet/livenetroot
@@ -1,41 +1,26 @@
 #!/bin/bash
 # livenetroot - fetch a live image from the network and run it
-#
-# TODO:
-# * HTTPS: arg to use --no-check-certificate with https (boo)
-#          args for --certificate, --ca-certificate
-# * NFS support?
 
 type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
 [ -f /tmp/root.info ] && . /tmp/root.info
 
+. /lib/url-lib.sh
+
 PATH=/usr/sbin:/usr/bin:/sbin:/bin
 
 # args get passed from 40network/netroot
-netroot=$2
-
-liveurl=${netroot#livenet:}
-
-if [ ${liveurl##*.} == "iso" ]; then
-    imgfile="/run/live.iso"
-else
-    imgfile="/run/rootfs.img"
-fi
-
-
-case "$liveurl" in
-  http://*|https://*|ftp://*)
-    wget -O $imgfile "$liveurl"
-  ;;
-  *) die "don't know how to handle URL: $liveurl" ;;
-esac
-[ $? == 0 ] || die "failed to download live image"
-
-
-if [ ${imgfile##*.} == "iso" ]; then
+netroot="$2"
+liveurl="${netroot#livenet:}"
+info "fetching $liveurl"
+imgfile=$(fetch_url "$liveurl")
+[ $? = 0 ] || die "failed to download live image: error $?"
+
+# TODO: couldn't dmsquash-live-root handle this?
+if [ ${imgfile##*.} = "iso" ]; then
     root=$(losetup -f)
     losetup $root $imgfile
 else
     root=$imgfile
 fi
+
 exec /sbin/dmsquash-live-root $root
diff --git a/modules.d/90livenet/module-setup.sh b/modules.d/90livenet/module-setup.sh
index 5cfbb75..31164fd 100755
--- a/modules.d/90livenet/module-setup.sh
+++ b/modules.d/90livenet/module-setup.sh
@@ -2,26 +2,15 @@
 # module-setup.sh for livenet
 
 check() {
-    # a live, host-only image doesn't really make a lot of sense
-    [[ $hostonly ]] && return 1
-    command -v wget >/dev/null || return 1
     return 255
 }
 
 depends() {
-    echo network dmsquash-live
+    echo network url-lib dmsquash-live
     return 0
 }
 
 install() {
-    dracut_install wget
-    mkdir -m 0755 -p "$initdir/etc/ssl/certs"
-    if ! inst_any -t /etc/ssl/certs/ca-bundle.crt \
-            /etc/ssl/certs/ca-bundle.crt \
-            /etc/ssl/certs/ca-certificates.crt; then
-        dwarn "Couldn't find SSL CA cert bundle; HTTPS won't work."
-    fi
-
     inst_hook cmdline 29 "$moddir/parse-livenet.sh"
     inst "$moddir/livenetroot" "/sbin/livenetroot"
 }
diff --git a/modules.d/90livenet/parse-livenet.sh b/modules.d/90livenet/parse-livenet.sh
index 2e6bc4b..042cc51 100755
--- a/modules.d/90livenet/parse-livenet.sh
+++ b/modules.d/90livenet/parse-livenet.sh
@@ -3,17 +3,20 @@
 # root=live:[url-to-backing-file]
 
 [ -z "$root" ] && root=$(getarg root=)
+. /lib/url-lib.sh
 
-str_starts $root "live:" && liveurl=$root
-str_starts $liveurl "live:" || return
+str_starts "$root" "live:" && liveurl="$root"
+str_starts "$liveurl" "live:" || return
 liveurl="${liveurl#live:}"
 
 # setting netroot to "livenet:..." makes "livenetroot" get run after ifup
-case "$liveurl" in
-    http://*|https://*|ftp://*)
-        netroot="livenet:$liveurl"
-        root="livenet" # quiet complaints from init
-        rootok=1 ;;
-esac
+if get_url_handler "$liveurl" >/dev/null; then
+    info "livenet: root image at $liveurl"
+    netroot="livenet:$liveurl"
+    root="livenet" # quiet complaints from init
+    rootok=1
+else
+    info "livenet: no url handler for $liveurl"
+fi
 
 wait_for_dev /dev/root
-- 
1.7.7.6

[PATCH 6/9] url-lib: add support for NFS

[Will Woods]

This adds nfs_fetch_url to allow fetching arbitrary files from NFS.

This means that livenet can now run using an NFS-mounted live image,
which reduces memory usage by a lot.

Signed-off-by: Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 modules.d/45url-lib/url-lib.sh |   28 ++++++++++++++++++++++++++++
 1 files changed, 28 insertions(+), 0 deletions(-)

diff --git a/modules.d/45url-lib/url-lib.sh b/modules.d/45url-lib/url-lib.sh
index 51340d5..03ee8d1 100755
--- a/modules.d/45url-lib/url-lib.sh
+++ b/modules.d/45url-lib/url-lib.sh
@@ -10,6 +10,8 @@ type mkuniqdir >/dev/null 2>&1 || . /lib/dracut-lib.sh
 #   fetch the given URL to a locally-visible location.
 #   if OUTFILE is given, the URL will be fetched to that filename,
 #   overwriting it if present.
+#   If the URL is something mountable (e.g. nfs://) and no OUTFILE is given,
+#   the server will be left mounted until pre-pivot.
 #   the return values are as follows:
 #   0: success
 #   253: unknown error (file missing)
@@ -47,6 +49,8 @@ add_url_handler() {
     url_handler_map="$@"
 }
 
+### HTTP, HTTPS, FTP #################################################
+
 curl_args="--location --retry 3 --fail --show-error"
 curl_fetch_url() {
     local url="$1" outloc="$2"
@@ -64,3 +68,27 @@ curl_fetch_url() {
     echo "$outloc"
 }
 add_url_handler curl_fetch_url http https ftp
+
+### NFS ##############################################################
+
+. /lib/nfs-lib.sh
+
+nfs_fetch_url() {
+    local url="$1" outloc="$2" nfs="" server="" path="" options=""
+    nfs_to_var "$url" || return 255
+    local filepath="${path%/*}" filename="${path##*/}"
+
+    # TODO: check to see if server:/filepath is already mounted
+    local mntdir="$(mkuniqdir /run nfs_mnt)"
+    mount_nfs $nfs:$server:$path${options:+:$options} $mntdir
+    # FIXME: schedule lazy unmount during pre-pivot hook
+
+    if [ -z "$outloc" ]; then
+        outloc="$mntdir/$filename"
+    else
+        cp -f "$mntdir/$filename" "$outloc" || return $?
+    fi
+    [ -f "$outloc" ] || return 253
+    echo "$outloc"
+}
+add_url_handler nfs_fetch_url nfs nfs4
-- 
1.7.7.6

[PATCH 7/9] add set_http_header to url-lib

[Will Woods]

This allows you to set custom headers for curl to send with HTTP
requests.

Signed-off-by: Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 modules.d/45url-lib/url-lib.sh |   34 +++++++++++++++++++++++++++++-----
 1 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/modules.d/45url-lib/url-lib.sh b/modules.d/45url-lib/url-lib.sh
index 03ee8d1..dd473d4 100755
--- a/modules.d/45url-lib/url-lib.sh
+++ b/modules.d/45url-lib/url-lib.sh
@@ -51,6 +51,8 @@ add_url_handler() {
 
 ### HTTP, HTTPS, FTP #################################################
 
+export CURL_HOME="/run/initramfs/url-lib"
+mkdir -p $CURL_HOME
 curl_args="--location --retry 3 --fail --show-error"
 curl_fetch_url() {
     local url="$1" outloc="$2"
@@ -69,19 +71,41 @@ curl_fetch_url() {
 }
 add_url_handler curl_fetch_url http https ftp
 
+set_http_header() {
+    echo "header = \"$1: $2\"" >> $CURL_HOME/.curlrc
+}
+
 ### NFS ##############################################################
 
 . /lib/nfs-lib.sh
 
+nfs_already_mounted() {
+    local server="$1" path="$2" localdir="" s="" p=""
+    cat /proc/mounts | while read src mnt rest; do
+        splitsep ":" "$src" s p
+        if [ "$server" = "$s" ]; then
+            if [ "$path" = "$p" ]; then
+                echo $mnt
+            elif str_starts "$path" "$p"; then
+                echo $mnt/${path#$p/}
+            fi
+        fi
+    done
+}
+
 nfs_fetch_url() {
     local url="$1" outloc="$2" nfs="" server="" path="" options=""
     nfs_to_var "$url" || return 255
-    local filepath="${path%/*}" filename="${path##*/}"
+    local filepath="${path%/*}" filename="${path##*/}" mntdir=""
 
-    # TODO: check to see if server:/filepath is already mounted
-    local mntdir="$(mkuniqdir /run nfs_mnt)"
-    mount_nfs $nfs:$server:$path${options:+:$options} $mntdir
-    # FIXME: schedule lazy unmount during pre-pivot hook
+    # skip mount if server:/filepath is already mounted
+    mntdir=$(nfs_already_mounted $server $path)
+    if [ -z "$mntdir" ]; then
+        local mntdir="$(mkuniqdir /run nfs_mnt)"
+        mount_nfs $nfs:$server:$path${options:+:$options} $mntdir
+        # lazy unmount during pre-pivot hook
+        inst_hook --hook pre-pivot --name 99url-lib-umount-nfs umount -l $mntdir
+    fi
 
     if [ -z "$outloc" ]; then
         outloc="$mntdir/$filename"
-- 
1.7.7.6

[PATCH 8/9] add img-lib: a library for handling filesystem images

[Will Woods]

img-lib handles identifying and unpacking archives (uncompressed or
compressed) and filesystem images.

Currently tar and gzip are required; cpio and xz are optional, and bzip2
is supported but unused.

Signed-off-by: Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 modules.d/99img-lib/img-lib.sh      |   72 +++++++++++++++++++++++++++++++++++
 modules.d/99img-lib/module-setup.sh |   23 +++++++++++
 2 files changed, 95 insertions(+), 0 deletions(-)
 create mode 100755 modules.d/99img-lib/img-lib.sh
 create mode 100755 modules.d/99img-lib/module-setup.sh

diff --git a/modules.d/99img-lib/img-lib.sh b/modules.d/99img-lib/img-lib.sh
new file mode 100755
index 0000000..582b416
--- /dev/null
+++ b/modules.d/99img-lib/img-lib.sh
@@ -0,0 +1,72 @@
+#!/bin/bash
+# img-lib.sh: utilities for dealing with archives and filesystem images.
+#
+# TODO: identify/unpack rpm, deb?
+
+
+# super-simple "file" that only identifies archives.
+# works with stdin if $1 is not set.
+det_archive() {
+    local bz="BZh" xz="$(echo -e '\xfd7zXZ')" gz="$(echo -e '\x1f\x8b')"
+    local headerblock="$(dd ${1:+if=$1} bs=262 count=1 2>/dev/null)"
+    case "$headerblock" in
+        $xz*) echo "xz" ;;
+        $gz*) echo "gzip" ;;
+        $bz*) echo "bzip2" ;;
+        07070*) echo "cpio" ;;
+        *ustar) echo "tar" ;;
+    esac
+}
+
+# determine filesystem type for a filesystem image
+det_fs_img() {
+    local dev=$(losetup --find --show "$1") rv=""
+    det_fs $dev; rv=$?
+    losetup -d $dev
+    return $rv
+}
+
+# unpack_archive ARCHIVE OUTDIR
+# unpack a (possibly compressed) cpio/tar archive
+unpack_archive() {
+    local img="$1" outdir="$2" archiver="" decompr=""
+    local ft="$(det_archive $img)"
+    case "$ft" in
+        xz|gzip|bzip2) decompr="$decompr -dc" ;;
+        cpio|tar) decompr="cat";;
+        *) return 1 ;;
+    esac
+    ft="$($decompr $img | det_archive)"
+    case "$ft" in
+        cpio) archiver="cpio -iumd" ;;
+        tar) archiver="tar -xf -" ;;
+        *) return 2 ;;
+    esac
+    mkdir -p $outdir
+    ( cd $outdir; $decompr | $archiver 2>/dev/null ) < $img
+}
+
+# unpack_fs FSIMAGE OUTDIR
+# unpack a filesystem image
+unpack_fs() {
+    local img="$1" outdir="$2" mnt="$(mkuniqdir /tmp unpack_fs.)"
+    mount -o loop $img $mnt || { rmdir $mnt; return 1; }
+    mkdir -p $outdir; outdir="$(cd $outdir; pwd)"
+    ( cd $mnt; cp -a -t $outdir . )
+    umount $mnt
+    rmdir $mnt
+}
+
+# unpack an image file - compressed/uncompressed cpio/tar, filesystem, whatever
+# unpack_img IMAGEFILE OUTDIR
+unpack_img() {
+    local img="$1" outdir="$2"
+    [ -r "$img" ] || { warn "can't read img!"; return 1; }
+    [ -n "$outdir" ] || { warn "unpack_img: no output dir given"; return 1; }
+
+    if [ "$(det_img $img)" ]; then
+        unpack_archive "$@" || { warn "can't unpack archive file!"; return 1; }
+    else
+        unpack_fs "$@" || { warn "can't unpack filesystem image!"; return 1; }
+    fi
+}
diff --git a/modules.d/99img-lib/module-setup.sh b/modules.d/99img-lib/module-setup.sh
new file mode 100755
index 0000000..eead2ab
--- /dev/null
+++ b/modules.d/99img-lib/module-setup.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+# module-setup for img-lib
+
+check() {
+    for cmd in tar gzip dd; do
+        command -v $cmd >/dev/null || return 1
+    done
+    return 255
+}
+
+depends() {
+    return 0
+}
+
+install() {
+    # NOTE/TODO: we require bash, but I don't know how to specify that..
+    dracut_install tar gzip dd
+    dracut_install -o cpio xz
+    # TODO: make this conditional on a cmdline flag / config option
+    # dracut_install -o bzip2
+    inst "$moddir/img-lib.sh" "/lib/img-lib.sh"
+}
+
-- 
1.7.7.6

[PATCH 9/9] skiproot - script for network setup without network root

[Will Woods]

This allows you to set up the network for non-netroot things (like
fetching updates for live images, etc).

The other parse-*.sh script can set 'netroot=skip', and anything that
should happen when the network comes up can be put into the 'netroot'
hook, something like this:

  echo "fetch-liveupdate $update_url" >> $hookdir/netroot/liveupdates.sh

Signed-off-by: Will Woods <wwoods-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
---
 modules.d/40network/module-setup.sh |    1 +
 modules.d/40network/skiproot        |   14 ++++++++++++++
 2 files changed, 15 insertions(+), 0 deletions(-)
 create mode 100755 modules.d/40network/skiproot

diff --git a/modules.d/40network/module-setup.sh b/modules.d/40network/module-setup.sh
index 2f4d93d..a224bc5 100755
--- a/modules.d/40network/module-setup.sh
+++ b/modules.d/40network/module-setup.sh
@@ -75,6 +75,7 @@ install() {
     dracut_install -o brctl ifenslave
     inst "$moddir/ifup" "/sbin/ifup"
     inst "$moddir/netroot" "/sbin/netroot"
+    inst "$moddir/skiproot" "/sbin/skiproot"
     inst "$moddir/dhclient-script" "/sbin/dhclient-script"
     inst "$moddir/net-lib.sh" "/lib/net-lib.sh"
     inst_simple "$moddir/dhclient.conf" "/etc/dhclient.conf"
diff --git a/modules.d/40network/skiproot b/modules.d/40network/skiproot
new file mode 100755
index 0000000..86793c7
--- /dev/null
+++ b/modules.d/40network/skiproot
@@ -0,0 +1,14 @@
+#!/bin/bash
+# skiproot - a netroot handler that doesn't do anything.
+#
+# useful if you want the network up for non-root things (like fetching
+# updates, kickstarts, drivers, etc.) but you're not actually using a
+# network root device.
+#
+# Your other parse-*.sh script should set "netroot=skip", and anything
+# that you want to happen when the network comes up should be put into
+# the "netroot" hook, e.g.:
+#echo "fetch-liveupdate $updateurl" > $hookdir/netroot/fetch-updates.sh
+
+# do nothing!
+true
-- 
1.7.7.6

Re: [PATCH 9/9] skiproot - script for network setup without network root

[Will Woods]

On Tue, 2012-02-14 at 12:38 -0500, Will Woods wrote:
> This allows you to set up the network for non-netroot things (like
> fetching updates for live images, etc).
> 
> The other parse-*.sh script can set 'netroot=skip', and anything that
> should happen when the network comes up can be put into the 'netroot'
> hook, something like this:
> 
>   echo "fetch-liveupdate $update_url" >> $hookdir/netroot/liveupdates.sh

This patch can be ignored - I like the rd.neednet solution better,
although I wish it ran through the setup in netroot and ran the
'netroot' hooks..

-w

Re: [PATCH 1/9] 40network: add net-lib.sh

[Harald Hoyer]

Am 14.02.2012 18:38, schrieb Will Woods:
> diff --git a/modules.d/40network/net-lib.sh b/modules.d/40network/net-lib.sh
> +#!/bin/bash


Does it have to be bash?

Re: [PATCH] add net-lib, nfs-lib, url-lib, and img-lib

[Harald Hoyer]

Am 14.02.2012 18:38, schrieb Will Woods:
> Hi all,
> 
> This set of patches adds a bunch of "libraries" to dracut:
> - net-lib adds some utility functions to module 40network
> - nfs-lib provides the special nfs handling that was in 95nfs/nfsroot
> - url-lib is a new module for handling http{s}/ftp URLs with curl
> - img-lib is a new module for dealing with archives & fs images.
> 
> They might need tweaking for style/GPL headers/etc - just let me know and
> I'll fix up the patches.
> 
> Only patch 3 and 5 change existing code - the rest should be very low-risk.
> 
> Patch 3 ports the existing nfsroot to nfs-lib. This is kind of a
> proof-of-concept (look how short it is! whee!) but it *has not been tested*.
> Feel free to take or leave that patch - but if someone could apply it and
> help test it, I would be *very* thankful!
> 
> Patch 5 moves 90livenet to url-lib, and I *have* tested this to confirm that
> it works.
> 
> If these are accepted, some next steps might be:
> - move things out of dracut-lib.sh to net-lib or nfs-lib
>   (wait_for_if_up, nfs_root_to_var, ip_to_var, etc.)
> - add rpm/deb support to img-lib
>   (is there a way to do optional dependencies, specified in dracut.conf?)
> - add support for live "updates" images to livenet
> 
> Thanks for looking these over,
> 
> -w
> 

Pushed all of them and corrected nfs-lib.sh