[Buildroot] Persistent dropbear keys

[Buildroot] Persistent dropbear keys

[Thomas De Schampheleire]

Hello,

Commit e7d04dd2df8bb935c61f7c814ee88eba7e75b5e4 (package/dropbear: fix
generating keys on RO file systems) (+ some subsequent commits)
changed the handling of the /etc/dropbear directory. Previously
/etc/dropbear was a real directory in the rootfs, now it initially is
a link to /var/run/dropbear. During S50dropboar, the link is replaced
with a real (empty) directory (if rootfs is writable) or a warning is
given.

I understand all this. However, what I do not understand is how you
are then creating persistent dropbear keys. From how I understand the
code, the keys are persistent across reboots, but not between upgrades
of the rootfs, because after an upgrade a new empty /etc/dropbear is
created.

In my case, the rootfs is an initramfs, but mounted rw at boot time.

The solution I have been using is with an S49dropbear_keys script that:
- at 'stop', verifies the correctness of the keys in /etc/dropbear
(with dropbearkey) and if ok copies them to a real persistent medium,
- at 'start', verifies if there are any keys on the persistent medium,
verify their correctness, and if ok copies them to /etc/dropbear.

I will now need to adapt this script S49dropbear_keys to first remove
the /etc/dropbear link, to keep its functionality.

But it pops up the question how other people are handling this.

Thanks,
Thomas

[Buildroot] Persistent dropbear keys

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

 > Hello,
 > Commit e7d04dd2df8bb935c61f7c814ee88eba7e75b5e4 (package/dropbear: fix
 > generating keys on RO file systems) (+ some subsequent commits)
 > changed the handling of the /etc/dropbear directory. Previously
 > /etc/dropbear was a real directory in the rootfs, now it initially is
 > a link to /var/run/dropbear. During S50dropboar, the link is replaced
 > with a real (empty) directory (if rootfs is writable) or a warning is
 > given.

 > I understand all this. However, what I do not understand is how you
 > are then creating persistent dropbear keys. From how I understand the
 > code, the keys are persistent across reboots, but not between upgrades
 > of the rootfs, because after an upgrade a new empty /etc/dropbear is
 > created.

If your upgrade overwrites /etc/dropbear, then yes.

E.G. I use a persistent writable unionfs on /etc, so changes to /etc are
not lost after an upgrade.


 > In my case, the rootfs is an initramfs, but mounted rw at boot time.

 > The solution I have been using is with an S49dropbear_keys script that:
 > - at 'stop', verifies the correctness of the keys in /etc/dropbear
 > (with dropbearkey) and if ok copies them to a real persistent medium,
 > - at 'start', verifies if there are any keys on the persistent medium,
 > verify their correctness, and if ok copies them to /etc/dropbear.

Why don't you just make /etc/dropbear a symlink to your persistent
medium?

-- 
Bye, Peter Korsgaard

[Buildroot] Persistent dropbear keys

[Arnout Vandecappelle]

On 08-01-16 18:45, Peter Korsgaard wrote:
>>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:
> 
>  > Hello,
>  > Commit e7d04dd2df8bb935c61f7c814ee88eba7e75b5e4 (package/dropbear: fix
>  > generating keys on RO file systems) (+ some subsequent commits)
>  > changed the handling of the /etc/dropbear directory. Previously
>  > /etc/dropbear was a real directory in the rootfs, now it initially is
>  > a link to /var/run/dropbear. During S50dropboar, the link is replaced
>  > with a real (empty) directory (if rootfs is writable) or a warning is
>  > given.
> 
>  > I understand all this. However, what I do not understand is how you
>  > are then creating persistent dropbear keys. From how I understand the
>  > code, the keys are persistent across reboots, but not between upgrades
>  > of the rootfs, because after an upgrade a new empty /etc/dropbear is
>  > created.
> 
> If your upgrade overwrites /etc/dropbear, then yes.
> 
> E.G. I use a persistent writable unionfs on /etc, so changes to /etc are
> not lost after an upgrade.
> 
> 
>  > In my case, the rootfs is an initramfs, but mounted rw at boot time.
> 
>  > The solution I have been using is with an S49dropbear_keys script that:
>  > - at 'stop', verifies the correctness of the keys in /etc/dropbear
>  > (with dropbearkey) and if ok copies them to a real persistent medium,
>  > - at 'start', verifies if there are any keys on the persistent medium,
>  > verify their correctness, and if ok copies them to /etc/dropbear.
> 
> Why don't you just make /etc/dropbear a symlink to your persistent
> medium?

 We should probably add some explanation in the help text about this possibility.

 I'll try to cook something up.


 Regards,
 Arnout
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Persistent dropbear keys

[Thomas De Schampheleire]

On Fri, Jan 8, 2016 at 6:45 PM, Peter Korsgaard <peter@korsgaard.com> wrote:
>>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:
>
>  > Hello,
>  > Commit e7d04dd2df8bb935c61f7c814ee88eba7e75b5e4 (package/dropbear: fix
>  > generating keys on RO file systems) (+ some subsequent commits)
>  > changed the handling of the /etc/dropbear directory. Previously
>  > /etc/dropbear was a real directory in the rootfs, now it initially is
>  > a link to /var/run/dropbear. During S50dropboar, the link is replaced
>  > with a real (empty) directory (if rootfs is writable) or a warning is
>  > given.
>
>  > I understand all this. However, what I do not understand is how you
>  > are then creating persistent dropbear keys. From how I understand the
>  > code, the keys are persistent across reboots, but not between upgrades
>  > of the rootfs, because after an upgrade a new empty /etc/dropbear is
>  > created.
>
> If your upgrade overwrites /etc/dropbear, then yes.
>
> E.G. I use a persistent writable unionfs on /etc, so changes to /etc are
> not lost after an upgrade.
>
>
>  > In my case, the rootfs is an initramfs, but mounted rw at boot time.
>
>  > The solution I have been using is with an S49dropbear_keys script that:
>  > - at 'stop', verifies the correctness of the keys in /etc/dropbear
>  > (with dropbearkey) and if ok copies them to a real persistent medium,
>  > - at 'start', verifies if there are any keys on the persistent medium,
>  > verify their correctness, and if ok copies them to /etc/dropbear.
>
> Why don't you just make /etc/dropbear a symlink to your persistent
> medium?

While I had not thought of that when designing my current solution,
doing that without additional checking can be a problem: if the key on
the persistent storage gets corrupted somehow, new connections would
no longer be possible. I.e. make a small change in the existing key in
/etc/dropbear, restart dropbear, and connection attempts fail with
following syslog entry:

Jan  3 06:41:58 hostname authpriv.info dropbear[14789]: Exit before
auth: Couldn't read or generate hostkey
/etc/dropbear/dropbear_rsa_host_key

Just like with my current solution that actually copies the keys
(rather than symlinking) I think you need to verify the keys
before/after reboot, using dropbearkey. However, the symlink solution
has the disadvantage in this case that an unexpected reboot (without
the additional key check) potentially results in a corrupted key in
/etc/dropbear, while the copy solution will only copy validated keys
(during controlled reboot).

So, I don't think my current solution is so bad after all. How do you see this?

/Thomas

[Buildroot] Persistent dropbear keys

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

Hi,

>> Why don't you just make /etc/dropbear a symlink to your persistent
 >> medium?

 > While I had not thought of that when designing my current solution,
 > doing that without additional checking can be a problem: if the key on
 > the persistent storage gets corrupted somehow, new connections would
 > no longer be possible. I.e. make a small change in the existing key in
 > /etc/dropbear, restart dropbear, and connection attempts fail with
 > following syslog entry:

 > Jan  3 06:41:58 hostname authpriv.info dropbear[14789]: Exit before
 > auth: Couldn't read or generate hostkey
 > /etc/dropbear/dropbear_rsa_host_key

Yeah :/

 > Just like with my current solution that actually copies the keys
 > (rather than symlinking) I think you need to verify the keys
 > before/after reboot, using dropbearkey. However, the symlink solution
 > has the disadvantage in this case that an unexpected reboot (without
 > the additional key check) potentially results in a corrupted key in
 > /etc/dropbear, while the copy solution will only copy validated keys
 > (during controlled reboot).

First of all, it is important to understand why these files might get
corrupted, E.G. what kind of issue you try to work around. Dropbear used
to have problems, but afaik does the correct fsync / rename dance to
safely create these files since 2015.67:

https://github.com/mkj/dropbear/commit/4ba830fc31c056aaada774ce29bb7d4e136b5dcd

But that naturally still relies on a working fs / block device. Is this
your concern or something else?

With the fix in 2015.67 I haven't seen corrupted host keys any more in
the projects I work on, and for the use cases we have that is good
enough (E.G. ssh is not a feature accessible to end users, and our
factory reset button throws away the /etc overlay, so this could be used
to fix corrupted host keys if needed).

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] Persistent dropbear keys

[Thomas De Schampheleire]

On Mon, Jan 11, 2016 at 10:49 AM, Peter Korsgaard <peter@korsgaard.com> wrote:
>>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:
>
> Hi,
>
>>> Why don't you just make /etc/dropbear a symlink to your persistent
>  >> medium?
>
>  > While I had not thought of that when designing my current solution,
>  > doing that without additional checking can be a problem: if the key on
>  > the persistent storage gets corrupted somehow, new connections would
>  > no longer be possible. I.e. make a small change in the existing key in
>  > /etc/dropbear, restart dropbear, and connection attempts fail with
>  > following syslog entry:
>
>  > Jan  3 06:41:58 hostname authpriv.info dropbear[14789]: Exit before
>  > auth: Couldn't read or generate hostkey
>  > /etc/dropbear/dropbear_rsa_host_key
>
> Yeah :/
>
>  > Just like with my current solution that actually copies the keys
>  > (rather than symlinking) I think you need to verify the keys
>  > before/after reboot, using dropbearkey. However, the symlink solution
>  > has the disadvantage in this case that an unexpected reboot (without
>  > the additional key check) potentially results in a corrupted key in
>  > /etc/dropbear, while the copy solution will only copy validated keys
>  > (during controlled reboot).
>
> First of all, it is important to understand why these files might get
> corrupted, E.G. what kind of issue you try to work around. Dropbear used
> to have problems, but afaik does the correct fsync / rename dance to
> safely create these files since 2015.67:
>
> https://github.com/mkj/dropbear/commit/4ba830fc31c056aaada774ce29bb7d4e136b5dcd
>
> But that naturally still relies on a working fs / block device. Is this
> your concern or something else?
>
> With the fix in 2015.67 I haven't seen corrupted host keys any more in
> the projects I work on, and for the use cases we have that is good
> enough (E.G. ssh is not a feature accessible to end users, and our
> factory reset button throws away the /etc overlay, so this could be used
> to fix corrupted host keys if needed).

While we haven't seen corrupted host keys in real life (but apparently
you did already), I think it is a good safety precaution to take it
into account. The fix in dropbear you refer to is very recent, so up
until now there was anyway a good reason to not use a direct symlink
/etc/dropbear -> /persistent/something.

But even with this fix, how much guarantee can you give that there is
not another bug in dropbear or the filesystem, or a script of our own
for that matter, that would inadvertently change the file or cause
some kind of corruption in it?

In our case, these boxes are 'out there' (and this can be in the
middle of a city or the middle of the desert). In case of trouble, SSH
access is the sole practical connection available and as such it needs
to be reliable. The alternative is having a technician driving to the
box, either attaching a serial connection or simply replacing the box
and sending the 'bad' one to skilled people. In either case, this is a
very expensive operation that should be avoided as much as possible.

For reference, if it can be useful to someone, I'm pasting the script
we're using (S49dropbear_keys):

#!/bin/sh
# Store/restore SSH host keys to/from persistent storage

PERSISTENT=/persistent/dropbear
VOLATILE=/etc/dropbear

# All created files should be forbidden by non-root users
umask 077

keys="dropbear_rsa_host_key dropbear_dss_host_key dropbear_ecdsa_host_key"

check_key() # (key)
{
     dropbearkey -y -f "$1" > /dev/null
}

case "$1" in
    start)
        # restore
        #
        # since buildroot 2015.11.1, /etc/dropbear (potentially read-only)
        # is created in the rootfs as a link to /var/run/dropbear (definitely
        # writable because /run is a tmpfs). In S50dropbear, it is checked
        # whether /etc/dropbear is writable in which case the link is replaced
        # by a real empty dir.
        # This mechanism does not work for this script (nor is it needed)
        # so first replace the link with an actual directory.
        if [ -L $VOLATILE ]; then
            rm -f $VOLATILE
        fi
        mkdir -p $VOLATILE
        for key in $keys; do
            if [ -f $PERSISTENT/$key ]; then
                if check_key $PERSISTENT/$key; then
                    cp $PERSISTENT/$key $VOLATILE/
                else
                    # corrupt key
                    rm -f $PERSISTENT/$key
                fi
            fi
        done
        ;;
    stop)
        # store
        mkdir -p $PERSISTENT
        for key in $keys; do
            if [ -f $VOLATILE/$key ]; then
                if check_key $VOLATILE/$key; then
                    cp $VOLATILE/$key $PERSISTENT/
                else
                    # corrupt key
                    rm -f $VOLATILE/$key
                fi
            fi
        done
        sync
        ;;
    *)
        echo "Usage: $0 {start|stop}" >&2
        exit 1
        ;;
esac



/Thomas

[Buildroot] Persistent dropbear keys

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

Hi,

>> With the fix in 2015.67 I haven't seen corrupted host keys any more in
 >> the projects I work on, and for the use cases we have that is good
 >> enough (E.G. ssh is not a feature accessible to end users, and our
 >> factory reset button throws away the /etc overlay, so this could be used
 >> to fix corrupted host keys if needed).

 > While we haven't seen corrupted host keys in real life (but apparently
 > you did already), I think it is a good safety precaution to take it
 > into account. The fix in dropbear you refer to is very recent, so up
 > until now there was anyway a good reason to not use a direct symlink
 > /etc/dropbear -> /persistent/something.

I've seen empty (not corrupted data) key files from the missing fsync,
yes.

Yes, the fix is quite recent, but the feature to let dropbear
automatically generate the keys is also relatively new (2013.62).


 > But even with this fix, how much guarantee can you give that there is
 > not another bug in dropbear or the filesystem, or a script of our own
 > for that matter, that would inadvertently change the file or cause
 > some kind of corruption in it?

Guarantees are hard to come by ;) Now, the code that writes the file is
fairly small and self contained, but with a writable /etc you could
indeed conceptually have all kinds of problems (not limited to dropbear
keys).


 > In our case, these boxes are 'out there' (and this can be in the
 > middle of a city or the middle of the desert). In case of trouble, SSH
 > access is the sole practical connection available and as such it needs
 > to be reliable. The alternative is having a technician driving to the
 > box, either attaching a serial connection or simply replacing the box
 > and sending the 'bad' one to skilled people. In either case, this is a
 > very expensive operation that should be avoided as much as possible.

Yes, like just about anything in engineering, it is a matter of
tradeoffs - And different projects may need different solutions.

The solution you describe is certainly fine. As I mentioned, I'm
currently using a reset-to-factory-defaults button in case stuff really
get corrupt. Other solutions I've used are:

- Generate / install (ssh keys, TLS certificates, ..) data once during
  production flow and write this to a file system. In operation this
  file system is always mounted read only.

- Prebake SSH keys in (RO) rootfs (!).

And I'm sure there's others as well.

-- 
Venlig hilsen,
Peter Korsgaard