Re: Remote management of embedded devices

[Alex J Lennon <ajlennon@dynamicdevices.co.uk>]

+
On 07/03/2014 13:41, Bryan Evenson wrote:
> Alex,
>
>> -----Original Message-----
>> From: yocto-bounces@yoctoproject.org [mailto:yocto-
>> bounces@yoctoproject.org] On Behalf Of Alex J Lennon
>> Sent: Friday, March 07, 2014 7:30 AM
>> To: yocto@yoctoproject.org
>> Subject: [yocto] Remote management of embedded devices
>>
>> Hi,
>>
>> I'm looking into remote management solutions for an upcoming headless
>> mesh edge router running Poky. I think, at least in the initial rollout we're
>> going to need something more than, say, a cron-based package update
>> facility.
>>
>> I'm currently thinking of going down the route of a cloud based server
>> providing SSH port forwarding to the embedded devices, and then perhaps
>> putting some scripting together on top of that to enable monitoring,
>> configuration, and control.
> This is similar to what we have running.  We're using opkg for the package management system and allow firmware upgrade through the USB stick or through remote access.  We have an SSH server for remote access, and each device has its own private key for access into the server.  If the device is in the middle of an SSH session with the remote access server, we can then SSH into the device from our server if we want to do some deeper diagnostics on an issue with a device.  We have separate HTTP server which each device queries to see if it needs to check in.  So instead of having to do an ssh login each time to check if there's a firmware upgrade available, it just needs to do a HTTP GET to see if there's firmware available or if we want to check on its status.
>
> The biggest issues we've had have been due to our network setup and handling upgrade both through the network and the USB stick.  We are using "opkg upgrade --download-only" as the first step of the upgrade process to make sure that we don't do a partial upgrade.  opkg-0.1.8 doesn't do --download-only for file:// sources; why download a file that is already on the filesystem?  So I had to add a patch so it would download files from the USB stick.  We also had an issue with DNS names because the server has a different name when the device finds it locally on our intranet then when it connects remotely, so we had to setup separate mirrors.  Other than that it's been working pretty well.  

Thanks Bryan, thanks really interesting. We do something similar with
HTTP requests on various projects and that might well be something to
apply here. I guess the limitation of this approach is that you are
limited in when you can obtain your back channel to the device to the
frequency with which they connect to the HTTP server. I do wonder just
how much load there would be on a server handling a lot of SSH
connections if almost all of those connections were just idling, sending
keep alives now and again. I keep meaning to try to put together some
metrics on that as, if I'm right and the load is low (as long as we
ensure connections from devices are never synchronised, e.g. power
failure) then I'd like to have an "always on" back-channel available

Interestesting about opkg too - thanks for that!

> I'd also love to hear other people's solutions to see if they have done something similar or came up with a different solution.

Me too