Web frameworks on embedded linux

Web frameworks on embedded linux

[Edward Wingate]

[-- Attachment #1: Type: text/plain, Size: 2598 bytes --]

I've been trying to get an existing python flask application running on an
embedded linux distro (Poky).  The web app was originally deployed on a
Debian distro.  I need to move the application to a less heftier board
and wanted to try a distro created by Yocto, rather than try to pare down
Debian.

I've been following the chain of dependencies for the web application and
adding them to Yocto to try and get the app to work (nginx, uwsgi, python,
pyserial, etc.), and hit a bump - Python's built-in logging and codecs
modules are not provided by Yocto.  Maybe I can add them myself, or maybe
I'll rip them out of the application, but I wanted to take a step back and
see if maybe Python/Flask is not the best app framework to use in an
embedded linux distro.

CPU is not a problem.  This board has ARM Cortex A9 processor running at
667MHz.  RAM should be OK as well at 512MB, but I am limited by the 32 MB
flash size on the board. I started with core-image-minimal on Poky (which
was ~3MB), and I'm up to almost 8MB now.  I guesstimate that I'm maybe 50%
of the way along the dependency chain. I haven't even gotten to flask yet,
just the webserver, uwsgi and plain python.  The size I'm referring to is
just the ramdisk image, not including FSBL nor u-boot (which are minimal
and shouldn't grow much, if at all, at this point).  I don't know what I'll
eventually end up at, but I do have to share the 32MB flash with other
data; it's not dedicated to the OS/Linux.  How much space the OS gets is
still up in the air.  I'd say at least 16 MB and maybe as much as 20-24 MB.

While it would be nice to reuse the already written web application, if the
Python/Flask stack turns out to be too heavy, a port/rewrite is not out of
the question.  This application just displays a web frontend to get/set
values on a device over a serial port.

So I wanted to get the list's thoughts and experience on:

1) Web application frameworks to use with an embedded linux constrained by
image size.  How well supported by Yocto recipes is the framework?  I've
been intrigued by the Lua language and the Kepler/Orbit web framework for
Lua, but except for Lua, there doesn't appear to be Yocto recipes for them.


2) Python modules that Yocto doesn't provide recipes for, like logging and
codecs modules.  How difficult is it to augment python-core?  Is it just a
matter of adding your own recipe in your own layer? Logging and codecs is
normally a part of a default Python install, so not sure yet how I would go
about adding this.

Thanks for your help.

Edward

[-- Attachment #2: Type: text/html, Size: 2749 bytes --]

Re: Web frameworks on embedded linux

[Maciek Borzecki]

On czw, 2014-11-13 at 23:12 -0800, Edward Wingate wrote:
> I've been trying to get an existing python flask application running
> on an embedded linux distro (Poky).  The web app was originally
> deployed on a Debian distro.  I need to move the application to a less
> heftier board and wanted to try a distro created by Yocto, rather than
> try to pare down Debian.
> 
> I've been following the chain of dependencies for the web application
> and adding them to Yocto to try and get the app to work (nginx, uwsgi,
> python, pyserial, etc.), and hit a bump - Python's built-in logging
> and codecs modules are not provided by Yocto.  Maybe I can add them
> myself, or maybe I'll rip them out of the application, but I wanted to
> take a step back and see if maybe Python/Flask is not the best app
> framework to use in an embedded linux distro.  
Have you installed python-modules pacakge? This package pulls in all
standard python modules as dependencies.

I've used Tornado to provide a REST-ful API for an Android application
and some basic web pages with status information and so on. Worked quite
well.



-- 
Maciej Borzęcki
Senior Software Developer at Open-RnD Sp. z o.o., Poland
www.open-rnd.pl
mobile: +48 889 117 365, fax: +48 42 657 9079

Niniejsza wiadomość wraz z załącznikami może zawierać chronione prawem
lub poufne informacje i została wysłana wyłącznie do wiadomości i
użytku osób, do których została zaadresowana. Jeśli wiadomość została
otrzymana przypadkowo zabrania się jej kopiowania lub rozsyłania do
osób trzecich. W takim przypadku uprasza się o natychmiastowe
zniszczenie wiadomości oraz poinformowanie nadawcy o zaistniałej
sytuacji za pomocą wiadomości zwrotnej. Dziękujemy.

This message, including any attachments hereto, may contain privileged
or confidential information and is sent solely for the attention and
use of the intended addressee(s). If you are not an intended addressee,
you may neither use this message nor copy or deliver it to anyone. In
such case, you should immediately destroy this message and kindly notify
the sender by reply email. Thank you.

Re: Web frameworks on embedded linux

[Edward Wingate]

[-- Attachment #1: Type: text/plain, Size: 1756 bytes --]

On Fri, Nov 14, 2014 at 12:13 AM, Maciek Borzecki <
maciej.borzecki@open-rnd.pl> wrote:

> On czw, 2014-11-13 at 23:12 -0800, Edward Wingate wrote:
> > I've been trying to get an existing python flask application running
> > on an embedded linux distro (Poky).  The web app was originally
> > deployed on a Debian distro.  I need to move the application to a less
> > heftier board and wanted to try a distro created by Yocto, rather than
> > try to pare down Debian.
> >
> > I've been following the chain of dependencies for the web application
> > and adding them to Yocto to try and get the app to work (nginx, uwsgi,
> > python, pyserial, etc.), and hit a bump - Python's built-in logging
> > and codecs modules are not provided by Yocto.  Maybe I can add them
> > myself, or maybe I'll rip them out of the application, but I wanted to
> > take a step back and see if maybe Python/Flask is not the best app
> > framework to use in an embedded linux distro.
> Have you installed python-modules pacakge? This package pulls in all
> standard python modules as dependencies.
>
>
No, I haven't.  I still have a lot to learn about Yocto and recipes.  I
went ahead and pulled python-modules in and and got a lot further in making
this app work.  Thanks, Maciek, for your help.

The ramdisk image ballooned to 22MB though.  It's expected, but due to
flash size constraints, Python may not be an option in this case.


> I've used Tornado to provide a REST-ful API for an Android application
> and some basic web pages with status information and so on. Worked quite
> well.
>

Do you remember what your final ramdisk image size was?

I would still love to hear what web frameworks others here have used on
embedded linux.

[-- Attachment #2: Type: text/html, Size: 2417 bytes --]

Re: Web frameworks on embedded linux

[Gary Thomas]

On 2014-11-14 10:10, Edward Wingate wrote:
> On Fri, Nov 14, 2014 at 12:13 AM, Maciek Borzecki <maciej.borzecki@open-rnd.pl <mailto:maciej.borzecki@open-rnd.pl>> wrote:
>
>     On czw, 2014-11-13 at 23:12 -0800, Edward Wingate wrote:
>     > I've been trying to get an existing python flask application running
>     > on an embedded linux distro (Poky).  The web app was originally
>     > deployed on a Debian distro.  I need to move the application to a less
>     > heftier board and wanted to try a distro created by Yocto, rather than
>     > try to pare down Debian.
>     >
>     > I've been following the chain of dependencies for the web application
>     > and adding them to Yocto to try and get the app to work (nginx, uwsgi,
>     > python, pyserial, etc.), and hit a bump - Python's built-in logging
>     > and codecs modules are not provided by Yocto.  Maybe I can add them
>     > myself, or maybe I'll rip them out of the application, but I wanted to
>     > take a step back and see if maybe Python/Flask is not the best app
>     > framework to use in an embedded linux distro.
>     Have you installed python-modules pacakge? This package pulls in all
>     standard python modules as dependencies.
>
>
> No, I haven't.  I still have a lot to learn about Yocto and recipes.  I went ahead and pulled python-modules in and and got a lot further in making this app work.  Thanks, Maciek,
> for your help.
>
> The ramdisk image ballooned to 22MB though.  It's expected, but due to flash size constraints, Python may not be an option in this case.

You can always just bring in the packages you need (python-modules is a
pretty heavy hammer!)  Python is split into ~90 separate packages, each
with a [small] set of modules/functionality.

>     I've used Tornado to provide a REST-ful API for an Android application
>     and some basic web pages with status information and so on. Worked quite well.
>
>
> Do you remember what your final ramdisk image size was?
>
> I would still love to hear what web frameworks others here have used on embedded linux.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Web frameworks on embedded linux

[Nikolay Dimitrov]

Hi Edward,

On 11/14/2014 07:10 PM, Edward Wingate wrote:
> On Fri, Nov 14, 2014 at 12:13 AM, Maciek Borzecki
> <maciej.borzecki@open-rnd.pl <mailto:maciej.borzecki@open-rnd.pl>> wrote:
>
>     On czw, 2014-11-13 at 23:12 -0800, Edward Wingate wrote:
>     > I've been trying to get an existing python flask application running
>     > on an embedded linux distro (Poky).  The web app was originally
>     > deployed on a Debian distro.  I need to move the application to a less
>     > heftier board and wanted to try a distro created by Yocto, rather than
>     > try to pare down Debian.
>     >
>     > I've been following the chain of dependencies for the web application
>     > and adding them to Yocto to try and get the app to work (nginx, uwsgi,
>     > python, pyserial, etc.), and hit a bump - Python's built-in logging
>     > and codecs modules are not provided by Yocto.  Maybe I can add them
>     > myself, or maybe I'll rip them out of the application, but I wanted to
>     > take a step back and see if maybe Python/Flask is not the best app
>     > framework to use in an embedded linux distro.
>     Have you installed python-modules pacakge? This package pulls in all
>     standard python modules as dependencies.
>
>
> No, I haven't.  I still have a lot to learn about Yocto and recipes.  I
> went ahead and pulled python-modules in and and got a lot further in
> making this app work.  Thanks, Maciek, for your help.
>
> The ramdisk image ballooned to 22MB though.  It's expected, but due to
> flash size constraints, Python may not be an option in this case.
>
>     I've used Tornado to provide a REST-ful API for an Android application
>     and some basic web pages with status information and so on. Worked
>     quite well.
>
>
> Do you remember what your final ramdisk image size was?
>
> I would still love to hear what web frameworks others here have used on
> embedded linux.

I remember that before several years we worked on a prototype, that had
to deliver dynamic content over ethernet. It had to run on ARM board,
first with QNX and later with Linux. We then found a small web-server,
Mongoose (https://code.google.com/p/mongoose/), which had good license,
was lightweight, with very few dependencies, and had very clean
interface for extending (actually looking at the license, its now GPL,
but I remember that the older versions were licensed under MIT or BSD).

As far as I rememer, the whole thing boils down to registering a
callback, which is called each time you have to service a HTTP request.
We used this to write multiple small extensions in C that generated the
dynamic content, each plugins was responsible for a separate domain of
functionality. It sounds strange to do this in C instead of
higher-level language, but once you start seeing your memory usage to
be in KBytes instead of MBytes range, you start to like this little
component :D.

You can give it a spin, it builds easily for x86 and arm.

Regards,
Nikolay

Re: Web frameworks on embedded linux

[Maciek Borzecki]

On pią, 2014-11-14 at 09:10 -0800, Edward Wingate wrote:
> No, I haven't.  I still have a lot to learn about Yocto and recipes.
> I went ahead and pulled python-modules in and and got a lot further in
> making this app work.  Thanks, Maciek, for your help.
>
> The ramdisk image ballooned to 22MB though.  It's expected, but due to
> flash size constraints, Python may not be an option in this case.
>
>
>         I've used Tornado to provide a REST-ful API for an Android
>         application
>         and some basic web pages with status information and so on.
>         Worked quite well.
>
>
> Do you remember what your final ramdisk image size was?
It was a beefy application image, with lots of dependencies.

>
>
>
> I would still love to hear what web frameworks others here have used
> on embedded linux.

In yet another project we used lighttpd as request handler, while the
actual app was running as fastcgi service. HTML templates were handled
by Clearsilver, see http://www.clearsilver.net (really neat library, New BSD
license). IIRC, Clearsilver pulls in bdwgc as dependency, so there's no
hassle with memory management.

Quite recently we had yet another custom work where HTTP JSON-RPC
requests were handled using libsoup.

--
Maciej Borzęcki
Senior Software Developer at Open-RnD Sp. z o.o., Poland
www.open-rnd.pl
mobile: +48 889 117 365, fax: +48 42 657 9079

Niniejsza wiadomość wraz z załącznikami może zawierać chronione prawem
lub poufne informacje i została wysłana wyłącznie do wiadomości i
użytku osób, do których została zaadresowana. Jeśli wiadomość została
otrzymana przypadkowo zabrania się jej kopiowania lub rozsyłania do
osób trzecich. W takim przypadku uprasza się o natychmiastowe
zniszczenie wiadomości oraz poinformowanie nadawcy o zaistniałej
sytuacji za pomocą wiadomości zwrotnej. Dziękujemy.

This message, including any attachments hereto, may contain privileged
or confidential information and is sent solely for the attention and
use of the intended addressee(s). If you are not an intended addressee,
you may neither use this message nor copy or deliver it to anyone. In
such case, you should immediately destroy this message and kindly notify
the sender by reply email. Thank you.

Re: Web frameworks on embedded linux

[Edward Wingate]

> From: Gary Thomas <gary@mlbassoc.com>
> To: yocto@yoctoproject.org
> Date: Fri, 14 Nov 2014 12:04:23 -0700
>
> On 2014-11-14 11:52, Edward Wingate wrote:
>>
>> On Fri, Nov 14, 2014 at 9:39 AM, <yocto-request@yoctoproject.org <mailto:yocto-request@yoctoproject.org>> wrote:
>>
>>     From: Gary Thomas <gary@mlbassoc.com <mailto:gary@mlbassoc.com>>
>>
>>     You can always just bring in the packages you need (python-modules is a
>>     pretty heavy hammer!)  Python is split into ~90 separate packages, each
>>     with a [small] set of modules/functionality.
>>
>> I would really like to try this. How do I find out what the various python packages are and what's in each package?  I tried
>> http://layers.openembedded.org/layerindex/branch/master/recipes/, but don't find what I'm looking for (right now, Python's built-in logging and codecs modules).
>
> Try looking in your build tree.  Depending on your target configuration,
> you'll have either RPM or IPK packages and the python modules end up in
> the architecture dependent collection.
>
> It can get a bit tricky to decide which package contains the module(s) you need
> as many of them are grouped together.  If you know the name of the module, you
> can typically find it via a quick search, e.g. to find the package that contains
> the htmllib module:
>   $ find tmp/work/cortexa7hf-vfp-neon-amltd-linux-gnueabi/python/2.7.3-r0.3/packages-split/ -name htmllib.py
>   tmp/work/cortexa7hf-vfp-neon-amltd-linux-gnueabi/python/2.7.3-r0.3/packages-split/python-html/usr/lib/python2.7/htmllib.py
> so, in this case I'm looking for the 'python-html' package.
>
> Once you determine the package name, just add that to your image (or if you
> have things set up for package updates, you can just add the packages directly
> on your board).

Thanks so much Gary!  I pulled in just logging and codecs modules and
my image went from 22 MB down to 9MB, and the app is still working.
I'm hopeful I can keep the ramdisk image under 16MB now.

Moving on to install and configure flask and uwsgi to serve the app
now.  Speaking of which, how do you assign static IP instead
defaulting to DHCP, and get ssh keys into the rootfs instead of
Drpbear generating them every bootup?  Can this be accomplished
through Yocto or do you have to mount and modify the rootfs manually?