Status Update : Deployment of DOSEMU Application Server

Status Update : Deployment of DOSEMU Application Server

[Shaw Tong Tan]

A while back I sent a request to the list for advice on running such a setup.  I received several kind suggestions from the list members.  I am now in 1st week of deployment and decide to submit a status update.

My thanks to all the people who pass me info and solutions.

Basic:
1.  Multi-user DOS clipper-based software.
2.  User access from LAN and WAN.
3.  DOS-based dot matrix printing for LAN and WAN clients.

There are currently 2 major ways to do this.
A.  Microsoft Terminal Server/Citrix Products.
B.  Open Source Software.

Option A is known to work.  It does incur higher cost, hardware and software and license management overhead.  I was asked to explore Option B as priority choice.  In case it doesn't work then fall back to Option A.

Observation
1.  Server hardware Quad-Core
2.  Server software CentOS 5.x (32-bit)
3.  DOSEMU with FreeDOS (32-bit) standard RPM download from website
4.  To address the 100% CPU utilization issue, I have used a combination of specialized dos program, Linux NICE value tuning and CPU affinity control to achieve acceptable response to enduser
5.  Initially I use PUTTY and REMOTE ANSi Printing with lpansi for printing
6.  I ran into problem where small size print jobs are ok, but big size print job always hang.
7.  After fruitless workaround attempts and decided against running custom kernels, I made the decision not to use REMOTE ANSI Printing, thus creating a new problem on how to pass print job from server to remote WAN client over the Internet.
8.  Implement OPENVPN for remote WAN clients with DSL connection.
9.  With VPN tunnel open, setup std CUPS printer with SMB config for remote client printer.
10.  Tune the setup to minimize packet fragment, timeout and firewall rules issues.

Re: Status Update : Deployment of DOSEMU Application Server

[Alain M.]

Maybe you are over complicating...

I use Dosemu on a remote machine through ssh, very easyly.
1) each clinet use an instance of Dosemu at the server
2) all use the same linux directory as C:
3) I only needed to instal some fonts in the client machine.
4) I use a (free) pcl emulation software, whica generates .ps and is 
printed via Linux with cups. there is also an epson emulator, but it was 
older and the pcl worked first...

What I don't know:
1) if clipper locking is working with dosemu+freedos, you can test that 
easyly on a single machine
2) doemu does not implement any kind of flush printer buffer at LPT1 
close. This existe in ALL other dos multi user dos system, since 
Novell2. So spool timing can be an issue.

I can send you some info, but it would be nice then, if you make a 
public how to for that :)

Alain

Shaw Tong Tan escreveu:
> A while back I sent a request to the list for advice on running such a setup.  I received several kind suggestions from the list members.  I am now in 1st week of deployment and decide to submit a status update.
> 
> My thanks to all the people who pass me info and solutions.
> 
> Basic:
> 1.  Multi-user DOS clipper-based software.
> 2.  User access from LAN and WAN.
> 3.  DOS-based dot matrix printing for LAN and WAN clients.
> 
> There are currently 2 major ways to do this.
> A.  Microsoft Terminal Server/Citrix Products.
> B.  Open Source Software.
> 
> Option A is known to work.  It does incur higher cost, hardware and software and license management overhead.  I was asked to explore Option B as priority choice.  In case it doesn't work then fall back to Option A.
> 
> Observation
> 1.  Server hardware Quad-Core
> 2.  Server software CentOS 5.x (32-bit)
> 3.  DOSEMU with FreeDOS (32-bit) standard RPM download from website
> 4.  To address the 100% CPU utilization issue, I have used a combination of specialized dos program, Linux NICE value tuning and CPU affinity control to achieve acceptable response to enduser
> 5.  Initially I use PUTTY and REMOTE ANSi Printing with lpansi for printing
> 6.  I ran into problem where small size print jobs are ok, but big size print job always hang.
> 7.  After fruitless workaround attempts and decided against running custom kernels, I made the decision not to use REMOTE ANSI Printing, thus creating a new problem on how to pass print job from server to remote WAN client over the Internet.
> 8.  Implement OPENVPN for remote WAN clients with DSL connection.
> 9.  With VPN tunnel open, setup std CUPS printer with SMB config for remote client printer.
> 10.  Tune the setup to minimize packet fragment, timeout and firewall rules issues.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Status Update : Deployment of DOSEMU Application Server

[Andrew Joakimsen]

I am doing the *EXACT* thing as Shaw, note that there is also an
"Option C" which is just share the files over the VPN... which is
slow, that is what we were doing hence the need to find a "better"
solution. The author of the software says to use "Option B" ... why
use a graphics remote access (RDC) to a text program? Its more of the
same issue as the original, what about for users on dialup or using
wireless (GSM and CDMA) access... SSH will work better Also "option d"
use cygwin (for SSH-server) or the MS telnet server, neither work
right and telnet is insecure.

On Wed, Oct 29, 2008 at 10:33 AM, Alain M. <alainm@pobox.com> wrote:
>
> What I don't know:
> 1) if clipper locking is working with dosemu+freedos, you can test that
> easyly on a single machine

Is there a generic answer to this? Because I can not get the locking
working right. What is odd that under openSUSE 10.3 where I did my
initial testing it did but under openSUSE 11.0 it did not. And I don't
use seLINUX or AppArmour. So the people in the office still use the
same way they were using before (Net use x: \\server\\share) to run it
locally. For the time being it is "ok" not to have the locking working
right.

Re: Status Update : Deployment of DOSEMU Application Server

[Alain M.]

Andrew Joakimsen escreveu:
>> 1) if clipper locking is working with dosemu+freedos, you can test that
>> easyly on a single machine
> 
> Is there a generic answer to this? Because I can not get the locking
> working right. What is odd that under openSUSE 10.3 where I did my
> initial testing it did but under openSUSE 11.0 it did not. And I don't
> use seLINUX or AppArmour.

FWIK, it is a FreeDOS related problem. The first time I have seen this 
was on real hardware with Windows server..

> So the people in the office still use the
> same way they were using before (Net use x: \\server\\share) to run it
> locally. For the time being it is "ok" not to have the locking working
> right.

This is annoying, I hope that someone has it working...
The reason is that most users-to-be for dosemu need Clipper

Alain

Re: Status Update : Deployment of DOSEMU Application Server

[Andrew Joakimsen]

On Wed, Oct 29, 2008 at 5:15 PM, Alain M. <alainm@pobox.com> wrote:
>
> Andrew Joakimsen escreveu:
>>>
>>> 1) if clipper locking is working with dosemu+freedos, you can test that
>>> easyly on a single machine
>>
>> Is there a generic answer to this? Because I can not get the locking
>> working right. What is odd that under openSUSE 10.3 where I did my
>> initial testing it did but under openSUSE 11.0 it did not. And I don't
>> use seLINUX or AppArmour.
>
> FWIK, it is a FreeDOS related problem. The first time I have seen this was
> on real hardware with Windows server..

What does FreeDOS have to do with Windows Server? Anyways I tried on
MS-DOS under openSUSE 10.3 for another issue but now that I think of
it I did not try it on 11.0. If you don't hear back from me, please
pester me to try MS-DOS + dosemu + openSUSE 11.0.

Re: Status Update : Deployment of DOSEMU Application Server

[Shaw Tong Tan]

Per the posted questions, the additional info below.

1.  DOSEMU+FreeDOS and CLIPPER Locking issue

Reply:  My specific scenario is as follow:

1a.  I do not have source code to the software.
1b.  From the original DOS batch file supplied by vendor, I can see that it setups the clipper path, so I assume it should be clipper-based.
1c.  The clipper-based software is also likely re-compiled with 3rd party tools.  I am not 100% sure on this.  I attempted this path initially because I wanted to solve the 100% CPU problem and there are several DOS tools out there that can help if you have a standard Clipper-based setup.  During the course of doing this I remember 1 or 2 tools that can check the compiled code (low-level) to see whether it has the loop present and change it, but the software I have gave negative result.  (according to info, that loop is pretty common on old dos program, the negative result likely means that the local software has some non-standard changes compared to straight clipper compile.)
1d.  Eventually I received an advice from list member to use a specialize tool and I did that.  However, I need to do additional nice level tuning and CPU affinity control to get to acceptable setup.

the above is to explain the clipper-based software background.  Now on to locking.

2.  For now, I have many users doing READ and upto 3 doing WRITE.
3.  Maybe that is why locking problem is minimal.  I am still observing since this is 1st week of deployment.  Prior to the deployment I have a test where I open 4 instances of the same data entry screen (key in PO)(so likely accessing the same tables and potentials locking issue)  However, they seemed to progress without any problems at the time of testing.

ST

Re: Status Update : Deployment of DOSEMU Application Server

[Andrew Joakimsen]

On Wed, Oct 29, 2008 at 9:57 PM, Shaw Tong Tan <shawtan@yahoo.com> wrote:
> 1b.  From the original DOS batch file supplied by vendor, I can see that it setups the clipper path, so I assume it should be clipper-based.
> 1c.  The clipper-based software is also likely re-compiled with 3rd party tools.  I am not 100% sure on this.

Try to run the .exe through the 'strings" command. In my case this
helped me to determine it was a CA-Clipper 5.3 application with
Blinker.... can not be decompiled with any off-the-shelf tool that I
know of.

> 2.  For now, I have many users doing READ and upto 3 doing WRITE.
> 3.  Maybe that is why locking problem is minimal.  I am still observing since this is 1st week of deployment.  Prior to the deployment I have a test where I open 4 instances of the same data entry screen (key in PO)(so likely accessing the same tables and potentials locking issue)  However, they seemed to progress without any problems at the time of testing.

Hopefully that is the case.

On Thu, Oct 30, 2008 at 5:02 AM, Shaw Tong Tan <shawtan@yahoo.com> wrote:
> Sorry I was in a hurry in previous post so I now continue with the remaining.
>
> A.  Printing.
>
> 1.  PUTTY + REMOTE ANSI Printer + LPANSI was originally preferred since it makes the access and printing very easy for WAN clients.
>
> 2.  The plan failed when a client try to print LARGE Print job 32-300Kbytes.  The DOS Application still response but client print queue hung waiting.  Example a 32Kbytes job, it receive approx 4K+ then hung.  I tried various combination sometimes it send close to 8K then hung.

I had no issues even with large print jobs over ANSI printing. The
issue comes when the DOS program runs in the same session as the
Clipper application, since there is no way to block one input or the
other, e.g. if your program prints on the screen some status between
pages this could be causing what you describe. Try this,

"Print" to a file in homedir/printjobs:

#!/bin/bash
#/usr/local/bin/file2print
TMPFILE=`mktemp ~/printjobs/dfprint.XXXXXXXXXX` || exit 1
cat "$@" >> $TMPFILE


ANSI Print from the printjobs directory:

#!/bin/bash
# file2print
# send ~/printjobs/* to tty printer
echo Processing Print Jobs for LPT1...
sleep 2
directory=~/printjobs/
for file in $( ls $directory )
do
printf "\033[5i"
cat $directory$file
printf "\033[4i"
rm $directory$file
done

Re: Status Update : Deployment of DOSEMU Application Server

[Shaw Tong Tan]

Continuation of previous post.

A.  100% CPU remedy.

Original background
1.  test environment Pentium E2180/CentOS 5.2 (32-bit)/DOSEMU+FreeDOS 1.4.0 from website.
2.  I found maximum 4 instances on single host, after that random delay and hangs
3.  Std OS environment without any modification.

To arrive at a solution
1.  tried several DOS utilities, example dosidles, even citrix utility
2.  I spent weeks trying many combination.  Finally I adopted a combined recommendations from list member Joe to run BX200FIX and setting nice value.  I added a CPU control as well. Example below

EXAMPLE:  taskset -c 1 nice -n 19 dosemu

run DOSEMU with nice value of 19 on CPU Core number 1.  Dualcore CPU core is 0,1 Quad Core is 0,1,2,3

After implementation, I tested up to 6 clipper-based app instances on 1 cpu core and found response reasonable and stable, no random hangs.  Based on my assessment of the overall situation, I feel 6 session/cpu-core is the maximum I am willing to commit.  Since I have quad-core host now, I am willing to spread the load across to gain a bit of buffer zone.  Note that try not to use Core-0 for dosemu because that's the default for OS and system services.  The high cache size on Quad Core also help I think in reducing the memory contention issue.

Re: Status Update : Deployment of DOSEMU Application Server

[Shaw Tong Tan]

Sorry I was in a hurry in previous post so I now continue with the remaining.

A.  Printing.

1.  PUTTY + REMOTE ANSI Printer + LPANSI was originally preferred since it makes the access and printing very easy for WAN clients.

2.  The plan failed when a client try to print LARGE Print job 32-300Kbytes.  The DOS Application still response but client print queue hung waiting.  Example a 32Kbytes job, it receive approx 4K+ then hung.  I tried various combination sometimes it send close to 8K then hung.

3.  I have 4 variations of lpansi scripts/compiled program for test. Due to the way how I pipe the lpansi program for printing, I also suspect Linux PIPE SIZE setting.  However, to change the setting I need to recompile the kernel, after some thought I gave up the idea.

4.  OPENVPN was eventually chosen so that I can have unrestricted network access between server and WAN clients.  Using CUPS setup, the large print job printed OK.

5.  One printing downside is CUPS printer do not tolerate long downtime,  Example: let say remote client printed some pages in application but failed to put papers in the printer, and nobody care to fix it.  After some failed attempts, CUPS daemon noted the problematic status and stop the printer.  When remote user later attempt to print again, the printing will have problem because the printer has been stopped on the CUPS side.  Admin basically have to access CUPS page to clear jobs and restart the printer.  A potential hassle for remote office with a lot of novice users and no onsite IT support, complaining about random failed printing.

B.  WAN considerations

1.  Having WAN clients add layers of complication due to the security protection issue.  This is also why I 2nd guessing on opening SSH port facing the Internet.

2.  If support spending is allowed, I would honestly prefer using a vendor-supplied low-cost VPN solution and let the vendor deal with all the complication.  Alternative open solutions (IPCOP/M0N0Wall) available but the VPN reading and related config issues are too numerous for me to address in a short timeframe.  I may revisit them in future.

Re: Status Update : Deployment of DOSEMU Application Server

[Ivan Baldo]

  Hello Shaw.
Thanks for sharing your experience!


El 30/10/08 07:02, Shaw Tong Tan escribió:
> 5.  One printing downside is CUPS printer do not tolerate long downtime,  Example: let say remote client printed some pages in application but failed to put papers in the printer, and nobody care to fix it.  After some failed attempts, CUPS daemon noted the problematic status and stop the printer.  When remote user later attempt to print again, the printing will have problem because the printer has been stopped on the CUPS side.  Admin basically have to access CUPS page to clear jobs and restart the printer.  A potential hassle for remote office with a lot of novice users and no onsite IT support, complaining about random failed printing.
>
>   
There is an option for every printer in the CUPS administration webpage 
that allows to select what to do when print jobs fail, I don't know why 
but the default is "stop-printer", you may change it to "abort-job" or 
"retry-job", that should fix that problem.


> B.  WAN considerations
>
> 1.  Having WAN clients add layers of complication due to the security protection issue.  This is also why I 2nd guessing on opening SSH port facing the Internet.
>
> 2.  If support spending is allowed, I would honestly prefer using a vendor-supplied low-cost VPN solution and let the vendor deal with all the complication.  Alternative open solutions (IPCOP/M0N0Wall) available but the VPN reading and related config issues are too numerous for me to address in a short timeframe.  I may revisit them in future.
>
>   
So, you are not comfortable with the VPN setup you made?
Well, maybe you can try FreeNX or NXServer from NoMachine, they are SSH 
based, easy to setup on the clients. I am using FreeNX with printing 
with DOSEmu and OpenOffice, speed is very good.

Hope this helps!!! Bye.

-- 
Ivan Baldo - ibaldo@adinet.com.uy - http://ibaldo.codigolibre.net/
From Montevideo, Uruguay, at the south of South America.
Freelance programmer and GNU/Linux system administrator, hire me!
Alternatives: ibaldo@codigolibre.net - http://go.to/ibaldo

--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status Update : Deployment of DOSEMU Application Server

[Alain M.]

Ther is only one topic that I can answer:

Shaw Tong Tan escreveu:
> 
> 5.  One printing downside is CUPS printer do not tolerate long downtime,  Example: let say remote client printed some pages in application but failed to put papers in the printer, and nobody care to fix it.  After some failed attempts, CUPS daemon noted the problematic status and stop the printer.  When remote user later attempt to print again, the printing will have problem because the printer has been stopped on the CUPS side.  Admin basically have to access CUPS page to clear jobs and restart the printer.  A potential hassle for remote office with a lot of novice users and no onsite IT support, complaining about random failed printing.

Normaly Dosemu (in my setup) sends a print file do a script. This 
happens whenever it (dosemu) decides to flush the spool.

So it would be possible to workaround any problem inside this script.

AAMOF, I never have problem with printers out of paper, whenever I 
insert paper it starts printing again :)

Alain

Re: Status Update : Deployment of DOSEMU Application Server

[Alain M.]

Ther is only one topic that I can answer:

Shaw Tong Tan escreveu:
> 
> 5.  One printing downside is CUPS printer do not tolerate long downtime,  Example: let say remote client printed some pages in application but failed to put papers in the printer, and nobody care to fix it.  After some failed attempts, CUPS daemon noted the problematic status and stop the printer.  When remote user later attempt to print again, the printing will have problem because the printer has been stopped on the CUPS side.  Admin basically have to access CUPS page to clear jobs and restart the printer.  A potential hassle for remote office with a lot of novice users and no onsite IT support, complaining about random failed printing.

Normaly Dosemu (in my setup) sends a print file do a script. This 
happens whenever it (dosemu) decides to flush the spool.

So it would be possible to workaround any problem inside this script.

AAMOF, I never have problem with printers out of paper, whenever I 
insert paper it starts printing again :)

Alain

Re: Status Update : Deployment of DOSEMU Application Server

[Manfred Scherer]

For the 100% CPU-load problem caused by clipper programs and other dos programs, I use
two small patches in dosemu 1.3.4:

--- ./base/bios/int16.c.ORIG    2006-10-31 21:28:54.000000000 +0100
+++ ./base/bios/int16.c 2006-12-31 13:37:10.000000000 +0100
@@ -118,7 +118,11 @@
       trigger_idle();
     else
       reset_idle(0);
-    idle(500, 20, 0, INT2F_IDLE_USECS, "int16");
+    /* 2006-12-13
+     * dbase3+, word4.0, .... CPU-load is to heavy during idle. --ms
+     * idle(500, 20, 0, INT2F_IDLE_USECS, "int16");
+     */
+    idle(10, 20, 0, INT2F_IDLE_USECS, "int16");
   } else {
     reset_idle(1);
   }

--- ./base/async/int.c.ORIG     2006-11-12 02:20:26.000000000 +0100
+++ ./base/async/int.c  2007-01-22 15:12:30.000000000 +0100
@@ -1232,7 +1232,13 @@
 #endif

   case 0x2C: {                   /* get time & date */
-      idle(2, 100, 0, INT2F_IDLE_USECS, "dos_time");
+      /*
+       * 2004/08/20, 2006/11/25
+       * waiting loops, like 'wait until time ...' call never a trigger_idle()
+       * but 90% cpu load. --ms
+       * idle(2, 100, 0, INT2F_IDLE_USECS, "dos_time");
+       */
+      idle(0, 20, 0, INT2F_IDLE_USECS, "dos_time");
       return 0;
     }

maybe this can solve your 100% CPU-load too.

Manfred

Re: Status Update : Deployment of DOSEMU Application Server

[Claudia Neumann]

Hi Manfred!

I tried your patches, but they didn't work out as they should on 1.3.4 and 
1.4.0. 

I had the same problem with some other program on Linux kernel 2.6.24. They 
changed kernel idle time handling somewhere between 2.6.18 and 2.6.24. Now in 
some programs, and dosemu is one of them, the CPU load runs high to over 90% 
on current Linux kernels. 

Perhaps someone knows where to change that in the source code.

Regards

Claudia

Am Donnerstag Oktober 30 2008 18:14 schrieben Sie:
> For the 100% CPU-load problem caused by clipper programs and other dos
> programs, I use two small patches in dosemu 1.3.4:
>
> --- ./base/bios/int16.c.ORIG    2006-10-31 21:28:54.000000000 +0100
> +++ ./base/bios/int16.c 2006-12-31 13:37:10.000000000 +0100
> @@ -118,7 +118,11 @@
>        trigger_idle();
>      else
>        reset_idle(0);
> -    idle(500, 20, 0, INT2F_IDLE_USECS, "int16");
> +    /* 2006-12-13
> +     * dbase3+, word4.0, .... CPU-load is to heavy during idle. --ms
> +     * idle(500, 20, 0, INT2F_IDLE_USECS, "int16");
> +     */
> +    idle(10, 20, 0, INT2F_IDLE_USECS, "int16");
>    } else {
>      reset_idle(1);
>    }
>
> --- ./base/async/int.c.ORIG     2006-11-12 02:20:26.000000000 +0100
> +++ ./base/async/int.c  2007-01-22 15:12:30.000000000 +0100
> @@ -1232,7 +1232,13 @@
>  #endif
>
>    case 0x2C: {                   /* get time & date */
> -      idle(2, 100, 0, INT2F_IDLE_USECS, "dos_time");
> +      /*
> +       * 2004/08/20, 2006/11/25
> +       * waiting loops, like 'wait until time ...' call never a
> trigger_idle() +       * but 90% cpu load. --ms
> +       * idle(2, 100, 0, INT2F_IDLE_USECS, "dos_time");
> +       */
> +      idle(0, 20, 0, INT2F_IDLE_USECS, "dos_time");
>        return 0;
>      }
>
> maybe this can solve your 100% CPU-load too.
>
> Manfred
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Status Update : Deployment of DOSEMU Application Server

[Manfred Scherer]

Hi Claudia!

actually my linux kernel is 2.6.27
AFAIK since kernel 2.6.23 a new scheduler is active:
-----------
Citation from kernel source:
sched-design-CFS.txt:
   the CFS scheduler has a much stronger handling of nice levels and
   SCHED_BATCH: both types of workloads should be isolated much more
   aggressively than under the vanilla scheduler.
-----------
Maybe this can bee a reason for the different behavoir between kernel 2.6.18 vs 2.6.24.

Anyway for my used programs this makes no different in the runtime behavior for all
kernels 2.6.*.
My patches works for for me in all kernel versions.

Which programs causes more than 90% CPU consumption in your case?

Regards
Manfred

Re: Status Update : Deployment of DOSEMU Application Server

[Claudia Neumann]

Hi Manfred!

Am Samstag November 8 2008 17:49 schrieben Sie:
> Hi Claudia!
>
> actually my linux kernel is 2.6.27
> AFAIK since kernel 2.6.23 a new scheduler is active:
> -----------
> Citation from kernel source:
> sched-design-CFS.txt:
>    the CFS scheduler has a much stronger handling of nice levels and
>    SCHED_BATCH: both types of workloads should be isolated much more
>    aggressively than under the vanilla scheduler.
> -----------
> Maybe this can bee a reason for the different behavoir between kernel
> 2.6.18 vs 2.6.24.
>
> Anyway for my used programs this makes no different in the runtime behavior
> for all kernels 2.6.*.
> My patches works for for me in all kernel versions.
>
> Which programs causes more than 90% CPU consumption in your case?

The programs are Clipper programs. Same problem as with some other members of 
this list. In Windows/real DOS they don't use up CPU load.

Viele Grüße

Claudia

--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html