High network usage over network filesystem

High network usage over network filesystem

[Javier Tarifa]

I have a weird problem. I'm using dosbox to access a mounted network 
partition. The problem is that every time a command or a program is 
launched, if your current working directory (of msdos, not linux) is in 
the novell partition, it takes forever to start. The weird thing was 
that the cpu was idle all the time, so I looked into it and saw that the 
network usage would skyrocket.
This happens specially when you're on a directories with lots of files 
in them, and I need to run a lot of commands in directories that have 
over 4000 files. This means that every time I try to launch something it 
will usually sit there for 5-6 seconds before doing anything, and that 
means that it's impossible to execute some bats we now execute under dos 
(they invoke  a program more than a thousand times).
Doing an strace, I can see it "hangs" after doing a getdents call, so I 
suppose that for some reason it's retrieving info of the files on the 
current directory or something.
Why it's doing that? Is it a dosemu thing or a freedos thing? Any idea 
how I can get around it?

Re: High network usage over network filesystem

[Frank Cox]

On Tue, 27 Jan 2009 10:13:28 +0100
Javier Tarifa wrote:

> I have a weird problem. I'm using dosbox to access a mounted network 
> partition.

1. Are you having problems using dosbox or dosemu?  Despite the similarity of
the names, they are two different dos emulators.

2. "mounted network partition".  Do you mean nfs?  Samba?  Something else?

>  The problem is that every time a command or a program is 
> launched, if your current working directory (of msdos, not linux) is in 
> the novell partition,

Are you talking about Novell Netware?

>  it takes forever to start. The weird thing was 
> that the cpu was idle all the time, so I looked into it and saw that the 
> network usage would skyrocket.

What if you just type dir at a dosemu prompt?

Describe more about your setup, how you're mounting your network partition and
what it consists of and how it's structured, and how you're accessing your data
files.

-- 
MELVILLE THEATRE ~ Melville Sask ~ http://www.melvilletheatre.com
DRY CLEANER BUSINESS FOR SALE ~ http://www.canadadrycleanerforsale.com

Re: High network usage over network filesystem

[Javier Tarifa]

> 1. Are you having problems using dosbox or dosemu?  Despite the similarity of
> the names, they are two different dos emulators.
>   
Sorry about the confusion dosbox/dosemu, i'm definitely using dosemu.
> 2. "mounted network partition".  Do you mean nfs?  Samba?  Something else?
>   
The computer dosemu is running on is a fedora that mounts a remote 
novell netware partition using ncpmount, and I use lredir to make it my f:
> What if you just type dir at a dosemu prompt?
>
> Describe more about your setup, how you're mounting your network partition and
> what it consists of and how it's structured, and how you're accessing your data
> files.
>   
It doesn't matter how I acces my files. After every command I type there 
are several seconds of high network usage before doing anything,
even running "dir" or running an "unix" command. The only thing that is 
not slowed down is changing directory and changing the device
This time of doing nothing seems to be proportional to the amount of 
files in the current directory.
Here's an example:
test.bat contains this:

call time /t
call dir > null
call time /t
call unix "echo hi"
call time /t

running it on the root of the network device, on f:, where there are 
only 44 files, gives this:

F:\>test.bat
F:\>call time /t
Current time is  6:47:34.97 pm
F:\>call dir > null
F:\>call time /t
Current time is  6:47:35.68 pm
F:\>call unix "echo hi"
hi
F:\>call time /t
Current time is  6:47:35.84 pm

But then running on the directory where everything needs to be run, with 
3830 files on the current directory, gives you this:
F:\somedir>test.bat
F:\somedir>call time /t
Current time is  6:51:20.16 pm
F:\somedir>call dir > null
F:\somedir>call time /t
Current time is  6:51:41.20 pm
F:\somedir>call unix "echo hi"
hi
F:\somedir>call time /t
Current time is  6:51:44.22 pm

On a windows prompt it runs like this:

F:\somedir>test.bat

F:\somedir>call time /t
18:53

F:\somedir>call dir  1>null

F:\somedir>call time /t
18:53

F:\soemdir>call unix "echo hi"
"unix" no se reconoce como un comando interno o externo,
programa o archivo por lotes ejecutable.

F:\somedir>call time /t
18:53


Any ideas why is this slow? I wouldn't mind if doing a dir or something 
like this lasted a little more than normal, but those 3-4 seconds that 
is "hanged" everytime anything is trying to execute
makes that, for example, compiling can take hours with dosemu and before 
were a matter of a couple of minutes or so (the compiler is called 2000 
times more or less, but it finishes at the moment)

If I can give any more info, please tell me.

Re: High network usage over network filesystem

[Frantisek Hanzlik]

Javier Tarifa wrote:
>
>> 1. Are you having problems using dosbox or dosemu? Despite the
>> similarity of
>> the names, they are two different dos emulators.
> Sorry about the confusion dosbox/dosemu, i'm definitely using dosemu.
>> 2. "mounted network partition". Do you mean nfs? Samba? Something else?
> The computer dosemu is running on is a fedora that mounts a remote
> novell netware partition using ncpmount, and I use lredir to make it my f:
>> What if you just type dir at a dosemu prompt?
>>
>> Describe more about your setup, how you're mounting your network
>> partition and
>> what it consists of and how it's structured, and how you're accessing
>> your data
>> files.
> It doesn't matter how I acces my files. After every command I type there
> are several seconds of high network usage before doing anything,
> even running "dir" or running an "unix" command. The only thing that is
> not slowed down is changing directory and changing the device
> This time of doing nothing seems to be proportional to the amount of
> files in the current directory.
> Here's an example:
> test.bat contains this:
>
> call time /t
> call dir > null
> call time /t
> call unix "echo hi"
> call time /t
>
> running it on the root of the network device, on f:, where there are
> only 44 files, gives this:
>
> F:\>test.bat
> F:\>call time /t
> Current time is 6:47:34.97 pm
> F:\>call dir > null
> F:\>call time /t
> Current time is 6:47:35.68 pm
> F:\>call unix "echo hi"
> hi
> F:\>call time /t
> Current time is 6:47:35.84 pm
>
> But then running on the directory where everything needs to be run, with
> 3830 files on the current directory, gives you this:
> F:\somedir>test.bat
> F:\somedir>call time /t
> Current time is 6:51:20.16 pm
> F:\somedir>call dir > null
> F:\somedir>call time /t
> Current time is 6:51:41.20 pm
> F:\somedir>call unix "echo hi"
> hi
> F:\somedir>call time /t
> Current time is 6:51:44.22 pm
>
> On a windows prompt it runs like this:
>
> F:\somedir>test.bat
>
> F:\somedir>call time /t
> 18:53
>
> F:\somedir>call dir 1>null
>
> F:\somedir>call time /t
> 18:53
>
> F:\soemdir>call unix "echo hi"
> "unix" no se reconoce como un comando interno o externo,
> programa o archivo por lotes ejecutable.
>
> F:\somedir>call time /t
> 18:53
>
>
> Any ideas why is this slow? I wouldn't mind if doing a dir or something
> like this lasted a little more than normal, but those 3-4 seconds that
> is "hanged" everytime anything is trying to execute
> makes that, for example, compiling can take hours with dosemu and before
> were a matter of a couple of minutes or so (the compiler is called 2000
> times more or less, but it finishes at the moment)
>
> If I can give any more info, please tell me.
-- 
It look as Novell NW access is slow - and as DOS (in contrast in Unix,
which look for command wo full path only in PATH environ var) first
look for command in current directory, then when there is several thousands
files, this search will take a while.
When You know about command full path, then it may help:

C:\bin\command CMDLINE

However I not know, what it does in case of internal command.com commands
(dir, cd, etc).

Franta Hanzlík
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: High network usage over network filesystem

[Frank Cox]

On Tue, 27 Jan 2009 19:02:34 +0100
Javier Tarifa wrote:

> If I can give any more info, please tell me.

It would be interesting to see what happens if you put your data on a nfs
filesystem and then try running your compiler on that.

Then put it on a local filesystem and see what happens there.



-- 
MELVILLE THEATRE ~ Melville Sask ~ http://www.melvilletheatre.com
DRY CLEANER BUSINESS FOR SALE ~ http://www.canadadrycleanerforsale.com

Re: High network usage over network filesystem

[Javier Tarifa]

>> If I can give any more info, please tell me.
>>     
>
> It would be interesting to see what happens if you put your data on a nfs
> filesystem and then try running your compiler on that.
>
> Then put it on a local filesystem and see what happens there.
>   
Nevermind, I've got it.
When you try to execute something, scan_dir scans all the files on your 
path and on your current directory searching for it.
Even more, it scans multiple times searching for different name 
combinations (exe/com/bat and with different capitalization), and that's 
what causing
huge bandwidth consumption and stalls when you're on a big directory. 
Luckily for me, as I said in another mail I'm mounting the remote novell
partition with ncpmount that makes all acceses to the mounted device 
case-insensitive, so I just made scan_dir return true always, and now I 
think it determines
if a file it's there or not by stating it in find_file. I have 
encountered no problems with this yet.


Another thing, I don't know if this is the proper place to ask for it, 
but I've found that dosemu doesn't work right with dtach.
When you reattach you simply get a black screen and all the text, 
windows, pictures, etc are gone. I can't use screen because for some
reason some programs just hang when they're executed in dosemu inside a 
screen, but they work flawlessly in dtach. It would be
great if you could do something about this. To get around it I added 
this at the end of sigwinch:

SLsmg_touch_lines(0,vga.text_height);
SLsmg_refresh();

It does work now, but I know nothing of slang and the docs say you 
should not use SLsmg_touch_lines, so it has to be a better way.
It would be great if in the next version dosemu and dtach could get along.

Re: High network usage over network filesystem

[Alain M.]

Javier Tarifa escreveu:
>>   
> Nevermind, I've got it.
> When you try to execute something, scan_dir scans all the files on your 
> path and on your current directory searching for it.
> Even more, it scans multiple times searching for different name 
> combinations (exe/com/bat and with different capitalization), and that's 
> what causing
> huge bandwidth consumption and stalls when you're on a big directory. 
> Luckily for me, as I said in another mail I'm mounting the remote novell
> partition with ncpmount that makes all acceses to the mounted device 
> case-insensitive, so I just made scan_dir return true always, and now I 
> think it determines
> if a file it's there or not by stating it in find_file. I have 
> encountered no problems with this yet.

Could you explain this a little more... It seems that you have found the 
reason why FreeDOS is slow sometimes.

Maybe that scan_dir could be re-written so as to scan only once ant test 
all name combinations for each file in turn.

Alain

> 
> 
> Another thing, I don't know if this is the proper place to ask for it, 
> but I've found that dosemu doesn't work right with dtach.
> When you reattach you simply get a black screen and all the text, 
> windows, pictures, etc are gone. I can't use screen because for some
> reason some programs just hang when they're executed in dosemu inside a 
> screen, but they work flawlessly in dtach. It would be
> great if you could do something about this. To get around it I added 
> this at the end of sigwinch:
> 
> SLsmg_touch_lines(0,vga.text_height);
> SLsmg_refresh();
> 
> It does work now, but I know nothing of slang and the docs say you 
> should not use SLsmg_touch_lines, so it has to be a better way.
> It would be great if in the next version dosemu and dtach could get along.
> --