File Limits

File Limits

[Xavier Tarifa]

So here's my problem: Until now I was using dosemu with ms-dos 6.22 to run an old database. A little while ago we had some big changes to the tables and they grew a lot.
Yesterday I realized that we have a database file that is about to hit 2 Gb. I remembered that some time ago, we hit a bug on our program and it endlessly inserted rows on a table until it hit the 2 gb mark. Then the file either disappeared or became a 0 bytes file, I don't remember well.
So I started reading and found that ms-dos 6.22 only supports FAT16 and apparently it only allows 2 gb volumes, and 2 gb files.
So I thought of using ms-dos 7.10 which supports fat32 and would give us 4 gb files. But the same happens. The files disappear as soon as they hit the 2 gb mark. And when I execute the dir command, it always reports 2 gb of free disk space, which is odd.

I don't know if this is a limitation of dosemu or I'm doing something wrong. Right now, I have about 140 users. On each $HOME/.dosemu/drive_c/ of the users there are symlinks to each file in /usr/share/dosemu/drive_z/ , where ms-dos is located. And I have the database files in /datos/ . I use lredir to make it appear as f: to every user.
All of this is in the same ext3 partition.

Any ideas? 

Re: File Limits

[Paul Crawford]

Dear Xavier,
> So here's my problem: Until now I was using dosemu with ms-dos 6.22 to run an old database. A little while ago we had some big changes to the tables and they grew a lot.
> Yesterday I realized that we have a database file that is about to hit 2 Gb. I remembered that some time ago, we hit a bug on our program and it endlessly inserted rows on a table until it hit the 2 gb mark. Then the file either disappeared or became a 0 bytes file, I don't remember well.
> So I started reading and found that ms-dos 6.22 only supports FAT16 and apparently it only allows 2 gb volumes, and 2 gb files.
> So I thought of using ms-dos 7.10 which supports fat32 and would give us 4 gb files. But the same happens. The files disappear as soon as they hit the 2 gb mark. And when I execute the dir command, it always reports 2 gb of free disk space, which is odd.
>
> I don't know if this is a limitation of dosemu or I'm doing something wrong. Right now, I have about 140 users. On each $HOME/.dosemu/drive_c/ of the users there are symlinks to each file in /usr/share/dosemu/drive_z/ , where ms-dos is located. And I have the database files in /datos/ . I use lredir to make it appear as f: to every user.
> All of this is in the same ext3 partition.
>
> Any ideas?

The standard limits for 32-bit Linux programs is 2GB per file, even 
though ext3 allows much, much more total space (and file size).

Have you tried using a 64-bit Linux and matching dosemu version?

I know it is not always as successful (e.g. I found the Microsoft C6 
compiler works fine on Ubuntu 10.04 32-bit, but crashes/closes dosemu on 
64-bit on the same hardware) but that would be my first suggestion to try.

Failing that, you may have to consider modifying the application to 
split your data set or to store pointers to other data sets, etc, all of 
which is a whole lot more trouble than just running it under dosemu!

Regards,
Paul
-- 
  Dr. Paul S. Crawford
  c/o Satellite Station
  University of Dundee
  Small's Wynd, Dundee, DD1 4HN
  Email: psc@sat.dundee.ac.uk
  Tel: +44 (0)1382 38 4687
  The University of Dundee is a Scottish Registered Charity, No. SC015096

RE: File Limits

[Xavier Tarifa]

> The standard limits for 32-bit Linux programs is 2GB per file, even though ext3 allows much, much more total space (and file size).

> Have you tried using a 64-bit Linux and matching dosemu version?

You're right, a 64-bits linux does work. I would have never thought that there was such a low limitation, though. It's the first program  I can think of that I've found problems with big files (but the only other big files that I've used are mainly backups and vídeos so not a lot of apps involved).
 
Thanks a lot, that would have been probably the last reason I would think of.
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File Limits

[Paul Crawford]

Dear Xavier,
Traditionally most UNIX / C language applications used a 32-bit signed 
integer for file offset operations, hence the 2GB limit. This applied 
even to DOS 16-bit applications as they also supported 32-bit 'long' 
integers by special arithmetic.

You can get 32-bit applications that support >2GB because they are using 
special arithmetic to implement 64-bit offsets, but that is of course 
the 'natural' size for 64-bit applications.

At the time this was started 2GB was huge! When DOS was first available 
in the early 1980s you could only get 10MB or so HDD. It was only around 
1995 that you could get a consumer product HDD with 2GB capacity, let 
alone consider single files of greater than this. And now my PC has 8GB 
of memory!

Even today, the likes of fat32 has the problem that a large file (e.g. 
video or backup archive, etc) can cause a write error if it exceeds 4GB.

Regards,
Paul

On 18/01/11 14:16, Xavier Tarifa wrote:
>> The standard limits for 32-bit Linux programs is 2GB per file, even
>> though ext3 allows much, much more total space (and file size).
>
>> Have you tried using a 64-bit Linux and matching dosemu version?
>
> You're right, a 64-bits linux does work. I would have never thought
> that there was such a low limitation, though. It's the first program
> I can think of that I've found problems with big files (but the only
> other big files that I've used are mainly backups and vídeos so not a
> lot of apps involved).
>
> Thanks a lot, that would have been probably the last reason I would
> think of.



-- 
Dr. Paul S. Crawford
Satellite Station
Dundee University
Small's Wynd, Dundee,
DD1 4HN, U.K.
Tel: +44 (0)1382 38 4687
Email: psc@sat.dundee.ac.uk
The University of Dundee is a Scottish Registered Charity, No. SC015096
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File Limits

[Bryan J Smith]

On 18/01/11 14:16, Xavier Tarifa wrote:  
> The standard limits for 32-bit Linux programs is 2GB per file, even
> though ext3 allows much, much more total space (and file size).

Large File Support (LFS) on 32-bit POSIX implementations was addressed almost a 
decade ago, including in Linux.

You must have three (3) things:  
1)  Kernel 2.4 (or a late 2.2 release with the backport, circa Alan Cox's 
2.2.18+, some earlier)
2)  GLibC 2.2 (or a late 2.1 release with the backport, circa 2.1.3+)
3)  A program or build solution that implements the LFS macros for 32-bit 
programs

For network file systems:  
- NFS v3 or later (including SGI+Trond backports to Alan Cox's 2.2.18)
- Samba 2.2

Most programs written in the last decade should implement #3, and it's fairly 
easy to do so in C code.  Most interpreters also do.  Unless you are explicitly 
typecasting 32-bit ints, it's fairly easy to do #3.  Any most any releases 
starting with Red Hat Linux 7 and later do all of the above.  I even had a 
GLibC/kernel running on Red Hat Linux 6.2 from VA Linux that offered such.

However, for Windows itself, you must be using the new xFAT for >2GiB file 
sizes, or >4GiB file sizes on NT kernels (a special call, not legacy compatible 
-- NT offered a 4GiB capable FAT16 file system as well).  Just having FAT32 is 
not enough.  xFAT is heavily patent encumbered though, and not Real Mode DOS 
compatible.

On 18/01/11 14:16, Xavier Tarifa wrote:  
> Have you tried using a 64-bit Linux and matching dosemu version?

I don't think that would work because x86-64 processors in Long Mode, as I 
understand it, do not support Virtual86 Memory Models.  Memory Models define 
library/program compatibility.

Long Mode uses a 48-bit "flat" (again, "long") instruction pointer where the 
segment register (16-bit) is concatenated with the offeset register (32-bit).

Protected386 and Virtual86 uses a 32-bit "normalized" instruction pointer where 
the segment register (16-bit) is the segment where the offset register (32-bit) 
is "flat."  Real86 uses a 20-bit "normalized" instruction pointer where the 
segment register (16-bit) is the segment where the offset register (16-bit) is 
"flat."

So as I understand it ...

Protected386 is compatible with Virtual86 memory models and can execute Real86 
programs.
Long Mode can execute Protected386, but cannot provide Virtual86 compatibility 
and execute Real86 programs.

SIDE NOTE:  This is also why x86-64 Long Mode cannot execute 386Enhanced models 
(shunts processor between Protected386 and Real86).

Long Mode actually uses Physical Address Extensions (PAE) paging, 52-bit to be 
exact.  It can provide up to 52-bit/1PiB addressing with flat pages of up to 
48-bit/256TiB sizes, although there are some flat 40-bit/1TiB and 36-bit/64GiB 
limits in some structures (especially on Intel processors, and AMD before the 
Processor 10h series).

PAE was originally introduced with the Pentium Pro and uses 36-bit, providing up 
to 36-bit/64GiB addressing with flat pages up to 32-bit/4GiB.  In other words, 
there is a "hard" limitation at 52-bit/1PiB "paging" and 48-bit/256TiB "flat" 
address models in the current x86-64 Long Mode memory model where it will break 
even Protected386 (32-bit) compatibility.

It's all about memory models.  I could be wrong though.  Does DOSEmu provide 
emulated Real86 compatibility when the processor is in x86-64 Long Mode?

-- Bryan

P.S.  HyperVisors and virtualization will likely be the solution when 48/52-bit 
limitations are hit in hardware.  HyperVisors will no longer be optional, but 
required and ubiquitous.

-- 
Bryan J  Smith     Professional, Technical Annoyance 
Linked Profile:   http://www.linkedin.com/in/bjsmith 
---------------------------------------------------- 
UCF Basketball:   AP #23, ESPN/USAToday #22, RPI #17
UCF Football:   AP #21, BCS #25, ESPN/USAToday #20

Re: File Limits

[Bryan J Smith]

From: Bryan J Smith <b.j.smith@ieee.org>

> - Samba 2.2

Just want to make 1 clarification ...

All Real86 (DOS), Virtual86 (DOS, DOS/Win and NTVDM) and 386Enhanced (DOS/Win, 
including through Windows Me) solutions still have client-side 2GiB file size 
limitations.  Samba 2.2+ just offers LFS support on 32-bit GNU/Linux platforms 
for NT clients and other, true 32-bit CIFS/SMB clients (e.g., Linux itself).  
Samba 2.2+ won't let you get around DOS, DOS/Win or NTVDM client limitations.


-- 
Bryan J  Smith     Professional, Technical Annoyance 
Linked Profile:   http://www.linkedin.com/in/bjsmith 
---------------------------------------------------- 
UCF Basketball:   AP #23, ESPN/USAToday #22, RPI #17
UCF Football:   AP #21, BCS #25, ESPN/USAToday #20
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File Limits

[Bryan J Smith]

All DOS (including DOS 7/Win 4 releases running in 386Enhanced mode -- Windows 
95, 98, Me) only support 2GiB files IIRC.  Even Real86 mode DOS 7 supports 
FAT32, which allows bigger than 2GiB _file_systems_, but still has the 2GiB-1 
(signed 32-bit int) _individual_file_ size limitation IIRC.


In fact, I don't think the 4GiB-1 (unsigned 32-bit int) support is in any 
DOS/Win releases.  I only think it is an option in NT kernels (FAT16 in 3.1, 
3.5, 3.51 and 4.0, FAT16 and FAT32 in 5.0/2000, 5.1/XP-2003, 6.0-Vista and 
6.1/7-2008).  And any program that uses it _must_ know about the NT-only Win32 
function calls that offer this.

I.e., the "Chicago" (Win9x) and NT/Win32 teams used _different_ DOS Int 21h and 
Win32[s] function calls, the former completely ignoring the pre-existing of the 
latter.  "Chicago" _never_ supported the native NT/Win32 calls, hence why they  
likely _never_ supported the extensions for 4GiB-1 support IIRC.[1]

One quick trip to TechNet and you'll note this is an issue in "bare metal" 
DOS/Win and even NT.  The legacy "Chicago" DOS Int 21h file services don't 
handle growing beyond 2GiB-1 proper, _unless_ the NT/Win32 function extensions 
are utilized.  Unfortunately even NT 5.0 (Windows 2000) implements the "Chicago" 
calls bug-for-bug for compatibility reasons.

I.e., your program is likely instigating this, and the OS is just letting it do 
what it wants (and causing the issue).

-- Bryan

NOTES:  
[1] Like many libraries in NT/Win32, things just "didn't work" or "work well" 
under Win9x -- OpenGL being my personal favorite, and the whole "DirectMM -- aka 
Direct [DOS] Memory Map" lineage now known as "DirectX".  It wasn't until NT 5.0 
(Windows 2000) that the NT/Win32 team started also supporting the 
"Chicago"/Win32 DOS Int 21h changes.



----- Original Message ----

From: Xavier Tarifa <Xavier.Tarifa@adparts.com>

So here's my problem: Until now I was using dosemu with ms-dos 6.22 to run an 
old database. A little while ago we had some big changes to the tables and they 
grew a lot.
Yesterday I realized that we have a database file that is about to hit 2 Gb. I 
remembered that some time ago, we hit a bug on our program and it endlessly 
inserted rows on a table until it hit the 2 gb mark. Then the file either 
disappeared or became a 0 bytes file, I don't remember well.
So I started reading and found that ms-dos 6.22 only supports FAT16 and 
apparently it only allows 2 gb volumes, and 2 gb files.
So I thought of using ms-dos 7.10 which supports fat32 and would give us 4 gb 
files. But the same happens. The files disappear as soon as they hit the 2 gb 
mark. And when I execute the dir command, it always reports 2 gb of free disk 
space, which is odd.

I don't know if this is a limitation of dosemu or I'm doing something wrong. 
Right now, I have about 140 users. On each $HOME/.dosemu/drive_c/ of the users 
there are symlinks to each file in /usr/share/dosemu/drive_z/ , where ms-dos is 
located. And I have the database files in /datos/ . I use lredir to make it 
appear as f: to every user.
All of this is in the same ext3 partition.

Any ideas? 

RE: File Limits

[Xavier Tarifa]

I just created a 3 gb file on a dos 7.10 and it worked (worked as in dir tells the correct filesize). I'll have to wait a day or two to test it against our database because I'm offsite, then I'll report back the results.

-----Mensaje original-----
De: linux-msdos-owner@vger.kernel.org [mailto:linux-msdos-owner@vger.kernel.org] En nombre de Bryan J Smith
Enviado el: martes, 18 de enero de 2011 18:35
Para: Xavier Tarifa; linux-msdos@vger.kernel.org
Asunto: Re: File Limits

All DOS (including DOS 7/Win 4 releases running in 386Enhanced mode -- Windows 95, 98, Me) only support 2GiB files IIRC.  Even Real86 mode DOS 7 supports FAT32, which allows bigger than 2GiB _file_systems_, but still has the 2GiB-1 (signed 32-bit int) _individual_file_ size limitation IIRC.


In fact, I don't think the 4GiB-1 (unsigned 32-bit int) support is in any DOS/Win releases.  I only think it is an option in NT kernels (FAT16 in 3.1, 3.5, 3.51 and 4.0, FAT16 and FAT32 in 5.0/2000, 5.1/XP-2003, 6.0-Vista and 6.1/7-2008).  And any program that uses it _must_ know about the NT-only Win32 function calls that offer this.

I.e., the "Chicago" (Win9x) and NT/Win32 teams used _different_ DOS Int 21h and Win32[s] function calls, the former completely ignoring the pre-existing of the latter.  "Chicago" _never_ supported the native NT/Win32 calls, hence why they likely _never_ supported the extensions for 4GiB-1 support IIRC.[1]

One quick trip to TechNet and you'll note this is an issue in "bare metal" 
DOS/Win and even NT.  The legacy "Chicago" DOS Int 21h file services don't handle growing beyond 2GiB-1 proper, _unless_ the NT/Win32 function extensions are utilized.  Unfortunately even NT 5.0 (Windows 2000) implements the "Chicago" 
calls bug-for-bug for compatibility reasons.

I.e., your program is likely instigating this, and the OS is just letting it do what it wants (and causing the issue).

-- Bryan

NOTES:  
[1] Like many libraries in NT/Win32, things just "didn't work" or "work well" 
under Win9x -- OpenGL being my personal favorite, and the whole "DirectMM -- aka Direct [DOS] Memory Map" lineage now known as "DirectX".  It wasn't until NT 5.0 (Windows 2000) that the NT/Win32 team started also supporting the
"Chicago"/Win32 DOS Int 21h changes.



----- Original Message ----

From: Xavier Tarifa <Xavier.Tarifa@adparts.com>

So here's my problem: Until now I was using dosemu with ms-dos 6.22 to run an old database. A little while ago we had some big changes to the tables and they grew a lot.
Yesterday I realized that we have a database file that is about to hit 2 Gb. I remembered that some time ago, we hit a bug on our program and it endlessly inserted rows on a table until it hit the 2 gb mark. Then the file either disappeared or became a 0 bytes file, I don't remember well.
So I started reading and found that ms-dos 6.22 only supports FAT16 and apparently it only allows 2 gb volumes, and 2 gb files.
So I thought of using ms-dos 7.10 which supports fat32 and would give us 4 gb files. But the same happens. The files disappear as soon as they hit the 2 gb mark. And when I execute the dir command, it always reports 2 gb of free disk space, which is odd.

I don't know if this is a limitation of dosemu or I'm doing something wrong. 
Right now, I have about 140 users. On each $HOME/.dosemu/drive_c/ of the users there are symlinks to each file in /usr/share/dosemu/drive_z/ , where ms-dos is located. And I have the database files in /datos/ . I use lredir to make it appear as f: to every user.
All of this is in the same ext3 partition.

Any ideas? 
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File Limits

[Bryan J Smith]

Then it's using the extended functions that legacy DOS Int 21h programs do not 
understand.  They may have merged in some NT/Win32 extensions in late Windows 98 
SE and Me.  They weren't in there for Windows 95, not even OEM 2 (MS-DOS 7.1) 
when I tested them.

-- 
Bryan J  Smith     Professional, Technical Annoyance 
Linked Profile:   http://www.linkedin.com/in/bjsmith 
---------------------------------------------------- 
UCF Basketball:   AP #23, ESPN/USAToday #22, RPI #17
UCF Football:   AP #21, BCS #25, ESPN/USAToday #20



----- Original Message ----
From: Xavier Tarifa <Xavier.Tarifa@adparts.com>
To: Bryan J Smith <b.j.smith@ieee.org>
Cc: "linux-msdos@vger.kernel.org" <linux-msdos@vger.kernel.org>
Sent: Tue, January 18, 2011 1:10:08 PM
Subject: RE: File Limits

I just created a 3 gb file on a dos 7.10 and it worked (worked as in dir tells 
the correct filesize). I'll have to wait a day or two to test it against our 
database because I'm offsite, then I'll report back the results.

-----Mensaje original-----
De: linux-msdos-owner@vger.kernel.org [mailto:linux-msdos-owner@vger.kernel.org] 
En nombre de Bryan J Smith
Enviado el: martes, 18 de enero de 2011 18:35
Para: Xavier Tarifa; linux-msdos@vger.kernel.org
Asunto: Re: File Limits

All DOS (including DOS 7/Win 4 releases running in 386Enhanced mode -- Windows 
95, 98, Me) only support 2GiB files IIRC.  Even Real86 mode DOS 7 supports 
FAT32, which allows bigger than 2GiB _file_systems_, but still has the 2GiB-1 
(signed 32-bit int) _individual_file_ size limitation IIRC.


In fact, I don't think the 4GiB-1 (unsigned 32-bit int) support is in any 
DOS/Win releases.  I only think it is an option in NT kernels (FAT16 in 3.1, 
3.5, 3.51 and 4.0, FAT16 and FAT32 in 5.0/2000, 5.1/XP-2003, 6.0-Vista and 
6.1/7-2008).  And any program that uses it _must_ know about the NT-only Win32 
function calls that offer this.

I.e., the "Chicago" (Win9x) and NT/Win32 teams used _different_ DOS Int 21h and 
Win32[s] function calls, the former completely ignoring the pre-existing of the 
latter.  "Chicago" _never_ supported the native NT/Win32 calls, hence why they 
likely _never_ supported the extensions for 4GiB-1 support IIRC.[1]

One quick trip to TechNet and you'll note this is an issue in "bare metal" 
DOS/Win and even NT.  The legacy "Chicago" DOS Int 21h file services don't 
handle growing beyond 2GiB-1 proper, _unless_ the NT/Win32 function extensions 
are utilized.  Unfortunately even NT 5.0 (Windows 2000) implements the "Chicago" 

calls bug-for-bug for compatibility reasons.

I.e., your program is likely instigating this, and the OS is just letting it do 
what it wants (and causing the issue).

-- Bryan

NOTES:  
[1] Like many libraries in NT/Win32, things just "didn't work" or "work well" 
under Win9x -- OpenGL being my personal favorite, and the whole "DirectMM -- aka 
Direct [DOS] Memory Map" lineage now known as "DirectX".  It wasn't until NT 5.0 
(Windows 2000) that the NT/Win32 team started also supporting the
"Chicago"/Win32 DOS Int 21h changes.



----- Original Message ----

From: Xavier Tarifa <Xavier.Tarifa@adparts.com>

So here's my problem: Until now I was using dosemu with ms-dos 6.22 to run an 
old database. A little while ago we had some big changes to the tables and they 
grew a lot.
Yesterday I realized that we have a database file that is about to hit 2 Gb. I 
remembered that some time ago, we hit a bug on our program and it endlessly 
inserted rows on a table until it hit the 2 gb mark. Then the file either 
disappeared or became a 0 bytes file, I don't remember well.
So I started reading and found that ms-dos 6.22 only supports FAT16 and 
apparently it only allows 2 gb volumes, and 2 gb files.
So I thought of using ms-dos 7.10 which supports fat32 and would give us 4 gb 
files. But the same happens. The files disappear as soon as they hit the 2 gb 
mark. And when I execute the dir command, it always reports 2 gb of free disk 
space, which is odd.

I don't know if this is a limitation of dosemu or I'm doing something wrong. 
Right now, I have about 140 users. On each $HOME/.dosemu/drive_c/ of the users 
there are symlinks to each file in /usr/share/dosemu/drive_z/ , where ms-dos is 
located. And I have the database files in /datos/ . I use lredir to make it 
appear as f: to every user.
All of this is in the same ext3 partition.

Any ideas? 
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in the 
body of a message to majordomo@vger.kernel.org More majordomo info at  
http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: File Limits

[Xavier Tarifa]

The file límit seems to grow to 4 gb, but no luck running the application. On a 32 bits dosemu it works flawlessly, but on the 64 bits version it dies horribly. It seems to have problems reading/modifying/validating the license file, and only one user can use the db engine. As soon as another one logs in, it says that the max user count has been exceded. Strange thing is that the program that gives you info about the license file identifies it correclty as a 200 users license.

As soon as I replace the 64 bits dosemu with a 32 bits one it works ok again, but we're back to the 2 gb file size limit.

I'll try compiling it on my 32 bits machine with large file support, although I see that there are places where  it's using int instead of off_t to seek files and this will need to be worked out. 

-----Mensaje original-----
De: linux-msdos-owner@vger.kernel.org [mailto:linux-msdos-owner@vger.kernel.org] En nombre de Bryan J Smith
Enviado el: martes, 18 de enero de 2011 20:18
Para: Xavier Tarifa
CC: linux-msdos@vger.kernel.org
Asunto: Re: File Limits

Then it's using the extended functions that legacy DOS Int 21h programs do not understand.  They may have merged in some NT/Win32 extensions in late Windows 98 SE and Me.  They weren't in there for Windows 95, not even OEM 2 (MS-DOS 7.1) when I tested them.

-- 
Bryan J  Smith     Professional, Technical Annoyance 
Linked Profile:   http://www.linkedin.com/in/bjsmith 
---------------------------------------------------- 
UCF Basketball:   AP #23, ESPN/USAToday #22, RPI #17
UCF Football:   AP #21, BCS #25, ESPN/USAToday #20



----- Original Message ----
From: Xavier Tarifa <Xavier.Tarifa@adparts.com>
To: Bryan J Smith <b.j.smith@ieee.org>
Cc: "linux-msdos@vger.kernel.org" <linux-msdos@vger.kernel.org>
Sent: Tue, January 18, 2011 1:10:08 PM
Subject: RE: File Limits

I just created a 3 gb file on a dos 7.10 and it worked (worked as in dir tells the correct filesize). I'll have to wait a day or two to test it against our database because I'm offsite, then I'll report back the results.

-----Mensaje original-----
De: linux-msdos-owner@vger.kernel.org [mailto:linux-msdos-owner@vger.kernel.org]
En nombre de Bryan J Smith
Enviado el: martes, 18 de enero de 2011 18:35
Para: Xavier Tarifa; linux-msdos@vger.kernel.org
Asunto: Re: File Limits

All DOS (including DOS 7/Win 4 releases running in 386Enhanced mode -- Windows 95, 98, Me) only support 2GiB files IIRC.  Even Real86 mode DOS 7 supports FAT32, which allows bigger than 2GiB _file_systems_, but still has the 2GiB-1 (signed 32-bit int) _individual_file_ size limitation IIRC.


In fact, I don't think the 4GiB-1 (unsigned 32-bit int) support is in any DOS/Win releases.  I only think it is an option in NT kernels (FAT16 in 3.1, 3.5, 3.51 and 4.0, FAT16 and FAT32 in 5.0/2000, 5.1/XP-2003, 6.0-Vista and 6.1/7-2008).  And any program that uses it _must_ know about the NT-only Win32 function calls that offer this.

I.e., the "Chicago" (Win9x) and NT/Win32 teams used _different_ DOS Int 21h and Win32[s] function calls, the former completely ignoring the pre-existing of the latter.  "Chicago" _never_ supported the native NT/Win32 calls, hence why they likely _never_ supported the extensions for 4GiB-1 support IIRC.[1]

One quick trip to TechNet and you'll note this is an issue in "bare metal" 
DOS/Win and even NT.  The legacy "Chicago" DOS Int 21h file services don't handle growing beyond 2GiB-1 proper, _unless_ the NT/Win32 function extensions are utilized.  Unfortunately even NT 5.0 (Windows 2000) implements the "Chicago" 

calls bug-for-bug for compatibility reasons.

I.e., your program is likely instigating this, and the OS is just letting it do what it wants (and causing the issue).

-- Bryan

NOTES:  
[1] Like many libraries in NT/Win32, things just "didn't work" or "work well" 
under Win9x -- OpenGL being my personal favorite, and the whole "DirectMM -- aka Direct [DOS] Memory Map" lineage now known as "DirectX".  It wasn't until NT 5.0 (Windows 2000) that the NT/Win32 team started also supporting the
"Chicago"/Win32 DOS Int 21h changes.



----- Original Message ----

From: Xavier Tarifa <Xavier.Tarifa@adparts.com>

So here's my problem: Until now I was using dosemu with ms-dos 6.22 to run an old database. A little while ago we had some big changes to the tables and they grew a lot.
Yesterday I realized that we have a database file that is about to hit 2 Gb. I remembered that some time ago, we hit a bug on our program and it endlessly inserted rows on a table until it hit the 2 gb mark. Then the file either disappeared or became a 0 bytes file, I don't remember well.
So I started reading and found that ms-dos 6.22 only supports FAT16 and apparently it only allows 2 gb volumes, and 2 gb files.
So I thought of using ms-dos 7.10 which supports fat32 and would give us 4 gb files. But the same happens. The files disappear as soon as they hit the 2 gb mark. And when I execute the dir command, it always reports 2 gb of free disk space, which is odd.

I don't know if this is a limitation of dosemu or I'm doing something wrong. 
Right now, I have about 140 users. On each $HOME/.dosemu/drive_c/ of the users there are symlinks to each file in /usr/share/dosemu/drive_z/ , where ms-dos is located. And I have the database files in /datos/ . I use lredir to make it appear as f: to every user.
All of this is in the same ext3 partition.

Any ideas? 
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-msdos" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html