2GB File limitation

2GB File limitation

[Bobby Hitt]

Hello,

Is someone working on a way to overcome the 2GB file limitation in Linux? I
currently backup several servers using a dedicated hard drive for the
backups. Recently I saw one backup die saying the the file size had been
exceeded. I've never had good luck with tape backups, yes they backup, but
whenever I really need a file, it can't be retrieved.

TIA,

Bobby

Re: 2GB File limitation

[Andreas Dilger]

On Sep 28, 2001  02:17 -0400, Bobby Hitt wrote:
> Is someone working on a way to overcome the 2GB file limitation in Linux? I
> currently backup several servers using a dedicated hard drive for the
> backups. Recently I saw one backup die saying the the file size had been
> exceeded.

This is your lucky day.  The 2GB limit has been fixed a year or two ago.
You need an "LFS" patch for 2.2 kernels (any vendor kernel will have this),
or any 2.4 kernel.

You ALSO need to ensure your libc supports large files (glibc-2.2
definitely will), and also that your tools can handle it (needs to be
compiled with special LARGEFILE defines, and open files with O_LARGEFILE).

Cheers, Andreas
--
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: 2GB File limitation

[Ingo Molnar]

On Fri, 28 Sep 2001, Bobby Hitt wrote:

> Is someone working on a way to overcome the 2GB file limitation in
> Linux? I currently backup several servers using a dedicated hard drive
> for the backups. Recently I saw one backup die saying the the file
> size had been exceeded. I've never had good luck with tape backups,
> yes they backup, but whenever I really need a file, it can't be
> retrieved.

file sizes up to ~ 2 TB are supported under the 2.4 kernel. (or 2.2 kernel
+ patches) Most utilities are updated to use O_LARGEFILE properly, in any
recent 2.4-based Linux distribution. I regularly use 6-10 GB files.

	Ingo

Re: 2GB File limitation

[Matti Aarnio]

On Fri, Sep 28, 2001 at 10:13:09AM +0200, Ingo Molnar wrote:
> On Fri, 28 Sep 2001, Bobby Hitt wrote:
> > Is someone working on a way to overcome the 2GB file limitation in
> > Linux? I currently backup several servers using a dedicated hard drive
> > for the backups. Recently I saw one backup die saying the the file
> > size had been exceeded. I've never had good luck with tape backups,
> > yes they backup, but whenever I really need a file, it can't be
> > retrieved.
> 
> file sizes up to ~ 2 TB are supported under the 2.4 kernel. (or 2.2 kernel
> + patches) Most utilities are updated to use O_LARGEFILE properly, in any
> recent 2.4-based Linux distribution. I regularly use 6-10 GB files.

  Like Ingo says, most of modern distributions begin to work with
  O_LARGEFILE  properly, but still some utilities fail it.

  Your older backup application program may fail it too.

  The solution could be as simple as piping the backup material
  into another program, which then pipes it out, e.g.:
	'|dd bs=1M of=/backups/fileNNN'

> 	Ingo

/Matti Aarnio

RE: 2GB File limitation

[M. Edward Borasky]

Do you know if they ever fixed this same issue in Microsoft Access? :-) Does
Linux support tape drives? :-)

Sure, Linux needs to be able to handle files > 2 GB, but I wonder if backups
over a network to a hard drive are giving you a false sense of "security".
In the "enterprise" world, you have to be *religious* about backups.
Incrementals at least daily, fulls at least weekly, all backups verified and
*off-site secure storage of backup media*. Dang! The soap company wants
their &$^%&$# box back :-).

--
M. Edward (Ed) Borasky, Chief Scientist, Borasky Research
http://www.borasky-research.net  http://www.aracnet.com/~znmeb
mailto:znmeb@borasky-research.net mailto:znmeb@aracnet.com

If there's nothing to astrology, how come so many famous men were born on
holidays?

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Bobby Hitt
> Sent: Thursday, September 27, 2001 11:17 PM
> To: linux-kernel@vger.kernel.org
> Subject: 2GB File limitation
>
>
> Hello,
>
> Is someone working on a way to overcome the 2GB file limitation
> in Linux? I
> currently backup several servers using a dedicated hard drive for the
> backups. Recently I saw one backup die saying the the file size had been
> exceeded. I've never had good luck with tape backups, yes they backup, but
> whenever I really need a file, it can't be retrieved.
>
> TIA,
>
> Bobby
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>

Re: 2GB File limitation

[Michael H. Warfield]

On Fri, Sep 28, 2001 at 02:17:07AM -0400, Bobby Hitt wrote:
> Hello,

> Is someone working on a way to overcome the 2GB file limitation in Linux? I
> currently backup several servers using a dedicated hard drive for the
> backups. Recently I saw one backup die saying the the file size had been
> exceeded. I've never had good luck with tape backups, yes they backup, but
> whenever I really need a file, it can't be retrieved.

	The problem has been fixed in the file system, the kernel and
the supporting libraries for ages.  Applications may need to be updated
to take advantage of 64 bit file offsets and file operations, though.
You didn't say how you were backing up those other servers.  Is it a
particular backup program?  If so, that program may need to be rebuilt
or updated.  Are you backing up over NFS or some other file transport
where the Linux box is just acting as a file server or is the backup
program running on that Linux box?

	Oh...  And what version of Linux?  If it's old enough, I
suppose you might still have a problem there.  Recent (RedHat 7.x
and contemporaries) distributions shouldn't be a problem.

> TIA,

> Bobby

	Mike
-- 
 Michael H. Warfield    |  (770) 985-6132   |  mhw@WittsEnd.com
  (The Mad Wizard)      |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!

Re: 2GB File limitation

[Tim Moore]

> Is someone working on a way to overcome the 2GB file limitation in Linux? I
> currently backup several servers using a dedicated hard drive for the
> backups. Recently I saw one backup die saying the the file size had been
> exceeded. I've never had good luck with tape backups, yes they backup, but
> whenever I really need a file, it can't be retrieved.

I use the -B flag to insure the archives are just under 2GB.  This
generates multiple volumes by appending a sequence number if necessary.

For example, on a /home dump of >2GB, the commands

ddate=`date +%m%d%y-%H%M%S`
dump -0uM -b 32 -B 2097142 -f /mnt/home.$ddate.L0.dump -L "abit:/home"
/home

generate

/mnt/home.092801-130439.L0.dump001
/mnt/home.092801-130439.L0.dump002

each guaranteed to be 10K under 2GB.  Fixes potential portability issues
also.

rgds,
tim.
--