Re: Confirmation request about new 2.4.x. kernel limits

Re: Confirmation request about new 2.4.x. kernel limits

[Tigran Aivazian]

On Thu, 4 Jan 2001, A.D.F. wrote:
> Max. RAM size:			64 GB	(any slowness accessing RAM over 4 GB
> 					 with 32 bit machines ?)

realistic benchmarks (unixbench) will show about 3%-6% performance
degradation with use of PAE. Note that this is not "accessing RAM over
4G" but (what you probably meant) "accessing any RAM in a machine with
over 4G of RAM" or even "accessing any RAM in a machine with less than 4G
or RAM but running kernel capable of accessing >4G". If you really meant
"accessing RAM over 4G" then you are probably talking about 36bit MTRR
support which is present in recent 2.4.x kernels and works very nicely!

You can construct artificial benchmarks that will show 10% performance
degradation. I haven't compared this with other PAE implementation,
e.g. that of UnixWare 7.1.1+ -- this would be a very useful exercise as I 
vaguely remember some people claiming that Linux PAE implementation is
not ideal (if it is true then it ought to be made ideal to be inline with
the rest of the kernel).

Regards,
Tigran

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Confirmation request about new 2.4.x. kernel limits

[A.D.F.]

Hi, I would like to know whether following limits are right for kernel
2.4.x:

Max. N. of CPU:			32	(SMP)
Max. CPU speed:			> 2 Ghz	(up to ?)
Max. RAM size:			64 GB	(any slowness accessing RAM over 4 GB
					 with 32 bit machines ?)
Max. file size:	 		1 TB	(?)
Max. file system size:		2 TB	(?)
Max. N. of files per FS:	2^32	(depending on max. n. of inodes ?)
Max. N. of users/groups:	2^32	(well, that's in theory, probably 						
practical limits, due to RAM usage,
					 access time, etc., are much lower).

Other upper limits, eventually compared with those of latest 2.2.x
kernels, would be appreciated.

	Thanks in advance for your patience.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Anton Blanchard]

> Hi, I would like to know whether following limits are right for kernel
> 2.4.x:
> 
> Max. N. of CPU:			32	(SMP)

Max CPUs is 64 on 64 bit architectures (well you have to change NR_CPUS).
I am told larger than 32 cpu ultrasparcs have booted linux already.

Anton
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Pavel Machek]

Hi!

> Hi, I would like to know whether following limits are right for kernel
> 2.4.x:
> 
> Max. N. of CPU:			32	(SMP)
> Max. CPU speed:			> 2 Ghz	(up to ?)
> Max. RAM size:			64 GB	(any slowness accessing RAM over 4 GB
> 					 with 32 bit machines ?)

64GB on i386. I believe you can get few terabytes with ultrasparc.

> Max. file size:	 		1 TB	(?)
> Max. file system size:		2 TB	(?)

Again, maybe on i386 with ext2.
								Pavel

-- 
I'm pavel@ucw.cz. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss@linmodems.org
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: Confirmation request about new 2.4.x. kernel limits

[Venkatesh Ramamurthy]

	> Max. RAM size:			64 GB	(any slowness
accessing RAM over 4 GB
*	with 32 bit machines ?)
	Imore than 4GB in RAM is bounce buffered, so there is performance
penalty as the data have to be copied into the 4GB RAM area
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Matti Aarnio]

On Mon, Jan 08, 2001 at 11:11:05PM -0500, Venkatesh Ramamurthy wrote:
> 
> > Max. RAM size:	64 GB	(any slowness accessing RAM over 4 GB
> >				 with 32 bit machines ?)
> 	more than 4GB in RAM is bounce buffered, so there is performance
> 	penalty as the data have to be copied into the 4GB RAM area

  Actual memory limit is lower, your run-of-the-mill Pentium-PAE36 capable
  system has PCI bus(es) for IO, and address space for that/those need to
  stay in area below 4G for bootup to access any devices, thus very likely 
  your system doesn't have more than, say, 3 GB of RAM below 4G.

  Pick your processors.  You need XEONs to have L1/L2 cacheing on memory
  above the 4 GB address (PAE36 mapped physical addresses.)

  For IO on usual systems you have 32 bit address space PCI busmasters,
  so those can access only the lowest 4GB of address space, and to have
  a block of data in upper area, it needs to be "bounced", that is, CPU
  must copy it.  Linux 2.4.0 system doesn't support 64-bit PCI addresses
  at 32-bit systems (not at 64-bit Alpha either, I recall.)
  On the other hand, Alpha systems and SPARC systems have IOMMU hardware,
  and we do support that (to some extent), but 32-bit intel world doesn't
  have similar things.

  For userspace, if parts of userspace are physically mapped above 4G,
  it might not be very harmfull at all -- presuming you have XEONs which
  cache the memory accesses there also.  The libc and similar multiply
  shared objects might as well reside in high memory.   Userspace process
  doesn't see, after all, where each page resides physically.

/Matti Aarnio
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Stephen C. Tweedie]

Hi,

On Fri, Jan 05, 2001 at 11:46:04PM +0100, Pavel Machek wrote:
> 
> > Max. file size:	 		1 TB	(?)
> > Max. file system size:		2 TB	(?)
> 
> Again, maybe on i386 with ext2.

Actually, the 2TB limit affects all architectures, as we assume that
block indexes fit into 32 bits.  Blocks are passed around as unsigned
longs in some cases, but even on 64-bit machines that doesn't help us
as the limit still persists in the filesystem (32-bit block numbers)
and device drivers (ints and 4-byte sector numbers used when
generating SCSI commands).

Auditing the whole driver path to allow 64-bit block numbers, and
adding the logic to generate the 5th sector address byte in the scsi
command when we're doing 10-byte commands, are all possible extensions
for 2.5.  For now, though, the 2TB device limit is with us for all
architectures and all filesystems on 2.4.

--Stephen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Stephen C. Tweedie]

Hi,

On Mon, Jan 08, 2001 at 11:11:05PM -0500, Venkatesh Ramamurthy wrote:
> 
> 	> Max. RAM size:			64 GB	(any slowness
> accessing RAM over 4 GB
> *	with 32 bit machines ?)
> 	Imore than 4GB in RAM is bounce buffered, so there is performance
> penalty as the data have to be copied into the 4GB RAM area

Any memory over 1GB is bounce-buffered, but we don't use that memory
for anything other than process data pages or file cache, so only
swapping and disk IO to regular files gets the extra copy.  In
particular, things like network buffers are still all kept in the low
1GB so never need to be buffered.

--Stephen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: Confirmation request about new 2.4.x. kernel limits

[Venkatesh Ramamurthy]

> Any memory over 1GB is bounce-buffered, but we don't use that memory
> for anything other than process data pages or file cache, so only
> swapping and disk IO to regular files gets the extra copy.  In
> particular, things like network buffers are still all kept in the low
> 1GB so never need to be buffered.
	[Venkatesh Ramamurthy]  If anything over 1GB is bounce buffered than
what is the purpose of setting the pci_dev->dma_mask field.  On a IA32
system we set it to 32 1's and IA64 to 64 1's - Venkat
>  
> 
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Andi Kleen]

On Tue, Jan 09, 2001 at 10:15:34AM -0500, Venkatesh Ramamurthy wrote:
> > Any memory over 1GB is bounce-buffered, but we don't use that memory
> > for anything other than process data pages or file cache, so only
> > swapping and disk IO to regular files gets the extra copy.  In
> > particular, things like network buffers are still all kept in the low
> > 1GB so never need to be buffered.
> 	[Venkatesh Ramamurthy]  If anything over 1GB is bounce buffered than
> what is the purpose of setting the pci_dev->dma_mask field.  On a IA32
> system we set it to 32 1's and IA64 to 64 1's - Venkat

On 2.4 on 32bit hosts it has no direct effect because the driver interface has not 
been revised.  It's only useful on 64bit atm.

2.5 will hopefully move the bounce test into your driver. Currently it tests
PageHighMem() which is bogus (and gives the 1GB boundary), because it should test 
the physical address against the dma_mask.

On 64bit PageHighMem always returns false, so the bounce buffer is never create in
the upper level and the final decision is down in your driver.

Currently create_bounce() in the block layer is at a very unfortunate place on the
top half of ll_rw_blk, because there it has not even a chance to know about the
final device (e.g. when LVM or MD come into play) 

I suspect someone will quickly do a limited 2.4 patch that moves create_bounce a bit
down for selected devices on 32bit hosts -- it is a very obvious optimization. 
It probably also could be done with minor changes.


-Andi

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Tim Wright]

On Tue, Jan 09, 2001 at 02:21:56PM +0200, Matti Aarnio wrote:
[...]
> 
>   For IO on usual systems you have 32 bit address space PCI busmasters,
>   so those can access only the lowest 4GB of address space, and to have
>   a block of data in upper area, it needs to be "bounced", that is, CPU
>   must copy it.  Linux 2.4.0 system doesn't support 64-bit PCI addresses
>   at 32-bit systems (not at 64-bit Alpha either, I recall.)
>   On the other hand, Alpha systems and SPARC systems have IOMMU hardware,
>   and we do support that (to some extent), but 32-bit intel world doesn't
>   have similar things.
> 

Hi Matti,
you are correct in saying that ia32 systems don't have IOMMU hardware, but
it's unfortunate that we don't support 64-bit PCI bus master cards, since
they're inexpensive and fairly common now. For instance, the Qlogic ISP SCSI
cards can do 64-bit addressing, as can many others. Has anybody taken a look
at enabling this ?

Tim

-- 
Tim Wright - timw@splhi.com or timw@aracnet.com or twright@us.ibm.com
IBM Linux Technology Center, Beaverton, Oregon
"Nobody ever said I was charming, they said "Rimmer, you're a git!"" RD VI
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Andi Kleen]

On Tue, Jan 09, 2001 at 08:27:49AM -0800, Tim Wright wrote:
> you are correct in saying that ia32 systems don't have IOMMU hardware, but
> it's unfortunate that we don't support 64-bit PCI bus master cards, since
> they're inexpensive and fairly common now. For instance, the Qlogic ISP SCSI
> cards can do 64-bit addressing, as can many others. Has anybody taken a look
> at enabling this ?

Problem is that it needs a driver interface change and cooperation from the
drivers. 

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Tim Wright]

On Tue, Jan 09, 2001 at 05:44:46PM +0100, Andi Kleen wrote:
> On Tue, Jan 09, 2001 at 08:27:49AM -0800, Tim Wright wrote:
> > you are correct in saying that ia32 systems don't have IOMMU hardware, but
> > it's unfortunate that we don't support 64-bit PCI bus master cards, since
> > they're inexpensive and fairly common now. For instance, the Qlogic ISP SCSI
> > cards can do 64-bit addressing, as can many others. Has anybody taken a look
> > at enabling this ?
> 
> Problem is that it needs a driver interface change and cooperation from the
> drivers. 
> 
> -Andi

I thought as much. Sounds like 2.5 material really, although it might be
interesting to see if I can make a patch to play with.

Regards,

Tim

-- 
Tim Wright - timw@splhi.com or timw@aracnet.com or twright@us.ibm.com
IBM Linux Technology Center, Beaverton, Oregon
"Nobody ever said I was charming, they said "Rimmer, you're a git!"" RD VI
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: Confirmation request about new 2.4.x. kernel limits

[Venkatesh Ramamurthy]

> Problem is that it needs a driver interface change and cooperation from
> the
> drivers. 
	[Venkatesh Ramamurthy]  Atleast the spec for this new interface,
that the driver has to support be prepared? Once this is done we can port
driver by driver to this new standard.

> -Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Andi Kleen]

On Tue, Jan 09, 2001 at 12:35:02PM -0500, Venkatesh Ramamurthy wrote:
> 
> > Problem is that it needs a driver interface change and cooperation from
> > the
> > drivers. 
> 	[Venkatesh Ramamurthy]  Atleast the spec for this new interface,
> that the driver has to support be prepared? Once this is done we can port
> driver by driver to this new standard.

AFAIK there is no spec yet. Just supporting 64bit DMA on 32bit hosts would 
probably only minor changes (like pushing the dma_mask flag a bit more out so
that it is visible by scsi or ll_rw_blk), but 2.5 will probably also see more
extensive changes for block drivers, like moving them to kiovecs.  

Your input would be probably welcome.

-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: Confirmation request about new 2.4.x. kernel limits

[Venkatesh Ramamurthy]

> On Tue, Jan 09, 2001 at 08:27:49AM -0800, Tim Wright wrote:
> > you are correct in saying that ia32 systems don't have IOMMU hardware,
> but
> > it's unfortunate that we don't support 64-bit PCI bus master cards,
> since
> > they're inexpensive and fairly common now. For instance, the Qlogic ISP
> SCSI
> > cards can do 64-bit addressing, as can many others. Has anybody taken a
> look
> > at enabling this ?
	[Venkatesh Ramamurthy]  Our AMI RAID controller and MegaRAID driver
( part of kernel) supports full blown 64 bit and we have successfully tested
with more than 4 GB RAM under IA32 (uses bounce buffer) and IA64 systems. We
are seeing a dramatic performance drop on IA32 with 8GB ram as it has to do
bounce buffering. I think we need this support to give our dear old linux a
lead compared to other OS'es.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: Confirmation request about new 2.4.x. kernel limits

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 843 bytes --]

On Tue, Jan 09, 2001 at 08:27:49AM -0800, Tim Wright wrote:
> On Tue, Jan 09, 2001 at 02:21:56PM +0200, Matti Aarnio wrote:
> [...]
> >   On the other hand, Alpha systems and SPARC systems have IOMMU hardware,
> >   and we do support that (to some extent), but 32-bit intel world doesn't
> >   have similar things.
> 
> you are correct in saying that ia32 systems don't have IOMMU hardware,

Well, there is: AGPGART does something like this. 
I don't know, if it support more than 32bits address space, though.

> but it's unfortunate that we don't support 64-bit PCI bus master cards,

Indeed.

Regards,
-- 
Kurt Garloff  <garloff@suse.de>                          Eindhoven, NL
GPG key: See mail header, key servers         Linux kernel development
SuSE GmbH, Nuernberg, FRG                               SCSI, Security

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]