memory capacity

memory capacity

[ronkhu]

Can anyone help me in increasing the maximum allowable memory usage for 
an application in HP/UX?

My superior instructed me to create an array of at least 7,140,000 
elements(statically)... that would be able to
store 8 bytes per index..

What compile option could i use to enhance the memory model (from small 
to huge?) or increase the memory
limit for an application?

-Ron

Re: memory capacity

[Stephen Satchell]

On Mon, 2003-12-08 at 02:38, ronkhu wrote:
> Can anyone help me in increasing the maximum allowable memory usage for 
> an application in HP/UX?
> 
> My superior instructed me to create an array of at least 7,140,000 
> elements(statically)... that would be able to
> store 8 bytes per index..
> 
> What compile option could i use to enhance the memory model (from small 
> to huge?) or increase the memory
> limit for an application?

So what is the limit currently?  I calculate the your array would
require 54.5 megabytes, and unless you have a VERY small swap store you
should be able to handle that easily on any modern computer with modern
operating system.  ("Modern" meaning less than four years old.)

Your other option is to learn how to do applications-level paging to
hard disk, so your RAM requirement is small -- I used to do such stuff
with minicomputers with 64 kilobytes without breaking a sweat.  By
breaking the data into fair-sized pages, you can have a decent working
set in a small footprint (512 KB if this is an old 286-based computer,
with 16-KB pages, as an example) and managing the page swapping
intelligenly, including doing some pre-sorting of input data to minimize
thrashing.

What are you going to do when those eight-byte records grow by a couple
of orders of magnitude?  Or you have to deal with multiple sets of
records in parallel?  You can't beat Parkinson's Law.

Satch

Re: memory capacity

[Glynn Clements]

ronkhu wrote:

> Can anyone help me in increasing the maximum allowable memory usage for 
> an application in HP/UX?

This is a Linux list. You might get some useful replies regarding
generic Unix memory management, but you are less likely to get
anything regarding HP/UX-specific features.

> My superior instructed me to create an array of at least 7,140,000 
> elements(statically)... that would be able to
> store 8 bytes per index..
> 
> What compile option could i use to enhance the memory model (from small 
> to huge?)

Unix isn't MS-DOS. The concept of "small", "huge" etc memory models
was a hack for dealing with the 16-bit limitations on the 8086; it
isn't applicable to Unix.

> or increase the memory limit for an application?

If you have bash, try "ulimit -aS" and "ulimit -aH". Those will report
the soft and hard per-process memory limits respectively. Similarly,
csh and zsh have the "limit" command.

If the soft limits are too low, you can just use "ulimit -d ...",
"ulimit -v ..." etc in the shell to increase them. Or use setrlimit()
in the application.

If the hard limits are too low, the administrator (i.e. root) will
have to increase them; the precise mechanism is platform-specific
(e.g. RedHat uses the pam_limits PAM module; I have no idea what HP/UX
does).

-- 
Glynn Clements <glynn.clements@virgin.net>

Splitter

[K.Anantha Kiran]

Hi,
 I want to modify the Linux code for " REDIRECTING PACKETS FROM ONE 
INTERFACE TO ANOTHER INTERFACE OF MY INTEREST OUT OF ALL AVAILABLE 
INTERFACES". What part of CODE , do i need to concentrate on.

Thanks in advance,
K.AnanthaKiran

Re: Splitter

[Jad Saklawi]

K.Anantha Kiran wrote:

>Hi,
> I want to modify the Linux code for " REDIRECTING PACKETS FROM ONE 
>INTERFACE TO ANOTHER INTERFACE OF MY INTEREST OUT OF ALL AVAILABLE 
>INTERFACES". What part of CODE , do i need to concentrate on.
>
>  
>
    This ain`t gonna be an easy task if you want to do it in kernel 
space. Why re invent the loop when iptables (netfilter) can do the job ? 
any way the networking part for the linux kernel is found the net 
directory in the root directory.

Greets,
Jad

-- 
Jad Saklawi
Lebanese American University
P.O.Box : 13-5053/492
Chouran Beirut : 1102 2801
Phone : 009613 98 98 75
http://Saklawi.info
Lebanon

Re: Splitter

[K.Anantha Kiran]

Hi,
 Thanks for ur suggetsion.
 But we are not clear about the sentece "re invent the loop".
 Can u give clear explanantion about the "iptables" solution.
 IMP: Our machine should not work like a router.I should be able to decide 
to which interface i want to redirect to , on the fly for each packet 
using a HASH like functions.

Thanks,
K.AnanthaKiran
On Mon, 22 Mar 2004, Jad Saklawi wrote:

> Date: Mon, 22 Mar 2004 08:42:28 +0200
> From: Jad Saklawi <Jad@Saklawi.info>
> To: K.Anantha Kiran <ananth@cse.iitk.ac.in>
> Cc: linux-c-programming@vger.kernel.org
> Subject: Re: Splitter
> 
> K.Anantha Kiran wrote:
> 
> >Hi,
> > I want to modify the Linux code for " REDIRECTING PACKETS FROM ONE 
> >INTERFACE TO ANOTHER INTERFACE OF MY INTEREST OUT OF ALL AVAILABLE 
> >INTERFACES". What part of CODE , do i need to concentrate on.
> >
> >  
> >
>     This ain`t gonna be an easy task if you want to do it in kernel 
> space. Why re invent the loop when iptables (netfilter) can do the job ? 
> any way the networking part for the linux kernel is found the net 
> directory in the root directory.
> 
> Greets,
> Jad
> 
> 

Re: Splitter

[Jad Saklawi]

K.Anantha Kiran wrote:

>Hi,
> Thanks for ur suggetsion.
> But we are not clear about the sentece "re invent the loop".
> Can u give clear explanantion about the "iptables" solution.
> IMP: Our machine should not work like a router.I should be able to decide 
>to which interface i want to redirect to , on the fly for each packet 
>using a HASH like functions.
>  
>
 It`s your choice then, check out net/core/dev.c net/core/dv.c 
net/ipv4/tcp_input.c net/ipv4/tcp_output.c net/ethernet/eth.c You have 
to take a look at the whole net directory and modify accordingly. I 
doubt you will be able to do any thing. Any way good luck!!!

Greets,
Jad

Re: Splitter

[K.Anantha Kiran]

Hi,
 Thanks a lot for ur guidance.
 I am following the same files(i mean NET directory).But i am facing 
following problem.
 " where the output interface variable (rtable->key->oif) is changed to 
the index of interface through which packet has to go out. In the function 
/net/route.c ip_route_input_slow() they are setting it to 0 what is it 
mean.

	When a packet has to be put on to an interface what are the fields 
will it use to decide that( in skbuff structure).

Thanks 
regards,
K. Anantha kiran


On Mon, 22 Mar 2004, Jad Saklawi wrote:

> Date: Mon, 22 Mar 2004 09:41:22 +0200
> From: Jad Saklawi <Jad@Saklawi.info>
> To: K.Anantha Kiran <ananth@cse.iitk.ac.in>
> Cc: linux-c-programming@vger.kernel.org
> Subject: Re: Splitter
> 
> K.Anantha Kiran wrote:
> 
> >Hi,
> > Thanks for ur suggetsion.
> > But we are not clear about the sentece "re invent the loop".
> > Can u give clear explanantion about the "iptables" solution.
> > IMP: Our machine should not work like a router.I should be able to decide 
> >to which interface i want to redirect to , on the fly for each packet 
> >using a HASH like functions.
> >  
> >
>  It`s your choice then, check out net/core/dev.c net/core/dv.c 
> net/ipv4/tcp_input.c net/ipv4/tcp_output.c net/ethernet/eth.c You have 
> to take a look at the whole net directory and modify accordingly. I 
> doubt you will be able to do any thing. Any way good luck!!!
> 
> Greets,
> Jad
> 

Re: Splitter

[syedk]

you can look into dev_queue_xmit(sk_buff) function in net/core/dev.c

 sk_buff->dev  has the information abt the device (can be any interface
registered using register_netdevice() for that matter)...

dev_queue_xmit just transfers the sk_buff for the device for
delivery....

I feel the best way to solve your problem would be use NF_IP_PRE_ROUTING
hook of netfilter by which u get all packets ( even before any routing
decisions has been made : that is what u wanted RIGHT??) and then call
dev_queue_xmit function by setting the sk_buff->dev as whatever
interface u wish to redirect....

I hope this would help....

Best regards
syed sk


On Tue, 2004-03-23 at 15:09, K.Anantha Kiran wrote:
> Hi,
>  Thanks a lot for ur guidance.
>  I am following the same files(i mean NET directory).But i am facing 
> following problem.
>  " where the output interface variable (rtable->key->oif) is changed to 
> the index of interface through which packet has to go out. In the function 
> /net/route.c ip_route_input_slow() they are setting it to 0 what is it 
> mean.
> 
> 	When a packet has to be put on to an interface what are the fields 
> will it use to decide that( in skbuff structure).
> 
> Thanks 
> regards,
> K. Anantha kiran
> 
> 
> On Mon, 22 Mar 2004, Jad Saklawi wrote:
> 
> > Date: Mon, 22 Mar 2004 09:41:22 +0200
> > From: Jad Saklawi <Jad@Saklawi.info>
> > To: K.Anantha Kiran <ananth@cse.iitk.ac.in>
> > Cc: linux-c-programming@vger.kernel.org
> > Subject: Re: Splitter
> > 
> > K.Anantha Kiran wrote:
> > 
> > >Hi,
> > > Thanks for ur suggetsion.
> > > But we are not clear about the sentece "re invent the loop".
> > > Can u give clear explanantion about the "iptables" solution.
> > > IMP: Our machine should not work like a router.I should be able to decide 
> > >to which interface i want to redirect to , on the fly for each packet 
> > >using a HASH like functions.
> > >  
> > >
> >  It`s your choice then, check out net/core/dev.c net/core/dv.c 
> > net/ipv4/tcp_input.c net/ipv4/tcp_output.c net/ethernet/eth.c You have 
> > to take a look at the whole net directory and modify accordingly. I 
> > doubt you will be able to do any thing. Any way good luck!!!
> > 
> > Greets,
> > Jad
> > 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Splitter

[K.Anantha Kiran]

Hi,
  Whenever we receive a packet( let it is stored in skb), can we directly 
change the skbuff->dev and call dev_queue_xmit(skbuff) without undergoing 
any processing in this system. That means this will simply take one packet and 
keep it in another interface( for suppose there are no packets destined 
for this system) . Doing this will cause any problems? 



On 23 Mar 2004, syedk wrote:

> Date: 23 Mar 2004 17:08:20 +0530
> From: syedk <syedk@intotoinc.com>
> To: K.Anantha Kiran <ananth@cse.iitk.ac.in>
> Cc: Jad Saklawi <Jad@Saklawi.info>, linux-c-programming@vger.kernel.org
> Subject: Re: Splitter
> 
> you can look into dev_queue_xmit(sk_buff) function in net/core/dev.c
> 
>  sk_buff->dev  has the information abt the device (can be any interface
> registered using register_netdevice() for that matter)...
> 
> dev_queue_xmit just transfers the sk_buff for the device for
> delivery....
> 
> I feel the best way to solve your problem would be use NF_IP_PRE_ROUTING
> hook of netfilter by which u get all packets ( even before any routing
> decisions has been made : that is what u wanted RIGHT??) and then call
> dev_queue_xmit function by setting the sk_buff->dev as whatever
> interface u wish to redirect....
> 
> I hope this would help....
> 
> Best regards
> syed sk
> 
> 
> On Tue, 2004-03-23 at 15:09, K.Anantha Kiran wrote:
> > Hi,
> >  Thanks a lot for ur guidance.
> >  I am following the same files(i mean NET directory).But i am facing 
> > following problem.
> >  " where the output interface variable (rtable->key->oif) is changed to 
> > the index of interface through which packet has to go out. In the function 
> > /net/route.c ip_route_input_slow() they are setting it to 0 what is it 
> > mean.
> > 
> > 	When a packet has to be put on to an interface what are the fields 
> > will it use to decide that( in skbuff structure).
> > 
> > Thanks 
> > regards,
> > K. Anantha kiran
> > 
> > 
> > On Mon, 22 Mar 2004, Jad Saklawi wrote:
> > 
> > > Date: Mon, 22 Mar 2004 09:41:22 +0200
> > > From: Jad Saklawi <Jad@Saklawi.info>
> > > To: K.Anantha Kiran <ananth@cse.iitk.ac.in>
> > > Cc: linux-c-programming@vger.kernel.org
> > > Subject: Re: Splitter
> > > 
> > > K.Anantha Kiran wrote:
> > > 
> > > >Hi,
> > > > Thanks for ur suggetsion.
> > > > But we are not clear about the sentece "re invent the loop".
> > > > Can u give clear explanantion about the "iptables" solution.
> > > > IMP: Our machine should not work like a router.I should be able to decide 
> > > >to which interface i want to redirect to , on the fly for each packet 
> > > >using a HASH like functions.
> > > >  
> > > >
> > >  It`s your choice then, check out net/core/dev.c net/core/dv.c 
> > > net/ipv4/tcp_input.c net/ipv4/tcp_output.c net/ethernet/eth.c You have 
> > > to take a look at the whole net directory and modify accordingly. I 
> > > doubt you will be able to do any thing. Any way good luck!!!
> > > 
> > > Greets,
> > > Jad
> > > 
> > 
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Splitter

[syedk]

hi ananth,
  yes u can directly send it to other interface...
but be very careful in this : u need to take care of lot of things
otherwise u may face lot of kernel panics:
1) u shud return NF_STOLEN; in the PRE_ROUTING hook... otherwise the
dev_queue_xmit will try to free the sk_buff (which might be already
freed if u use NF_DROP)...
2)u need to fill( or rather change existing ones) the relevant
information in the sk_buff...  that might include the
  i) source and destination address in the IP ..
  ii) changing the pkt_type to PACKET_OUTGOING,
  iii) u also need to take care of changing datalink layer stuff in
sk_buff.. for example if packet came on ppp or loopback and u want to
redirect it to ethernet interface.....

  all u need to care of is this : it shud be a correct sk_buff while
xmiting it....
 probably u wont succeed first time itself .... u may find some problems
or worst some crashes... but u will succeed finally...

Better read this document: 
http://uqconnect.net/~zzoklan/documents/netfilter.html

i hope this would help...

best regards
Syed SK

On Tue, 2004-03-23 at 17:24, K.Anantha Kiran wrote:
> Hi,
>   Whenever we receive a packet( let it is stored in skb), can we directly 
> change the skbuff->dev and call dev_queue_xmit(skbuff) without undergoing 
> any processing in this system. That means this will simply take one packet and 
> keep it in another interface( for suppose there are no packets destined 
> for this system) . Doing this will cause any problems? 
> 
> 
> 
> On 23 Mar 2004, syedk wrote:
> 
> > Date: 23 Mar 2004 17:08:20 +0530
> > From: syedk <syedk@intotoinc.com>
> > To: K.Anantha Kiran <ananth@cse.iitk.ac.in>
> > Cc: Jad Saklawi <Jad@Saklawi.info>, linux-c-programming@vger.kernel.org
> > Subject: Re: Splitter
> > 
> > you can look into dev_queue_xmit(sk_buff) function in net/core/dev.c
> > 
> >  sk_buff->dev  has the information abt the device (can be any interface
> > registered using register_netdevice() for that matter)...
> > 
> > dev_queue_xmit just transfers the sk_buff for the device for
> > delivery....
> > 
> > I feel the best way to solve your problem would be use NF_IP_PRE_ROUTING
> > hook of netfilter by which u get all packets ( even before any routing
> > decisions has been made : that is what u wanted RIGHT??) and then call
> > dev_queue_xmit function by setting the sk_buff->dev as whatever
> > interface u wish to redirect....
> > 
> > I hope this would help....
> > 
> > Best regards
> > syed sk
> > 
> > 
> > On Tue, 2004-03-23 at 15:09, K.Anantha Kiran wrote:
> > > Hi,
> > >  Thanks a lot for ur guidance.
> > >  I am following the same files(i mean NET directory).But i am facing 
> > > following problem.
> > >  " where the output interface variable (rtable->key->oif) is changed to 
> > > the index of interface through which packet has to go out. In the function 
> > > /net/route.c ip_route_input_slow() they are setting it to 0 what is it 
> > > mean.
> > > 
> > > 	When a packet has to be put on to an interface what are the fields 
> > > will it use to decide that( in skbuff structure).
> > > 
> > > Thanks 
> > > regards,
> > > K. Anantha kiran
> > > 
> > > 
> > > On Mon, 22 Mar 2004, Jad Saklawi wrote:
> > > 
> > > > Date: Mon, 22 Mar 2004 09:41:22 +0200
> > > > From: Jad Saklawi <Jad@Saklawi.info>
> > > > To: K.Anantha Kiran <ananth@cse.iitk.ac.in>
> > > > Cc: linux-c-programming@vger.kernel.org
> > > > Subject: Re: Splitter
> > > > 
> > > > K.Anantha Kiran wrote:
> > > > 
> > > > >Hi,
> > > > > Thanks for ur suggetsion.
> > > > > But we are not clear about the sentece "re invent the loop".
> > > > > Can u give clear explanantion about the "iptables" solution.
> > > > > IMP: Our machine should not work like a router.I should be able to decide 
> > > > >to which interface i want to redirect to , on the fly for each packet 
> > > > >using a HASH like functions.
> > > > >  
> > > > >
> > > >  It`s your choice then, check out net/core/dev.c net/core/dv.c 
> > > > net/ipv4/tcp_input.c net/ipv4/tcp_output.c net/ethernet/eth.c You have 
> > > > to take a look at the whole net directory and modify accordingly. I 
> > > > doubt you will be able to do any thing. Any way good luck!!!
> > > > 
> > > > Greets,
> > > > Jad
> > > > 
> > > 
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

System hang at high bandwidth

[K.Anantha Kiran]

Hi, 
  I am using linux 2.4.20-8 smp kernel on a dual processor Xeon 2GHz 
processor with 4 Intel PRO 1000 MT Desktop adapters. I am using latest 
version of e1000 drivers with NAPI enabled. I am facing a problem. I want 
to read packets from one NIC and send them to one of the other three NICs 
based on some hash function. When I try this experiment at high speeds 
(>250 Mbps) using either raw sockets (application level programming) or by 
writing a kernel module for IP_PREROUTING_HOOK, my system is hanging. Can any one please help me out 
by telling some possible reason for this and how to get rid of this.
   Is it OK to use 4 NICs simultaneously at such high speeds.
Thanks in advance,
Ananth.    

Re: System hang at high bandwidth

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

On Fri, 2004-03-26 01:05:28 +0530, K.Anantha Kiran <ananth@cse.iitk.ac.in>
wrote in message <Pine.LNX.4.44.0403260056310.4797-100000@csews72.cse.iitk.ac.in>:
> based on some hash function. When I try this experiment at high speeds 
> (>250 Mbps) using either raw sockets (application level programming) or by 
> writing a kernel module for IP_PREROUTING_HOOK, my system is hanging. Can any one please help me out 
> by telling some possible reason for this and how to get rid of this.

The possible reason is: "There's a bug somewhere". However, nobody can
help because you messed to pass along some URL where we can download
your userspace application and/or your kernel module's sources.

>    Is it OK to use 4 NICs simultaneously at such high speeds.

Yes.

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
   ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Can i use dev_queue_xmit

[K.Anantha Kiran]

 We are supposed to build a kernel module which takes packets in the
 internet through one interface and forward them to another interface of
 my choice based on the IP,Port feilds in the packet.

Our module is working fine for less rate(200Mbps) of traffic.If speed is 
increasing(>200Mbps), for some packets "dev_queue_xmit()" is returning NET_XMIT_DROP due to which packets are being dropped.

 Can u suggest any solution. Can u give the reason for that return value
 only for high traffic.Is there any speed restiction to dev_queue_xmit() 
We are using Gigbyte traffic ethrtnet cards.

Thanks in advance,
K.AnanthaKiran

Re: Can i use dev_queue_xmit

[Armen Kaleshian]

Why don't you look at the source for iptables? Or use iptables instead of writing the kernel module from scratch?

Good luck!

On Tue, Apr 06, 2004 at 07:50:50PM +0530, K.Anantha Kiran wrote:
: 
:  We are supposed to build a kernel module which takes packets in the
:  internet through one interface and forward them to another interface of
:  my choice based on the IP,Port feilds in the packet.
: 
: Our module is working fine for less rate(200Mbps) of traffic.If speed is 
: increasing(>200Mbps), for some packets "dev_queue_xmit()" is returning NET_XMIT_DROP due to which packets are being dropped.
: 
:  Can u suggest any solution. Can u give the reason for that return value
:  only for high traffic.Is there any speed restiction to dev_queue_xmit() 
: We are using Gigbyte traffic ethrtnet cards.
: 
: Thanks in advance,
: K.AnanthaKiran
: 
: 
: 
: -
: To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
: the body of a message to majordomo@vger.kernel.org
: More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Splitter

[Michael Taylor]

K.Anantha Kiran wrote:
> Hi,
>  Thanks for ur suggetsion.
>  But we are not clear about the sentece "re invent the loop".
>  Can u give clear explanantion about the "iptables" solution.

<http://www.google.com/search?q=iptables>

iptables/netfilter sounds like it will do what you need (and more)
<http://www.netfilter.org/>

>  IMP: Our machine should not work like a router.I should be able to decide
> to which interface i want to redirect to , on the fly for each packet
> using a HASH like functions.

Like a router.

Re: Splitter

[K.Anantha Kiran]

Hi,
  I am giving some what better description of my problem.I want to modify 
the Linux code for " REDIRECTING PACKETS REACHING ONE INTERFACE , AS IT IS 
TO ANOTHER INTERFACE OF MY INTEREST OUT OF ALL AVAILABLE 
 INTERFACES".Note PACKETS here should not be allowed to go beyond IP 
layer .That means System should not do any changes to the packet 
before outputting.What part of CODE , do i need to concentrate on.

 Thanks in advance,
 K.AnanthaKiran