From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from localhost (localhost.localdomain [127.0.0.1]) by einstein.tteng.com.br (Postfix) with ESMTP id 44D3E1205CA for ; Thu, 11 Nov 2004 12:33:10 -0200 (BRDT) Received: from [192.168.0.141] (luciano.tteng.com.br [192.168.0.141]) by einstein.tteng.com.br (Postfix) with ESMTP id 356F01205C4 for ; Thu, 11 Nov 2004 12:33:09 -0200 (BRDT) Message-ID: <41937940.9070001@tteng.com.br> Date: Thu, 11 Nov 2004 12:37:52 -0200 From: "Luciano A. Stertz" MIME-Version: 1.0 Subject: [Fwd: Page allocator doubt] Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: linux-mm@kvack.org List-ID: May someone help me? I tried kernelnewbies, but got no answers. TIA, Luciano Stertz -------- Original Message -------- Subject: Page allocator doubt Date: Wed, 10 Nov 2004 15:31:41 -0200 From: Luciano A. Stertz To: KERNEL I have a doubt about the page allocation. I need to allocate a number of contiguous pages to later initialize and add them to the page cache. I'm doing something like that: struct address_space *x = &area->vm_file->f_dentry->d_inode->i_data; struct page *page = alloc_pages(mapping_gfp_mask(x)|__GFP_COLD, order); for (i = 0; i< 1< aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e31.co.us.ibm.com (8.12.10/8.12.9) with ESMTP id iABJAoCB490106 for ; Thu, 11 Nov 2004 14:10:50 -0500 Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by westrelay02.boulder.ibm.com (8.12.10/NCO/VER6.6) with ESMTP id iABJAo1p269022 for ; Thu, 11 Nov 2004 12:10:50 -0700 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id iABJAnxj004008 for ; Thu, 11 Nov 2004 12:10:50 -0700 Subject: Re: [Fwd: Page allocator doubt] From: Dave Hansen In-Reply-To: <41937940.9070001@tteng.com.br> References: <41937940.9070001@tteng.com.br> Content-Type: text/plain Message-Id: <1100200247.932.1145.camel@localhost> Mime-Version: 1.0 Date: Thu, 11 Nov 2004 11:10:47 -0800 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: "Luciano A. Stertz" Cc: linux-mm List-ID: On Thu, 2004-11-11 at 06:37, Luciano A. Stertz wrote: > Only the first page got it page counter incremented. Is this expected? Yes. -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4193BD07.5010100@tteng.com.br> Date: Thu, 11 Nov 2004 17:27:03 -0200 From: "Luciano A. Stertz" MIME-Version: 1.0 Subject: Re: [Fwd: Page allocator doubt] References: <41937940.9070001@tteng.com.br> <1100200247.932.1145.camel@localhost> In-Reply-To: <1100200247.932.1145.camel@localhost> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: Dave Hansen Cc: linux-mm List-ID: Dave Hansen wrote: > On Thu, 2004-11-11 at 06:37, Luciano A. Stertz wrote: > >>Only the first page got it page counter incremented. Is this expected? > > > Yes. But... are they allocated to me, even with page_count zeroed? Do I need to do get_page on the them? Sorry if it's a too lame question, but I still didn't understand and found no place to read about this. Luciano Stertz -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e34.co.us.ibm.com (8.12.10/8.12.9) with ESMTP id iABJaxAD317142 for ; Thu, 11 Nov 2004 14:36:59 -0500 Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by westrelay04.boulder.ibm.com (8.12.10/NCO/VER6.6) with ESMTP id iABJb2b4147530 for ; Thu, 11 Nov 2004 12:37:02 -0700 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id iABJaxLQ026496 for ; Thu, 11 Nov 2004 12:36:59 -0700 Subject: Re: [Fwd: Page allocator doubt] From: Dave Hansen In-Reply-To: <4193BD07.5010100@tteng.com.br> References: <41937940.9070001@tteng.com.br> <1100200247.932.1145.camel@localhost> <4193BD07.5010100@tteng.com.br> Content-Type: text/plain Message-Id: <1100201816.7883.22.camel@localhost> Mime-Version: 1.0 Date: Thu, 11 Nov 2004 11:36:56 -0800 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: "Luciano A. Stertz" Cc: linux-mm List-ID: On Thu, 2004-11-11 at 11:27, Luciano A. Stertz wrote: > But... are they allocated to me, even with page_count zeroed? Do I need > to do get_page on the them? Sorry if it's a too lame question, but I > still didn't understand and found no place to read about this. Do you see anywhere in the page allocator where it does a loop like yours? for (i = 1; i< 1< aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4193CA1B.1090409@tteng.com.br> Date: Thu, 11 Nov 2004 18:22:51 -0200 From: "Luciano A. Stertz" MIME-Version: 1.0 Subject: Re: [Fwd: Page allocator doubt] References: <41937940.9070001@tteng.com.br> <1100200247.932.1145.camel@localhost> <4193BD07.5010100@tteng.com.br> <1100201816.7883.22.camel@localhost> In-Reply-To: <1100201816.7883.22.camel@localhost> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: Dave Hansen Cc: linux-mm List-ID: Dave Hansen wrote: > On Thu, 2004-11-11 at 11:27, Luciano A. Stertz wrote: > >> But... are they allocated to me, even with page_count zeroed? Do I need >>to do get_page on the them? Sorry if it's a too lame question, but I >>still didn't understand and found no place to read about this. > > > Do you see anywhere in the page allocator where it does a loop like > yours? > > for (i = 1; i< 1< get_page(page + i); Actually this loop isn't mine. It's part of the page allocator, but it's only executed on systems without a MMU. > When you do a multi-order allocation, the first page represents the > whole group and they're treated as a whole. As you've noticed, breaking > them up requires a little work. > > Why don't you post all of the code that you're using so that we can tell > what you're doing? There might be a better way. Drivers probably > shouldn't be putting stuff in the page cache all by themselves. Unhappily I can't post any code yet, but I'll try to give an insight of what we're trying to do. It's not a driver. We're doing an implementation to allow the kernel to execute compressed files, decompressing pages on demand. These files will usually be compressed in small blocks, typically 4kb. But if they got compressed in blocks bigger then a page (say 8kb blocks on a 4kb page system), the kernel will have more than one decompressed page each time a block have to be decompressed; and I'd like to add them both to the page cache. So, seems I would have to break multi-order allocated pages. Is this possible / viable? If not, maybe I'll have to work only with small blocks, but I wouldn't like to... > > -- Dave > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: aart@kvack.org > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e33.co.us.ibm.com (8.12.10/8.12.9) with ESMTP id iABKYEJT541392 for ; Thu, 11 Nov 2004 15:34:15 -0500 Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by westrelay02.boulder.ibm.com (8.12.10/NCO/VER6.6) with ESMTP id iABKYE1p240234 for ; Thu, 11 Nov 2004 13:34:14 -0700 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id iABKYE0u030070 for ; Thu, 11 Nov 2004 13:34:14 -0700 Subject: Re: [Fwd: Page allocator doubt] From: Dave Hansen In-Reply-To: <4193CA1B.1090409@tteng.com.br> References: <41937940.9070001@tteng.com.br> <1100200247.932.1145.camel@localhost> <4193BD07.5010100@tteng.com.br> <1100201816.7883.22.camel@localhost> <4193CA1B.1090409@tteng.com.br> Content-Type: text/plain Message-Id: <1100205251.7883.90.camel@localhost> Mime-Version: 1.0 Date: Thu, 11 Nov 2004 12:34:11 -0800 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: "Luciano A. Stertz" Cc: linux-mm List-ID: On Thu, 2004-11-11 at 12:22, Luciano A. Stertz wrote: > Dave Hansen wrote: > > On Thu, 2004-11-11 at 11:27, Luciano A. Stertz wrote: > > > >> But... are they allocated to me, even with page_count zeroed? Do I need > >>to do get_page on the them? Sorry if it's a too lame question, but I > >>still didn't understand and found no place to read about this. > > > > > > Do you see anywhere in the page allocator where it does a loop like > > yours? > > > > for (i = 1; i< 1< > get_page(page + i); > Actually this loop isn't mine. It's part of the page allocator, but > it's only executed on systems without a MMU. Well, what does that tell you? How can page_count(page[i]) be non-zero unless someone goes and sets it like that for pages other than the first (0th) one? > Unhappily I can't post any code yet, but I'll try to give an insight of > what we're trying to do. > It's not a driver. We're doing an implementation to allow the kernel to > execute compressed files, decompressing pages on demand. > These files will usually be compressed in small blocks, typically 4kb. > But if they got compressed in blocks bigger then a page (say 8kb blocks > on a 4kb page system), the kernel will have more than one decompressed > page each time a block have to be decompressed; and I'd like to add them > both to the page cache. Why do 2 *uncompressed* blocks have to be physically contiguous? If you're decompressing and you need more than one page, just allocate another one. I understand that your algorithms may not be optimized for this right now, but that's what you get for doing it in the kernel. :) > So, seems I would have to break multi-order allocated pages. Is this > possible / viable? If not, maybe I'll have to work only with small > blocks, but I wouldn't like to... It's possible, but you shouldn't do it. Multi-order pages are a very valuable commodity and should be reserved for things that *ABSOLUTELY* need them, like DMA buffers. If you ever get a system up for a while and under a lot of memory pressure, those non-order-zero allocations are going to start failing all over the place. Make your code handle all order-0 pages now. You'll need to do it eventually. -- Dave -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Date: Thu, 11 Nov 2004 19:21:01 -0200 From: Marcelo Tosatti Subject: Re: [Fwd: Page allocator doubt] Message-ID: <20041111212101.GA18822@logos.cnet> References: <41937940.9070001@tteng.com.br> <1100200247.932.1145.camel@localhost> <4193BD07.5010100@tteng.com.br> <1100201816.7883.22.camel@localhost> <4193CA1B.1090409@tteng.com.br> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4193CA1B.1090409@tteng.com.br> Sender: owner-linux-mm@kvack.org Return-Path: To: "Luciano A. Stertz" Cc: Dave Hansen , linux-mm List-ID: On Thu, Nov 11, 2004 at 06:22:51PM -0200, Luciano A. Stertz wrote: > Dave Hansen wrote: > >On Thu, 2004-11-11 at 11:27, Luciano A. Stertz wrote: > > > >> But... are they allocated to me, even with page_count zeroed? Do I > >> need to do get_page on the them? Sorry if it's a too lame question, but I > >>still didn't understand and found no place to read about this. > > > > > >Do you see anywhere in the page allocator where it does a loop like > >yours? > > > > for (i = 1; i< 1< > get_page(page + i); > Actually this loop isn't mine. It's part of the page allocator, but > it's only executed on systems without a MMU. > > >When you do a multi-order allocation, the first page represents the > >whole group and they're treated as a whole. As you've noticed, breaking > >them up requires a little work. > > > >Why don't you post all of the code that you're using so that we can tell > >what you're doing? There might be a better way. Drivers probably > >shouldn't be putting stuff in the page cache all by themselves. > Unhappily I can't post any code yet, but I'll try to give an insight > of what we're trying to do. > It's not a driver. We're doing an implementation to allow the kernel > to execute compressed files, decompressing pages on demand. > These files will usually be compressed in small blocks, typically > 4kb. But if they got compressed in blocks bigger then a page (say 8kb > blocks on a 4kb page system), the kernel will have more than one > decompressed page each time a block have to be decompressed; and I'd like > to add them both to the page cache. > So, seems I would have to break multi-order allocated pages. Is this > possible / viable? If not, maybe I'll have to work only with small > blocks, but I wouldn't like to... Why do you need the pages to be physically contiguous? I dont see any reason for that requirement - you can use discontiguous physical pages which are virtually contiguous (so your decompression code wont need to care about non adjacent pieces of memory). -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: aart@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4194A05C.2090103@tteng.com.br> Date: Fri, 12 Nov 2004 09:37:00 -0200 From: "Luciano A. Stertz" MIME-Version: 1.0 Subject: Re: [Fwd: Page allocator doubt] References: <41937940.9070001@tteng.com.br> <1100200247.932.1145.camel@localhost> <4193BD07.5010100@tteng.com.br> <1100201816.7883.22.camel@localhost> <4193CA1B.1090409@tteng.com.br> <20041111212101.GA18822@logos.cnet> In-Reply-To: <20041111212101.GA18822@logos.cnet> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org Return-Path: To: Marcelo Tosatti Cc: Dave Hansen , linux-mm List-ID: Marcelo Tosatti wrote: > On Thu, Nov 11, 2004 at 06:22:51PM -0200, Luciano A. Stertz wrote: > >>Dave Hansen wrote: >> >>>On Thu, 2004-11-11 at 11:27, Luciano A. Stertz wrote: >>> >>> >>>> But... are they allocated to me, even with page_count zeroed? Do I >>>> need to do get_page on the them? Sorry if it's a too lame question, but I >>>>still didn't understand and found no place to read about this. >>> >>> >>>Do you see anywhere in the page allocator where it does a loop like >>>yours? >>> >>> for (i = 1; i< 1<>> get_page(page + i); >> >> Actually this loop isn't mine. It's part of the page allocator, but >>it's only executed on systems without a MMU. >> >> >>>When you do a multi-order allocation, the first page represents the >>>whole group and they're treated as a whole. As you've noticed, breaking >>>them up requires a little work. >>> >>>Why don't you post all of the code that you're using so that we can tell >>>what you're doing? There might be a better way. Drivers probably >>>shouldn't be putting stuff in the page cache all by themselves. >> >> Unhappily I can't post any code yet, but I'll try to give an insight >> of what we're trying to do. >> It's not a driver. We're doing an implementation to allow the kernel >> to execute compressed files, decompressing pages on demand. >> These files will usually be compressed in small blocks, typically >> 4kb. But if they got compressed in blocks bigger then a page (say 8kb >>blocks on a 4kb page system), the kernel will have more than one >>decompressed page each time a block have to be decompressed; and I'd like >>to add them both to the page cache. >> So, seems I would have to break multi-order allocated pages. Is this >>possible / viable? If not, maybe I'll have to work only with small >>blocks, but I wouldn't like to... > > > Why do you need the pages to be physically contiguous? > > I dont see any reason for that requirement - you can use discontiguous physical > pages which are virtually contiguous (so your decompression code wont need to > care about non adjacent pieces of memory). > Thanks Dave and Marcelo. You're obviously right. I'll do it. Luciano -- Luciano A. Stertz luciano@tteng.com.br T&T Engenheiros Associados Ltda http://www.tteng.com.br Fone/Fax (51) 3224 8425 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: aart@kvack.org