1004K MT paging issue

1004K MT paging issue

[Mikael Starvik]

Hi!

We have a 1004K core with two VPEs with two TCs per VPE. We have a problem that is hard to debug and would like to know if anyone has seen or solved such an issue.

A multithreaded application is running. 
Twoapplication threads are running in one TC each on the same VPE.
A piece of code has been paged out.
Application thread 1 tries to execute the code and thus gets a page fault.
While the page fault is being handled the second application thread enters the same code.
For some reason it looks like application thread 2 is allowed to execute even if the page fault handling has not been finished yet.
Thread 2 executes the wrong code and typically gets a reserved instruction exception.

Any thougts?

BR
/Mikael
From kevink@paralogos.com Tue May 27 00:58:05 2014
Received: with ECARTIS (v1.0.0; list linux-mips); Tue, 27 May 2014 00:58:07 +0200 (CEST)
Received: from gateway09.websitewelcome.com ([67.18.14.9]:50725 "EHLO
        gateway09.websitewelcome.com" rhost-flags-OK-OK-OK-OK)
        by eddie.linux-mips.org with ESMTP id S6821443AbaEZW6Fxt29- (ORCPT
        <rfc822;linux-mips@linux-mips.org>); Tue, 27 May 2014 00:58:05 +0200
Received: by gateway09.websitewelcome.com (Postfix, from userid 507)
        id AC86B75A91E30; Mon, 26 May 2014 17:58:03 -0500 (CDT)
Received: from gator3163.hostgator.com (gator3163.hostgator.com [50.87.144.199])
        by gateway09.websitewelcome.com (Postfix) with ESMTP id 04A6575A916F6
        for <linux-mips@linux-mips.org>; Mon, 26 May 2014 17:58:02 -0500 (CDT)
Received: from [98.234.48.184] (port=52665 helo=[10.0.0.13])
        by gator3163.hostgator.com with esmtpa (Exim 4.82)
        (envelope-from <kevink@paralogos.com>)
        id 1Wp3qD-0005gN-Hr
        for linux-mips@linux-mips.org; Mon, 26 May 2014 17:58:01 -0500
Message-ID: <5383C6DD.4090107@paralogos.com>
Date:   Mon, 26 May 2014 15:57:33 -0700
From:   "Kevin D. Kissell" <kevink@paralogos.com>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To:     linux-mips@linux-mips.org
Subject: Re: 1004K MT paging issue
References: <498838AF-48B0-4244-95C0-F590040E5E08@axis.com>
In-Reply-To: <498838AF-48B0-4244-95C0-F590040E5E08@axis.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - gator3163.hostgator.com
X-AntiAbuse: Original Domain - linux-mips.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - paralogos.com
X-BWhitelist: no
X-Source-IP: 98.234.48.184
X-Exim-ID: 1Wp3qD-0005gN-Hr
X-Source: 
X-Source-Args: 
X-Source-Dir: 
X-Source-Sender: ([10.0.0.13]) [98.234.48.184]:52665
X-Source-Auth: kevink@kevink.net
X-Email-Count: 2
X-Source-Cap: a2tpc3NlbGw7a2tpc3NlbGw7Z2F0b3IzMTYzLmhvc3RnYXRvci5jb20=
Return-Path: <kevink@paralogos.com>
X-Envelope-To: <"|/home/ecartis/ecartis -s linux-mips"> (uid 0)
X-Orcpt: rfc822;linux-mips@linux-mips.org
Original-Recipient: rfc822;linux-mips@linux-mips.org
X-archive-position: 40270
X-ecartis-version: Ecartis v1.0.0
Sender: linux-mips-bounce@linux-mips.org
Errors-to: linux-mips-bounce@linux-mips.org
X-original-sender: kevink@paralogos.com
Precedence: bulk
List-help: <mailto:ecartis@linux-mips.org?Subject=help>
List-unsubscribe: <mailto:ecartis@linux-mips.org?subject=unsubscribe%20linux-mips>
List-software: Ecartis version 1.0.0
List-Id: linux-mips <linux-mips.eddie.linux-mips.org>
X-List-ID: linux-mips <linux-mips.eddie.linux-mips.org>
List-subscribe: <mailto:ecartis@linux-mips.org?subject=subscribe%20linux-mips>
List-owner: <mailto:ralf@linux-mips.org>
List-post: <mailto:linux-mips@linux-mips.org>
List-archive: <http://www.linux-mips.org/archives/linux-mips/>
X-list: linux-mips
Content-Length: 1239
Lines: 29

When a VPE hits an exception and sets EXL, thread scheduling stops and
the VPE is single-threaded coming into the exception.  With dual VPEs,
one of them hitting a fault won't prevent another from doing so, but all
the usual rules of locking and ordering that are needed for any SMP
kernel should apply and "just work".  Is there any SMP support in the VM
subsystem that's conditionally modfied or excluded for your 1004K kernel
build?

/K.


On 5/26/2014 1:56 PM, Mikael Starvik wrote:
> Hi!
>
> We have a 1004K core with two VPEs with two TCs per VPE. We have a problem that is hard to debug and would like to know if anyone has seen or solved such an issue.
>
> A multithreaded application is running. 
> Twoapplication threads are running in one TC each on the same VPE.
> A piece of code has been paged out.
> Application thread 1 tries to execute the code and thus gets a page fault.
> While the page fault is being handled the second application thread enters the same code.
> For some reason it looks like application thread 2 is allowed to execute even if the page fault handling has not been finished yet.
> Thread 2 executes the wrong code and typically gets a reserved instruction exception.
>
> Any thougts?
>
> BR
> /Mikael

1004K MT paging issue

[Mikael Starvik]

Hi!

We have a 1004K core with two VPEs with two TCs per VPE. We have a problem that is hard to debug and would like to know if anyone has seen or solved such an issue.

A multithreaded application is running. 
Twoapplication threads are running in one TC each on the same VPE.
A piece of code has been paged out.
Application thread 1 tries to execute the code and thus gets a page fault.
While the page fault is being handled the second application thread enters the same code.
For some reason it looks like application thread 2 is allowed to execute even if the page fault handling has not been finished yet.
Thread 2 executes the wrong code and typically gets a reserved instruction exception.

Any thougts?

BR
/Mikael

Re: 1004K MT paging issue

[Kevin D. Kissell]

When a VPE hits an exception and sets EXL, thread scheduling stops and
the VPE is single-threaded coming into the exception.  With dual VPEs,
one of them hitting a fault won't prevent another from doing so, but all
the usual rules of locking and ordering that are needed for any SMP
kernel should apply and "just work".  Is there any SMP support in the VM
subsystem that's conditionally modfied or excluded for your 1004K kernel
build?

/K.


On 5/26/2014 1:56 PM, Mikael Starvik wrote:
> Hi!
>
> We have a 1004K core with two VPEs with two TCs per VPE. We have a problem that is hard to debug and would like to know if anyone has seen or solved such an issue.
>
> A multithreaded application is running. 
> Twoapplication threads are running in one TC each on the same VPE.
> A piece of code has been paged out.
> Application thread 1 tries to execute the code and thus gets a page fault.
> While the page fault is being handled the second application thread enters the same code.
> For some reason it looks like application thread 2 is allowed to execute even if the page fault handling has not been finished yet.
> Thread 2 executes the wrong code and typically gets a reserved instruction exception.
>
> Any thougts?
>
> BR
> /Mikael

RE: 1004K MT paging issue

[Lars Persson]

Hi

I have found a MIPS-specific race window in __do_fault() between these two calls:
set_pte_at(mm, address, page_table, entry);

/* no need to invalidate: a not-present page won't be cached */
update_mmu_cache(vma, address, page_table);
 
When the block device uses PIO-mode to populate the page in the page-cache, then the main memory still contains stale data when set_pte_at() makes the mapping live. Another VPE hitting the same page will execute garbage code.

Please see the ARM port for hints how to fix this by flushing the data cache already in set_pte_at.

BR,
 Lars


> -----Original Message-----
> From: Mikael Starvik
> Sent: den 26 maj 2014 22:57
> To: linux-mips@linux-mips.org
> Cc: Lars Persson; Martin Santesson
> Subject: 1004K MT paging issue
> 
> Hi!
> 
> We have a 1004K core with two VPEs with two TCs per VPE. We have a
> problem that is hard to debug and would like to know if anyone has seen
> or solved such an issue.
> 
> A multithreaded application is running.
> Twoapplication threads are running in one TC each on the same VPE.
> A piece of code has been paged out.
> Application thread 1 tries to execute the code and thus gets a page
> fault.
> While the page fault is being handled the second application thread
> enters the same code.
> For some reason it looks like application thread 2 is allowed to
> execute even if the page fault handling has not been finished yet.
> Thread 2 executes the wrong code and typically gets a reserved
> instruction exception.
> 
> Any thougts?
> 
> BR
> /Mikael