From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: weird memory access problem running on dom0 Date: Tue, 22 Oct 2013 17:28:11 +0100 Message-ID: <5266A79B.4050608@citrix.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3436097767251685569==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Alice Wan Cc: "xen-devel@lists.xensource.com" , keir@xen.org, jbeulich@suse.com List-Id: xen-devel@lists.xenproject.org --===============3436097767251685569== Content-Type: multipart/alternative; boundary="------------010405030205040006000404" --------------010405030205040006000404 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit On 22/10/13 16:41, Alice Wan wrote: > hi all, > > recently we met an unbelievable weird memory problem running on > dom0, test case is very simple, code is as following: > > #define BUF_SIZE 4096 > #define IO_PATTERN 0xab > > int main(int argc, char *argv[]) > { > void *buf; > char cmp_buf[BUF_SIZE]; > int err = 0; > > buf = malloc(BUF_SIZE); > if (!buf) { > fprintf(stderr, "error %s during %s\n", > strerror(-err), > "malloc"); > return 1; > } > memset(buf, IO_PATTERN, BUF_SIZE); > memset(cmp_buf, IO_PATTERN, BUF_SIZE); > > if (memcmp(buf, cmp_buf, BUF_SIZE)) { > unsigned long long *ubuf = (unsigned long long *)buf; > int i; > > for (i = 0; i < BUF_SIZE / sizeof(unsigned long long); > i++) > printf("%d: 0x%llx\n", i, ubuf[i]); > > return 2; > } > > return 0; > } > > memcmp failure occurs while the case is running on 500 machines > with Xen, each for billion times. > error log has two results, one is 0x0, it shows buf is zero, the > other one is 0xabababa...ababa, it shows cmp_buf isn't 0xabab..ab > > both of error log shows either buf or cmp_buf is all incorrect. > > However, this case pass when we run on native linux kernel(2.6.32) > without Xen. > > we suspect maybe it's relevent to pvops behavior of dom0. > > we're not sure whether it's a bug fixed in newer version of kernel > and xen, so we have tried diffrent version of Xen and dom0 including > Xen4.0.1+kernel2.6.32/3.0/3.11 and Xen4.2 + kernel2.6.32, > unfortunately, all of these failed. > > we found PAT behaves differenly between linux and xen, so we try > to add nopat into command line of kernel 3.11, and it also failed. > > now we're blocked, realy need some help. > > any advice will be appreciated > > thanks in advance > > > > regards, > wanjia Picking randomly at some ideas: Do you have ballooning enabled? At the time of a failure, is there anything interesting in the Linux or Xen dmesg? Are you running a debug version of Linux or Xen? ~Andrew --------------010405030205040006000404 Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit
On 22/10/13 16:41, Alice Wan wrote:
hi all,

    recently we met an unbelievable weird memory problem running on dom0, test case is very simple, code is as following:

#define BUF_SIZE        4096
#define IO_PATTERN      0xab

int main(int argc, char *argv[])
{
        void *buf;
        char cmp_buf[BUF_SIZE];
        int err = 0;

        buf = malloc(BUF_SIZE);
        if (!buf) {
                fprintf(stderr, "error %s during %s\n",
                        strerror(-err),
                        "malloc");
                return 1;
        }
        memset(buf, IO_PATTERN, BUF_SIZE);
        memset(cmp_buf, IO_PATTERN, BUF_SIZE);

        if (memcmp(buf, cmp_buf, BUF_SIZE)) {
                unsigned long long *ubuf = (unsigned long long *)buf;
                int i;

                for (i = 0; i < BUF_SIZE / sizeof(unsigned long long); i++)
                        printf("%d: 0x%llx\n", i, ubuf[i]);
                        
                return 2;
        }

        return 0;
}

    memcmp failure occurs while the case is running on 500 machines with Xen, each for billion times. 
    error log has two results, one is 0x0, it shows buf is zero,  the other one is 0xabababa...ababa,  it shows cmp_buf isn't 0xabab..ab

    both of error log shows either buf or cmp_buf is all incorrect.

    However, this case pass when we run on native linux kernel(2.6.32) without Xen.

    we suspect maybe it's relevent to pvops behavior of dom0.

    we're not sure whether it's a bug fixed in newer version of kernel and xen, so we have tried diffrent version of Xen and dom0 including Xen4.0.1+kernel2.6.32/3.0/3.11 and Xen4.2 + kernel2.6.32, unfortunately, all of these failed.

    we found PAT behaves differenly between linux and xen, so we try to add nopat into command line of kernel 3.11, and it also failed.

    now we're blocked, realy need some help.

    any advice will be appreciated

    thanks in advance



regards,
wanjia

Picking randomly at some ideas:

Do you have ballooning enabled?

At the time of a failure, is there anything interesting in the Linux or Xen dmesg?

Are you running a debug version of Linux or Xen?

~Andrew
--------------010405030205040006000404-- --===============3436097767251685569== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============3436097767251685569==--