回复： XFS direct IO problem

[YeYin <eyniy@qq.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4307 bytes --]

Dave,
Thank you for your explanation. I got the reason, and I write some code to  simulate the MySQL.It will reproduce the progress:‍


open file without direct flag
read file   //cause kernel readahead 4 pages, and inode->i_mapping->nrpages > 0
close file


open file with direct flag
lseek 4*4096 // skip 4 readahead pages
read  file //cause xfs_flushinval_pages to do nothing
...


I'd like to ask XFS how to resovle this problem?


Attach code：‍
 
/* gcc -o test_read test_read.c
 
 * dd if=/dev/zero of=/data1/fo.dat bs=4096 count=10
 
 * ./test_read /data1/fo.dat 2 direct
 
 * */
 
#define _GNU_SOURCE
 
#include <stdio.h>
 
#include <unistd.h>
 
#include <sys/types.h>
 
#include <sys/stat.h>
 
#include <fcntl.h>
 
#include <errno.h>
 
#include <string.h>
 
#include <stdlib.h>
 


 
#define BUFSIZE 4096
 


 
int read_count = 2;
 


 
int main(int argc, char *argv[]){
 
	if(argc < 3){
 
		fprintf(stderr, "usage: %s <file> <count> [buffer|direct]\n", argv[0]);
 
		exit(1);
 
	}
 
	char *buf = memalign(BUFSIZE - 1, BUFSIZE);
 
	char *file = argv[1];
 
	read_count = atoi(argv[2]);
 
	int ret = 0,sum = 0, i = 0, fd = -1;
 
	if(argc == 4 && strncmp(argv[3], "direct",6) == 0){
 
		//fd = open(file, O_RDONLY|O_DIRECT);
 
		fd = open(file, O_RDONLY);
 
		if(fd < 0){
 
			fprintf(stderr, "open read only file failed\n");
 
			exit(1);
 
		}
 
		ret = read(fd, buf, BUFSIZE);
 
		if(ret < 0){
 
			fprintf(stderr, "buffer read error\n");
 
		}
 
		close(fd);
 
		fd = open(file, O_RDWR);
 
		if(fd < 0){
 
			fprintf(stderr, "open read only file failed\n");
 
			exit(1);
 
		}
 


 
		if (fcntl(fd, F_SETFL, O_RDONLY|O_DIRECT) == -1) {
 
			fprintf(stderr, "set direct error\n");
 
			exit(1);
 
		}
 


 
	}else{
 
		fd = open(file, O_RDONLY);
 
		if(fd < 0){
 
			fprintf(stderr, "open buf file failed\n");
 
			exit(1);
 
		}
 
	}
 


 
	while(i++ < read_count){
 
		//memset(buf, 0, BUFSIZE);
 
		if(buf == NULL){
 
			fprintf(stderr, "memory allocate failed\n");
 
			exit(1);
 
		}
 
		if(lseek(fd, 4*4096, SEEK_SET) < 0){
 
			fprintf(stderr, "seek error!\n");
 
			break;
 
		}
 
		ret = read(fd, buf, BUFSIZE);
 
		if(ret > 0){
 
			sum += ret;
 
		}else if(ret == 0){
 
			printf("read end\n");
 
			break;
 
		}
 
		else{
 
			printf("error:%d\n", errno);
 
			break;
 
		}
 
		sleep(1);
 
	}
 
	printf("read sum: %d\n", sum);
 
	close(fd);
 
	free(buf);
 
	return 0;
 
}‍





------------------ 原始邮件 ------------------
发件人: "Dave Chinner";<david@fromorbit.com>;
发送时间: 2015年4月8日(星期三) 中午12:49
收件人: "YeYin"<eyniy@qq.com>; 
抄送: "xfs"<xfs@oss.sgi.com>; 
主题: Re: XFS direct IO problem



On Wed, Apr 08, 2015 at 12:21:45PM +0800, YeYin wrote:
> Hi, About 2 months ago, I asked one problem in XFS, see
> here(http://oss.sgi.com/archives/xfs/2015-02/msg00197.html).
> 
> 
> After that, I use direct IO in MySQL, see
> here(https://dev.mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_flush_method).‍
> 
> 
> However, I found that MySQL performance is still poor sometimes. I
> use some tools(https://github.com/brendangregg/perf-tools‍) to
> trace the kernel, I found some problems:

<snip>

> This will cause bad performance, even direct IO. I still don't
> understand why not truncate_inode_page called?

Because the cached page must be outside the range of the direct IO
that is in progress - direct IO only tries to flush pages over the
range it is doing the IO over.

> Every time, after I run this: echo 1 > /proc/sys/vm/drop_caches
> 
> Immediately enhance performance.

Because that flushes whatever page is in the cache. Can you identify
what offset that cached page is at? Tracing the xfs events will tell
you what pages that operation invalidates on each inode, and knowing
the offset may tell us why that page is not getting flushed.

Alternatively, write a simple C program that deomnstrates the same
problem so we can reproduce it easily, fix the problem and turn it
into a regression test....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[-- Attachment #1.2: Type: text/html, Size: 13464 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs