* Question on blocks periodic writes @ 2016-11-09 17:20 Theophanis Kontogiannis 2016-11-10 2:00 ` NeilBrown 0 siblings, 1 reply; 6+ messages in thread From: Theophanis Kontogiannis @ 2016-11-09 17:20 UTC (permalink / raw) To: Linux RAID Hello All, I am in the middle of bringing my server's power consumption to an absolute minimum. Have already reduced idle power from 110W to 52W (including 10W consumed by the UPS). Having enabled spin down to all 5 x 2TB disks, with 5 seconds idle times, I noticed that the disks wake up quite often without the server doing anything actual. Following 'echo 1 > /proc/sys/vm/block_dump' I can not miss that 'dmesg -c' reports frequent messages like: [ 8662.496150] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) [ 8662.496185] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors) [ 8662.496253] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors) [ 8662.496269] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors) [ 8662.496282] md0_raid6(631): WRITE block 16 on sde3 (8 sectors) [ 8664.849252] md0_raid6(631): WRITE block 8 on sda3 (1 sectors) [ 8664.849276] md0_raid6(631): WRITE block 8 on sdf3 (1 sectors) [ 8664.849287] md0_raid6(631): WRITE block 8 on sdg3 (1 sectors) [ 8664.849298] md0_raid6(631): WRITE block 8 on sdd3 (1 sectors) [ 8664.849352] md0_raid6(631): WRITE block 8 on sde3 (1 sectors) [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors) [ 8664.902688] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) [ 8664.902705] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors) [ 8664.902715] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors) [ 8664.902725] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors) [ 8664.902735] md0_raid6(631): WRITE block 16 on sde3 (8 sectors) [ 8665.100056] md1_raid6(630): WRITE block 8 on sda1 (1 sectors) [ 8665.100107] md1_raid6(630): WRITE block 8 on sdf1 (1 sectors) [ 8665.100164] md1_raid6(630): WRITE block 8 on sde1 (1 sectors) [ 8665.100217] md1_raid6(630): WRITE block 8 on sdg1 (1 sectors) [ 8665.100467] md1_raid6(630): WRITE block 8 on sdd1 (1 sectors) [ 8665.177050] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) [ 8665.177098] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors) [ 8665.177154] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors) [ 8665.177207] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors) [ 8665.177431] md0_raid6(631): WRITE block 16 on sde3 (8 sectors) [ 8665.225978] md0_raid6(631): WRITE block 8 on sda3 (1 sectors) [ 8665.225996] md0_raid6(631): WRITE block 8 on sdf3 (1 sectors) [ 8665.226064] md0_raid6(631): WRITE block 8 on sdg3 (1 sectors) [ 8665.226111] md0_raid6(631): WRITE block 8 on sdd3 (1 sectors) [ 8665.226191] md0_raid6(631): WRITE block 8 on sde3 (1 sectors) [ 8670.269138] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) [ 8670.269180] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors) [ 8670.269237] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors) [ 8670.269291] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors) [ 8670.269344] md0_raid6(631): WRITE block 16 on sde3 (8 sectors) [ 8680.269050] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) [ 8680.269092] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors) [ 8680.269147] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors) [ 8680.269249] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors) [ 8680.269428] md0_raid6(631): WRITE block 16 on sde3 (8 sectors) I guess those messages are the reason for the frequent disks spin up. What is the reason behind those writes? Can I affect it? Should I touch it? --- Best regards, ΜΦΧ, Theophanis Kontogiannis ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes 2016-11-09 17:20 Question on blocks periodic writes Theophanis Kontogiannis @ 2016-11-10 2:00 ` NeilBrown 2016-11-10 16:10 ` Wols Lists 0 siblings, 1 reply; 6+ messages in thread From: NeilBrown @ 2016-11-10 2:00 UTC (permalink / raw) To: Theophanis Kontogiannis, Linux RAID [-- Attachment #1: Type: text/plain, Size: 2261 bytes --] On Thu, Nov 10 2016, Theophanis Kontogiannis wrote: > Hello All, > > I am in the middle of bringing my server's power consumption to an > absolute minimum. > > Have already reduced idle power from 110W to 52W (including 10W > consumed by the UPS). > > Having enabled spin down to all 5 x 2TB disks, with 5 seconds idle > times, I noticed that the disks wake up quite often without the server > doing anything actual. > > Following > > 'echo 1 > /proc/sys/vm/block_dump' > > I can not miss that > > 'dmesg -c' > > reports frequent messages like: > > [ 8662.496150] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) > [ 8662.496185] md0_raid6(631): WRITE block 16 on sdf3 (8 sectors) > [ 8662.496253] md0_raid6(631): WRITE block 16 on sdg3 (8 sectors) > [ 8662.496269] md0_raid6(631): WRITE block 16 on sdd3 (8 sectors) > [ 8662.496282] md0_raid6(631): WRITE block 16 on sde3 (8 sectors) These are probably the md bitmap being updated. If you provided some basic detail about you array like mdadm --detail /dev/md1 mdadm --examine /dev/sda3 it would be easier to be sure. > [ 8664.849252] md0_raid6(631): WRITE block 8 on sda3 (1 sectors) > [ 8664.849276] md0_raid6(631): WRITE block 8 on sdf3 (1 sectors) > [ 8664.849287] md0_raid6(631): WRITE block 8 on sdg3 (1 sectors) > [ 8664.849298] md0_raid6(631): WRITE block 8 on sdd3 (1 sectors) > [ 8664.849352] md0_raid6(631): WRITE block 8 on sde3 (1 sectors) This is probably the superblock being updated. > [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors) This is XFS doing something. md cannot possibly stop all IO while the filesystem performs occasional IO. If these continue, you need to discuss with xfs developers how to stop it. If the writes to individual drives continue after there are no writes to 'md1', then it is worth coming back here to ask. > [ 8664.902688] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) > [ 8665.177050] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) > [ 8670.269138] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) > [ 8680.269050] md0_raid6(631): WRITE block 16 on sda3 (8 sectors) The delay here is 270ms, then 5 seconds, then 10 seconds. Does it reach a stable state? What is the period in the stable state? NeilBrown [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 800 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes 2016-11-10 2:00 ` NeilBrown @ 2016-11-10 16:10 ` Wols Lists 2016-11-11 1:52 ` NeilBrown 0 siblings, 1 reply; 6+ messages in thread From: Wols Lists @ 2016-11-10 16:10 UTC (permalink / raw) To: NeilBrown, Theophanis Kontogiannis, Linux RAID On 10/11/16 02:00, NeilBrown wrote: >> [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors) > This is XFS doing something. md cannot possibly stop all IO while the > filesystem performs occasional IO. If these continue, you need to > discuss with xfs developers how to stop it. If the writes to individual > drives continue after there are no writes to 'md1', then it is worth > coming back here to ask. > > Would the new journal feature be any help? I haven't dug in enough to understand it properly, and it would increase the vulnerability of the system to a journal failure, but the feature itself seems almost perfect for batching writes and enabling the disks to spin down for extended periods. Cheers, Wol ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes 2016-11-10 16:10 ` Wols Lists @ 2016-11-11 1:52 ` NeilBrown 2016-11-11 12:11 ` Theophanis Kontogiannis 2016-11-11 12:13 ` Theophanis Kontogiannis 0 siblings, 2 replies; 6+ messages in thread From: NeilBrown @ 2016-11-11 1:52 UTC (permalink / raw) To: Wols Lists, Theophanis Kontogiannis, Linux RAID [-- Attachment #1: Type: text/plain, Size: 1070 bytes --] On Fri, Nov 11 2016, Wols Lists wrote: > On 10/11/16 02:00, NeilBrown wrote: >>> [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors) >> This is XFS doing something. md cannot possibly stop all IO while the >> filesystem performs occasional IO. If these continue, you need to >> discuss with xfs developers how to stop it. If the writes to individual >> drives continue after there are no writes to 'md1', then it is worth >> coming back here to ask. >> >> > Would the new journal feature be any help? Probably not, though until we know what is causing the writes, it is hard to say. > > I haven't dug in enough to understand it properly, and it would increase > the vulnerability of the system to a journal failure, but the feature > itself seems almost perfect for batching writes and enabling the disks > to spin down for extended periods. You might be able to build functionality onto the journal which allows the drives in the main array to stay idle for longer, but it doesn't try to do that at present. NeilBrown [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 800 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes 2016-11-11 1:52 ` NeilBrown @ 2016-11-11 12:11 ` Theophanis Kontogiannis 2016-11-11 12:13 ` Theophanis Kontogiannis 1 sibling, 0 replies; 6+ messages in thread From: Theophanis Kontogiannis @ 2016-11-11 12:11 UTC (permalink / raw) To: NeilBrown; +Cc: Wols Lists, Linux RAID Hi Neil, Hi Community, Regarding XFS we can ignore it. That file system will be moved to an ext2 file system on a CF. So we are left with the rest: -bash-4.2# mdadm --detail /dev/md1 /dev/md1: Version : 1.2 Creation Time : Thu Jun 16 18:02:57 2016 Raid Level : raid6 Array Size : 9397248 (8.96 GiB 9.62 GB) Used Dev Size : 3132416 (2.99 GiB 3.21 GB) Raid Devices : 5 Total Devices : 5 Persistence : Superblock is persistent Update Time : Fri Nov 11 14:05:33 2016 State : clean Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : tweety.example.com:1 (local to host tweety.example.com) UUID : 98e2af83:dc074310:d1639adb:3f19f0d3 Events : 127 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 49 1 active sync /dev/sdd1 2 8 65 2 active sync /dev/sde1 3 8 81 3 active sync /dev/sdf1 4 8 97 4 active sync /dev/sdg1 -bash-4.2# mdadm --examine /dev/sda1 /dev/sda1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 98e2af83:dc074310:d1639adb:3f19f0d3 Name : tweety.example.com:1 (local to host tweety.example.com) Creation Time : Thu Jun 16 18:02:57 2016 Raid Level : raid6 Raid Devices : 5 Avail Dev Size : 6264832 (2.99 GiB 3.21 GB) Array Size : 9397248 (8.96 GiB 9.62 GB) Data Offset : 4096 sectors Super Offset : 8 sectors Unused Space : before=4008 sectors, after=0 sectors State : clean Device UUID : 5ff290a3:68faf9d0:22edd403:abbaf970 Update Time : Fri Nov 11 14:05:59 2016 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 55812945 - correct Events : 127 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing) Thank you ALL --- Best regards, ΜΦΧ, Theophanis Kontogiannis On Fri, Nov 11, 2016 at 3:52 AM, NeilBrown <neilb@suse.com> wrote: > On Fri, Nov 11 2016, Wols Lists wrote: > >> On 10/11/16 02:00, NeilBrown wrote: >>>> [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors) >>> This is XFS doing something. md cannot possibly stop all IO while the >>> filesystem performs occasional IO. If these continue, you need to >>> discuss with xfs developers how to stop it. If the writes to individual >>> drives continue after there are no writes to 'md1', then it is worth >>> coming back here to ask. >>> >>> >> Would the new journal feature be any help? > > Probably not, though until we know what is causing the writes, it is > hard to say. > >> >> I haven't dug in enough to understand it properly, and it would increase >> the vulnerability of the system to a journal failure, but the feature >> itself seems almost perfect for batching writes and enabling the disks >> to spin down for extended periods. > > You might be able to build functionality onto the journal which allows > the drives in the main array to stay idle for longer, but it doesn't try > to do that at present. > > NeilBrown ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Question on blocks periodic writes 2016-11-11 1:52 ` NeilBrown 2016-11-11 12:11 ` Theophanis Kontogiannis @ 2016-11-11 12:13 ` Theophanis Kontogiannis 1 sibling, 0 replies; 6+ messages in thread From: Theophanis Kontogiannis @ 2016-11-11 12:13 UTC (permalink / raw) To: NeilBrown; +Cc: Wols Lists, Linux RAID Have also run the following for about 6 hours: -bash-4.2# cat iotop-results Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s PID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 696 be/3 root 0.00 B 956.00 K 0.00 % 0.31 % [jbd2/md0-8] 750 be/4 root 0.00 B 0.00 B 0.00 % 0.05 % smartd -n -q never 2134 be/4 mysql 624.00 K 344.85 M 0.00 % 0.01 % mysqld --basedir=/usr --datadir=/mnt/md/md0/DBs/mys~riadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock 14110 be/4 root 0.00 B 0.00 B 0.00 % 0.01 % udisksd --no-debug 5217 be/4 root 0.00 B 0.00 B 0.00 % 0.29 % [kworker/1:0] 2454 be/4 root 0.00 B 0.00 B 0.00 % 0.04 % [kworker/0:2] 31764 be/4 root 4.00 K 0.00 B 0.00 % 0.00 % -bash 2368 be/4 root 0.00 B 0.00 B 0.00 % 0.01 % [kworker/1:3] 4584 be/4 root 0.00 B 0.00 B 0.00 % 0.00 % [kworker/1:1] 5265 be/4 root 0.00 B 0.00 B 0.00 % 0.00 % [kworker/0:0] 4595 be/4 root 0.00 B 0.00 B 0.00 % 0.00 % [kworker/0:1] 31747 be/4 root 0.00 B 8.00 K 0.00 % 0.00 % sshd: root@pts/3 761 be/4 root 0.00 B 10.20 M 0.00 % 0.00 % rsyslogd -n 7437 be/4 apache 0.00 B 24.00 K 0.00 % 0.00 % httpd -DFOREGROUND 14625 be/4 apache 0.00 B 20.00 K 0.00 % 0.00 % httpd -DFOREGROUND 10059 be/4 apache 0.00 B 8.00 K 0.00 % 0.00 % httpd -DFOREGROUND 10062 be/4 apache 0.00 B 24.00 K 0.00 % 0.00 % httpd -DFOREGROUND 10094 be/4 apache 0.00 B 16.00 K 0.00 % 0.00 % httpd -DFOREGROUND 10095 be/4 apache 0.00 B 20.00 K 0.00 % 0.00 % httpd -DFOREGROUND 10141 be/4 apache 0.00 B 24.00 K 0.00 % 0.00 % httpd -DFOREGROUND 12220 be/4 root 0.00 B 9.99 M 0.00 % 0.00 % python /usr/bin/weewxd --daemon --pidfile=/var/run/weewx.pid /etc/weewx/weewx.conf --- Best regards, ΜΦΧ, Theophanis Kontogiannis On Fri, Nov 11, 2016 at 3:52 AM, NeilBrown <neilb@suse.com> wrote: > On Fri, Nov 11 2016, Wols Lists wrote: > >> On 10/11/16 02:00, NeilBrown wrote: >>>> [ 8664.858104] xfsaild/md1(658): WRITE block 0 on md1 (8 sectors) >>> This is XFS doing something. md cannot possibly stop all IO while the >>> filesystem performs occasional IO. If these continue, you need to >>> discuss with xfs developers how to stop it. If the writes to individual >>> drives continue after there are no writes to 'md1', then it is worth >>> coming back here to ask. >>> >>> >> Would the new journal feature be any help? > > Probably not, though until we know what is causing the writes, it is > hard to say. > >> >> I haven't dug in enough to understand it properly, and it would increase >> the vulnerability of the system to a journal failure, but the feature >> itself seems almost perfect for batching writes and enabling the disks >> to spin down for extended periods. > > You might be able to build functionality onto the journal which allows > the drives in the main array to stay idle for longer, but it doesn't try > to do that at present. > > NeilBrown ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-11-11 12:13 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-11-09 17:20 Question on blocks periodic writes Theophanis Kontogiannis 2016-11-10 2:00 ` NeilBrown 2016-11-10 16:10 ` Wols Lists 2016-11-11 1:52 ` NeilBrown 2016-11-11 12:11 ` Theophanis Kontogiannis 2016-11-11 12:13 ` Theophanis Kontogiannis
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).