* RE: RAID 5,6 sequential writing seems slower in newer kernels
From: Robert Kierski @ 2015-12-02 15:28 UTC (permalink / raw)
To: Phil Turmel, Dallas Clement; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <565F03F2.3070803@turmel.org>
Thanks for the response.
Nice try... But, the reason I’m using the 3.18.4 kernel is that it has the parallelization. I've got group_thread_cnt set to 32. I'm watching the CPU's with mpstat, and they're pretty much idle. I'm also watching the system traces with perf. It claims that only 11.9% of my time is spent doing the xor.
I've got my CS set at 128k. I have noticed that if I set the CS to 32k, the TP is about 2x. I'm pretty sure the problem is that the 1M writes I'm doing are being broken into 4K pages, and then reassembled before going to disk.
Also, this is independent of the IO Scheduler. I've tried all 3 and got the same results.
Bob Kierski
Senior Storage Performance Engineer
Cray Inc.
380 Jackson Street
Suite 210
St. Paul, MN 55101
Tele: 651-967-9590
Fax: 651-605-9001
Cell: 651-890-7461
-----Original Message-----
From: Phil Turmel [mailto:philip@turmel.org]
Sent: Wednesday, December 02, 2015 8:45 AM
To: Robert Kierski; Dallas Clement
Cc: linux-raid@vger.kernel.org
Subject: Re: RAID 5,6 sequential writing seems slower in newer kernels
On 12/02/2015 09:18 AM, Robert Kierski wrote:
> Sorry... I should have mentioned that I'm running the 3.18.4 kernel
> with a 32 core xeon and 128G of memory. I'm not using a FS, I'm going
> directly to the raid block device.
I'm not sure if the parallelization of raid parity has been merged yet, but I'm pretty sure it isn't in 3.18. With one core tied up computing parity and the rest idle, that'd be 96.875% idle.
Phil
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Phil Turmel @ 2015-12-02 14:45 UTC (permalink / raw)
To: Robert Kierski, Dallas Clement; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <F7761B9B1D11B64BBB666019E9378117FDDB18@CFWEX01.americas.cray.com>
On 12/02/2015 09:18 AM, Robert Kierski wrote:
> Sorry... I should have mentioned that I'm running the 3.18.4 kernel
> with a 32 core xeon and 128G of memory. I'm not using a FS, I'm
> going directly to the raid block device.
I'm not sure if the parallelization of raid parity has been merged yet,
but I'm pretty sure it isn't in 3.18. With one core tied up computing
parity and the rest idle, that'd be 96.875% idle.
Phil
^ permalink raw reply
* RE: RAID 5,6 sequential writing seems slower in newer kernels
From: Robert Kierski @ 2015-12-02 14:18 UTC (permalink / raw)
To: Dallas Clement; +Cc: linux-raid@vger.kernel.org
In-Reply-To: <20151202010745.GC9812@www5.open-std.org>
Sorry... I should have mentioned that I'm running the 3.18.4 kernel with a 32 core xeon and 128G of memory. I'm not using a FS, I'm going directly to the raid block device.
Bob Kierski
Senior Storage Performance Engineer
Cray Inc.
380 Jackson Street
Suite 210
St. Paul, MN 55101
Tele: 651-967-9590
Fax: 651-605-9001
Cell: 651-890-7461
^ permalink raw reply
* RE: RAID 5,6 sequential writing seems slower in newer kernels
From: Robert Kierski @ 2015-12-02 14:15 UTC (permalink / raw)
To: Dallas Clement, linux-raid@vger.kernel.org
In-Reply-To: <CAE9DZURyZKmnUmnn5ELAhLUq8nhXD1eBx4g8hDmCzS-MYLrT+Q@mail.gmail.com>
Wow!!! You've stole my thunder. But I can do you one better.
I've got DDR3 RAM disks that are capable of sustained performance of about 6G/s each. When combined into a 8+2 RAID6, I get only about 3G/s doing sequential writes, and about 42G/s on reads. CPU is 96.8% idle for the writes.
In iostat, I'm seeing about 10x the amount of data going to each individual disk compared to what my user app is writing to the raid device.
Bob Kierski
Senior Storage Performance Engineer
Cray Inc.
380 Jackson Street
Suite 210
St. Paul, MN 55101
Tele: 651-967-9590
Fax: 651-605-9001
Cell: 651-890-7461
^ permalink raw reply
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
From: Semyon Enskiy @ 2015-12-02 13:35 UTC (permalink / raw)
To: philip, linux-raid@vger.kernel.org
Hello, Phil, thank you for help.
Writing to report, that array and LVM/ext4 on it successfully restored.
# Before rebooting and committing changes, dump of first 10 MB was done,
now restore headers.
# for i in f i h g d e b c a j; do dd if=devsd"$i"3-head-bigger.raw
of=/dev/sd"$i"3; done
# Without revert-reshape assembling.
# mdadm --create --assume-clean --level=5 --raid-devices=10
--metadata=1.2 /dev/md3 /dev/sd{f,i,h,g,d,e,b,c,a,j}3
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md3 : active raid5 sdj3[9] sda3[8] sdc3[7] sdb3[6] sde3[5] sdd3[4]
sdg3[3] sdh3[2] sdi3[1] sdf3[0]
35095924224 blocks super 1.2 level 5, 512k chunk, algorithm 2
[10/10] [UUUUUUUUUU]
bitmap: 1/30 pages [4KB], 65536KB chunk
# blkid /dev/md3
/dev/md3: UUID="4bz7kl-2QWk-4YZX-IDmH-IFXL-eM3l-F2dB2n" TYPE="LVM2_member"
# vgchange -aay
# blkid /dev/vg/data
/dev/vg/data: UUID="60e11554-074d-4156-a4d3-ed43318f5446" TYPE="ext4"
# fsck.ext4 /dev/vg/data
e2fsck 1.42.9 (4-Feb-2014)
/dev/vg/data: clean, 464/548112384 files, 4798924497/8769786880 blocks
# mount /dev/vg/data
# echo check > /sys/block/md3/md/sync_action
# mdadm --detail /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 3899547136 (3718.90 GiB 3993.14 GB)
Raid Devices : 10
Total Devices : 10
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Dec 2 13:40:04 2015
State : active
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : ubuntu:3 (local to host ubuntu)
UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Events : 2
Number Major Minor RaidDevice State
0 8 83 0 active sync /dev/sdf3
1 8 131 1 active sync /dev/sdi3
2 8 115 2 active sync /dev/sdh3
3 8 99 3 active sync /dev/sdg3
4 8 51 4 active sync /dev/sdd3
5 8 67 5 active sync /dev/sde3
6 8 19 6 active sync /dev/sdb3
7 8 35 7 active sync /dev/sdc3
8 8 3 8 active sync /dev/sda3
9 8 147 9 active sync /dev/sdj3
# mdadm --examine /dev/sd?3
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 074ebbe7:81662e37:cc777096:3d50d481
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d47becef - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 4adc3cf2:9d813d1e:ab8044ab:0cc5afe8
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 41bc13fb - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : ddc9c73b:007f6c0f:35a03e5c:e5fc8157
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9c425653 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 73077551:029e51dc:6d036bf4:b81195a4
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 64142af5 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 1edcf3a7:9decc271:262271a6:54cff640
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9e6c2a90 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 0438b536:b5943d93:70280dd9:70952d14
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 547afaef - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdg3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : e16e0db6:5d8e9016:84d2034b:7519f24d
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2e15990 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdh3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 32bdd4dd:934c3ba5:adc8bfc2:0f1f323e
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 214f61da - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdi3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : bd0d56ba:78e5ee76:d600d587:ab65cbc9
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2032ca0e - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdj3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 35d4e323:8e2485cb:2c6492a1:e9f8e215
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 442bc637 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
^ permalink raw reply
* Re: [PATCH 0/2] Introduce the request handling for dm-crypt
From: Baolin Wang @ 2015-12-02 12:46 UTC (permalink / raw)
To: Mark Brown
Cc: Jens Axboe, Mike Snitzer, Alasdair G Kergon, dm-devel, neilb,
linux-raid, Jan Kara, Arnd Bergmann, LKML, keith.busch, jmoyer,
tj, bart.vanassche, Garg, Dinesh
In-Reply-To: <20151113115144.GR12392@sirena.org.uk>
Hi All,
These are the benchmarks for request based dm-crypt. Please check it.
一、Environment
1. Hardware configuration
Board: Beaglebone black
Processor: Am335x 1GHz ARM Cortex-A8
RAM:512M
SD card:8G
Kernel version:4.4-rc1
2. Encryption method
(1) Use cbc(aes) cipher to encrypt the block device with dmsetup tool
dmsetup create dm-0 --table “0 `blockdev --getsize /dev/mmcblk0p1`
crypt aes-cbc-plain:sha256 babebabebabebabebabebabebabebabe 0
/dev/mmcblk0p1 0”
(2) Enable the AES engine by config
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_OMAP_AES=y
(3) Limitation
We want to test it on ramdisk devices rather than slow media devices
(SD card) firstly, but here we can't use ramdisk to be mapped with
dmsetup tool. Cause ramdisk device is non-request-stackable device, it
can not be used for request-based dm.
二. Result summary
1. Results table
-----------------------------------------------------------------------------------------------
| Test | size | bio | request
| % change |
-----------------------------------------------------------------------------------------------
| dd sequential read | 1G | 5.6Mb/s | 11.3Mb/s | +101.8% |
-----------------------------------------------------------------------------------------------
| dd sequential write | 1G | 4.2Mb/s | 6.8Mb/s | +61.9% |
-----------------------------------------------------------------------------------------------
| fio sequential read | 1G | 5336KB/s | 10928KB/s | +104.8% |
-----------------------------------------------------------------------------------------------
| fio sequential write | 1G | 4049KB/s | 6574KB/s | +62.4% |
-----------------------------------------------------------------------------------------------
2. Summary
From all the test data with the dd/fio tools, the test results
basically are coincident though the different tools, so it can
basically reflect the IO performance effection by request based
opimization.
It has a larger IO performance effection for reading speed, and it'll
have a larger improvement (at least a double improvement) when
enabling the request based opimization.
Also it will have a big improvement for writing speed, and it is
increased about 50% when enabling the request based opimization. But
for random writing, it has a litle difference limited by slow hardware
random accessing.
三. DD test procedure
dd can be used for simplified copying of data at the low level with
operating the raw devices.
dd can provide good basic coverage but isn't very realistic, and only
provide sequential IO accessing.
But we can use dd to read/write the raw devices without the
filesystem's caches effection, test result as below:
1. Sequential read:
(1) Sequential read 1G with bio based:
time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct
1073741824 bytes (1.1 GB) copied, 192.091 s, 5.6 MB/s
real 3m12.112s
user 0m0.070s
sys 0m3.820s
(2) Sequential read 1G with requset based:
time dd if=/dev/dm-0 of=/dev/null bs=64K count=16384 iflag=direct
1073741824 bytes (1.1 GB) copied, 94.8922 s, 11.3 MB/s
real 1m34.908s
user 0m0.030s
sys 0m4.000s
(3) Sequential read 1G without encryption:
time dd if=/dev/mmcblk0p1 of=/dev/null bs=64K count=16384 iflag=direct
1073741824 bytes (1.1 GB) copied, 58.49 s, 18.4 MB/s
real 0m58.505s
user 0m0.040s
sys 0m3.050s
2. Sequential write:
(1) Sequential write 1G with bio based:
time dd if=/dev/zero of=/dev/dm-0 bs=64K count=16384 oflag=direct
1073741824 bytes (1.1 GB) copied, 253.477 s, 4.2 MB/s
real 4m13.497s
user 0m0.130s
sys 0m3.990s
(2) Sequential write 1G with requset based:
time dd if=/dev/zero of=/dev/dm-0 bs=64K count=16384 oflag=direct
1073741824 bytes (1.1 GB) copied, 157.396 s, 6.8 MB/s
real 2m37.414s
user 0m0.130s
sys 0m4.190s
(3) Sequential write 1G without encryption:
time dd if=/dev/zero of=/dev/mmcblk0p1 bs=64K count=16384 oflag=direct
1073741824 bytes (1.1 GB) copied, 120.452 s, 8.9 MB/s
real 2m0.471s
user 0m0.050s
sys 0m3.820s
3. Summary:
we can see the sequential read/write speed with bio based is: 5.6MB/s
and 4.2MB/s, but when encrypting the block device with request based
things, the sequential read/write speed can be increased to: 11.3MB/s
and 6.8MBs. So sequential reading and writing speed have a big
different with request based, speed are increased by 101.8% and 61.9%.
Meanwhile we also can see the difference in 'sys' time with request
based optimizations.
三、Fio test procedure
We specify the block size is 64K, and command like:
fio --filename=/dev/dm-0 --direct=1 --iodepth=1 --rw=read --bs=64K
--size=1G --group_reporting --numjobs=1 --name=test_read
1. Sequential read 1G with bio based:
READ: io=1024.0MB, aggrb=5336KB/s, minb=5336KB/s, maxb=5336KB/s,
mint=196494msec, maxt=196494msec
2. Sequential write 1G with bio based:
WRITE: io=1024.0MB, aggrb=4049KB/s, minb=4049KB/s, maxb=4049KB/s,
mint=258954msec, maxt=258954msec
3. Sequential read 1G with request based:
READ: io=1024.0MB, aggrb=10928KB/s, minb=10928KB/s, maxb=10928KB/s,
mint=95947msec, maxt=95947msec
4. Sequential write 1G with request based:
WRITE: io=1024.0MB, aggrb=6574KB/s, minb=6574KB/s, maxb=6574KB/s,
mint=159493msec, maxt=159493msec
5. Summary:
(1) read:
The sequential read speed has a big improvment with reuest based
things, which is increased by 104.8% when the reuest based things are
enabled for dm-crypt. It can not be a really random read if we specify
the block size, so the data doesn't list the random read speed
improvements, though it show big improvements.
(2) write:
The sequential write speed has some improvements with request based,
which is increased about by 62.4%. But for random write, this part is
very hard to measure on an SD card though, because any random write
smaller than the underlying block size will cause long I/O latencies
at some point, which is can not show the improvements.
四、IO block size test
We also change the block size from 4K to 1M (most IO block size in
practice are much smaller than 1M) to see the block size influences
with reuest based for dm-crypt.
1. Sequential read 1G
(1) block size = 4k
time dd if=/dev/dm-0 of=/dev/null bs=4k count=262144 iflag=direct
1073741824 bytes (1.1 GB) copied, 310.598 s, 3.5 MB/s
real 5m10.614s
user 0m0.610s
sys 0m36.040s
(2) block size = 64k
1073741824 bytes (1.1 GB) copied, 95.0489 s, 11.3 MB/s
real 1m35.071s
user 0m0.040s
sys 0m4.030s
(3) block size = 256k
1073741824 bytes (1.1 GB) copied, 84.3311 s, 12.7 MB/s
real 1m24.347s
user 0m0.050s
sys 0m1.950s
(4) block size = 1M
1073741824 bytes (1.1 GB) copied, 80.8778 s, 13.3 MB/s
real 1m20.893s
user 0m0.010s
sys 0m1.390s
2. Sequential write 1G
(1) block size = 4k
time dd if=/dev/zero of=/dev/dm-0 bs=4K count=262144 oflag=direct
1073741824 bytes (1.1 GB) copied, 629.656 s, 1.7 MB/s
real 10m29.671s
user 0m0.790s
sys 0m33.550s
(2) block size = 64k
1073741824 bytes (1.1 GB) copied, 155.697 s, 6.9 MB/s
real 2m35.713s
user 0m0.040s
sys 0m4.110s
(3) block size = 256k
1073741824 bytes (1.1 GB) copied, 143.682 s, 7.5 MB/s
real 2m23.698s
user 0m0.040s
sys 0m2.500s
(4) block size = 1M
1073741824 bytes (1.1 GB) copied, 140.654 s, 7.6 MB/s
real 2m20.670s
user 0m0.040s
sys 0m2.090s
3. Summary
For request based things, some sequential bios/requests can merged
into one request to expand the IO size to be a big block handled by
hardware engine at one time. With the hardware acceleration, it can
improve the encryption/decryption speed, so the hardware engine can
play the best performance with big block size.
From the data, we also can see the reading/writing speed can be
increased by expanding the block size, which means that it doesn't
help much for small sizes in. But when the block size is above 64K,
the speed dose not get the corresponding performance benefits, I think
the speed limitation is also in cryto that lets the bigger bios can't
get the similar performance benefits, which is need more
investigations.
On 13 November 2015 at 19:51, Mark Brown <broonie@kernel.org> wrote:
> On Thu, Nov 12, 2015 at 08:26:26AM -0700, Jens Axboe wrote:
>> On 11/12/2015 03:04 AM, Mark Brown wrote:
>
>> >Android now wants to encrypt phones and tablets by default and have been
>> >seeing substantial performance hits as a result, we can try to get
>> >people to share performance data from productionish systems but it might
>> >be difficult.
>
>> Well, shame on them for developing out-of-tree, looks like they are reaping
>> all the benefits of that.
>
>> Guys, we need some numbers, enough with the hand waving. There's no point
>> discussing this further until we know how much of a difference it makes to
>> handle X MB chunks instead of Y MB chunks. As was previously stated, unless
>> there's a _substantial_ performance benefit, this patchset isn't going
>> anywhere.
>
> Yeah, what I'm saying here is that there will issues getting the numbers
> from relevant production systems - we are most likely to be looking at
> proxies which are hopefully reasonably representative but there's likely
> to be more divergence than you'd see just running benchmark workloads on
> similar systems to those used in production.
--
Baolin.wang
Best Regards
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: Roman Mamedov @ 2015-12-02 5:22 UTC (permalink / raw)
To: Dallas Clement; +Cc: linux-raid
In-Reply-To: <CAE9DZURyZKmnUmnn5ELAhLUq8nhXD1eBx4g8hDmCzS-MYLrT+Q@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 522 bytes --]
On Tue, 1 Dec 2015 17:02:27 -0600
Dallas Clement <dallas.a.clement@gmail.com> wrote:
> I am frankly surprised at these results. Perhaps there is some
> configuration or tunable settings that have changed since the 2.6
> kernel that I am unaware of that affect RAID 5, 6 performance. Please
> comment if you have any ideas which might explain what I am seeing.
Do you use a write intent bitmap (internal?), what is your bitmap chunk size?
What is your stripe_cache_size set to?
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* [PATCH 2/3] [mdadm] recreate journal in mdadm
From: Song Liu @ 2015-12-02 1:13 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1449018805-320020-1-git-send-email-songliubraving@fb.com>
Example work flow:
./mdadm --fail /dev/md1 /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md1
./mdadm --stop /dev/md1
mdadm: stopped /dev/md1
./mdadm -A --scan --force
mdadm: Journal is missing or stale, starting array read only.
mdadm: /dev/md/1 has been started with 15 drives.
./mdadm --add-journal /dev/md1 /dev/sdb2
mdadm: Journal created on /dev/sdb2.
mdadm: Please restart /dev/md1.
./mdadm --stop /dev/md1
mdadm: stopped /dev/md1
./mdadm -A --scan
mdadm: clearing FAULTY flag for new journal in /dev/md/1 for /dev/sdb2
mdadm: Marking array /dev/md/1 as 'clean'
mdadm: /dev/md/1 has been started with 15 drives and 1 journal.
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
Assemble.c | 8 +++++++-
Manage.c | 32 +++++++++++++++++++++++++++++++-
ReadMe.c | 1 +
mdadm.c | 8 ++++++++
mdadm.h | 1 +
super1.c | 3 ++-
6 files changed, 50 insertions(+), 3 deletions(-)
diff --git a/Assemble.c b/Assemble.c
index 90c9918..6b9a6da 100644
--- a/Assemble.c
+++ b/Assemble.c
@@ -1688,7 +1688,13 @@ try_again:
if (st->ss->update_super(st, &devices[j].i, "assemble", NULL,
c->verbose, 0, NULL)) {
- if (c->force) {
+ if (devices[j].i.disk.state & (1<<MD_DISK_JOURNAL)) {
+ if (c->verbose >= 0)
+ pr_err("clearing FAULTY flag for new journal in %s for %s\n",
+ mddev, devices[j].devname);
+ change = 1;
+ content->journal_clean = 1;
+ } else if (c->force) {
if (c->verbose >= 0)
pr_err("clearing FAULTY flag for device %d in %s for %s\n",
j, mddev, devices[j].devname);
diff --git a/Manage.c b/Manage.c
index 2df303d..930865a 100644
--- a/Manage.c
+++ b/Manage.c
@@ -825,7 +825,8 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
}
/* Make sure device is large enough */
- if (tst->sb &&
+ if (dv->disposition != 'j' && /* skip size check for Journal */
+ tst->sb &&
tst->ss->avail_size(tst, ldsize/512, INVALID_SECTORS) <
array_size) {
if (dv->disposition == 'M')
@@ -929,8 +930,30 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
else
disc.number = raid_slot;
disc.state = 0;
+
+ /* only add journal to array that supports journaling */
+ if (dv->disposition == 'j') {
+ struct mdinfo mdi;
+ struct mdinfo *mdp;
+
+ mdp = sysfs_read(fd, NULL, GET_ARRAY_STATE);
+
+ if (strncmp(mdp->sysfs_array_state, "readonly", 8) != 0) {
+ pr_err("%s is not readonly, cannot add journal.\n", devname);
+ return -1;
+ }
+
+ tst->ss->getinfo_super(tst, &mdi, NULL);
+ if (mdi.journal_device_required == 0) {
+ pr_err("%s does not support journal device.\n", devname);
+ return -1;
+ }
+ }
+
if (array->not_persistent==0) {
int dfd;
+ if (dv->disposition == 'j')
+ disc.state |= 1 << MD_DISK_JOURNAL;
if (dv->writemostly == 1)
disc.state |= 1 << MD_DISK_WRITEMOSTLY;
dfd = dev_open(dv->devname, O_RDWR | O_EXCL|O_DIRECT);
@@ -1038,6 +1061,11 @@ int Manage_add(int fd, int tfd, struct mddev_dev *dv,
ping_monitor(devnm);
sysfs_free(sra);
close(container_fd);
+ } else if (dv->disposition == 'j') {
+ /* do not call ioctl(ADD_NEW_DISK) for journal */
+ pr_err("Journal created on %s.\n", dv->devname);
+ pr_err("Please restart %s.\n", devname);
+ return 1;
} else {
tst->ss->free_super(tst);
if (ioctl(fd, ADD_NEW_DISK, &disc)) {
@@ -1277,6 +1305,7 @@ int Manage_subdevs(char *devname, int fd,
* try HOT_ADD_DISK
* If that fails EINVAL, try ADD_NEW_DISK
* 'S' - add the device as a spare - don't try re-add
+ * 'j' - add the device as a journal device
* 'A' - re-add the device
* 'r' - remove the device: HOT_REMOVE_DISK
* device can be 'faulty' or 'detached' in which case all
@@ -1509,6 +1538,7 @@ int Manage_subdevs(char *devname, int fd,
goto abort;
case 'a':
case 'S': /* --add-spare */
+ case 'j': /* --add-journal */
case 'A':
case 'M': /* --re-add missing */
case 'F': /* --re-add faulty */
diff --git a/ReadMe.c b/ReadMe.c
index fb5a671..566fc8d 100644
--- a/ReadMe.c
+++ b/ReadMe.c
@@ -157,6 +157,7 @@ struct option long_options[] = {
/* Management */
{"add", 0, 0, Add},
{"add-spare", 0, 0, AddSpare},
+ {"add-journal", 0, 0, AddJournal},
{"remove", 0, 0, Remove},
{"fail", 0, 0, Fail},
{"set-faulty",0, 0, Fail},
diff --git a/mdadm.c b/mdadm.c
index f56a8cf..feec3b7 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -190,6 +190,7 @@ int main(int argc, char *argv[])
case 'a':
case Add:
case AddSpare:
+ case AddJournal:
case 'r':
case Remove:
case Replace:
@@ -925,6 +926,13 @@ int main(int argc, char *argv[])
case O(MANAGE,AddSpare): /* add drive - never re-add */
devmode = 'S';
continue;
+ case O(MANAGE,AddJournal): /* add journal */
+ if (s.journaldisks && (s.level < 4 || s.level > 6)) {
+ pr_err("--add-journal is only supported for RAID level 4/5/6.\n");
+ exit(2);
+ }
+ devmode = 'j';
+ continue;
case O(MANAGE,ReAdd):
devmode = 'A';
continue;
diff --git a/mdadm.h b/mdadm.h
index 1b027bb..c16bcd7 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -345,6 +345,7 @@ enum special_options {
ManageOpt,
Add,
AddSpare,
+ AddJournal,
Remove,
Fail,
Replace,
diff --git a/super1.c b/super1.c
index 1735c2d..893f9bf 100644
--- a/super1.c
+++ b/super1.c
@@ -1713,7 +1713,8 @@ static int write_init_super1(struct supertype *st)
if (rfd >= 0)
close(rfd);
- sb->events = 0;
+ if (!(di->disk.state & (1<<MD_DISK_JOURNAL)))
+ sb->events = 0;
refst = dup_super(st);
if (load_super1(refst, di->fd, NULL)==0) {
--
2.4.6
^ permalink raw reply related
* [PATCH 1/3] add sysfs_array_state to struct mdinfo
From: Song Liu @ 2015-12-02 1:13 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
In-Reply-To: <1449018805-320020-1-git-send-email-songliubraving@fb.com>
Add sysfs_array_state to struct mdinfo, and add GET_ARRAY_STATE to
options of sysfs_read.
Signed-off-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Shaohua Li <shli@fb.com>
---
mdadm.h | 4 ++++
sysfs.c | 7 +++++++
2 files changed, 11 insertions(+)
diff --git a/mdadm.h b/mdadm.h
index 5d5e97f..1b027bb 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -280,6 +280,9 @@ struct mdinfo {
#define DS_REMOVE 1024
#define DS_UNBLOCK 2048
int prev_state, curr_state, next_state;
+
+ /* info read from sysfs */
+ char sysfs_array_state[20];
};
struct createinfo {
@@ -563,6 +566,7 @@ enum sysfs_read_flags {
GET_SIZE = (1 << 22),
GET_STATE = (1 << 23),
GET_ERROR = (1 << 24),
+ GET_ARRAY_STATE = (1 << 25),
};
/* If fd >= 0, get the array it is open on,
diff --git a/sysfs.c b/sysfs.c
index f1fd610..2600343 100644
--- a/sysfs.c
+++ b/sysfs.c
@@ -230,6 +230,13 @@ struct mdinfo *sysfs_read(int fd, char *devnm, unsigned long options)
goto abort;
}
+ if (options & GET_ARRAY_STATE) {
+ strcpy(base, "array_state");
+ if (load_sys(fname, sra->sysfs_array_state))
+ goto abort;
+ } else
+ sra->sysfs_array_state[0] = 0;
+
if (! (options & GET_DEVS))
return sra;
--
2.4.6
^ permalink raw reply related
* [PATCH 0/3] mdadm: re-add journal to array with bad journal
From: Song Liu @ 2015-12-02 1:13 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, hch, kernel-team, Song Liu
These patches try to recreate missing/faulty journal in mdadm. Since
the array is readonly with journal missing, it is necessary to
restart the array after recreating journal.
Song Liu (3):
add sysfs_array_state to struct mdinfo
[mdadm] recreate journal in mdadm
make .spec file work
Assemble.c | 8 +++++++-
Manage.c | 32 +++++++++++++++++++++++++++++++-
ReadMe.c | 3 ++-
mdadm.c | 8 ++++++++
mdadm.h | 5 +++++
mdadm.spec | 8 ++++----
super1.c | 3 ++-
sysfs.c | 7 +++++++
udev-md-raid-assembly.rules | 2 +-
9 files changed, 67 insertions(+), 9 deletions(-)
--
2.4.6
^ permalink raw reply
* Re: RAID 5,6 sequential writing seems slower in newer kernels
From: keld @ 2015-12-02 1:07 UTC (permalink / raw)
To: Dallas Clement; +Cc: linux-raid
In-Reply-To: <CAE9DZURyZKmnUmnn5ELAhLUq8nhXD1eBx4g8hDmCzS-MYLrT+Q@mail.gmail.com>
Hi Dallas
Did you test the performance of other raid types, such as RAID1 and the various
layouts of RAID10 for the newer kernels?
Best regards
keld
On Tue, Dec 01, 2015 at 05:02:27PM -0600, Dallas Clement wrote:
> Hi,
>
> I have a NAS system with 12 spinning disks that has been running with
> the 2.6.39.4 kernel. It has a 4 core xeon processor (E31275 @
> 3.40GHz), with 8 GB of RAM. The 12 disks in my RAID array are Hitachi
> 4 TB, 7200 RPM SATA drives. The filesystem is XFS.
>
> Recently I have been evaluating RAID performance on newer kernels 3.10
> and 4.2. I have observed that with the same settings, I am seeing
> much slower RAID 5 and 6 sequential write speeds with newer kernels
> compared to what I was seeing with the 2.6.39.4 kernel. However, the
> 4.2 kernel has much better read speeds for both sequential and random
> patterns. I understand that there have been many improvements to RAID
> 5 and 6 in the 4.1 kernel. I definitely am seeing improvement with
> reads but not writes.
>
> If I observe disk and array throughput with iostat, the individual
> disk utilization and wMB/s is much lower in the newer kernels. With
> the older 2.6.39.4 kernel, disk utilization seems to stay above 80%
> with wMB/s around 74 MB/s, whereas the newer kernel disk utilization
> seems to vary between 20-70% with wMB/s around 9-38 MB/s. CPU iowait
> gets up to about 10% much of the time. These Hitachi disks are
> capable of sustaining around 170 MB/s, which is just about what I see
> when doing sequential writes to all 12 disks concurrently in a JBOD
> configuration, i.e no RAID. The iowait for 12 disks of JBOD gets up
> to about 97% - which makes the system very unresponsive.
>
> One other observation is that RAID 0 sequential write speeds in newer
> kernels are only slightly less than what I was seeing in 2.6.39.4.
>
> I am frankly surprised at these results. Perhaps there is some
> configuration or tunable settings that have changed since the 2.6
> kernel that I am unaware of that affect RAID 5, 6 performance. Please
> comment if you have any ideas which might explain what I am seeing.
>
> Thanks,
>
> Dalla
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 17/17] In generated udev rules, skip RAID members
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
Add the following
ENV{ID_FS_TYPE}=="ddf_raid_member|isw_raid_member|linux_raid_member", GOTO="end"
...
LABEL="end"
to generated udev rules.
Signed-off-by: Song Liu <songliubraving@fb.com>
---
policy.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/policy.c b/policy.c
index 177544a..7341659 100644
--- a/policy.c
+++ b/policy.c
@@ -810,7 +810,10 @@ int policy_check_path(struct mdinfo *disk, struct map_ent *array)
/* invocation of udev rule file */
char udev_template_start[] =
"# do not edit this file, it is automatically generated by mdadm\n"
-"\n";
+"ENV{ID_FS_TYPE}==\"ddf_raid_member|isw_raid_member|linux_raid_member\", "
+"GOTO=\"end\"\n";
+
+char udev_template_end[] = "LABEL=\"end\"\n";
/* find rule named rule_type and return its value */
char *find_rule(struct rule *rule, char *rule_type)
@@ -962,6 +965,11 @@ int Write_rules(char *rule_name)
if (generate_entries(fd) < 0)
goto abort;
+ if (write(fd, udev_template_end,
+ sizeof(udev_template_end) - 1)
+ != (int)sizeof(udev_template_end)-1)
+ goto abort;
+
fsync(fd);
if (rule_name) {
close(fd);
--
2.4.6
^ permalink raw reply related
* [PATCH 16/17] Clear MBR in when Kill superblock
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
When killing the superblock, also clear MBR.
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Kill.c | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/Kill.c b/Kill.c
index f2fdb85..06ced78 100644
--- a/Kill.c
+++ b/Kill.c
@@ -28,6 +28,23 @@
#include "mdadm.h"
#include "md_u.h"
#include "md_p.h"
+#include "part.h"
+
+int KillMBR(int fd)
+{
+ struct MBR *null_mbr;
+
+ null_mbr = xmalloc(sizeof(*null_mbr));
+ memset(null_mbr, 0, sizeof(*null_mbr));
+ lseek(fd, 0, 0);
+ if (write(fd, null_mbr, sizeof(*null_mbr)) != sizeof(*null_mbr)) {
+ free(null_mbr);
+ return 1;
+ }
+ free(null_mbr);
+ fsync(fd);
+ return 0;
+}
int Kill(char *dev, struct supertype *st, int force, int verbose, int noexcl)
{
@@ -76,6 +93,11 @@ int Kill(char *dev, struct supertype *st, int force, int verbose, int noexcl)
rv = 0;
}
}
+
+ if (KillMBR(fd))
+ if (verbose >=0)
+ pr_err("Cannot clear MBR on %s\n", dev);
+
close(fd);
return rv;
}
--
2.4.6
^ permalink raw reply related
* [PATCH 15/17] SCSIMODE: configuration file support for mode page settings.
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
Before a a drive is incorporated into an array we want to be sure that settings
like write-cache-disable and error-recovery-response are set consistently for
each drive in the array. Add a "SCSIMODE" configuration parameter that:
1/ identifies the disks to apply the policy to by domain name. When
domain= is specified multiple times the scsi mode page settings are
applied to the union of those domains.
2/ Modifies the mode page bytes identified by a tuple of (page, subpage,
byte). For now only the '=' operator is supported to assign the
entire byte.
_____ page
/ __ subpage
| / _ byte
| | /
v v v
x.y.z=val
Example disable the write cache on all drives in domain0 and domain1:
SCSIMODE domain=domain0 domain=domain1
# disable write cache
8.0.2=0
# enable automatic read/write re-assignment
1.0.2=0xc0
For multi-byte values specifying something like:
8.0.0=0
8.0.1=0
8.0.2=0
8.0.3=0
...will result in one write to mode-page-8. Settings to the same page
are aggregated and the values are set in specified order within a page.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Incremental.c | 9 +++
Makefile | 2 +-
config.c | 191 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
mdadm.h | 1 +
4 files changed, 201 insertions(+), 2 deletions(-)
diff --git a/Incremental.c b/Incremental.c
index 781d27d..88a945a 100644
--- a/Incremental.c
+++ b/Incremental.c
@@ -194,6 +194,15 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
policy = disk_policy(&dinfo);
have_target = policy_check_path(&dinfo, &target_array);
+ /* before taking any action on the disk check if we need to modify settings */
+ if (conf_apply_scsimode(&dinfo, policy) != 0) {
+ if (c->verbose >= 0)
+ pr_err("%s failed to apply scsi mode settings\n",
+ devname);
+ rv = 2;
+ goto out;
+ }
+
if (st == NULL && (st = guess_super_type(dfd, guess_array)) == NULL) {
if (c->verbose >= 0)
pr_err("no recognisable superblock on %s.\n",
diff --git a/Makefile b/Makefile
index 009073c..3a5f28d 100644
--- a/Makefile
+++ b/Makefile
@@ -143,7 +143,7 @@ SRCS = $(patsubst %.o,%.c,$(OBJS))
INCL = mdadm.h part.h bitmap.h
MON_OBJS = mdmon.o monitor.o managemon.o util.o maps.o mdstat.o sysfs.o \
- policy.o lib.o \
+ policy.o lib.o config.o mapfile.o \
Kill.o sg_io.o dlink.o ReadMe.o super-intel.o \
super-mbr.o super-gpt.o \
super-ddf.o sha1.o crc32.o msg.o bitmap.o xmalloc.o \
diff --git a/config.c b/config.c
index d06928b..f1ab62b 100644
--- a/config.c
+++ b/config.c
@@ -24,6 +24,7 @@
#include "mdadm.h"
#include "dlink.h"
+#include "sg_io.h"
#include <dirent.h>
#include <glob.h>
#include <fnmatch.h>
@@ -77,7 +78,8 @@ char DefaultAltConfFile[] = CONFFILE2;
char DefaultAltConfDir[] = CONFFILE2 ".d";
enum linetype { Devices, Array, Mailaddr, Mailfrom, Program, CreateDev,
- Homehost, HomeCluster, AutoMode, Policy, PartPolicy, Enclosures, LTEnd };
+ Homehost, HomeCluster, AutoMode, Policy, PartPolicy, Enclosures,
+ ScsiMode, LTEnd };
char *keywords[] = {
[Devices] = "devices",
[Array] = "array",
@@ -91,6 +93,7 @@ char *keywords[] = {
[Policy] = "policy",
[PartPolicy]="part-policy",
[Enclosures] = "enclosure",
+ [ScsiMode] = "scsimode",
[LTEnd] = NULL
};
@@ -123,6 +126,22 @@ struct conf_enclosure {
char *id;
} *edevlist = NULL;
+struct conf_scsimode {
+ struct conf_scsimode *next;
+ int strict;
+ struct mode_domain {
+ struct mode_domain *next;
+ char *name;
+ } *domains;
+ struct mode_setting {
+ struct mode_setting *next;
+ int page;
+ int subpage;
+ int byte;
+ unsigned char val;
+ } *settings;
+} *scsimodelist;
+
struct mddev_dev *load_partitions(void)
{
FILE *f = fopen("/proc/partitions", "r");
@@ -535,6 +554,99 @@ static void enclosureline(char *line)
}
}
+static void scsimodeline(char *line)
+{
+ char *w;
+ struct conf_scsimode *mode = xmalloc(sizeof(*mode));
+ struct mode_setting **s_pos = &mode->settings;
+
+ mode->domains = NULL;
+ mode->settings = NULL;
+ mode->strict = 0;
+ for (w = dl_next(line); w != line; w = dl_next(w)) {
+ if (strncasecmp(w, "domain=", 7) == 0) {
+ struct mode_domain *d = xmalloc(sizeof(*d));
+
+ d->name = xstrdup(w+7);
+ d->next = mode->domains;
+ mode->domains = d;
+ } else if (strncasecmp(w, "strict", 6) == 0) {
+ mode->strict = 1;
+ } else {
+ char *start, *end;
+ char sep[] = "..=";
+ int fields[4];
+ size_t i;
+
+ start = w;
+ for (i = 0; i < sizeof(sep); i++) {
+ fields[i] = strtoul(start, &end, 0);
+ if (end && end > start && *end == sep[i]) {
+ start = end+1;
+ } else
+ break;
+ }
+
+ if (i < sizeof(sep))
+ pr_err("unrecognized format for SCSIMODE line: %s\n", w);
+ else {
+ struct mode_setting *s = xmalloc(sizeof(*s));
+ struct mode_setting *insert;
+
+ s->page = fields[0];
+ s->subpage = fields[1];
+ s->byte = fields[2];
+ s->val = fields[3];
+
+ /* keep pages adjacent, as we will apply
+ * entire pages at a time
+ */
+ for (insert = mode->settings; insert; insert = insert->next) {
+ if (insert->next == NULL)
+ break;
+ if (insert->next->page == s->page &&
+ insert->next->subpage == s->subpage)
+ continue;
+ if (insert->page == s->page &&
+ insert->subpage == s->subpage)
+ break;
+ }
+
+ if (insert) {
+ s->next = insert->next;
+ insert->next = s;
+ } else {
+ s->next = NULL;
+ while (*s_pos)
+ s_pos = &(*s_pos)->next;
+ *s_pos = s;
+ s_pos = &s->next;
+ }
+ }
+ }
+ }
+
+ if (mode->domains && mode->settings) {
+ mode->next = scsimodelist;
+ scsimodelist = mode;
+ } else {
+ struct mode_domain *d;
+ struct mode_setting *s;
+
+ while (mode->domains) {
+ d = mode->domains->next;
+ free(mode->domains);
+ mode->domains = d;
+ }
+ while (mode->settings) {
+ s = mode->settings->next;
+ free(mode->settings);
+ mode->settings = s;
+ }
+ free(mode);
+ }
+}
+
static char *alert_email = NULL;
void mailline(char *line)
{
@@ -757,6 +869,9 @@ void conf_file(FILE *f)
case Enclosures:
enclosureline(line);
break;
+ case ScsiMode:
+ scsimodeline(line);
+ break;
case Mailaddr:
mailline(line);
break;
@@ -1259,6 +1374,80 @@ struct mddev_ident *conf_match(struct supertype *st,
return match;
}
+static int apply_mode(struct mdinfo *info, struct mode_setting *settings)
+{
+ struct mode_setting *s = settings;
+ struct scsi_mode_sense_data data;
+ __u8 mode_buf[512];
+ int errors = 0;
+
+ for (s = settings; s; s = s->next) {
+ __u8 *mode_page;
+ char devt[22];
+ int fd, rc;
+
+ sprintf(devt, "%d:%d",
+ info->disk.major,
+ info->disk.minor);
+ fd = dev_open(devt, O_RDWR);
+
+ rc = scsi_mode_sense(fd, s->page, s->subpage, &data,
+ mode_buf, sizeof(mode_buf));
+ mode_page = to_mode_page(mode_buf, &data);
+ for (;;) {
+
+ mode_page[s->byte] = s->val;
+ if (s->next &&
+ s->next->page == s->page &&
+ s->next->subpage == s->subpage)
+ s = s->next;
+ else
+ break;
+ }
+ if (rc == 0)
+ rc = scsi_mode_select(fd, 1, 1, &data, mode_buf);
+ if (rc)
+ errors++;
+ close(fd);
+ }
+
+ return errors;
+}
+
+int conf_apply_scsimode(struct mdinfo *info, struct dev_policy *pol)
+{
+ struct conf_scsimode *mode;
+ int err = 0;
+
+ load_conffile();
+
+ pol = pol_find(pol, pol_domain);
+ if (!scsimodelist || !pol)
+ return 0;
+
+ for (mode = scsimodelist; mode; mode = mode->next) {
+ struct mode_domain *domain = mode->domains;
+ struct dev_policy *p;
+ int found = 0;
+
+ for (domain = mode->domains; domain; domain = domain->next) {
+ pol_for_each(p, pol, NULL) {
+ if (strcmp(p->value, domain->name) == 0) {
+ err += apply_mode(info, mode->settings);
+ found = 1;
+ break;
+ }
+ }
+ if (found)
+ break;
+ }
+ if (mode->strict && err)
+ return err;
+ }
+
+ return 0;
+}
+
int conf_verify_devnames(struct mddev_ident *array_list)
{
struct mddev_ident *a1, *a2;
diff --git a/mdadm.h b/mdadm.h
index 4f6c733..e0b1a07 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -1376,6 +1376,7 @@ extern struct mddev_ident *conf_match(struct supertype *st,
struct mdinfo *info,
char *devname,
int verbose, int *rvp);
+extern int conf_apply_scsimode(struct mdinfo *info, struct dev_policy *pol);
extern int experimental(void);
extern void free_line(char *line);
--
2.4.6
^ permalink raw reply related
* [PATCH 14/17] scsi mode sense/select support
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
Add scsi_mode_{sense|select} support in preparation for allowing drive
settings to be specified in the configuration file.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
sg_io.c | 170 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
sg_io.h | 27 +++++++++++
2 files changed, 195 insertions(+), 2 deletions(-)
create mode 100644 sg_io.h
diff --git a/sg_io.c b/sg_io.c
index 50ad180..813cbdd 100644
--- a/sg_io.c
+++ b/sg_io.c
@@ -16,15 +16,20 @@
* this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
*/
+#include "mdadm.h"
+#include <stdio.h>
#include <string.h>
#include <scsi/scsi.h>
#include <scsi/sg.h>
#include <sys/ioctl.h>
+#include <stdlib.h>
+
+#include "sg_io.h"
int scsi_get_serial(int fd, void *buf, size_t buf_len)
{
- unsigned char inq_cmd[] = {INQUIRY, 1, 0x80, 0, buf_len, 0};
- unsigned char sense[32];
+ __u8 inq_cmd[] = {INQUIRY, 1, 0x80, 0, buf_len, 0};
+ __u8 sense[32];
struct sg_io_hdr io_hdr;
memset(&io_hdr, 0, sizeof(io_hdr));
@@ -40,3 +45,164 @@ int scsi_get_serial(int fd, void *buf, size_t buf_len)
return ioctl(fd, SG_IO, &io_hdr);
}
+
+static int process_response(const char *cmd, struct sg_io_hdr *io)
+{
+ if ((io->info & SG_INFO_OK_MASK) != SG_INFO_OK) {
+ if (io->sb_len_wr > 0) {
+ int i;
+
+ dprintf("resp: %x sense: %x asc: %x asq: %x\n",
+ io->sbp[0], io->sbp[2] & 0xf, io->sbp[12],
+ io->sbp[13]);
+ dprintf("%s sense data:\n", cmd);
+ for (i = 0; i < io->sb_len_wr; ++i) {
+ if ((i > 0) && (0 == (i % 10)))
+ printf("\n ");
+ dprintf("0x%02x ", io->sbp[i]);
+ }
+ dprintf("\n");
+ }
+ if (io->masked_status)
+ dprintf("%s SCSI status=0x%x\n", cmd, io->status);
+ if (io->host_status)
+ dprintf("%s host_status=0x%x\n", cmd, io->host_status);
+ if (io->driver_status)
+ dprintf("%s driver_status=0x%x\n", cmd, io->driver_status);
+ return -1;
+ }
+ return 0;
+}
+
+/**
+ * scsi_mode_sense - fetch mode page(s) and a description
+ *
+ * cdb len based on buffer size
+ */
+static int __scsi_mode_sense(int fd, __u8 pg, __u8 spg,
+ struct scsi_mode_sense_data *data,
+ __u8 *buf, size_t buf_len, int force6)
+{
+ struct sg_io_hdr io_hdr;
+ int rc, cdb_len;
+ __u8 sense[32];
+ __u8 cmd[10];
+
+ if (buf_len < 8)
+ return -1;
+
+ memset(data, 0, sizeof(data));
+ memset(cmd, 0, sizeof(cmd));
+ cmd[2] = pg;
+ cmd[3] = spg;
+
+ if (buf_len > 255 || !force6) {
+ cdb_len = 10;
+ cmd[0] = MODE_SENSE_10;
+ cmd[7] = buf_len >> 8;
+ cmd[8] = buf_len;
+ } else {
+ cdb_len = 6;
+ cmd[0] = MODE_SENSE;
+ cmd[4] = buf_len;
+ }
+ memset(&io_hdr, 0, sizeof(io_hdr));
+ memset(sense, 0, sizeof(sense));
+ io_hdr.interface_id = 'S';
+ io_hdr.cmdp = cmd;
+ io_hdr.cmd_len = cdb_len;
+ io_hdr.dxferp = buf;
+ io_hdr.dxfer_len = buf_len;
+ io_hdr.dxfer_direction = SG_DXFER_FROM_DEV;
+ io_hdr.sbp = sense;
+ io_hdr.mx_sb_len = sizeof(sense);
+ io_hdr.timeout = 30000;
+
+ rc = ioctl(fd, SG_IO, &io_hdr);
+ if (rc)
+ return rc;
+
+ if (cdb_len == 10) {
+ data->data_len = buf[0]*256 + buf[1] + 2;
+ data->media_type = buf[2];
+ data->device_specific = buf[3];
+ data->long_lba = buf[4] & 1;
+ data->block_len = buf[6]*256 + buf[7];
+ } else {
+ data->data_len = buf[0] + 1;
+ data->media_type = buf[1];
+ data->device_specific = buf[2];
+ data->block_len = buf[3];
+ }
+ data->hdr_len = cdb_len - 2;
+
+ if ((size_t) (to_mode_page(buf, data) - buf) > buf_len)
+ return -1;
+
+ return process_response("MODE SENSE", &io_hdr);
+}
+
+int scsi_mode_sense(int fd, __u8 pg, __u8 spg,
+ struct scsi_mode_sense_data *data,
+ __u8 *buf, size_t buf_len)
+{
+ return __scsi_mode_sense(fd, pg, spg, data, buf, buf_len, 0);
+}
+
+int scsi_mode_select(int fd, enum page_format pf, int save,
+ const struct scsi_mode_sense_data *data,
+ __u8 *param_list)
+{
+ struct sg_io_hdr io_hdr;
+ int rc, cdb_len;
+ __u8 sense[32];
+ __u8 cmd[10];
+ __u8 *page;
+
+ memset(cmd, 0, sizeof(cmd));
+ page = to_mode_page(param_list, data);
+ cmd[1] = (!!pf << 4) | ((page[0] >> 7) & !!save);
+
+ /* ps reserved */
+ page[0] &= 0x7f;
+
+ if (data->hdr_len > 4) {
+ cdb_len = 10;
+ cmd[0] = MODE_SELECT_10;
+ cmd[7] = data->data_len >> 8;
+ cmd[8] = data->data_len;
+
+ param_list[0] = 0; /* data len reserved */
+ param_list[1] = 0; /* " " " */
+ param_list[2] = data->media_type;
+ param_list[3] = 0; /* whole disk parameter */
+ param_list[6] = data->block_len >> 8;
+ param_list[7] = data->block_len;
+ } else {
+ cdb_len = 6;
+ cmd[0] = MODE_SELECT;
+ cmd[4] = data->data_len;
+
+ param_list[0] = 0; /* data len reserved */
+ param_list[1] = data->media_type;
+ param_list[2] = 0; /* whole disk parameter */
+ param_list[3] = data->block_len;
+ }
+ memset(&io_hdr, 0, sizeof(io_hdr));
+ memset(sense, 0, sizeof(sense));
+ io_hdr.interface_id = 'S';
+ io_hdr.cmdp = cmd;
+ io_hdr.cmd_len = cdb_len;
+ io_hdr.dxferp = param_list;
+ io_hdr.dxfer_len = data->data_len;
+ io_hdr.dxfer_direction = SG_DXFER_TO_DEV;
+ io_hdr.sbp = sense;
+ io_hdr.mx_sb_len = sizeof(sense);
+ io_hdr.timeout = 30000;
+
+ rc = ioctl(fd, SG_IO, &io_hdr);
+ if (rc)
+ return rc;
+
+ return process_response("MODE SELECT", &io_hdr);
+}
diff --git a/sg_io.h b/sg_io.h
new file mode 100644
index 0000000..f0988a5
--- /dev/null
+++ b/sg_io.h
@@ -0,0 +1,27 @@
+#include <asm/types.h>
+
+struct scsi_mode_sense_data {
+ int hdr_len;
+ int block_len;
+ int media_type;
+ int data_len;
+ int device_specific;
+ char long_lba;
+};
+
+enum page_format {
+ PF_VENDOR = 0,
+ PF_STANDARD = 1,
+};
+
+static inline __u8 *to_mode_page(__u8 *buf, const struct scsi_mode_sense_data *data)
+{
+ return buf + data->hdr_len + data->block_len;
+}
+
+int scsi_mode_sense(int fd, __u8 pg, __u8 spg,
+ struct scsi_mode_sense_data *data,
+ __u8 *buf, size_t buf_len);
+int scsi_mode_select(int fd, enum page_format pf, int save,
+ const struct scsi_mode_sense_data *data,
+ __u8 *page);
--
2.4.6
^ permalink raw reply related
* [PATCH 13/17] Toggle enclosure leds in response to raid events
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
The configuration file lists the domains that the system is monitoring
for events. When an array arrives all the leds for the active member
drives are set to 'normal' status. Any device that fits a domain
description but is not active (in-sync) is marked faulty.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Makefile | 5 +-
Monitor.c | 3 +
contrib/md-ses | 310 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 317 insertions(+), 1 deletion(-)
create mode 100755 contrib/md-ses
diff --git a/Makefile b/Makefile
index 3f9c7d2..009073c 100644
--- a/Makefile
+++ b/Makefile
@@ -270,7 +270,7 @@ $(MON_OBJS) : $(INCL) mdmon.h
sha1.o : sha1.c sha1.h md5.h
$(CC) $(CFLAGS) -DHAVE_STDINT_H -o sha1.o -c sha1.c
-install : mdadm mdmon install-man install-udev
+install : mdadm mdmon install-man install-udev install-contrib
$(INSTALL) -D $(STRIP) -m 755 mdadm $(DESTDIR)$(BINDIR)/mdadm
$(INSTALL) -D $(STRIP) -m 755 mdmon $(DESTDIR)$(BINDIR)/mdmon
@@ -292,6 +292,9 @@ install-man: mdadm.8 md.4 mdadm.conf.5 mdmon.8
$(INSTALL) -D -m 644 md.4 $(DESTDIR)$(MAN4DIR)/md.4
$(INSTALL) -D -m 644 mdadm.conf.5 $(DESTDIR)$(MAN5DIR)/mdadm.conf.5
+install-contrib: contrib/md-ses
+ $(INSTALL) -D -m 755 contrib/md-ses $(DESTDIR)/usr/share/mdadm/md-ses
+
udev-enclosure-slot.rules: udev-enclosure-slot.rules.in udev-workaround-sysfs-deprecated.in Makefile
@echo "$@: generate"
@/bin/echo -e "# do not edit this file, it will be overwritten on update\n" > $@
diff --git a/Monitor.c b/Monitor.c
index f19c2e5..3f3d02e 100644
--- a/Monitor.c
+++ b/Monitor.c
@@ -666,6 +666,9 @@ static int check_array(struct state *st, struct mdstat_ent *mdstat,
alert("FailSpare", dev, dv, ainfo);
else if ((newstate&change)&(1<<MD_DISK_SYNC))
alert("SpareActive", dev, dv, ainfo);
+ else if ((st->devstate[i]&change)&(1<<MD_DISK_REMOVED))
+ alert("SpareSync", dev, dv, ainfo);
+
}
st->devstate[i] = newstate;
st->devid[i] = makedev(disc.major, disc.minor);
diff --git a/contrib/md-ses b/contrib/md-ses
new file mode 100755
index 0000000..9255e96
--- /dev/null
+++ b/contrib/md-ses
@@ -0,0 +1,310 @@
+#!/usr/bin/env python
+import os
+import sys
+import stat
+from glob import glob
+from optparse import OptionParser
+
+"""
+md-ses: update ses leds in response to mdadm --monitor events
+
+1/ at startup: set the "fault" indicator for every block device in a
+ ses slot that is also in a defined domain
+2/ at failure: set the "fault" indicator for the specified device
+3/ at spare active: clear the "fault" indicator for the specified
+ device
+
+partitioned devices are ignored
+"""
+
+LOGFILE = None
+
+def log(msg):
+ if LOGFILE != None:
+ LOGFILE.write('%s: %s' % (sys.argv[0], msg))
+
+def devt_to_devpath(devt):
+ # where devt is a string of "<major:minor>"
+ path = "/sys/dev/block/%s/device" % devt
+ if not os.path.exists(path):
+ return None
+ path = os.path.realpath(path)
+ return path
+
+def devname_to_devpath(name):
+ if not name:
+ return None
+
+ st = os.stat(name)
+ if not stat.S_ISBLK(st.st_mode):
+ return None
+ major = os.major(st.st_rdev)
+ minor = os.minor(st.st_rdev)
+ # note this path will be missing for partitions
+ return devt_to_devpath('%s:%s' % (major, minor))
+
+def devpath_to_sespath(path):
+ if not path:
+ return None
+ slot = glob('%s/enclosure_device:*' % path)
+ if len(slot) != 1:
+ return None
+ return os.path.realpath(slot[0])
+
+def devname_to_sespath(name):
+ dp = devname_to_devpath(name)
+ sp = devpath_to_sespath(dp)
+ return sp
+
+def parse_config(config_name, header, field_name):
+ """
+ Take mdadm config lines like:
+ POLICY domain=domain0 slot=X
+ POLICY domain=domain0 slot=Y
+ ENCLOSURE enclosure0 id=X
+
+ ...and create a dictionary of name and field_name values
+ For example parse_config(cfg, 'POLICY', 'slot=') for the above example
+ yields:
+ { 'POLICY': [ X, Y ] }
+
+ ...for parse_config(cfg, 'ENCLOSURE', 'id=') yields:
+ { 'enclosure0': [ X ] }
+ """
+
+ def parse_field(field_name, line):
+ (key, val) = (None, None)
+ for f in line.split():
+ if f.startswith(field_name):
+ (tempkey, val) = f.split('=')
+ elif not '=' in f:
+ key = f
+ return (key, val)
+
+
+ def finish_line(line, field_name, items):
+ if line != None and field_name in line:
+ (key, val) = parse_field(field_name, line)
+ if key == None or val == None:
+ return
+ if key in items:
+ items[key].append(val)
+ else:
+ items[key] = [val]
+
+ items = {}
+ with open(config_name, 'r') as config_file:
+ line = None
+ for l in config_file:
+ l = l.rstrip()
+ if l.startswith(header):
+ finish_line(line, field_name, items)
+ line = l
+ continue
+ if not line:
+ continue
+ # continue policy lines
+ if len(l) and l[0].isspace():
+ line = line + l
+ else:
+ finish_line(line, field_name, items)
+ line = None
+ finish_line(line, field_name, items)
+ return items
+
+"""
+assumes that POLICY lines only include named enclosures or ids, not paths
+"""
+def enclosure_ids_from_domains(config_name):
+ slots = parse_config(config_name, 'POLICY', 'slot=')
+ enclosures = parse_config(config_name, 'ENCLOSURE', 'id=')
+
+ all_slots = []
+ for s in slots.values():
+ all_slots = all_slots + s
+
+ ids = []
+ for s in all_slots:
+ if s in enclosures:
+ s = enclosures[s][0]
+ ids.append(s)
+ return ids
+
+def parse_domains(config_name):
+ """
+ Return a set of all disks (by "major:minor") that the configuration file cares about
+ to be compared against the rdevs read from the active md devices
+ """
+ device_list = []
+ for enc_id in enclosure_ids_from_domains(config_name):
+ by_slot = 'enclosure-%s-slot*' % enc_id
+ for f in glob('/dev/disk/by-slot/%s' % by_slot):
+ st = os.stat(f)
+ dev = "%s:%s" % (os.major(st.st_rdev), os.minor(st.st_rdev))
+ device_list.append(dev)
+
+ return set(device_list)
+
+def parse_rdevs(check_state):
+ device_list = []
+ for md in glob('/sys/block/md*'):
+ for rd in glob('%s/md/rd*' % md):
+ with open('%s/block/dev' % rd, 'r') as _dev:
+ dev = _dev.readline().rstrip()
+ with open('%s/state' % rd , 'r') as _state:
+ state = _state.readline().rstrip()
+ if check_state == None or state == check_state:
+ device_list.append(dev)
+ return set(device_list)
+
+def update_ses(device, fault, sp=None):
+ if not sp:
+ sp = devname_to_sespath(device)
+ if not sp:
+ log('unable to determine slot for %s\n' % device)
+ return 1
+
+ log('write \'%d\' to %s\n' % (fault, sp))
+ with open('%s/fault' % sp, 'w') as f:
+ f.write('%d\n' % fault)
+
+ return 0
+
+def debug_event(event):
+ dp = devname_to_devpath(event.device)
+ sp = devpath_to_sespath(dp)
+ log('name: %s array: %s, dev: %s ses: %s\n' % (
+ event.name, event.array, event.device,
+ sp
+ )
+ )
+ device_set = parse_domains(event.config)
+ active_set = parse_rdevs('in_sync')
+ print device_set - active_set
+
+ return 0
+
+def parse_enclosures(config_name):
+ path = '/sys/class/enclosure'
+ slots = []
+ ids = enclosure_ids_from_domains(config_name)
+ for e in glob('%s/*' % path):
+ enc_id = None
+ with open('%s/id' % e) as _id:
+ enc_id = _id.readline().rstrip()
+ if not enc_id in ids:
+ continue
+ for slot in glob('%s/*' % e):
+ if os.path.exists('%s/slot' % slot):
+ slots.append(os.path.realpath(slot))
+ return set(slots)
+
+def ignore_event(event):
+ return 0
+
+def handle_active(event):
+ return update_ses(event.device, 0)
+
+def handle_fail(event):
+ return update_ses(event.device, 1)
+
+def handle_new(event):
+ device_set = parse_domains(event.config)
+ active_devs = parse_rdevs('in_sync')
+
+ active_slots = []
+ for devt in device_set & active_devs:
+ dp = devt_to_devpath(devt)
+ sp = devpath_to_sespath(dp)
+ active_slots.append(sp)
+ update_ses(None, 0, sp)
+
+ for sp in parse_enclosures(event.config) - set(active_slots):
+ update_ses(None, 1, sp)
+
+ return 0
+
+def handle_missing_hdd(event):
+ for sp in parse_enclosures(event.config):
+ if not os.path.exists(sp + '/device'):
+ update_ses(None, 1, sp)
+ return 0
+
+class MDEvent:
+ event_fn = {
+ 'DeviceDisappeared': ignore_event,
+ 'HDDDisappeared': handle_missing_hdd,
+ 'RebuildStarted': ignore_event,
+ 'RebuildNN': ignore_event,
+ 'RebuildFinished': ignore_event,
+ 'Fail': handle_fail,
+ 'FailSpare': handle_fail,
+ 'SpareActive': handle_active,
+ 'SpareSync': handle_active,
+ 'NewArray': handle_new,
+ 'DegradedArray': ignore_event,
+ 'MoveSpare': ignore_event,
+ 'SparesMissing': ignore_event,
+ 'TestMessage': ignore_event,
+ }
+ def __init__(self, event, array, device, conf):
+ self.name = event
+ self.array = array
+ self.device = device
+ self.config = conf
+ def handle(self):
+ try:
+ handler = MDEvent.event_fn[self.name]
+ if handler == ignore_event:
+ log('ignoring %s for array %s (dev: %s)\n' % (self.name, self.array, self.device))
+ else:
+ log('handling %s for array %s (dev: %s)\n' % (self.name, self.array, self.device))
+ return handler(self)
+ except KeyError:
+ log('unsupported event \'%s\'\n' % self.name)
+ return 1
+
+CONF1 = '/etc/mdadm.conf'
+CONF2 = '/etc/mdadm/mdadm.conf'
+
+def main():
+ parser = OptionParser()
+ parser.add_option("-c", "--config", type="string", dest="config",
+ help="override configuration file location", default='<default>')
+ parser.add_option("-l", "--log", type="string", dest="log",
+ help="log actions to <file>", default=None)
+ (options, args) = parser.parse_args()
+
+ if len(args) <= 1:
+ sys.stderr.write('usage: %s <md event name> <array> [device]\n' % sys.argv[0])
+ return 1
+
+ if len(args) > 1:
+ event = args[0]
+ array = args[1]
+ device = None
+
+ if len(args) > 2:
+ device = args[2]
+
+ if options.config == '<default>':
+ if os.path.exists(CONF1):
+ options.config = CONF1
+ elif os.path.exists(CONF2):
+ options.config = CONF2
+
+ global LOGFILE
+ if LOGFILE != None:
+ LOGFILE = open(LOGFILE, 'a')
+ elif options.log == 'stderr':
+ LOGFILE = sys.stderr
+ elif options.log == 'stdout':
+ LOGFILE = sys.stdout
+ elif options.log != None:
+ LOGFILE = open(options.log, 'a')
+
+ md = MDEvent(event, array, device, options.config)
+ return md.handle()
+
+if __name__ == "__main__":
+ sys.exit(main())
--
2.4.6
^ permalink raw reply related
* [PATCH 12/17] invoke hot-add policy on "change" events
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
For enclosure managed disks the links identifying the enclosure topology
may not be available until after a change event. Modify the dynamic
udev rules to run on change events emitted on enclosure attach.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
policy.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/policy.c b/policy.c
index 0558351..177544a 100644
--- a/policy.c
+++ b/policy.c
@@ -841,12 +841,12 @@ static struct rule *find_path_rule(struct rule *rule)
}
#define UDEV_RULE_FORMAT \
-"ACTION==\"add\", SUBSYSTEM==\"block\", " \
+"ACTION==\"add|change\", SUBSYSTEM==\"block\", " \
"ENV{DEVTYPE}==\"%s\", ENV{%s}==\"%s\", " \
"RUN+=\"" BINDIR "/mdadm --incremental $env{DEVNAME}\"\n"
#define UDEV_RULE_FORMAT_NOTYPE \
-"ACTION==\"add\", SUBSYSTEM==\"block\", " \
+"ACTION==\"add|change\", SUBSYSTEM==\"block\", " \
"ENV{%s}==\"%s\", " \
"RUN+=\"" BINDIR "/mdadm --incremental $env{DEVNAME}\"\n"
--
2.4.6
^ permalink raw reply related
* [PATCH 11/17] policy: enable enclosure names for domain definitions
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
The argument to "slot=" in a POLICY line can either be a path glob
relative to /dev/disk/by-slot, or an enclosure name.
In other words, now an enclosure name can be used as a domain boundary:
mdadm --detail-platform --enclosure --brief >> /etc/mdadm.conf
cat /etc/mdadm.conf
ENCLOSURE enclosure0 id=0x5f80f41f1d6a60ff
ENCLOSURE enclosure1 id=0x5f80f41f1d6e80ff
echo "POLICY domain=domain0 slot=enclosure0" >> /etc/mdadm.conf
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
config.c | 26 ++++++++++++++++++++++++++
mdadm.h | 1 +
policy.c | 18 +++++++++++++++---
3 files changed, 42 insertions(+), 3 deletions(-)
diff --git a/config.c b/config.c
index 6ea5dd1..d06928b 100644
--- a/config.c
+++ b/config.c
@@ -1045,6 +1045,32 @@ void conf_put_enclosure_devs(struct mddev_dev *devs)
}
}
+/* this succeeds whether the enclosure is currently attached or not, just
+ * translates enclosure names to ids from the configuration file
+ */
+int conf_get_enclosure_glob(const char *slot_val, char *buf, int len)
+{
+ struct conf_enclosure *e;
+
+ load_conffile();
+ for (e = edevlist; e; e = e->next) {
+ /* slot=<enclosure_name> */
+ if (strcmp(e->name, slot_val) == 0 &&
+ snprintf(buf, len, "enclosure-%s-slot*", e->id) < len)
+ return 1;
+
+ /* slot=enclosure-<enclosure_name>[-<slot-glob>] */
+ if (snprintf(buf, len, "enclosure-%s", e->name) < len &&
+ strncmp(slot_val, buf, strlen(buf)) == 0 &&
+ snprintf(buf, len, "enclosure-%s%s", e->id,
+ &slot_val[strlen(buf)]) < len)
+ return 1;
+ }
+
+ /* slot= is already a glob */
+ return 0;
+}
+
int conf_test_dev(char *devname)
{
struct conf_dev *cd;
diff --git a/mdadm.h b/mdadm.h
index f318f1d..4f6c733 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -1353,6 +1353,7 @@ extern struct mddev_ident *conf_get_ident(char *dev);
extern struct mddev_dev *conf_get_devs(void);
extern struct mddev_dev *conf_get_enclosure_devs(char *id);
extern void conf_put_enclosure_devs(struct mddev_dev *devs);
+extern int conf_get_enclosure_glob(const char *slot_val, char *buf, int len);
extern int conf_test_dev(char *devname);
extern int conf_test_metadata(const char *version, struct dev_policy *pol, int is_homehost);
extern struct createinfo *conf_get_create_info(void);
diff --git a/policy.c b/policy.c
index 6afafad..0558351 100644
--- a/policy.c
+++ b/policy.c
@@ -274,9 +274,17 @@ static int pol_match(struct rule *rule, char *path, char *type)
while (rule) {
if (rule_is_path(rule)) {
+ char *value = rule->value;
+ char enc_desc[256];
+
+ if (rule->name == rule_slot &&
+ conf_get_enclosure_glob(value, enc_desc,
+ sizeof(enc_desc)))
+ value = enc_desc;
+
if (pathok == 0)
pathok = -1;
- if (path && fnmatch(rule->value, path, 0) == 0)
+ if (path && fnmatch(value, path, 0) == 0)
pathok = 1;
}
if (rule->name == rule_type) {
@@ -848,6 +856,7 @@ static struct rule *find_path_rule(struct rule *rule)
int write_rule(struct rule *rule, int fd, int force_part)
{
char line[1024];
+ char enc_desc[256];
struct rule *path_rule = find_path_rule(rule);
char *typ = find_rule(rule, rule_type);
char *slot_path, *pth;
@@ -856,9 +865,12 @@ int write_rule(struct rule *rule, int fd, int force_part)
return -1;
pth = path_rule->value;
- if (path_rule->name == rule_slot)
+ if (path_rule->name == rule_slot) {
+ if (conf_get_enclosure_glob(path_rule->value, enc_desc,
+ sizeof(enc_desc)))
+ pth = enc_desc;
slot_path = "ID_SLOT";
- else
+ } else
slot_path = "ID_PATH";
if (force_part)
--
2.4.6
^ permalink raw reply related
* [PATCH 10/17] config: ENCLOSURE keyword and enclosure device list expansion
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
Parse the output of "--detail-platform --brief --enclosure" to generate
a mapping of of enclosure names to enclosure ids.
When a single sub-device name is specified try to interpret it as an
enclosure id/name and grab the device list from the enclosure to scsi
device mapping.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
config.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
enclosure.c | 27 +++++++++++++++++++++++++
mdadm.c | 53 +++++++++++++++++++++++++++++++------------------
mdadm.h | 2 ++
4 files changed, 128 insertions(+), 20 deletions(-)
diff --git a/config.c b/config.c
index c58c8fe..6ea5dd1 100644
--- a/config.c
+++ b/config.c
@@ -77,7 +77,7 @@ char DefaultAltConfFile[] = CONFFILE2;
char DefaultAltConfDir[] = CONFFILE2 ".d";
enum linetype { Devices, Array, Mailaddr, Mailfrom, Program, CreateDev,
- Homehost, HomeCluster, AutoMode, Policy, PartPolicy, LTEnd };
+ Homehost, HomeCluster, AutoMode, Policy, PartPolicy, Enclosures, LTEnd };
char *keywords[] = {
[Devices] = "devices",
[Array] = "array",
@@ -90,6 +90,7 @@ char *keywords[] = {
[AutoMode] = "auto",
[Policy] = "policy",
[PartPolicy]="part-policy",
+ [Enclosures] = "enclosure",
[LTEnd] = NULL
};
@@ -116,6 +117,12 @@ struct conf_dev {
char *name;
} *cdevlist = NULL;
+struct conf_enclosure {
+ struct conf_enclosure *next;
+ char *name;
+ char *id;
+} *edevlist = NULL;
+
struct mddev_dev *load_partitions(void)
{
FILE *f = fopen("/proc/partitions", "r");
@@ -506,6 +513,28 @@ void arrayline(char *line)
}
}
+static void enclosureline(char *line)
+{
+ char *name = NULL, *id = NULL;
+ char *w;
+
+ for (w = dl_next(line); w != line; w = dl_next(w)) {
+ if (strchr(w, '=') == NULL)
+ name = w;
+ if (strncasecmp(w, "id=", 3) == 0)
+ id = w+3;
+ }
+
+ if (name && id) {
+ struct conf_enclosure *e = xmalloc(sizeof(*e));
+
+ e->name = xstrdup(name);
+ e->id = xstrdup(id);
+ e->next = edevlist;
+ edevlist = e;
+ }
+}
+
static char *alert_email = NULL;
void mailline(char *line)
{
@@ -725,6 +754,9 @@ void conf_file(FILE *f)
case Array:
arrayline(line);
break;
+ case Enclosures:
+ enclosureline(line);
+ break;
case Mailaddr:
mailline(line);
break;
@@ -981,6 +1013,38 @@ struct mddev_dev *conf_get_devs()
return dlist;
}
+extern struct mddev_dev *enclosure_get_devs(char *id);
+
+struct mddev_dev *conf_get_enclosure_devs(char *enclosure_id)
+{
+ struct conf_enclosure *e;
+ struct mddev_dev *dv;
+
+ load_conffile();
+
+ for (e = edevlist; e; e = e->next)
+ if (strcmp(e->name, enclosure_id) == 0)
+ break;
+
+ dv = enclosure_get_devs(e ? e->id : enclosure_id);
+ if (!e && !dv && strncmp("encl", enclosure_id, 4) == 0)
+ pr_err("No enclosure name '%s' defined in configuration file\n",
+ enclosure_id);
+ return dv;
+}
+
+void conf_put_enclosure_devs(struct mddev_dev *devs)
+{
+ while (devs) {
+ struct mddev_dev *next;
+
+ next = devs->next;
+ free(devs->devname);
+ free(devs);
+ devs = next;
+ }
+}
+
int conf_test_dev(char *devname)
{
struct conf_dev *cd;
diff --git a/enclosure.c b/enclosure.c
index bb5e88e..9c569ea 100644
--- a/enclosure.c
+++ b/enclosure.c
@@ -242,6 +242,33 @@ static struct enclosure *parse_enclosures(void)
return enc_list;
}
+/* return a mddev_dev for each populated slot in an enclosure */
+struct mddev_dev *enclosure_get_devs(char *id)
+{
+ struct enclosure *enclosure = parse_enclosures(), *e;
+ struct mddev_dev *dlist = NULL, **pos = &dlist;
+ struct slot *s;
+
+ for (e = enclosure; e; e = e->next)
+ if (strcmp(e->id, id) == 0)
+ break;
+
+ for (s = e ? e->slot : NULL; s; s = s->next) {
+ struct mddev_dev *d;
+
+ if (!s->devname)
+ continue;
+ d = xcalloc(1, sizeof(*d));
+ xasprintf(&d->devname, "/dev/%s", s->devname);
+ *pos = d;
+ pos = &d->next;
+ }
+ free_enclosure(enclosure);
+
+ return dlist;
+
+}
+
static int detail_platform_enclosure(int verbose, int enumerate, char *enclosure_name)
{
struct enclosure *enclosure = parse_enclosures();
diff --git a/mdadm.c b/mdadm.c
index 7eeeb4f..066fb35 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -298,16 +298,23 @@ int main(int argc, char *argv[])
/* If first option is a device, don't force the mode yet */
if (opt == 1) {
if (devs_found == 0) {
- dv = xmalloc(sizeof(*dv));
- dv->devname = optarg;
- dv->disposition = devmode;
- dv->writemostly = writemostly;
- dv->used = 0;
- dv->next = NULL;
- *devlistend = dv;
- devlistend = &dv->next;
-
- devs_found++;
+ dv = conf_get_enclosure_devs(optarg);
+ if (!dv) {
+ dv = xmalloc(sizeof(*dv));
+ dv->devname = optarg;
+ dv->next = NULL;
+ }
+
+ for (; dv; dv = dv->next) {
+ dv->disposition = devmode;
+ dv->writemostly = writemostly;
+ dv->used = 0;
+ *devlistend = dv;
+ devlistend = &dv->next;
+
+ devs_found++;
+ }
+
continue;
}
/* No mode yet, and this is the second device ... */
@@ -354,16 +361,24 @@ int main(int argc, char *argv[])
pr_err("Must give -a/--add for devices to add: %s\n", optarg);
exit(2);
}
- dv = xmalloc(sizeof(*dv));
- dv->devname = optarg;
- dv->disposition = devmode;
- dv->writemostly = writemostly;
- dv->used = 0;
- dv->next = NULL;
- *devlistend = dv;
- devlistend = &dv->next;
- devs_found++;
+ dv = conf_get_enclosure_devs(optarg);
+ if (!dv) {
+ dv = xmalloc(sizeof(*dv));
+ dv->devname = optarg;
+ dv->next = NULL;
+ }
+
+ for (; dv; dv = dv->next) {
+ dv->disposition = devmode;
+ dv->writemostly = writemostly;
+ dv->used = 0;
+ *devlistend = dv;
+ devlistend = &dv->next;
+
+ devs_found++;
+ }
+
continue;
}
diff --git a/mdadm.h b/mdadm.h
index 7ac0b95..f318f1d 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -1351,6 +1351,8 @@ extern void enable_fds(int devices);
extern int parse_auto(char *str, char *msg, int config);
extern struct mddev_ident *conf_get_ident(char *dev);
extern struct mddev_dev *conf_get_devs(void);
+extern struct mddev_dev *conf_get_enclosure_devs(char *id);
+extern void conf_put_enclosure_devs(struct mddev_dev *devs);
extern int conf_test_dev(char *devname);
extern int conf_test_metadata(const char *version, struct dev_policy *pol, int is_homehost);
extern struct createinfo *conf_get_create_info(void);
--
2.4.6
^ permalink raw reply related
* [PATCH 09/17] Add --brief support to --enclosure
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
Generate simple names to locally identify enclosures by id. This is a
step towards defining enclosure names in the configuration file.
Ex:
ENCLOSURE enclosure0 id=0x5f80f41f1d6a60ff
ENCLOSURE enclosure1 id=0x5f80f41f1d6e80ff
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Detail.c | 6 +++++-
enclosure.c | 14 ++++++++++++++
mdadm.c | 5 ++++-
mdadm.h | 3 ++-
super-intel.c | 14 +++++++++++++-
5 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/Detail.c b/Detail.c
index 8fb61cc..03764ad 100644
--- a/Detail.c
+++ b/Detail.c
@@ -717,7 +717,7 @@ out:
return rv;
}
-int Detail_Platform(const struct platform_ops *platform, int scan, int verbose, int export, char *hwdevice)
+int Detail_Platform(const struct platform_ops *platform, int scan, int verbose, int export, int brief, char *hwdevice)
{
/* display platform capabilities
* 'scan' in this context means iterate over all platform types
@@ -731,6 +731,8 @@ int Detail_Platform(const struct platform_ops *platform, int scan, int verbose,
if (platform && export)
err = platform->export_detail(verbose, hwdevice);
+ else if (platform && brief)
+ err = platform->brief_detail(verbose, hwdevice);
else if (platform)
err = platform->detail(verbose, 0, hwdevice);
else if (!scan) {
@@ -752,6 +754,8 @@ int Detail_Platform(const struct platform_ops *platform, int scan, int verbose,
if (export)
err |= (*p)->export_detail(verbose, hwdevice);
+ else if (brief)
+ err |= (*p)->brief_detail(verbose, hwdevice);
else
err |= (*p)->detail(verbose, 0, hwdevice);
diff --git a/enclosure.c b/enclosure.c
index 76ed30d..bb5e88e 100644
--- a/enclosure.c
+++ b/enclosure.c
@@ -307,8 +307,22 @@ static int export_detail_enclosure(int verbose, char *enclosure_name)
return 0;
}
+static int brief_detail_enclosure(int verbose, char *enclosure_name)
+{
+ struct enclosure *e, *enclosure = parse_enclosures();
+ int i = 0;
+
+ for (e = enclosure; e; e = e->next)
+ printf("ENCLOSURE enclosure%d id=%s\n", i++, e->id);
+
+ free_enclosure(enclosure);
+
+ return 0;
+}
+
const struct platform_ops enclosure_platform = {
.detail = detail_platform_enclosure,
.export_detail = export_detail_enclosure,
+ .brief_detail = brief_detail_enclosure,
.name = "enclosure",
};
diff --git a/mdadm.c b/mdadm.c
index 0e46177..7eeeb4f 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -1511,7 +1511,8 @@ int main(int argc, char *argv[])
if (scan) {
rv = Detail_Platform(NULL, scan, c.verbose,
- c.export, controller);
+ c.export, c.brief,
+ controller);
} else {
/* metadata format specific details that
* may be tied to a given controller
@@ -1520,6 +1521,7 @@ int main(int argc, char *argv[])
platform = ss->ss->platform;
rv = Detail_Platform(platform, scan,
c.verbose, c.export,
+ c.brief,
controller);
}
/* detail given enclosure or scan all
@@ -1529,6 +1531,7 @@ int main(int argc, char *argv[])
platform = &enclosure_platform;
rv |= Detail_Platform(platform, scan,
c.verbose, c.export,
+ c.brief,
c.enclosure);
}
}
diff --git a/mdadm.h b/mdadm.h
index 881c0fd..7ac0b95 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -699,6 +699,7 @@ struct reshape {
extern const struct platform_ops {
int (*detail)(int verbose, int enumerate_only, char *hwdevice);
int (*export_detail)(int verbose, char *hwdevice);
+ int (*brief_detail)(int verbose, char *hwdevice);
char *name;
} *platform_list[];
@@ -1281,7 +1282,7 @@ extern int Create(struct supertype *st, char *mddev,
extern int Detail(char *dev, struct context *c);
extern int Detail_Platform(const struct platform_ops *platform, int scan, int verbose,
- int export, char *controller_path);
+ int export, int brief, char *controller_path);
extern int Query(char *dev);
extern int ExamineBadblocks(char *devname, int brief, struct supertype *forcest);
extern int Examine(struct mddev_dev *devlist, struct context *c,
diff --git a/super-intel.c b/super-intel.c
index 4ce46ef..c4a8ac7 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -1985,7 +1985,7 @@ static int detail_platform_imsm(int verbose, int enumerate_only, char *controlle
return result;
}
-static int export_detail_platform_imsm(int verbose, char *controller_path)
+static int export_detail_print(int verbose, int print, char *controller_path)
{
struct sys_dev *list, *hba;
int result=1;
@@ -2015,6 +2015,11 @@ static int export_detail_platform_imsm(int verbose, char *controller_path)
return result;
}
+static int export_detail_platform_imsm(int verbose, char *controller_path)
+{
+ return export_detail_print(0, 1, controller_path);
+}
+
#endif
static int match_home_imsm(struct supertype *st, char *homehost)
@@ -10604,9 +10609,16 @@ abort:
return ret_val;
}
+
+static int brief_detail_platform_imsm(int verbose, char *controller)
+{
+ return export_detail_print(verbose, 0, controller);
+}
+
const struct platform_ops imsm_platform = {
.detail = detail_platform_imsm,
.export_detail = export_detail_platform_imsm,
+ .brief_detail = brief_detail_platform_imsm,
.name = "imsm",
};
#endif /* MDASSEMBLE */
--
2.4.6
^ permalink raw reply related
* [PATCH 08/17] enclosure detection/enumeration
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
enumerate enclosures via --detail-platform --enclosure
Add the plumbing for the new command line parameter to
--detail-platform, and implement simple enumeration of an enclosure via
the attributes exported by ses and the /dev/disk/by-slot infrastructure.
Ex:
Enclosure0:
Id : 0x5f80f41f1d6a60ff
Vendor : facebook
Model : Knox2U
Slots : 15
Slot00 : "OK" /dev/sdk YHJ9RP2D
Slot01 : "OK" /dev/sdj YHJ6ZWJD
Slot02 : "OK" /dev/sdi YHJ6BTLD
Slot03 : "OK" /dev/sdg YHJ9WG2D
Slot04 : "OK" /dev/sdf YHJ6WYGD
Slot05 : "OK" /dev/sdl YHJ9S47D
Slot06 : "OK" /dev/sdh YHJ9PZED
Slot07 : "OK" /dev/sde YHJ7LW8D
Slot08 : "OK" /dev/sdd YHJ7SSDD
Slot09 : "OK" /dev/sdc YHJ7SRYD
Slot10 : "OK" /dev/sdaa CVCV153103Z5120BGN
Slot11 : "OK" /dev/sdx CVCV213103AK120BGN
Slot12 : "OK" /dev/sdy CVCV203506EU120BGN
Slot13 : "OK" /dev/sdz CVCV203602X2120BGN
Slot14 : "OK" /dev/sdb YHJ9S3PD
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Detail.c | 12 +--
Makefile | 4 +-
ReadMe.c | 2 +
enclosure.c | 314 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
mdadm.c | 42 ++++++--
mdadm.h | 5 +
super-intel.c | 2 -
sysfs.c | 11 +-
util.c | 1 +
9 files changed, 374 insertions(+), 19 deletions(-)
create mode 100644 enclosure.c
diff --git a/Detail.c b/Detail.c
index f71afb9..8fb61cc 100644
--- a/Detail.c
+++ b/Detail.c
@@ -717,7 +717,7 @@ out:
return rv;
}
-int Detail_Platform(const struct platform_ops *platform, int scan, int verbose, int export, char *controller_path)
+int Detail_Platform(const struct platform_ops *platform, int scan, int verbose, int export, char *hwdevice)
{
/* display platform capabilities
* 'scan' in this context means iterate over all platform types
@@ -730,9 +730,9 @@ int Detail_Platform(const struct platform_ops *platform, int scan, int verbose,
platform->name);
if (platform && export)
- err = platform->export_detail(verbose, controller_path);
+ err = platform->export_detail(verbose, hwdevice);
else if (platform)
- err = platform->detail(verbose, 0, controller_path);
+ err = platform->detail(verbose, 0, hwdevice);
else if (!scan) {
if (verbose > 0)
pr_err("no platform components found\n");
@@ -741,7 +741,7 @@ int Detail_Platform(const struct platform_ops *platform, int scan, int verbose,
if (!scan)
return err;
- for (p = platform_list; p; p++) {
+ for (p = platform_list; *p; p++) {
/* enumerated above */
if (*p == platform)
continue;
@@ -751,9 +751,9 @@ int Detail_Platform(const struct platform_ops *platform, int scan, int verbose,
(*p)->name);
if (export)
- err |= (*p)->export_detail(verbose, controller_path);
+ err |= (*p)->export_detail(verbose, hwdevice);
else
- err |= (*p)->detail(verbose, 0, controller_path);
+ err |= (*p)->detail(verbose, 0, hwdevice);
}
diff --git a/Makefile b/Makefile
index 2963884..3f9c7d2 100644
--- a/Makefile
+++ b/Makefile
@@ -134,7 +134,7 @@ OBJS = mdadm.o config.o policy.o mdstat.o ReadMe.o util.o maps.o lib.o \
mdopen.o super0.o super1.o super-ddf.o super-intel.o bitmap.o \
super-mbr.o super-gpt.o \
restripe.o sysfs.o sha1.o mapfile.o crc32.o sg_io.o msg.o xmalloc.o \
- platform-intel.o probe_roms.o crc32c.o
+ platform-intel.o probe_roms.o crc32c.o enclosure.o
CHECK_OBJS = restripe.o sysfs.o maps.o lib.o xmalloc.o dlink.o
@@ -147,7 +147,7 @@ MON_OBJS = mdmon.o monitor.o managemon.o util.o maps.o mdstat.o sysfs.o \
Kill.o sg_io.o dlink.o ReadMe.o super-intel.o \
super-mbr.o super-gpt.o \
super-ddf.o sha1.o crc32.o msg.o bitmap.o xmalloc.o \
- platform-intel.o probe_roms.o
+ platform-intel.o probe_roms.o enclosure.o
MON_SRCS = $(patsubst %.o,%.c,$(MON_OBJS))
diff --git a/ReadMe.c b/ReadMe.c
index fb5a671..8989e5c 100644
--- a/ReadMe.c
+++ b/ReadMe.c
@@ -97,6 +97,7 @@ struct option long_options[] = {
{"examine-bitmap", 0, 0, 'X'},
{"auto-detect", 0, 0, AutoDetect},
{"detail-platform", 0, 0, DetailPlatform},
+ {"enclosure", 2, 0, Enclosure},
{"kill-subarray", 1, 0, KillSubarray},
{"update-subarray", 1, 0, UpdateSubarray},
{"udev-rules", 2, 0, UdevRules},
@@ -502,6 +503,7 @@ char Help_misc[] =
" device relates to the md driver\n"
" --detail -D : Display details of an array\n"
" --detail-platform : Display hardware/firmware details\n"
+" --enclosure : Display disks by physical/bay location\n"
" --examine -E : Examine superblock on an array component\n"
" --examine-bitmap -X: Display contents of a bitmap file\n"
" --examine-badblocks: Display list of known bad blocks on device\n"
diff --git a/enclosure.c b/enclosure.c
new file mode 100644
index 0000000..76ed30d
--- /dev/null
+++ b/enclosure.c
@@ -0,0 +1,314 @@
+/*
+ * mdadm - Enclosure management support
+ *
+ * Copyright (C) 2013 Facebook
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc.,
+ * 51 Franklin St - Fifth Floor, Boston, MA 02110-1301 USA.
+ */
+#include "mdadm.h"
+
+#include <ctype.h>
+#include <sys/types.h>
+#include <dirent.h>
+
+
+#define ENCLOSURE_STRING 40
+struct slot {
+ struct slot *next;
+ char *devname;
+ int slot_num;
+};
+
+const char ENC_BASE[] = "/sys/class/enclosure";
+
+struct enclosure {
+ char vendor[ENCLOSURE_STRING];
+ char model[ENCLOSURE_STRING];
+ char id[ENCLOSURE_STRING];
+ struct slot *slot;
+ char *devname;
+ int num_slots;
+ struct enclosure *next;
+};
+
+static void free_slot(struct slot *slot)
+{
+ while (slot) {
+ struct slot *next = slot->next;
+
+ free(slot->devname);
+ free(slot);
+ slot = next;
+ }
+}
+
+static void free_enclosure(struct enclosure *enclosure)
+{
+ while (enclosure) {
+ struct enclosure *next = enclosure->next;
+
+ free_slot(enclosure->slot);
+ free(enclosure->devname);
+ free(enclosure);
+ enclosure = next;
+ }
+}
+
+static char *devpath_to_diskname_deprecated(const char *devpath)
+{
+ struct dirent *de = NULL;
+ char *r_devpath;
+ DIR *b_dir;
+ char *diskname = NULL;
+
+ r_devpath = realpath(devpath, NULL);
+ if (!r_devpath)
+ return NULL;
+
+ b_dir = opendir("/sys/block");
+ if (!b_dir)
+ goto out;
+
+ for (de = readdir(b_dir); de; de = readdir(b_dir)) {
+ char diskpath[100];
+ char *r_diskpath;
+ int done;
+
+ if (de->d_name[0] == '.')
+ continue;
+ sprintf(diskpath, "/sys/block/%s/device", de->d_name);
+ r_diskpath = realpath(diskpath, NULL);
+ if (!r_diskpath)
+ continue;
+
+ done = strcmp(r_devpath, r_diskpath) == 0;
+ free(r_diskpath);
+ if (done) {
+ diskname = xstrdup(de->d_name);
+ break;
+ }
+ }
+ closedir(b_dir);
+ out:
+ free(r_devpath);
+ return de ? diskname : NULL;
+}
+
+static char *device_to_diskname(const char *devpath)
+{
+ struct dirent *de = NULL;
+ struct stat st;
+ char *b_path;
+ DIR *b_dir;
+
+ if (stat(devpath, &st))
+ return NULL; /* no device */
+
+ xasprintf(&b_path, "%s/block", devpath);
+ b_dir = opendir(b_path);
+
+ /* on sysfs-deprecated hosts we need to do this the long way... */
+ if (!b_dir)
+ goto out;
+
+ for (de = readdir(b_dir); de; de = readdir(b_dir)) {
+ if (de->d_name[0] == '.')
+ continue;
+ if (de->d_type != DT_DIR)
+ continue;
+ /* 1:1 mapping of scsi device to block device */
+ break;
+ }
+ closedir(b_dir);
+ out:
+ free(b_path);
+
+ return de ? xstrdup(de->d_name) : devpath_to_diskname_deprecated(devpath);
+}
+
+static struct slot *parse_slots(struct enclosure *enclosure, DIR *slot_dir)
+{
+ struct slot *slot_list = NULL, **pos = &slot_list;
+ char path[PATH_MAX];
+ struct dirent *de;
+
+ if (!slot_dir)
+ return NULL;
+
+ for (de = readdir(slot_dir); de; de = readdir(slot_dir)) {
+ char slot_id[ENCLOSURE_STRING];
+ struct slot *slot;
+ char *e;
+
+ if (de->d_name[0] == '.')
+ continue;
+ if (de->d_type != DT_DIR)
+ continue;
+
+ snprintf(path, sizeof(path), "%s/%s/%s/slot",
+ ENC_BASE, enclosure->devname, de->d_name);
+ if (load_sys_n(path, slot_id, sizeof(slot_id)) != 0)
+ continue;
+
+ slot = xcalloc(1, sizeof(*slot));
+ *pos = slot;
+ pos = &slot->next;
+ slot->slot_num = strtoul(slot_id, &e, 10);
+
+ snprintf(path, sizeof(path), "%s/%s/%s/device",
+ ENC_BASE, enclosure->devname, de->d_name);
+ slot->devname = device_to_diskname(path);
+ }
+
+ if (de) {
+ free_slot(slot_list);
+ slot_list = NULL;
+ }
+
+ return slot_list;
+}
+
+static struct enclosure *parse_enclosures(void)
+{
+ struct enclosure *enc_list = NULL, **pos = &enc_list;
+ struct dirent *de;
+ DIR *enc_dir;
+
+ enc_dir = opendir(ENC_BASE);
+ if (!enc_dir)
+ return NULL;
+
+ for (de = readdir(enc_dir); de; de = readdir(enc_dir)) {
+ char num_slots[ENCLOSURE_STRING];
+ struct enclosure *enclosure;
+ char path[PATH_MAX];
+ DIR *slots;
+ char *e;
+
+ if (de->d_name[0] == '.')
+ continue;
+
+ enclosure = xcalloc(1, sizeof(*enclosure));
+ enclosure->devname = xstrdup(de->d_name);
+ *pos = enclosure;
+ pos = &enclosure->next;
+
+ snprintf(path, sizeof(path), "%s/%s/id", ENC_BASE, de->d_name);
+ if (load_sys_n(path, enclosure->id, sizeof(enclosure->id)) != 0)
+ break;
+
+ snprintf(path, sizeof(path), "%s/%s/components", ENC_BASE, de->d_name);
+ if (load_sys_n(path, num_slots, sizeof(num_slots)) != 0)
+ break;
+ enclosure->num_slots = strtoul(num_slots, &e, 10);
+
+ snprintf(path, sizeof(path), "%s/%s/device/model", ENC_BASE, de->d_name);
+ load_sys_n(path, enclosure->model, sizeof(enclosure->model));
+
+ snprintf(path, sizeof(path), "%s/%s/device/vendor", ENC_BASE, de->d_name);
+ load_sys_n(path, enclosure->vendor, sizeof(enclosure->vendor));
+
+ snprintf(path, sizeof(path), "%s/%s", ENC_BASE, de->d_name);
+ slots = opendir(path);
+ if (!slots)
+ break;
+
+ enclosure->slot = parse_slots(enclosure, slots);
+ closedir(slots);
+
+ if (!enclosure->slot)
+ break;
+
+ }
+
+ if (de) {
+ free_enclosure(enc_list);
+ enc_list = NULL;
+ }
+
+ closedir(enc_dir);
+
+ return enc_list;
+}
+
+static int detail_platform_enclosure(int verbose, int enumerate, char *enclosure_name)
+{
+ struct enclosure *enclosure = parse_enclosures();
+ struct enclosure *e;
+ int i;
+
+ if (!enclosure)
+ return 0;
+
+ for (e = enclosure, i = 0; e; e = e->next, i++) {
+ struct slot *s;
+
+ if (i)
+ printf("\n");
+ printf("Enclosure%d:\n", i);
+ printf(" Id : %s\n", e->id);
+ if (e->vendor[0])
+ printf(" Vendor : %s\n", e->vendor);
+ if (e->model[0])
+ printf(" Model : %s\n", e->model);
+ printf(" Slots : %d\n", e->num_slots);
+ if (e->slot)
+ printf("\n");
+
+ for (s = e->slot; s; s = s->next) {
+ int fd;
+ char buf[255];
+
+ printf(" Slot%02d :", s->slot_num);
+ snprintf(buf, sizeof(buf), "/dev/%s", s->devname ? : "");
+ printf("%s%s", buf[0] ? " " : "", buf);
+ if ((fd = dev_open(buf, O_RDONLY)) >= 0) {
+ memset(buf, 0, sizeof(buf));
+ if (scsi_get_serial(fd, buf, sizeof(buf)) == 0
+ && buf[3]) {
+ char *serial;
+ char *end;
+
+ serial = &buf[4];
+
+ while (isspace(*serial))
+ serial++;
+ end = serial;
+ while (!isspace(*end) && *end != '\0')
+ end++;
+ if (end - &buf[4] > buf[3])
+ end = &buf[4] + buf[3] - 1;
+ *end = '\0';
+ printf(" %s", serial);
+ }
+ close(fd);
+ }
+ printf("\n");
+ }
+ }
+
+ free_enclosure(enclosure);
+ return 0;
+}
+
+static int export_detail_enclosure(int verbose, char *enclosure_name)
+{
+ return 0;
+}
+
+const struct platform_ops enclosure_platform = {
+ .detail = detail_platform_enclosure,
+ .export_detail = export_detail_enclosure,
+ .name = "enclosure",
+};
diff --git a/mdadm.c b/mdadm.c
index 98ef898..0e46177 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -154,6 +154,14 @@ int main(int argc, char *argv[])
case 'Y': c.export++;
continue;
+ case Enclosure:
+ if (c.enclosure) {
+ pr_err("--enclosure already specified\n");
+ exit(2);
+ }
+ c.enclosure = optarg ? optarg : "scan";
+ continue;
+
case HomeHost:
if (strcasecmp(optarg, "<ignore>") == 0)
c.require_homehost = 0;
@@ -1492,16 +1500,38 @@ int main(int argc, char *argv[])
}
rv = Examine(devlist, &c, ss);
} else if (devmode == DetailPlatform) {
+ char *controller = devlist ? devlist->devname : NULL;
const struct platform_ops *platform;
+ int scan, rv = 0;
- if (ss)
- platform = ss->ss->platform;
+ if (ss || c.enclosure)
+ scan = c.scan;
else
- platform = NULL;
+ scan = 1;
- rv = Detail_Platform(platform, platform ? c.scan : 1,
- c.verbose, c.export,
- devlist ? devlist->devname : NULL);
+ if (scan) {
+ rv = Detail_Platform(NULL, scan, c.verbose,
+ c.export, controller);
+ } else {
+ /* metadata format specific details that
+ * may be tied to a given controller
+ */
+ if (ss) {
+ platform = ss->ss->platform;
+ rv = Detail_Platform(platform, scan,
+ c.verbose, c.export,
+ controller);
+ }
+ /* detail given enclosure or scan all
+ * enclosures
+ */
+ if (c.enclosure) {
+ platform = &enclosure_platform;
+ rv |= Detail_Platform(platform, scan,
+ c.verbose, c.export,
+ c.enclosure);
+ }
+ }
} else if (devlist == NULL) {
if (devmode == 'S' && c.scan)
rv = stop_scan(c.verbose);
diff --git a/mdadm.h b/mdadm.h
index f3b24b3..881c0fd 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -333,6 +333,7 @@ enum special_options {
AutoDetect,
Waitclean,
DetailPlatform,
+ Enclosure,
KillSubarray,
UpdateSubarray,
IncrementalPath,
@@ -459,6 +460,7 @@ struct context {
char *action;
int nodes;
char *homecluster;
+ char *enclosure;
};
struct shape {
@@ -599,10 +601,12 @@ extern int sysfs_set_safemode(struct mdinfo *sra, unsigned long ms);
extern int sysfs_set_array(struct mdinfo *info, int vers);
extern int sysfs_add_disk(struct mdinfo *sra, struct mdinfo *sd, int resume);
extern int sysfs_disk_to_scsi_id(int fd, __u32 *id);
+extern int scsi_get_serial(int fd, void *buf, size_t buf_len);
extern int sysfs_unique_holder(char *devnm, long rdev);
extern int sysfs_freeze_array(struct mdinfo *sra);
extern int sysfs_wait(int fd, int *msec);
extern int load_sys(char *path, char *buf);
+extern int load_sys_n(char *path, char *buf, int len);
extern int reshape_prepare_fdlist(char *devname,
struct mdinfo *sra,
int raid_disks,
@@ -1019,6 +1023,7 @@ extern struct superswitch super_imsm, super_ddf;
extern struct superswitch mbr, gpt;
extern const struct platform_ops imsm_platform;
+extern const struct platform_ops enclosure_platform;
struct metadata_update {
int len;
diff --git a/super-intel.c b/super-intel.c
index 53a9238..4ce46ef 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -3234,8 +3234,6 @@ static void fd2devname(int fd, char *name)
}
}
-extern int scsi_get_serial(int fd, void *buf, size_t buf_len);
-
static int imsm_read_serial(int fd, char *devname,
__u8 serial[MAX_RAID_SERIAL_LEN])
{
diff --git a/sysfs.c b/sysfs.c
index f1fd610..d20cf34 100644
--- a/sysfs.c
+++ b/sysfs.c
@@ -27,15 +27,15 @@
#include <dirent.h>
#include <ctype.h>
-int load_sys(char *path, char *buf)
+int load_sys_n(char *path, char *buf, int len)
{
int fd = open(path, O_RDONLY);
int n;
if (fd < 0)
return -1;
- n = read(fd, buf, 1024);
+ n = read(fd, buf, len);
close(fd);
- if (n <0 || n >= 1024)
+ if (n <0 || n >= len)
return -1;
buf[n] = 0;
if (n && buf[n-1] == '\n')
@@ -43,6 +43,11 @@ int load_sys(char *path, char *buf)
return 0;
}
+int load_sys(char *path, char *buf)
+{
+ return load_sys_n(path, buf, 1024);
+}
+
void sysfs_free(struct mdinfo *sra)
{
while (sra) {
diff --git a/util.c b/util.c
index e81c45d..bdc84e3 100644
--- a/util.c
+++ b/util.c
@@ -1150,6 +1150,7 @@ struct superswitch *superlist[] =
const struct platform_ops *platform_list[] =
{
&imsm_platform,
+ &enclosure_platform,
NULL,
};
--
2.4.6
^ permalink raw reply related
* [PATCH 07/17] prepare for enclosure platform details
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@localhost.localdomain>
Make platform description entry points independent of the superswitch
for platform details like enclosures that have no metadata ties. This
allows generically looping through a list of platform_detail operations.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Create.c | 2 +-
Detail.c | 59 ++++++++++++++++++++++++++++-------------------------------
mdadm.c | 9 ++++++++-
mdadm.h | 19 ++++++++++++++++---
super-intel.c | 8 ++++++--
util.c | 6 ++++++
6 files changed, 65 insertions(+), 38 deletions(-)
diff --git a/Create.c b/Create.c
index 2642b36..fda7f67 100644
--- a/Create.c
+++ b/Create.c
@@ -535,7 +535,7 @@ int Create(struct supertype *st, char *mddev,
warn = 1;
}
- if (st->ss->detail_platform && st->ss->detail_platform(0, 1, NULL) != 0) {
+ if (st->ss->platform && st->ss->platform->detail(0, 1, NULL) != 0) {
if (c->runstop != 1 || c->verbose >= 0)
pr_err("%s unable to enumerate platform support\n"
" array may not be compatible with hardware/firmware\n",
diff --git a/Detail.c b/Detail.c
index 200f65f..f71afb9 100644
--- a/Detail.c
+++ b/Detail.c
@@ -717,47 +717,44 @@ out:
return rv;
}
-int Detail_Platform(struct superswitch *ss, int scan, int verbose, int export, char *controller_path)
+int Detail_Platform(const struct platform_ops *platform, int scan, int verbose, int export, char *controller_path)
{
- /* display platform capabilities for the given metadata format
- * 'scan' in this context means iterate over all metadata types
+ /* display platform capabilities
+ * 'scan' in this context means iterate over all platform types
*/
- int i;
- int err = 1;
-
- if (ss && export && ss->export_detail_platform)
- err = ss->export_detail_platform(verbose, controller_path);
- else if (ss && ss->detail_platform)
- err = ss->detail_platform(verbose, 0, controller_path);
- else if (ss) {
+ struct platform_ops const **p;
+ int err = 0;
+
+ if (platform && verbose > 0)
+ pr_err("checking for %s platform components\n",
+ platform->name);
+
+ if (platform && export)
+ err = platform->export_detail(verbose, controller_path);
+ else if (platform)
+ err = platform->detail(verbose, 0, controller_path);
+ else if (!scan) {
if (verbose > 0)
- pr_err("%s metadata is platform independent\n",
- ss->name ? : "[no name]");
- } else if (!scan) {
- if (verbose > 0)
- pr_err("specify a metadata type or --scan\n");
+ pr_err("no platform components found\n");
}
if (!scan)
return err;
- err = 0;
- for (i = 0; superlist[i]; i++) {
- struct superswitch *meta = superlist[i];
-
- if (meta == ss)
+ for (p = platform_list; p; p++) {
+ /* enumerated above */
+ if (*p == platform)
continue;
+
if (verbose > 0)
- pr_err("checking metadata %s\n",
- meta->name ? : "[no name]");
- if (!meta->detail_platform) {
- if (verbose > 0)
- pr_err("%s metadata is platform independent\n",
- meta->name ? : "[no name]");
- } else if (export && meta->export_detail_platform) {
- err |= meta->export_detail_platform(verbose, controller_path);
- } else
- err |= meta->detail_platform(verbose, 0, controller_path);
+ pr_err("checking for %s platform components\n",
+ (*p)->name);
+
+ if (export)
+ err |= (*p)->export_detail(verbose, controller_path);
+ else
+ err |= (*p)->detail(verbose, 0, controller_path);
+
}
return err;
diff --git a/mdadm.c b/mdadm.c
index f56a8cf..98ef898 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -1492,7 +1492,14 @@ int main(int argc, char *argv[])
}
rv = Examine(devlist, &c, ss);
} else if (devmode == DetailPlatform) {
- rv = Detail_Platform(ss ? ss->ss : NULL, ss ? c.scan : 1,
+ const struct platform_ops *platform;
+
+ if (ss)
+ platform = ss->ss->platform;
+ else
+ platform = NULL;
+
+ rv = Detail_Platform(platform, platform ? c.scan : 1,
c.verbose, c.export,
devlist ? devlist->devname : NULL);
} else if (devlist == NULL) {
diff --git a/mdadm.h b/mdadm.h
index 167ca0c..f3b24b3 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -687,6 +687,17 @@ struct reshape {
unsigned long long new_size; /* New size of array in sectors */
};
+/* Platform details describe hardware / firmware capabilites and
+ * limitations. They can be tied to a metadata format. For example,
+ * system firmware with a set of raid capabilities and expected
+ * controller, or describe the topology disk enclosures.
+ */
+extern const struct platform_ops {
+ int (*detail)(int verbose, int enumerate_only, char *hwdevice);
+ int (*export_detail)(int verbose, char *hwdevice);
+ char *name;
+} *platform_list[];
+
/* A superswitch provides entry point the a metadata handler.
*
* The superswitch primarily operates on some "metadata" that
@@ -734,8 +745,7 @@ extern struct superswitch {
void (*export_detail_super)(struct supertype *st);
/* Optional: platform hardware / firmware details */
- int (*detail_platform)(int verbose, int enumerate_only, char *controller_path);
- int (*export_detail_platform)(int verbose, char *controller_path);
+ const struct platform_ops *platform;
/* Used:
* to get uuid to storing in bitmap metadata
@@ -1008,6 +1018,8 @@ extern struct superswitch super0, super1;
extern struct superswitch super_imsm, super_ddf;
extern struct superswitch mbr, gpt;
+extern const struct platform_ops imsm_platform;
+
struct metadata_update {
int len;
char *buf;
@@ -1263,7 +1275,8 @@ extern int Create(struct supertype *st, char *mddev,
unsigned long long data_offset);
extern int Detail(char *dev, struct context *c);
-extern int Detail_Platform(struct superswitch *ss, int scan, int verbose, int export, char *controller_path);
+extern int Detail_Platform(const struct platform_ops *platform, int scan, int verbose,
+ int export, char *controller_path);
extern int Query(char *dev);
extern int ExamineBadblocks(char *devname, int brief, struct supertype *forcest);
extern int Examine(struct mddev_dev *devlist, struct context *c,
diff --git a/super-intel.c b/super-intel.c
index 330b24f..53a9238 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -10606,6 +10606,11 @@ abort:
return ret_val;
}
+const struct platform_ops imsm_platform = {
+ .detail = detail_platform_imsm,
+ .export_detail = export_detail_platform_imsm,
+ .name = "imsm",
+};
#endif /* MDASSEMBLE */
struct superswitch super_imsm = {
@@ -10620,8 +10625,6 @@ struct superswitch super_imsm = {
.validate_geometry = validate_geometry_imsm,
.add_to_super = add_to_super_imsm,
.remove_from_super = remove_from_super_imsm,
- .detail_platform = detail_platform_imsm,
- .export_detail_platform = export_detail_platform_imsm,
.kill_subarray = kill_subarray_imsm,
.update_subarray = update_subarray_imsm,
.load_container = load_container_imsm,
@@ -10631,6 +10634,7 @@ struct superswitch super_imsm = {
.manage_reshape = imsm_manage_reshape,
.recover_backup = recover_backup_imsm,
.copy_metadata = copy_metadata_imsm,
+ .platform = &imsm_platform,
#endif
.match_home = match_home_imsm,
.uuid_from_super= uuid_from_super_imsm,
diff --git a/util.c b/util.c
index 8217e11..e81c45d 100644
--- a/util.c
+++ b/util.c
@@ -1147,6 +1147,12 @@ struct superswitch *superlist[] =
#if !defined(MDASSEMBLE) || defined(MDASSEMBLE) && defined(MDASSEMBLE_AUTO)
+const struct platform_ops *platform_list[] =
+{
+ &imsm_platform,
+ NULL,
+};
+
struct supertype *super_by_fd(int fd, char **subarrayp)
{
mdu_array_info_t array;
--
2.4.6
^ permalink raw reply related
* [PATCH 06/17] ses: workaround sysfs deprecated
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
sysfs deprecated makes forwarding udev events from a scsi device to its
block device difficult. Provide a workalike for the udev rule:
ACTION=="change", SUBSYSTEM=="scsi", ENV{DEVTYPE}=="scsi_device", \
TEST=="block", ATTR{block/*/uevent}="change"
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Makefile | 17 +++++++++++++++--
forward-scsi-uevents | 14 ++++++++++++++
...enclosure-slot.rules => udev-enclosure-slot.rules.in | 2 --
udev-workaround-sysfs-deprecated.in | 2 ++
4 files changed, 31 insertions(+), 4 deletions(-)
create mode 100755 forward-scsi-uevents
rename udev-enclosure-slot.rules => udev-enclosure-slot.rules.in (87%)
create mode 100644 udev-workaround-sysfs-deprecated.in
diff --git a/Makefile b/Makefile
index 4eb2dad..2963884 100644
--- a/Makefile
+++ b/Makefile
@@ -292,7 +292,17 @@ install-man: mdadm.8 md.4 mdadm.conf.5 mdmon.8
$(INSTALL) -D -m 644 md.4 $(DESTDIR)$(MAN4DIR)/md.4
$(INSTALL) -D -m 644 mdadm.conf.5 $(DESTDIR)$(MAN5DIR)/mdadm.conf.5
-install-udev: udev-md-raid-arrays.rules udev-md-raid-assembly.rules
+udev-enclosure-slot.rules: udev-enclosure-slot.rules.in udev-workaround-sysfs-deprecated.in Makefile
+ @echo "$@: generate"
+ @/bin/echo -e "# do not edit this file, it will be overwritten on update\n" > $@
+ @if [ ! -d /sys/class/block ]; then \
+ echo "$@: applying sysfs-deprecated workaround"; \
+ cat udev-workaround-sysfs-deprecated.in >> $@; \
+ fi
+ @cat udev-enclosure-slot.rules.in >> $@
+
+install-udev: udev-md-raid-arrays.rules udev-md-raid-assembly.rules udev-enclosure-slot.rules \
+ forward-scsi-uevents
$(INSTALL) -D -m 655 ses_slot_id $(DESTDIR)$(LIBDIR)/ses_slot_id
$(INSTALL) -D -m 644 udev-enclosure-slot.rules $(DESTDIR)$(UDEVDIR)/rules.d/62-md-enclosure-slot.rules
@for file in 63-md-raid-arrays.rules 64-md-raid-assembly.rules ; \
@@ -301,6 +311,9 @@ install-udev: udev-md-raid-arrays.rules udev-md-raid-assembly.rules
$(INSTALL) -D -m 644 .install.tmp.1 $(DESTDIR)$(UDEVDIR)/rules.d/$$file ; \
rm -f .install.tmp.1; \
done
+ @if [ ! -d /sys/class/block ]; then \
+ $(INSTALL) -D -m 655 forward-scsi-uevents $(DESTDIR)$(UDEVDIR)/forward-scsi-uevents; \
+ fi
install-systemd: systemd/mdmon@.service
@for file in mdmon@.service mdmonitor.service mdadm-last-resort@.timer \
@@ -331,7 +344,7 @@ clean :
mdassemble mdassemble.static mdassemble.auto mdassemble.uclibc \
mdassemble.klibc swap_super \
init.cpio.gz mdadm.uclibc.static test_stripe raid6check raid6check.o mdmon \
- mdadm.8
+ mdadm.8 udev-enclosure-slot.rules
dist : clean
./makedist
diff --git a/forward-scsi-uevents b/forward-scsi-uevents
new file mode 100755
index 0000000..d43687f
--- /dev/null
+++ b/forward-scsi-uevents
@@ -0,0 +1,14 @@
+#!/bin/sh
+
+# on sysfs-deprecated hosts the default udev rules are unable to forward
+# udev events to the block device, this is a hack to enable this
+# forwarding by searching for the matching device path
+
+for i in /sys/block/sd*
+do
+ DEV=$(readlink -f $i/device)
+ if [ $DEV = /sys$1 ]; then
+ echo "change" > $i/uevent
+ break
+ fi
+done
diff --git a/udev-enclosure-slot.rules b/udev-enclosure-slot.rules.in
similarity index 87%
rename from udev-enclosure-slot.rules
rename to udev-enclosure-slot.rules.in
index 75c6376..50e3c45 100644
--- a/udev-enclosure-slot.rules
+++ b/udev-enclosure-slot.rules.in
@@ -1,5 +1,3 @@
-# do not edit this file, it will be overwritten on update
-
SUBSYSTEM!="block", GOTO="ses_slot_end"
ACTION!="add|change", GOTO="ses_slot_end"
diff --git a/udev-workaround-sysfs-deprecated.in b/udev-workaround-sysfs-deprecated.in
new file mode 100644
index 0000000..65a21b3
--- /dev/null
+++ b/udev-workaround-sysfs-deprecated.in
@@ -0,0 +1,2 @@
+ACTION=="change", SUBSYSTEM=="scsi", ENV{DEVTYPE}=="scsi_device", TEST=="scsi_disk", RUN="forward-scsi-uevents %p"
+
--
2.4.6
^ permalink raw reply related
* [PATCH 05/17] imsm: quiet detail platform
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid
Cc: neilb, dan.j.williams, shli, Dan Williams, Lukasz Dorau, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
Don't squawk when platform capabilities missing, in preparation for
generic SES platform support. Undo some indentation as well.
Cc: Lukasz Dorau <lukasz.dorau@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
super-intel.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/super-intel.c b/super-intel.c
index e609e0c..330b24f 100644
--- a/super-intel.c
+++ b/super-intel.c
@@ -1935,8 +1935,9 @@ static int detail_platform_imsm(int verbose, int enumerate_only, char *controlle
if (controller_path && (compare_paths(hba->path, controller_path) != 0))
continue;
if (!find_imsm_capability(hba)) {
- pr_err("imsm capabilities not found for controller: %s (type %s)\n",
- hba->path, get_sys_dev_type(hba->type));
+ if (verbose > 0)
+ pr_err("imsm capabilities not found for controller: %s (type %s)\n",
+ hba->path, get_sys_dev_type(hba->type));
continue;
}
result = 0;
--
2.4.6
^ permalink raw reply related
* [PATCH 04/17] fix variable offset when ':' is present in the device name
From: Song Liu @ 2015-12-02 0:25 UTC (permalink / raw)
To: linux-raid; +Cc: neilb, dan.j.williams, shli, Dan Williams, Song Liu
In-Reply-To: <1449015933-255689-1-git-send-email-songliubraving@fb.com>
From: Dan Williams <djbw@fb.com>
Stopping at the first ':' precludes using device names like
/dev/disk/by-path/pci-0000:04:00.0-sas-0x500605b005846060:1:0-0x5000cca01a7c5605:10
so, check that the name is not ambiguous before chopping off the offset at the end
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Create.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/Create.c b/Create.c
index 21d1374..2642b36 100644
--- a/Create.c
+++ b/Create.c
@@ -306,7 +306,13 @@ int Create(struct supertype *st, char *mddev,
continue;
}
if (data_offset == VARIABLE_OFFSET) {
- doff = strchr(dname, ':');
+ doff = strrchr(dname, ':');
+ if (stat(dname, &stb) == 0) {
+ pr_err("cannot determine if %s is a device name, or a device with a data-offset argument of '%s'\n",
+ dname, doff+1);
+ pr_err("check that you have specified a data-offset for all array members\n");
+ exit(2);
+ }
if (doff) {
*doff++ = 0;
dv->data_offset = parse_size(doff);
--
2.4.6
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox