* raid10 resync hangs in 4.2.6, 4.3
From: Andre Tomt @ 2015-11-20 18:14 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 5582 bytes --]
[resend with compressed attachments, first may have gotten eaten by a grue]
Hi
I'm seeing hangs with RAID10 resyncs on my system. RAID5/6 recovery on
the same drive set works without any problems. BTRFS RAID6 is problem
free on a different set of (very busy) drives on the same controllers as
well.
It happens shortly after array creation, from seconds to a couple
minutes in.
wchan for md0_resync kernel thread shows it sitting in raise_barrier()
forever, while md0_raid10 keeps a CPU core 100% busy (but shows no
wchan), and no resyncing or I/O to the array is getting done anymore.
After a short while kernel starts spitting out rcu_sched self-detected
stall on CPU warnings, and other rcu use starts getting iffy (I think).
I/O directly to the RAID member disk (below md layer, eg /dev/sdX
directly) continues to work after the hang, and there are no I/O errors
in the kernel log.
The array is a 8 drive array spread over 3 HBAs, created with:
mdadm --create /dev/md0 --level=10 --chunk=128 --bitmap=none
--raid-devices=8 /dev/sda1 /dev/sdg1 /dev/sdl1 /dev/sdm1 /dev/sdi1
/dev/sdj1 /dev/sdp1 /dev/sds1
The HBAs are LSI SAS2008 in IT mode (mpt2sas driver), oldish 2TB SATA
drives. Dual socket Xeon E5 v3 system with both sockets populated.
This happens on at least 4.2.6 and 4.3
I'm going to test some earlier kernels.
attached some more info.
md0_resync stack:
> root@mental:~# cat /proc/1663/stack
> [<ffffffffc042dd02>] raise_barrier+0x11b/0x14d [raid10]
> [<ffffffffc0432830>] sync_request+0x193/0x14fc [raid10]
> [<ffffffffc0559ac6>] md_do_sync+0x7d2/0xd78 [md_mod]
> [<ffffffffc0556df9>] md_thread+0x12f/0x145 [md_mod]
> [<ffffffff9d061db2>] kthread+0xcd/0xd5
> [<ffffffff9d3b7a8f>] ret_from_fork+0x3f/0x70
> [<ffffffffffffffff>] 0xffffffffffffffff
md0_raid10 stack:
> root@mental:~# cat /proc/1662/stack
> [<ffffffffffffffff>] 0xffffffffffffffff
cat stack trying to read /dev/md0 after hang:
> root@mental:~# cat /proc/1737/stack
> [<ffffffffc042de2b>] wait_barrier+0xd8/0x118 [raid10]
> [<ffffffffc042f83c>] __make_request+0x3e/0xb17 [raid10]
> [<ffffffffc0430399>] make_request+0x84/0xdc [raid10]
> [<ffffffffc0557aab>] md_make_request+0xf6/0x1cc [md_mod]
> [<ffffffff9d1a369a>] generic_make_request+0x97/0xd6
> [<ffffffff9d1a37d1>] submit_bio+0xf8/0x140
> [<ffffffff9d140e71>] mpage_bio_submit+0x25/0x2c
> [<ffffffff9d141499>] mpage_readpages+0x10e/0x11f
> [<ffffffff9d13c76c>] blkdev_readpages+0x18/0x1a
> [<ffffffff9d0d09e4>] __do_page_cache_readahead+0x13c/0x1e0
> [<ffffffff9d0d0c67>] ondemand_readahead+0x1df/0x1f2
> [<ffffffff9d0d0da0>] page_cache_sync_readahead+0x38/0x3a
> [<ffffffff9d0c70b3>] generic_file_read_iter+0x184/0x50b
> [<ffffffff9d13c8f7>] blkdev_read_iter+0x33/0x38
> [<ffffffff9d1132fd>] __vfs_read+0x8d/0xb1
> [<ffffffff9d113820>] vfs_read+0x95/0x120
> [<ffffffff9d11402d>] SyS_read+0x49/0x84
> [<ffffffff9d3b772e>] entry_SYSCALL_64_fastpath+0x12/0x71
> [<ffffffffffffffff>] 0xffffffffffffffff
First OOPS (more in dmesg.txt)
> [ 150.183473] md0: detected capacity change from 0 to 8001054310400
> [ 150.183647] md: resync of RAID array md0
> [ 150.183652] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> [ 150.183654] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
> [ 150.183678] md: using 128k window, over a total of 7813529600k.
> [ 233.068271] INFO: rcu_sched self-detected stall on CPU
> [ 233.068308] 5: (17999 ticks this GP) idle=695/140000000000001/0 softirq=1235/1235 fqs=8999
> [ 233.068335] (t=18000 jiffies g=935 c=934 q=37)
> [ 233.068354] Task dump for CPU 5:
> [ 233.068356] md0_raid10 R running task 0 1662 2 0x00000008
> [ 233.068358] 0000000000000000 ffff88103fca3de0 ffffffff9d069f07 0000000000000005
> [ 233.068360] ffffffff9d63d0c0 ffff88103fca3df8 ffffffff9d06beb9 ffffffff9d63d0c0
> [ 233.068361] ffff88103fca3e28 ffffffff9d08c836 ffffffff9d63d0c0 ffff88103fcb4e80
> [ 233.068363] Call Trace:
> [ 233.068364] <IRQ> [<ffffffff9d069f07>] sched_show_task+0xb9/0xbe
> [ 233.068372] [<ffffffff9d06beb9>] dump_cpu_task+0x32/0x35
> [ 233.068375] [<ffffffff9d08c836>] rcu_dump_cpu_stacks+0x71/0x8c
> [ 233.068378] [<ffffffff9d08f32c>] rcu_check_callbacks+0x20f/0x5a3
> [ 233.068382] [<ffffffff9d0b72da>] ? acct_account_cputime+0x17/0x19
> [ 233.068384] [<ffffffff9d0911a2>] update_process_times+0x2a/0x4f
> [ 233.068387] [<ffffffff9d09cd55>] tick_sched_handle.isra.5+0x31/0x33
> [ 233.068388] [<ffffffff9d09cd8f>] tick_sched_timer+0x38/0x60
> [ 233.068390] [<ffffffff9d0917e1>] __hrtimer_run_queues+0xa1/0x10c
> [ 233.068392] [<ffffffff9d091c52>] hrtimer_interrupt+0xa0/0x172
> [ 233.068395] [<ffffffff9d0367a4>] smp_trace_apic_timer_interrupt+0x76/0x88
> [ 233.068397] [<ffffffff9d0367bf>] smp_apic_timer_interrupt+0x9/0xb
> [ 233.068400] [<ffffffff9d3b8402>] apic_timer_interrupt+0x82/0x90
> [ 233.068401] <EOI> [<ffffffff9d19e9aa>] ? bio_copy_data+0xce/0x2af
> [ 233.068410] [<ffffffffc04320e5>] raid10d+0x974/0xf2c [raid10]
> [ 233.068417] [<ffffffffc0556df9>] md_thread+0x12f/0x145 [md_mod]
> [ 233.068421] [<ffffffffc0556df9>] ? md_thread+0x12f/0x145 [md_mod]
> [ 233.068424] [<ffffffff9d07ad2e>] ? wait_woken+0x6d/0x6d
> [ 233.068428] [<ffffffffc0556cca>] ? md_wait_for_blocked_rdev+0x102/0x102 [md_mod]
> [ 233.068431] [<ffffffff9d061db2>] kthread+0xcd/0xd5
> [ 233.068434] [<ffffffff9d061ce5>] ? kthread_worker_fn+0x13f/0x13f
> [ 233.068436] [<ffffffff9d3b7a8f>] ret_from_fork+0x3f/0x70
> [ 233.068438] [<ffffffff9d061ce5>] ? kthread_worker_fn+0x13f/0x13f
[-- Attachment #2: config.txt.gz --]
[-- Type: application/gzip, Size: 31864 bytes --]
[-- Attachment #3: dmesg.txt.gz --]
[-- Type: application/gzip, Size: 31060 bytes --]
[-- Attachment #4: wchan.txt.gz --]
[-- Type: application/gzip, Size: 3051 bytes --]
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Serge E. Hallyn @ 2015-11-20 17:33 UTC (permalink / raw)
To: Octavian Purdila
Cc: Seth Forshee, Richard Weinberger, Al Viro, Eric W. Biederman,
linux-bcache, device-mapper development,
linux-raid@vger.kernel.org, linux-mtd@lists.infradead.org,
linux-fsdevel, LSM, selinux, Serge Hallyn, Andy Lutomirski, LKML,
Theodore Ts'o
In-Reply-To: <CAE1zot+G=SohXzuV6Gs7APdCo5pu9xLwE38kZKNUuWRnSh++BA@mail.gmail.com>
On Thu, Nov 19, 2015 at 06:19:10PM +0200, Octavian Purdila wrote:
> On Thu, Nov 19, 2015 at 5:23 PM, Seth Forshee
> > And for the second:
> >
> > stress-ng --class filesystem --seq 0 -v -t 60
> >
> > There really wasn't anything interesting in the lklfuse output for the
> > first run, but for the second run I pasted the output here:
> > http://paste.ubuntu.com/13346993/
>
> lklfuse allocates a fixed 100MB to the kernel and this is probably not
> enough. For the short term I can add a parameter to lklfuse that
> allows the user to specify the amount of memory to allocate to lkl. A
> better fix would probably be to dynamically adjust the memory size of
> lkl. I am thinking of using the ballon virtio driver or the memory
> hotplug infrastructure. Any other suggestions?
Hi Octavian,
Like Seth said this was very nice to install and run. But I think I'm
doing something wrong. I cloned the lkl/linux git tree into a 9G ext4
loopback file, both one mounted with lkl and one mounted as loopback.
The results were worse than I expected - loopback was 0m10.078s for
clone and 0m0.877s for rm -rf; lkl was 5m5.625s for clone and 0m44.332s
for rm -rf. I assume this has to do with the 100MB buffer and subsequent
thrashing? How should I tune this to make it perform better?
No crashes from this, though.
thanks,
-serge
^ permalink raw reply
* Re: Ubuntu crashed during RAID6 grow
From: Phil Turmel @ 2015-11-20 14:22 UTC (permalink / raw)
To: Mikael Abrahamsson, Nathan Peterson; +Cc: linux-raid
In-Reply-To: <alpine.DEB.2.02.1511200745170.24520@uplift.swm.pp.se>
On 11/20/2015 01:48 AM, Mikael Abrahamsson wrote:
> On Thu, 19 Nov 2015, Nathan Peterson wrote:
>
>> Details after the grow:
>>
>> md126 : active (auto-read-only) raid6 sdj1[2] sdl1[3] sdf1[9] sdn1[11]
>> (S) sdi1[1] sdg1[10] sde1[8] sdh1[0] sdk1[4] sdo1[12]
>>
>> Delta Devices : 1, (8->9)
>
> Please post mdadm --examine from all component drives, but I'd say if
> you try to access your data everything should be there and the grow
> should continue once the array is not auto-read-only anymore (guess from
> my side though).
Concur. There's nothing wrong with your array, and yes, mdadm knows
that it was in the middle of a --grow operation. Mount it and carry on.
Phil
^ permalink raw reply
* [PATCH 3/3] md-cluster: update the documentation
From: Guoqing Jiang @ 2015-11-20 8:35 UTC (permalink / raw)
To: neilb; +Cc: rgoldwyn, linux-raid, Guoqing Jiang
In-Reply-To: <1448008525-17095-1-git-send-email-gqjiang@suse.com>
Update design documentation based on recent development.
original version comes from Neil.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Documentation/md-cluster.txt | 314 +++++++++++++++++++++++++++++++------------
1 file changed, 228 insertions(+), 86 deletions(-)
diff --git a/Documentation/md-cluster.txt b/Documentation/md-cluster.txt
index 1b79436..c100c71 100644
--- a/Documentation/md-cluster.txt
+++ b/Documentation/md-cluster.txt
@@ -3,7 +3,7 @@ The cluster MD is a shared-device RAID for a cluster.
1. On-disk format
-Separate write-intent-bitmap are used for each cluster node.
+Separate write-intent-bitmaps are used for each cluster node.
The bitmaps record all writes that may have been started on that node,
and may not yet have finished. The on-disk layout is:
@@ -14,117 +14,161 @@ and may not yet have finished. The on-disk layout is:
| bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
| bm bits [3, contd] | | |
-During "normal" functioning we assume the filesystem ensures that only one
-node writes to any given block at a time, so a write
-request will
+During "normal" functioning we assume the filesystem ensures that only
+one node writes to any given block at a time, so a write request will
+
- set the appropriate bit (if not already set)
- commit the write to all mirrors
- schedule the bit to be cleared after a timeout.
-Reads are just handled normally. It is up to the filesystem to
-ensure one node doesn't read from a location where another node (or the same
+Reads are just handled normally. It is up to the filesystem to ensure
+one node doesn't read from a location where another node (or the same
node) is writing.
2. DLM Locks for management
-There are two locks for managing the device:
+There are three groups of locks for managing the device:
2.1 Bitmap lock resource (bm_lockres)
- The bm_lockres protects individual node bitmaps. They are named in the
- form bitmap001 for node 1, bitmap002 for node and so on. When a node
- joins the cluster, it acquires the lock in PW mode and it stays so
- during the lifetime the node is part of the cluster. The lock resource
- number is based on the slot number returned by the DLM subsystem. Since
- DLM starts node count from one and bitmap slots start from zero, one is
- subtracted from the DLM slot number to arrive at the bitmap slot number.
+ The bm_lockres protects individual node bitmaps. They are named in
+ the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a
+ node joins the cluster, it acquires the lock in PW mode and it stays
+ so during the lifetime the node is part of the cluster. The lock
+ resource number is based on the slot number returned by the DLM
+ subsystem. Since DLM starts node count from one and bitmap slots
+ start from zero, one is subtracted from the DLM slot number to arrive
+ at the bitmap slot number.
+
+ The LVB of the bitmap lock for a particular node records the range
+ of sectors that are being re-synced by that node. No other
+ node may write to those sectors. This is used when a new nodes
+ joins the cluster.
+
+2.2 Message passing locks
+
+ Each node has to communicate with other nodes when starting or ending
+ resync, and for metadata superblock updates. This communication is
+ managed through three locks: "token", "message", and "ack", together
+ with the Lock Value Block (LVB) of one of the "message" lock.
+
+2.3 new-device management
+
+ A single lock: "no-new-dev" is used to co-ordinate the addition of
+ new devices - this must be synchronized across the array.
+ Normally all nodes hold a concurrent-read lock on this device.
3. Communication
-Each node has to communicate with other nodes when starting or ending
-resync, and metadata superblock updates.
+ Messages can be broadcast to all nodes, and the sender waits for all
+ other nodes to acknowledge the message before proceeding. Only one
+ message can be processed at a time.
3.1 Message Types
- There are 3 types, of messages which are passed
+ There are six types of messages which are passed:
- 3.1.1 METADATA_UPDATED: informs other nodes that the metadata has been
- updated, and the node must re-read the md superblock. This is performed
- synchronously.
+ 3.1.1 METADATA_UPDATED: informs other nodes that the metadata has
+ been updated, and the node must re-read the md superblock. This is
+ performed synchronously. It is primarily used to signal device
+ failure.
- 3.1.2 RESYNC: informs other nodes that a resync is initiated or ended
- so that each node may suspend or resume the region.
+ 3.1.2 RESYNCING: informs other nodes that a resync is initiated or
+ ended so that each node may suspend or resume the region. Each
+ RESYNCING message identifies a range of the devices that the
+ sending node is about to resync. This over-rides any pervious
+ notification from that node: only one ranged can be resynced at a
+ time per-node.
+
+ 3.1.3 NEWDISK: informs other nodes that a device is being added to
+ the array. Message contains an identifier for that device. See
+ below for further details.
+
+ 3.1.4 REMOVE: A failed or spare device is being removed from the
+ array. The slot-number of the device is included in the message.
+
+ 3.1.5 RE_ADD: A failed device is being re-activated - the assumption
+ is that it has been determined to be working again.
+
+ 3.1.6 BITMAP_NEEDS_SYNC: if a node is stopped locally but the bitmap
+ isn't clean, then another node is informed to take the ownership of
+ resync.
3.2 Communication mechanism
The DLM LVB is used to communicate within nodes of the cluster. There
are three resources used for the purpose:
- 3.2.1 Token: The resource which protects the entire communication
+ 3.2.1 token: The resource which protects the entire communication
system. The node having the token resource is allowed to
communicate.
- 3.2.2 Message: The lock resource which carries the data to
+ 3.2.2 message: The lock resource which carries the data to
communicate.
- 3.2.3 Ack: The resource, acquiring which means the message has been
+ 3.2.3 ack: The resource, acquiring which means the message has been
acknowledged by all nodes in the cluster. The BAST of the resource
- is used to inform the receive node that a node wants to communicate.
+ is used to inform the receiving node that a node wants to
+ communicate.
The algorithm is:
- 1. receive status
+ 1. receive status - all nodes have concurrent-reader lock on "ack".
- sender receiver receiver
- ACK:CR ACK:CR ACK:CR
+ sender receiver receiver
+ "ack":CR "ack":CR "ack":CR
- 2. sender get EX of TOKEN
- sender get EX of MESSAGE
+ 2. sender get EX on "token"
+ sender get EX on "message"
sender receiver receiver
- TOKEN:EX ACK:CR ACK:CR
- MESSAGE:EX
- ACK:CR
+ "token":EX "ack":CR "ack":CR
+ "message":EX
+ "ack":CR
- Sender checks that it still needs to send a message. Messages received
- or other events that happened while waiting for the TOKEN may have made
- this message inappropriate or redundant.
+ Sender checks that it still needs to send a message. Messages
+ received or other events that happened while waiting for the
+ "token" may have made this message inappropriate or redundant.
- 3. sender write LVB.
- sender down-convert MESSAGE from EX to CW
- sender try to get EX of ACK
- [ wait until all receiver has *processed* the MESSAGE ]
+ 3. sender writes LVB.
+ sender down-convert "message" from EX to CW
+ sender try to get EX of "ack"
+ [ wait until all receivers have *processed* the "message" ]
- [ triggered by bast of ACK ]
- receiver get CR of MESSAGE
+ [ triggered by bast of "ack" ]
+ receiver get CR on "message"
receiver read LVB
receiver processes the message
[ wait finish ]
- receiver release ACK
-
- sender receiver receiver
- TOKEN:EX MESSAGE:CR MESSAGE:CR
- MESSAGE:CR
- ACK:EX
-
- 4. triggered by grant of EX on ACK (indicating all receivers have processed
- message)
- sender down-convert ACK from EX to CR
- sender release MESSAGE
- sender release TOKEN
- receiver upconvert to PR of MESSAGE
- receiver get CR of ACK
- receiver release MESSAGE
+ receiver releases "ack"
+ receiver tries to get PR on "message"
+
+ sender receiver receiver
+ "token":EX "message":CR "message":CR
+ "message":CW
+ "ack":EX
+
+ 4. triggered by grant of EX on "ack" (indicating all receivers
+ have processed message)
+ sender down-converts "ack" from EX to CR
+ sender releases "message"
+ sender releases "token"
+ receiver upconvert to PR on "message"
+ receiver get CR of "ack"
+ receiver release "message"
sender receiver receiver
- ACK:CR ACK:CR ACK:CR
+ "ack":CR "ack":CR "ack":CR
4. Handling Failures
4.1 Node Failure
- When a node fails, the DLM informs the cluster with the slot. The node
- starts a cluster recovery thread. The cluster recovery thread:
+
+ When a node fails, the DLM informs the cluster with the slot
+ number. The node starts a cluster recovery thread. The cluster
+ recovery thread:
+
- acquires the bitmap<number> lock of the failed node
- opens the bitmap
- reads the bitmap of the failed node
@@ -132,45 +176,143 @@ The algorithm is:
- cleans the bitmap of the failed node
- releases bitmap<number> lock of the failed node
- initiates resync of the bitmap on the current node
+ md_check_recovery is invoked within recover_bitmaps,
+ then md_check_recovery -> metadata_update_start/finish,
+ it will lock the communication by lock_comm.
+ Which means when one node is resyncing it blocks all
+ other nodes from writing anywhere on the array.
- The resync process, is the regular md resync. However, in a clustered
+ The resync process is the regular md resync. However, in a clustered
environment when a resync is performed, it needs to tell other nodes
of the areas which are suspended. Before a resync starts, the node
- send out RESYNC_START with the (lo,hi) range of the area which needs
- to be suspended. Each node maintains a suspend_list, which contains
- the list of ranges which are currently suspended. On receiving
- RESYNC_START, the node adds the range to the suspend_list. Similarly,
- when the node performing resync finishes, it send RESYNC_FINISHED
- to other nodes and other nodes remove the corresponding entry from
- the suspend_list.
+ send out RESYNCING with the (lo,hi) range of the area which needs to
+ be suspended. Each node maintains a suspend_list, which contains the
+ list of ranges which are currently suspended. On receiving RESYNCING,
+ the node adds the range to the suspend_list. Similarly, when the node
+ performing resync finishes, it sends RESYNCING with an empty range to
+ other nodes and other nodes remove the corresponding entry from the
+ suspend_list.
- A helper function, should_suspend() can be used to check if a particular
- I/O range should be suspended or not.
+ A helper function, ->area_resyncing() can be used to check if a
+ particular I/O range should be suspended or not.
4.2 Device Failure
+
Device failures are handled and communicated with the metadata update
- routine.
+ routine. When a node detects a device failure it does not allow
+ any further writes to that device until the failure has been
+ acknowledged by all other nodes.
5. Adding a new Device
-For adding a new device, it is necessary that all nodes "see" the new device
-to be added. For this, the following algorithm is used:
+
+ For adding a new device, it is necessary that all nodes "see" the new
+ device to be added. For this, the following algorithm is used:
1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
- ioctl(ADD_NEW_DISC with disc.state set to MD_DISK_CLUSTER_ADD)
- 2. Node 1 sends NEWDISK with uuid and slot number
+ ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD)
+ 2. Node 1 sends a NEWDISK message with uuid and slot number
3. Other nodes issue kobject_uevent_env with uuid and slot number
(Steps 4,5 could be a udev rule)
4. In userspace, the node searches for the disk, perhaps
using blkid -t SUB_UUID=""
- 5. Other nodes issue either of the following depending on whether the disk
- was found:
+ 5. Other nodes issue either of the following depending on whether
+ the disk was found:
ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
- disc.number set to slot number)
+ disc.number set to slot number)
ioctl(CLUSTERED_DISK_NACK)
- 6. Other nodes drop lock on no-new-devs (CR) if device is found
- 7. Node 1 attempts EX lock on no-new-devs
- 8. If node 1 gets the lock, it sends METADATA_UPDATED after unmarking the disk
- as SpareLocal
- 9. If not (get no-new-dev lock), it fails the operation and sends METADATA_UPDATED
- 10. Other nodes get the information whether a disk is added or not
- by the following METADATA_UPDATED.
+ 6. Other nodes drop lock on "no-new-devs" (CR) if device is found
+ 7. Node 1 attempts EX lock on "no-new-dev"
+ 8. If node 1 gets the lock, it sends METADATA_UPDATED after
+ unmarking the disk as SpareLocal
+ 9. If not (get "no-new-dev" lock), it fails the operation and sends
+ METADATA_UPDATED.
+ 10. Other nodes get the information whether a disk is added or not
+ by the following METADATA_UPDATED.
+
+6. Module interface.
+
+ There are 17 call-backs which the md core can make to the cluster
+ module. Understanding these can give a good overview of the whole
+ process.
+
+6.1 join(nodes) and leave()
+
+ These are called when an array is started with a clustered bitmap,
+ and when the array is stopped. join() ensures the cluster is
+ available and initializes the various resources.
+ Only the first 'nodes' nodes in the cluster can use the array.
+
+6.2 slot_number()
+
+ Reports the slot number advised by the cluster infrastructure.
+ Range is from 0 to nodes-1.
+
+6.3 resync_info_update()
+
+ This updates the resync range that is stored in the bitmap lock.
+ The starting point is updated as the resync progresses. The
+ end point is always the end of the array.
+ It does *not* send a RESYNCING message.
+
+6.4 resync_start(), resync_finish()
+
+ These are called when resync/recovery/reshape starts or stops.
+ They update the resyncing range in the bitmap lock and also
+ send a RESYNCING message. resync_start reports the whole
+ array as resyncing, resync_finish reports none of it.
+
+ resync_finish() also sends a BITMAP_NEEDS_SYNC message which
+ allows some other node to take over.
+
+6.5 metadata_update_start(), metadata_update_finish(),
+ metadata_update_cancel().
+
+ metadata_update_start is used to get exclusive access to
+ the metadata. If a change is still needed once that access is
+ gained, metadata_update_finish() will send a METADATA_UPDATE
+ message to all other nodes, otherwise metadata_update_cancel()
+ can be used to release the lock.
+
+6.6 area_resyncing()
+
+ This combines two elements of functionality.
+
+ Firstly, it will check if any node is currently resyncing
+ anything in a given range of sectors. If any resync is found,
+ then the caller will avoid writing or read-balancing in that
+ range.
+
+ Secondly, while node recovery is happening it reports that
+ all areas are resyncing for READ requests. This avoids races
+ between the cluster-filesystem and the cluster-RAID handling
+ a node failure.
+
+6.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack()
+
+ These are used to manage the new-disk protocol described above.
+ When a new device is added, add_new_disk_start() is called before
+ it is bound to the array and, if that succeeds, add_new_disk_finish()
+ is called the device is fully added.
+
+ When a device is added in acknowledgement to a previous
+ request, or when the device is declared "unavailable",
+ new_disk_ack() is called.
+
+6.8 remove_disk()
+
+ This is called when a spare or failed device is removed from
+ the array. It causes a REMOVE message to be send to other nodes.
+
+6.9 gather_bitmaps()
+
+ This sends a RE_ADD message to all other nodes and then
+ gathers bitmap information from all bitmaps. This combined
+ bitmap is then used to recovery the re-added device.
+
+6.10 lock_all_bitmaps() and unlock_all_bitmaps()
+
+ These are called when change bitmap to none. If a node plans
+ to clear the cluster raid's bitmap, it need to make sure no other
+ nodes are using the raid which is achieved by lock all bitmap
+ locks within the cluster, and also those locks are unlocked
+ accordingly.
--
2.1.4
^ permalink raw reply related
* [PATCH 2/3] md-cluster: introduce clear_clusterinfo_from_sb
From: Guoqing Jiang @ 2015-11-20 8:35 UTC (permalink / raw)
To: neilb; +Cc: rgoldwyn, linux-raid, Guoqing Jiang
In-Reply-To: <1448008525-17095-1-git-send-email-gqjiang@suse.com>
To change the bitmap from clustered to none, we also
need to clean related info about cluster from sb, such
as version, nodes and cluster_name.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/bitmap.c | 25 +++++++++++++++++++++++++
drivers/md/bitmap.h | 1 +
drivers/md/md.c | 1 +
3 files changed, 27 insertions(+)
diff --git a/drivers/md/bitmap.c b/drivers/md/bitmap.c
index 4f22e91..c11390f 100644
--- a/drivers/md/bitmap.c
+++ b/drivers/md/bitmap.c
@@ -417,6 +417,31 @@ out:
/*
* bitmap file superblock operations
*/
+void clear_clusterinfo_from_sb(struct bitmap *bitmap)
+{
+ bitmap_super_t *sb;
+
+ if (!bitmap || !bitmap->mddev) /* no bitmap for this array */
+ return;
+ if (bitmap->mddev->bitmap_info.external)
+ return;
+ if (!bitmap->storage.sb_page) /* no superblock */
+ return;
+ sb = kmap_atomic(bitmap->storage.sb_page);
+ sb->nodes = 0;
+ sb->version = cpu_to_le32(BITMAP_MAJOR_HI);
+ memset(sb->cluster_name, NULL, 64);
+
+ kunmap_atomic(sb);
+ /* Don't write until any other writes have completed */
+ if (bitmap->storage.file)
+ wait_event(bitmap->write_wait,
+ atomic_read(&bitmap->pending_writes)==0);
+ else
+ md_super_wait(bitmap->mddev);
+
+ write_page(bitmap, bitmap->storage.sb_page, 1);
+}
/* update the event counter and sync the superblock to disk */
void bitmap_update_sb(struct bitmap *bitmap)
diff --git a/drivers/md/bitmap.h b/drivers/md/bitmap.h
index 7d5c3a6..1d0975b 100644
--- a/drivers/md/bitmap.h
+++ b/drivers/md/bitmap.h
@@ -241,6 +241,7 @@ void bitmap_flush(struct mddev *mddev);
void bitmap_destroy(struct mddev *mddev);
void bitmap_print_sb(struct bitmap *bitmap);
+void clear_clusterinfo_from_sb(struct bitmap *bitmap);
void bitmap_update_sb(struct bitmap *bitmap);
void bitmap_status(struct seq_file *seq, struct bitmap *bitmap);
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 47ee8ae..f06dc43 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -6598,6 +6598,7 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
}
mddev->bitmap_info.nodes = 0;
+ clear_clusterinfo_from_sb(mddev->bitmap);
md_cluster_ops->leave(mddev);
}
mddev->pers->quiesce(mddev, 1);
--
2.1.4
^ permalink raw reply related
* [PATCH 1/3] md-cluster: append some actions when change bitmap from clustered to none
From: Guoqing Jiang @ 2015-11-20 8:35 UTC (permalink / raw)
To: neilb; +Cc: rgoldwyn, linux-raid, Guoqing Jiang
For clustered raid, we need to do extra actions when change
bitmap to none.
1. check if all the bitmap lock could be get or not, if yes then
we can continue the change since cluster raid is only active
in current node. Otherwise return fail and unlock the related
bitmap locks
2. set nodes to 0 and then leave cluster environment.
3. release other nodes's bitmap lock.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
drivers/md/md-cluster.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++
drivers/md/md-cluster.h | 2 ++
drivers/md/md.c | 14 ++++++++++++
3 files changed, 73 insertions(+)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 430b139..148cd85 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -69,6 +69,7 @@ struct md_cluster_info {
struct completion completion;
struct mutex recv_mutex;
struct dlm_lock_resource *bitmap_lockres;
+ struct dlm_lock_resource **other_bitmap_lockres;
struct dlm_lock_resource *resync_lockres;
struct list_head suspend_list;
spinlock_t suspend_lock;
@@ -835,6 +836,7 @@ static void resync_bitmap(struct mddev *mddev)
__func__, __LINE__, err);
}
+static void unlock_all_bitmaps(struct mddev *mddev);
static int leave(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
@@ -855,6 +857,7 @@ static int leave(struct mddev *mddev)
lockres_free(cinfo->ack_lockres);
lockres_free(cinfo->no_new_dev_lockres);
lockres_free(cinfo->bitmap_lockres);
+ unlock_all_bitmaps(mddev);
dlm_release_lockspace(cinfo->lockspace, 2);
return 0;
}
@@ -1054,6 +1057,58 @@ static int remove_disk(struct mddev *mddev, struct md_rdev *rdev)
return sendmsg(cinfo, &cmsg);
}
+static int lock_all_bitmaps(struct mddev *mddev)
+{
+ int slot, my_slot, ret, held = 1, i = 0;
+ char str[64];
+ struct md_cluster_info *cinfo = mddev->cluster_info;
+
+ cinfo->other_bitmap_lockres = kzalloc((mddev->bitmap_info.nodes - 1) *
+ sizeof(struct dlm_lock_resource *),
+ GFP_KERNEL);
+ if (!cinfo->other_bitmap_lockres) {
+ pr_err("%s:%d: can't alloc mem for other bitmap locks\n", __func__, __LINE__);
+ return 0;
+ }
+
+ my_slot = slot_number(mddev);
+ for (slot = 0; slot < mddev->bitmap_info.nodes; slot++) {
+ if (slot == my_slot)
+ continue;
+
+ memset(str, '\0', 64);
+ snprintf(str, 64, "bitmap%04d", slot);
+ cinfo->other_bitmap_lockres[i] = lockres_init(mddev, str, NULL, 1);
+ if (!cinfo->other_bitmap_lockres[i])
+ return -ENOMEM;
+
+ cinfo->other_bitmap_lockres[i]->flags |= DLM_LKF_NOQUEUE;
+ ret = dlm_lock_sync(cinfo->other_bitmap_lockres[i], DLM_LOCK_PW);
+ if (ret)
+ held = -1;
+ i++;
+ }
+
+ return held;
+}
+
+static void unlock_all_bitmaps(struct mddev *mddev)
+{
+ struct md_cluster_info *cinfo = mddev->cluster_info;
+ int i;
+
+ /* release other node's bitmap lock if they are existed */
+ if (cinfo->other_bitmap_lockres) {
+ for (i = 0; i < mddev->bitmap_info.nodes - 1; i++) {
+ if (cinfo->other_bitmap_lockres[i]) {
+ dlm_unlock_sync(cinfo->other_bitmap_lockres[i]);
+ lockres_free(cinfo->other_bitmap_lockres[i]);
+ }
+ }
+ kfree(cinfo->other_bitmap_lockres);
+ }
+}
+
static int gather_bitmaps(struct md_rdev *rdev)
{
int sn, err;
@@ -1099,6 +1154,8 @@ static struct md_cluster_operations cluster_ops = {
.new_disk_ack = new_disk_ack,
.remove_disk = remove_disk,
.gather_bitmaps = gather_bitmaps,
+ .lock_all_bitmaps = lock_all_bitmaps,
+ .unlock_all_bitmaps = unlock_all_bitmaps,
};
static int __init cluster_init(void)
diff --git a/drivers/md/md-cluster.h b/drivers/md/md-cluster.h
index e75ea26..45ce6c9 100644
--- a/drivers/md/md-cluster.h
+++ b/drivers/md/md-cluster.h
@@ -24,6 +24,8 @@ struct md_cluster_operations {
int (*new_disk_ack)(struct mddev *mddev, bool ack);
int (*remove_disk)(struct mddev *mddev, struct md_rdev *rdev);
int (*gather_bitmaps)(struct md_rdev *rdev);
+ int (*lock_all_bitmaps)(struct mddev *mddev);
+ void (*unlock_all_bitmaps)(struct mddev *mddev);
};
#endif /* _MD_CLUSTER_H */
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 06066e67..47ee8ae 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -6586,6 +6586,20 @@ static int update_array_info(struct mddev *mddev, mdu_array_info_t *info)
rv = -EINVAL;
goto err;
}
+ if (mddev->bitmap_info.nodes) {
+ /* hold PW on all the bitmap lock */
+ if (md_cluster_ops->lock_all_bitmaps(mddev) <= 0) {
+ printk("%s:%d can't change bitmap to none since the"
+ " array is using by more than one node\n",
+ __func__, __LINE__);
+ rv = -EPERM;
+ md_cluster_ops->unlock_all_bitmaps(mddev);
+ goto err;
+ }
+
+ mddev->bitmap_info.nodes = 0;
+ md_cluster_ops->leave(mddev);
+ }
mddev->pers->quiesce(mddev, 1);
bitmap_destroy(mddev);
mddev->pers->quiesce(mddev, 0);
--
2.1.4
^ permalink raw reply related
* [v2 PATCH 2/5] md-cluster: Protect communication with mutexes
From: Guoqing Jiang @ 2015-11-20 8:27 UTC (permalink / raw)
To: neilb, rgoldwyn; +Cc: linux-raid, Guoqing Jiang, Goldwyn Rodrigues
In-Reply-To: <87fv0ariab.fsf@notabene.neil.brown.name>
Communication can happen through multiple threads. It is possible that
one thread steps over another threads sequence. So, we use mutexes to
protect both the send and receive sequences.
Send communication is locked through state bit, MD_CLUSTER_SEND_LOCK.
Communication is locked with bit manipulation in order to allow
"lock and hold" for the add operation. In case of an add operation,
if the lock is held, MD_CLUSTER_SEND_LOCKED_ALREADY is set.
When md_update_sb() calls metadata_update_start(), it checks
(in a single statement to avoid races), if the communication
is already locked. If yes, it merely returns zero, else it
locks the token lockresource.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
v2 changes:
1. replace send_mutex with wait_queue.
2. change MD_CLUSTER_COMM_LOCKED to MD_CLUSTER_SEND_LOCKED_ALREADY,
and set it later with add_new_disk.
3. add a new lock_token and change lock_comm to make it better
suit for the specific cases of locking communication.
drivers/md/md-cluster.c | 72 ++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 62 insertions(+), 10 deletions(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 89343c3..430b139 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -48,12 +48,26 @@ struct resync_info {
#define MD_CLUSTER_SUSPEND_READ_BALANCING 2
#define MD_CLUSTER_BEGIN_JOIN_CLUSTER 3
+/* Lock the send communication. This is done through
+ * bit manipulation as opposed to a mutex in order to
+ * accomodate lock and hold. See next comment.
+ */
+#define MD_CLUSTER_SEND_LOCK 4
+/* If cluster operations must lock the communication channel,
+ * so as to perform extra operations (and no other operation
+ * is allowed on the MD, such as adding a disk. Token needs
+ * to be locked and held until the operation completes with
+ * a md_update_sb(), which would eventually release the lock.
+ */
+#define MD_CLUSTER_SEND_LOCKED_ALREADY 5
+
struct md_cluster_info {
/* dlm lock space and resources for clustered raid. */
dlm_lockspace_t *lockspace;
int slot_number;
struct completion completion;
+ struct mutex recv_mutex;
struct dlm_lock_resource *bitmap_lockres;
struct dlm_lock_resource *resync_lockres;
struct list_head suspend_list;
@@ -67,6 +81,7 @@ struct md_cluster_info {
struct dlm_lock_resource *no_new_dev_lockres;
struct md_thread *recv_thread;
struct completion newdisk_completion;
+ wait_queue_head_t wait;
unsigned long state;
};
@@ -507,6 +522,7 @@ static void recv_daemon(struct md_thread *thread)
struct cluster_msg msg;
int ret;
+ mutex_lock(&cinfo->recv_mutex);
/*get CR on Message*/
if (dlm_lock_sync(message_lockres, DLM_LOCK_CR)) {
pr_err("md/raid1:failed to get CR on MESSAGE\n");
@@ -533,33 +549,45 @@ static void recv_daemon(struct md_thread *thread)
ret = dlm_unlock_sync(message_lockres);
if (unlikely(ret != 0))
pr_info("unlock msg failed return %d\n", ret);
+ mutex_unlock(&cinfo->recv_mutex);
}
-/* lock_comm()
+/* lock_token()
* Takes the lock on the TOKEN lock resource so no other
* node can communicate while the operation is underway.
- * If called again, and the TOKEN lock is alread in EX mode
- * return success. However, care must be taken that unlock_comm()
- * is called only once.
*/
-static int lock_comm(struct md_cluster_info *cinfo)
+static int lock_token(struct md_cluster_info *cinfo)
{
int error;
- if (cinfo->token_lockres->mode == DLM_LOCK_EX)
- return 0;
-
error = dlm_lock_sync(cinfo->token_lockres, DLM_LOCK_EX);
if (error)
pr_err("md-cluster(%s:%d): failed to get EX on TOKEN (%d)\n",
__func__, __LINE__, error);
+
+ /* Lock the receive sequence */
+ mutex_lock(&cinfo->recv_mutex);
return error;
}
+/* lock_comm()
+ * Sets the MD_CLUSTER_SEND_LOCK bit to lock the send channel.
+ */
+static int lock_comm(struct md_cluster_info *cinfo)
+{
+ wait_event(cinfo->wait,
+ !test_and_set_bit(MD_CLUSTER_SEND_LOCK, &cinfo->state));
+
+ return lock_token(cinfo);
+}
+
static void unlock_comm(struct md_cluster_info *cinfo)
{
WARN_ON(cinfo->token_lockres->mode != DLM_LOCK_EX);
+ mutex_unlock(&cinfo->recv_mutex);
dlm_unlock_sync(cinfo->token_lockres);
+ clear_bit(MD_CLUSTER_SEND_LOCK, &cinfo->state);
+ wake_up(&cinfo->wait);
}
/* __sendmsg()
@@ -712,6 +740,8 @@ static int join(struct mddev *mddev, int nodes)
spin_lock_init(&cinfo->suspend_lock);
init_completion(&cinfo->completion);
set_bit(MD_CLUSTER_BEGIN_JOIN_CLUSTER, &cinfo->state);
+ init_waitqueue_head(&cinfo->wait);
+ mutex_init(&cinfo->recv_mutex);
mddev->cluster_info = cinfo;
@@ -840,9 +870,25 @@ static int slot_number(struct mddev *mddev)
return cinfo->slot_number - 1;
}
+/*
+ * Check if the communication is already locked, else lock the communication
+ * channel.
+ * If it is already locked, token is in EX mode, and hence lock_token()
+ * should not be called.
+ */
static int metadata_update_start(struct mddev *mddev)
{
- return lock_comm(mddev->cluster_info);
+ struct md_cluster_info *cinfo = mddev->cluster_info;
+
+ wait_event(cinfo->wait,
+ !test_and_set_bit(MD_CLUSTER_SEND_LOCK, &cinfo->state) ||
+ test_and_clear_bit(MD_CLUSTER_SEND_LOCKED_ALREADY, &cinfo->state));
+
+ /* If token is already locked, return 0 */
+ if (cinfo->token_lockres->mode == DLM_LOCK_EX)
+ return 0;
+
+ return lock_token(cinfo);
}
static int metadata_update_finish(struct mddev *mddev)
@@ -867,6 +913,7 @@ static int metadata_update_finish(struct mddev *mddev)
ret = __sendmsg(cinfo, &cmsg);
} else
pr_warn("md-cluster: No good device id found to send\n");
+ clear_bit(MD_CLUSTER_SEND_LOCKED_ALREADY, &cinfo->state);
unlock_comm(cinfo);
return ret;
}
@@ -874,6 +921,7 @@ static int metadata_update_finish(struct mddev *mddev)
static void metadata_update_cancel(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
+ clear_bit(MD_CLUSTER_SEND_LOCKED_ALREADY, &cinfo->state);
unlock_comm(cinfo);
}
@@ -967,14 +1015,18 @@ static int add_new_disk(struct mddev *mddev, struct md_rdev *rdev)
ret = -ENOENT;
if (ret)
unlock_comm(cinfo);
- else
+ else {
dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR);
+ set_bit(MD_CLUSTER_SEND_LOCKED_ALREADY, &cinfo->state);
+ wake_up(&cinfo->wait);
+ }
return ret;
}
static void add_new_disk_cancel(struct mddev *mddev)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
+ clear_bit(MD_CLUSTER_SEND_LOCKED_ALREADY, &cinfo->state);
unlock_comm(cinfo);
}
--
2.1.4
^ permalink raw reply related
* [v2 PATCH] md-cluster: Defer MD reloading to mddev->thread
From: Guoqing Jiang @ 2015-11-20 8:25 UTC (permalink / raw)
To: neilb, rgoldwyn; +Cc: linux-raid, Guoqing Jiang, Goldwyn Rodrigues
In-Reply-To: <564163ED.9090709@suse.de>
Reloading of superblock must be performed under reconfig_mutex. However,
this cannot be done with md_reload_sb because it would deadlock with
the message DLM lock. So, we defer it in md_check_recovery() which is
executed by mddev->thread.
This introduces a new flag, MD_RELOAD_SB, which if set, will reload the
superblock. And good_device_nr is also added to 'struct mddev' which is
used to get the num of the good device within cluster raid.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
v2 changes: remove atomic and add good_device_nr to struct mddev
drivers/md/md-cluster.c | 4 +++-
drivers/md/md.c | 4 ++++
drivers/md/md.h | 4 ++++
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index b58374d..89343c3 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -431,8 +431,10 @@ static void process_add_new_disk(struct mddev *mddev, struct cluster_msg *cmsg)
static void process_metadata_update(struct mddev *mddev, struct cluster_msg *msg)
{
struct md_cluster_info *cinfo = mddev->cluster_info;
- md_reload_sb(mddev, le32_to_cpu(msg->raid_slot));
+ mddev->good_device_nr = le32_to_cpu(msg->raid_slot);
+ set_bit(MD_RELOAD_SB, &mddev->flags);
dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR);
+ md_wakeup_thread(mddev->thread);
}
static void process_remove_disk(struct mddev *mddev, struct cluster_msg *msg)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 7ab9ed9..06066e67 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -8260,6 +8260,7 @@ void md_check_recovery(struct mddev *mddev)
(mddev->flags & MD_UPDATE_SB_FLAGS & ~ (1<<MD_CHANGE_PENDING)) ||
test_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
test_bit(MD_RECOVERY_DONE, &mddev->recovery) ||
+ test_bit(MD_RELOAD_SB, &mddev->flags) ||
(mddev->external == 0 && mddev->safemode == 1) ||
(mddev->safemode == 2 && ! atomic_read(&mddev->writes_pending)
&& !mddev->in_sync && mddev->recovery_cp == MaxSector)
@@ -8308,6 +8309,9 @@ void md_check_recovery(struct mddev *mddev)
rdev->raid_disk < 0)
md_kick_rdev_from_array(rdev);
}
+
+ if (test_and_clear_bit(MD_RELOAD_SB, &mddev->flags))
+ md_reload_sb(mddev, mddev->good_device_nr);
}
if (!mddev->external) {
diff --git a/drivers/md/md.h b/drivers/md/md.h
index f5b9aad..7a6da5c 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -235,6 +235,9 @@ struct mddev {
*/
#define MD_JOURNAL_CLEAN 5 /* A raid with journal is already clean */
#define MD_HAS_JOURNAL 6 /* The raid array has journal feature set */
+#define MD_RELOAD_SB 7 /* Reload the superblock because another node
+ * updated it.
+ */
int suspended;
atomic_t active_io;
@@ -465,6 +468,7 @@ struct mddev {
struct work_struct event_work; /* used by dm to report failure event */
void (*sync_super)(struct mddev *mddev, struct md_rdev *rdev);
struct md_cluster_info *cluster_info;
+ unsigned int good_device_nr; /* good device num within cluster raid */
};
static inline int __must_check mddev_lock(struct mddev *mddev)
--
2.1.4
^ permalink raw reply related
* Re: Ubuntu crashed during RAID6 grow
From: Mikael Abrahamsson @ 2015-11-20 6:48 UTC (permalink / raw)
To: Nathan Peterson; +Cc: linux-raid
In-Reply-To: <loom.20151119T222250-428@post.gmane.org>
On Thu, 19 Nov 2015, Nathan Peterson wrote:
> Details after the grow:
>
> md126 : active (auto-read-only) raid6 sdj1[2] sdl1[3] sdf1[9] sdn1[11] (S) sdi1[1] sdg1[10] sde1[8] sdh1[0] sdk1[4] sdo1[12]
>
> Delta Devices : 1, (8->9)
Please post mdadm --examine from all component drives, but I'd say if you
try to access your data everything should be there and the grow should
continue once the array is not auto-read-only anymore (guess from my side
though).
If you look here:
http://www.linuxquestions.org/questions/linux-server-73/raid-5-mdadm-grow-interrupted-what-to-do-next-602671/
It's a bit old, but here there is also "delta devices" and on the
component drive there is a "reshape position". If you see that on your
component drives as well, just try accessing the array and see if the
reshape continues.
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply
* Ubuntu crashed during RAID6 grow
From: Nathan Peterson @ 2015-11-19 21:37 UTC (permalink / raw)
To: linux-raid
Hello All,
I was attempting to add another hdd to my RAID6. At about 4% into the
operation the computer froze and crashed. After rebooting and finding
out it was a CPU overheating issue I fixed that. The array now shows as
the new drive has been officially added. That cannot be possible. I am
hoping to get help on assembling this array properly so I can mount and
see my data again.
The details:
Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-31-generic x86_64)
mdadm - v3.2.5 - 18th May 2012
Details before the grow:
cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
md127 : active raid6 sdo1[12](S) sdn1[11](S) sdg1[10] sdi1[1] sdj1[2]
sdh1[0] sdk1[4] sdl1[3] sdf1[9] sde1[8]
11720291328 blocks super 1.2 level 6, 512k chunk, algorithm 2
[8/8] [UUUUUUUU]
/dev/md127:
Version : 1.2
Creation Time : Fri Nov 1 07:52:30 2013
Raid Level : raid6
Array Size : 11720291328 (11177.34 GiB 12001.58 GB)
Used Dev Size : 1953381888 (1862.89 GiB 2000.26 GB)
Raid Devices : 8
Total Devices : 10
Persistence : Superblock is persistent
Update Time : Wed Nov 11 17:20:05 2015
State : clean
Active Devices : 8
Working Devices : 10
Failed Devices : 0
Spare Devices : 2
Layout : left-symmetric
Chunk Size : 512K
Name : lithia:6 (local to host lithia)
UUID : 62373b29:22c2640c:33b5a642:9b079bff
Events : 656743
Number Major Minor RaidDevice State
0 8 113 0 active sync /dev/sdh1
1 8 129 1 active sync /dev/sdi1
2 8 145 2 active sync /dev/sdj1
3 8 177 3 active sync /dev/sdl1
4 8 161 4 active sync /dev/sdk1
10 8 97 5 active sync /dev/sdg1
8 8 65 6 active sync /dev/sde1
9 8 81 7 active sync /dev/sdf1
11 8 209 - spare /dev/sdn1
12 8 225 - spare /dev/sdo1
Details after the grow:
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
md126 : active (auto-read-only) raid6 sdj1[2] sdl1[3] sdf1[9] sdn1[11]
(S) sdi1[1] sdg1[10] sde1[8] sdh1[0] sdk1[4] sdo1[12]
11720291328 blocks super 1.2 level 6, 512k chunk, algorithm 2
[9/9] [UUUUUUUUU]
/dev/md126:
Version : 1.2
Creation Time : Fri Nov 1 07:52:30 2013
Raid Level : raid6
Array Size : 11720291328 (11177.34 GiB 12001.58 GB)
Used Dev Size : 1953381888 (1862.89 GiB 2000.26 GB)
Raid Devices : 9
Total Devices : 10
Persistence : Superblock is persistent
Update Time : Sat Nov 14 05:14:17 2015
State : clean
Active Devices : 9
Working Devices : 10
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Delta Devices : 1, (8->9)
Name : lithia:6 (local to host lithia)
UUID : 62373b29:22c2640c:33b5a642:9b079bff
Events : 657251
Number Major Minor RaidDevice State
0 8 113 0 active sync /dev/sdh1
1 8 129 1 active sync /dev/sdi1
2 8 145 2 active sync /dev/sdj1
3 8 177 3 active sync /dev/sdl1
4 8 161 4 active sync /dev/sdk1
10 8 97 5 active sync /dev/sdg1
8 8 65 6 active sync /dev/sde1
9 8 81 7 active sync /dev/sdf1
12 8 225 8 active sync /dev/sdo1
11 8 209 - spare /dev/sdn1
Sadly, there was no --backup-file created during the grow.
Please let me know if you need more details/information. Anything would
be greatly appreciated. Sorry if this has been asked before or I come
off as a dunce.
-Nathan
^ permalink raw reply
* [RFC PATCH] block: introduce poison tracking for block devices
From: Vishal Verma @ 2015-11-19 21:15 UTC (permalink / raw)
To: linux-nvdimm
Cc: Vishal Verma, linux-block, linux-raid, linux-scsi, Jens Axboe,
NeilBrown
This patch copies the badblock management code from md-raid to use it
for tracking bad/'poison' sectors on a per-block device level.
NVDIMM devices, which behave more like DRAM, may develop bad cache
lines, or 'poison'. A block device exposed by the pmem driver can
then consume poison via a read (or write), and cause a machine check.
On platforms without machine check recovery features, this would
mean a crash.
The block device maintaining a runtime list of all known poison can
directly avoid this, and also provide a path forward to enable proper
handling/recovery for DAX faults on such a device.
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
---
This really is a copy-paste + a few modifications of the badblock management
code + sysfs representation from md.
In this RFC, I want to make sure this path sounds acceptable for the use case
described above, for NVDIMMs. Eventually, I think the md badblock management
and this should be refactored to use the same code - I think this should be
easy to do:
- move the badblocks struct and associated functions into a header file (along
the lines of include/linux/list.h)
- embed the structure into whatever needs to use this list (in case of md, this
would be 'rdev', in the nvdimm case, the gendisk)
- call the functions from badblocks.h as needed to manipulate the list.
- The sysfs show/store functions in badblocks.h would be generic variants, with
wrappers being present in md and gendisk to fit into their respective sysfs
layouts
If this looks generally reasonable, I'll post a v2 with this refactoring done.
block/genhd.c | 502 ++++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/genhd.h | 26 +++
2 files changed, 528 insertions(+)
diff --git a/block/genhd.c b/block/genhd.c
index 0c706f3..de99d28 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -23,6 +23,15 @@
#include "blk.h"
+#define BB_LEN_MASK (0x00000000000001FFULL)
+#define BB_OFFSET_MASK (0x7FFFFFFFFFFFFE00ULL)
+#define BB_ACK_MASK (0x8000000000000000ULL)
+#define BB_MAX_LEN 512
+#define BB_OFFSET(x) (((x) & BB_OFFSET_MASK) >> 9)
+#define BB_LEN(x) (((x) & BB_LEN_MASK) + 1)
+#define BB_ACK(x) (!!((x) & BB_ACK_MASK))
+#define BB_MAKE(a, l, ack) (((a)<<9) | ((l)-1) | ((u64)(!!(ack)) << 63))
+
static DEFINE_MUTEX(block_class_lock);
struct kobject *block_depr;
@@ -670,6 +679,496 @@ void del_gendisk(struct gendisk *disk)
}
EXPORT_SYMBOL(del_gendisk);
+int disk_poison_list_init(struct gendisk *disk)
+{
+ disk->plist = kmalloc(sizeof(*disk->plist), GFP_KERNEL);
+ if (!disk->plist)
+ return -ENOMEM;
+ disk->plist->count = 0;
+ disk->plist->shift = 0;
+ disk->plist->page = kmalloc(PAGE_SIZE, GFP_KERNEL);
+ seqlock_init(&disk->plist->lock);
+ if (disk->plist->page == NULL)
+ return -ENOMEM;
+
+ return 0;
+}
+EXPORT_SYMBOL(disk_poison_list_init);
+
+/* Bad block management.
+ * We can record which blocks on each device are 'bad' and so just
+ * fail those blocks, or that stripe, rather than the whole device.
+ * Entries in the bad-block table are 64bits wide. This comprises:
+ * Length of bad-range, in sectors: 0-511 for lengths 1-512
+ * Start of bad-range, sector offset, 54 bits (allows 8 exbibytes)
+ * A 'shift' can be set so that larger blocks are tracked and
+ * consequently larger devices can be covered.
+ * 'Acknowledged' flag - 1 bit. - the most significant bit.
+ *
+ * Locking of the bad-block table uses a seqlock so md_is_badblock
+ * might need to retry if it is very unlucky.
+ * We will sometimes want to check for bad blocks in a bi_end_io function,
+ * so we use the write_seqlock_irq variant.
+ *
+ * When looking for a bad block we specify a range and want to
+ * know if any block in the range is bad. So we binary-search
+ * to the last range that starts at-or-before the given endpoint,
+ * (or "before the sector after the target range")
+ * then see if it ends after the given start.
+ * We return
+ * 0 if there are no known bad blocks in the range
+ * 1 if there are known bad block which are all acknowledged
+ * -1 if there are bad blocks which have not yet been acknowledged in metadata.
+ * plus the start/length of the first bad section we overlap.
+ */
+int disk_check_poison(struct gendisk *disk, sector_t s, int sectors,
+ sector_t *first_bad, int *bad_sectors)
+{
+ struct disk_poison *bb = disk->plist;
+ int hi;
+ int lo;
+ u64 *p = bb->page;
+ int rv;
+ sector_t target = s + sectors;
+ unsigned seq;
+
+ if (bb->shift > 0) {
+ /* round the start down, and the end up */
+ s >>= bb->shift;
+ target += (1<<bb->shift) - 1;
+ target >>= bb->shift;
+ sectors = target - s;
+ }
+ /* 'target' is now the first block after the bad range */
+
+retry:
+ seq = read_seqbegin(&bb->lock);
+ lo = 0;
+ rv = 0;
+ hi = bb->count;
+
+ /* Binary search between lo and hi for 'target'
+ * i.e. for the last range that starts before 'target'
+ */
+ /* INVARIANT: ranges before 'lo' and at-or-after 'hi'
+ * are known not to be the last range before target.
+ * VARIANT: hi-lo is the number of possible
+ * ranges, and decreases until it reaches 1
+ */
+ while (hi - lo > 1) {
+ int mid = (lo + hi) / 2;
+ sector_t a = BB_OFFSET(p[mid]);
+ if (a < target)
+ /* This could still be the one, earlier ranges
+ * could not. */
+ lo = mid;
+ else
+ /* This and later ranges are definitely out. */
+ hi = mid;
+ }
+ /* 'lo' might be the last that started before target, but 'hi' isn't */
+ if (hi > lo) {
+ /* need to check all range that end after 's' to see if
+ * any are unacknowledged.
+ */
+ while (lo >= 0 &&
+ BB_OFFSET(p[lo]) + BB_LEN(p[lo]) > s) {
+ if (BB_OFFSET(p[lo]) < target) {
+ /* starts before the end, and finishes after
+ * the start, so they must overlap
+ */
+ if (rv != -1 && BB_ACK(p[lo]))
+ rv = 1;
+ else
+ rv = -1;
+ *first_bad = BB_OFFSET(p[lo]);
+ *bad_sectors = BB_LEN(p[lo]);
+ }
+ lo--;
+ }
+ }
+
+ if (read_seqretry(&bb->lock, seq))
+ goto retry;
+
+ return rv;
+}
+EXPORT_SYMBOL_GPL(disk_check_poison);
+
+/*
+ * Add a range of bad blocks to the table.
+ * This might extend the table, or might contract it
+ * if two adjacent ranges can be merged.
+ * We binary-search to find the 'insertion' point, then
+ * decide how best to handle it.
+ */
+int disk_add_poison(struct gendisk *disk, sector_t s, int sectors,
+ int acknowledged)
+{
+ struct disk_poison *bb = disk->plist;
+ u64 *p;
+ int lo, hi;
+ int rv = 1;
+ unsigned long flags;
+
+ if (bb->shift < 0)
+ /* badblocks are disabled */
+ return 0;
+
+ if (bb->shift) {
+ /* round the start down, and the end up */
+ sector_t next = s + sectors;
+ s >>= bb->shift;
+ next += (1<<bb->shift) - 1;
+ next >>= bb->shift;
+ sectors = next - s;
+ }
+
+ write_seqlock_irqsave(&bb->lock, flags);
+
+ p = bb->page;
+ lo = 0;
+ hi = bb->count;
+ /* Find the last range that starts at-or-before 's' */
+ while (hi - lo > 1) {
+ int mid = (lo + hi) / 2;
+ sector_t a = BB_OFFSET(p[mid]);
+ if (a <= s)
+ lo = mid;
+ else
+ hi = mid;
+ }
+ if (hi > lo && BB_OFFSET(p[lo]) > s)
+ hi = lo;
+
+ if (hi > lo) {
+ /* we found a range that might merge with the start
+ * of our new range
+ */
+ sector_t a = BB_OFFSET(p[lo]);
+ sector_t e = a + BB_LEN(p[lo]);
+ int ack = BB_ACK(p[lo]);
+ if (e >= s) {
+ /* Yes, we can merge with a previous range */
+ if (s == a && s + sectors >= e)
+ /* new range covers old */
+ ack = acknowledged;
+ else
+ ack = ack && acknowledged;
+
+ if (e < s + sectors)
+ e = s + sectors;
+ if (e - a <= BB_MAX_LEN) {
+ p[lo] = BB_MAKE(a, e-a, ack);
+ s = e;
+ } else {
+ /* does not all fit in one range,
+ * make p[lo] maximal
+ */
+ if (BB_LEN(p[lo]) != BB_MAX_LEN)
+ p[lo] = BB_MAKE(a, BB_MAX_LEN, ack);
+ s = a + BB_MAX_LEN;
+ }
+ sectors = e - s;
+ }
+ }
+ if (sectors && hi < bb->count) {
+ /* 'hi' points to the first range that starts after 's'.
+ * Maybe we can merge with the start of that range */
+ sector_t a = BB_OFFSET(p[hi]);
+ sector_t e = a + BB_LEN(p[hi]);
+ int ack = BB_ACK(p[hi]);
+ if (a <= s + sectors) {
+ /* merging is possible */
+ if (e <= s + sectors) {
+ /* full overlap */
+ e = s + sectors;
+ ack = acknowledged;
+ } else
+ ack = ack && acknowledged;
+
+ a = s;
+ if (e - a <= BB_MAX_LEN) {
+ p[hi] = BB_MAKE(a, e-a, ack);
+ s = e;
+ } else {
+ p[hi] = BB_MAKE(a, BB_MAX_LEN, ack);
+ s = a + BB_MAX_LEN;
+ }
+ sectors = e - s;
+ lo = hi;
+ hi++;
+ }
+ }
+ if (sectors == 0 && hi < bb->count) {
+ /* we might be able to combine lo and hi */
+ /* Note: 's' is at the end of 'lo' */
+ sector_t a = BB_OFFSET(p[hi]);
+ int lolen = BB_LEN(p[lo]);
+ int hilen = BB_LEN(p[hi]);
+ int newlen = lolen + hilen - (s - a);
+ if (s >= a && newlen < BB_MAX_LEN) {
+ /* yes, we can combine them */
+ int ack = BB_ACK(p[lo]) && BB_ACK(p[hi]);
+ p[lo] = BB_MAKE(BB_OFFSET(p[lo]), newlen, ack);
+ memmove(p + hi, p + hi + 1,
+ (bb->count - hi - 1) * 8);
+ bb->count--;
+ }
+ }
+ while (sectors) {
+ /* didn't merge (it all).
+ * Need to add a range just before 'hi' */
+ if (bb->count >= DISK_MAX_POISON) {
+ /* No room for more */
+ rv = 0;
+ break;
+ } else {
+ int this_sectors = sectors;
+ memmove(p + hi + 1, p + hi,
+ (bb->count - hi) * 8);
+ bb->count++;
+
+ if (this_sectors > BB_MAX_LEN)
+ this_sectors = BB_MAX_LEN;
+ p[hi] = BB_MAKE(s, this_sectors, acknowledged);
+ sectors -= this_sectors;
+ s += this_sectors;
+ }
+ }
+
+ bb->changed = 1;
+ if (!acknowledged)
+ bb->unacked_exist = 1;
+ write_sequnlock_irqrestore(&bb->lock, flags);
+
+ /* Make sure they get written out promptly */
+ /* TODO sysfs_notify_dirent_safe(disk->sysfs_state); */
+
+ return rv;
+}
+EXPORT_SYMBOL_GPL(disk_add_poison);
+
+/*
+ * Remove a range of bad blocks from the table.
+ * This may involve extending the table if we spilt a region,
+ * but it must not fail. So if the table becomes full, we just
+ * drop the remove request.
+ */
+int disk_clear_poison(struct gendisk *disk, sector_t s, int sectors)
+{
+ struct disk_poison *bb = disk->plist;
+ u64 *p;
+ int lo, hi;
+ sector_t target = s + sectors;
+ int rv = 0;
+
+ if (bb->shift > 0) {
+ /* When clearing we round the start up and the end down.
+ * This should not matter as the shift should align with
+ * the block size and no rounding should ever be needed.
+ * However it is better the think a block is bad when it
+ * isn't than to think a block is not bad when it is.
+ */
+ s += (1<<bb->shift) - 1;
+ s >>= bb->shift;
+ target >>= bb->shift;
+ sectors = target - s;
+ }
+
+ write_seqlock_irq(&bb->lock);
+
+ p = bb->page;
+ lo = 0;
+ hi = bb->count;
+ /* Find the last range that starts before 'target' */
+ while (hi - lo > 1) {
+ int mid = (lo + hi) / 2;
+ sector_t a = BB_OFFSET(p[mid]);
+ if (a < target)
+ lo = mid;
+ else
+ hi = mid;
+ }
+ if (hi > lo) {
+ /* p[lo] is the last range that could overlap the
+ * current range. Earlier ranges could also overlap,
+ * but only this one can overlap the end of the range.
+ */
+ if (BB_OFFSET(p[lo]) + BB_LEN(p[lo]) > target) {
+ /* Partial overlap, leave the tail of this range */
+ int ack = BB_ACK(p[lo]);
+ sector_t a = BB_OFFSET(p[lo]);
+ sector_t end = a + BB_LEN(p[lo]);
+
+ if (a < s) {
+ /* we need to split this range */
+ if (bb->count >= DISK_MAX_POISON) {
+ rv = -ENOSPC;
+ goto out;
+ }
+ memmove(p+lo+1, p+lo, (bb->count - lo) * 8);
+ bb->count++;
+ p[lo] = BB_MAKE(a, s-a, ack);
+ lo++;
+ }
+ p[lo] = BB_MAKE(target, end - target, ack);
+ /* there is no longer an overlap */
+ hi = lo;
+ lo--;
+ }
+ while (lo >= 0 &&
+ BB_OFFSET(p[lo]) + BB_LEN(p[lo]) > s) {
+ /* This range does overlap */
+ if (BB_OFFSET(p[lo]) < s) {
+ /* Keep the early parts of this range. */
+ int ack = BB_ACK(p[lo]);
+ sector_t start = BB_OFFSET(p[lo]);
+ p[lo] = BB_MAKE(start, s - start, ack);
+ /* now low doesn't overlap, so.. */
+ break;
+ }
+ lo--;
+ }
+ /* 'lo' is strictly before, 'hi' is strictly after,
+ * anything between needs to be discarded
+ */
+ if (hi - lo > 1) {
+ memmove(p+lo+1, p+hi, (bb->count - hi) * 8);
+ bb->count -= (hi - lo - 1);
+ }
+ }
+
+ bb->changed = 1;
+out:
+ write_sequnlock_irq(&bb->lock);
+ return rv;
+}
+EXPORT_SYMBOL_GPL(disk_clear_poison);
+
+/*
+ * Acknowledge all bad blocks in a list.
+ * This only succeeds if ->changed is clear. It is used by
+ * in-kernel metadata updates
+ */
+void disk_ack_all_poison(struct gendisk *disk)
+{
+ struct disk_poison *bb = disk->plist;
+
+ if (bb->page == NULL || bb->changed)
+ /* no point even trying */
+ return;
+ write_seqlock_irq(&bb->lock);
+
+ if (bb->changed == 0 && bb->unacked_exist) {
+ u64 *p = bb->page;
+ int i;
+ for (i = 0; i < bb->count ; i++) {
+ if (!BB_ACK(p[i])) {
+ sector_t start = BB_OFFSET(p[i]);
+ int len = BB_LEN(p[i]);
+ p[i] = BB_MAKE(start, len, 1);
+ }
+ }
+ bb->unacked_exist = 0;
+ }
+ write_sequnlock_irq(&bb->lock);
+}
+EXPORT_SYMBOL_GPL(disk_ack_all_poison);
+
+/* sysfs access to bad-blocks list.
+ * We present two files.
+ * 'bad-blocks' lists sector numbers and lengths of ranges that
+ * are recorded as bad. The list is truncated to fit within
+ * the one-page limit of sysfs.
+ * Writing "sector length" to this file adds an acknowledged
+ * bad block list.
+ * 'unacknowledged-bad-blocks' lists bad blocks that have not yet
+ * been acknowledged. Writing to this file adds bad blocks
+ * without acknowledging them. This is largely for testing.
+ */
+
+static ssize_t poison_list_show(struct device *dev,
+ struct device_attribute *attr,
+ char *page)
+{
+ struct gendisk *disk = dev_to_disk(dev);
+ struct disk_poison *bb = disk->plist;
+ size_t len;
+ int i;
+ u64 *p = bb->page;
+ unsigned seq;
+
+ if (bb->shift < 0)
+ return 0;
+
+retry:
+ seq = read_seqbegin(&bb->lock);
+
+ len = 0;
+ i = 0;
+
+ while (len < PAGE_SIZE && i < bb->count) {
+ sector_t s = BB_OFFSET(p[i]);
+ unsigned int length = BB_LEN(p[i]);
+
+ i++;
+ len += snprintf(page+len, PAGE_SIZE-len, "%llu %u\n",
+ (unsigned long long)s << bb->shift,
+ length << bb->shift);
+ }
+
+ if (read_seqretry(&bb->lock, seq))
+ goto retry;
+
+ return len;
+}
+
+#define DO_DEBUG 1
+
+static ssize_t poison_list_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *page, size_t len)
+{
+ struct gendisk *disk = dev_to_disk(dev);
+ unsigned long long sector;
+ int length;
+ char newline;
+#ifdef DO_DEBUG
+ /* Allow clearing via sysfs *only* for testing/debugging.
+ * Normally only a successful write may clear a badblock
+ */
+ int clear = 0;
+ if (page[0] == '-') {
+ clear = 1;
+ page++;
+ }
+#endif /* DO_DEBUG */
+
+ switch (sscanf(page, "%llu %d%c", §or, &length, &newline)) {
+ case 3:
+ if (newline != '\n')
+ return -EINVAL;
+ case 2:
+ if (length <= 0)
+ return -EINVAL;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+#ifdef DO_DEBUG
+ if (clear) {
+ disk_clear_poison(disk, sector, length);
+ return len;
+ }
+#endif /* DO_DEBUG */
+ if (disk_add_poison(disk, sector, length, 1))
+ return len;
+ else
+ return -ENOSPC;
+}
+
/**
* get_gendisk - get partitioning information for a given device
* @devt: device to get partitioning information for
@@ -988,6 +1487,8 @@ static DEVICE_ATTR(discard_alignment, S_IRUGO, disk_discard_alignment_show,
static DEVICE_ATTR(capability, S_IRUGO, disk_capability_show, NULL);
static DEVICE_ATTR(stat, S_IRUGO, part_stat_show, NULL);
static DEVICE_ATTR(inflight, S_IRUGO, part_inflight_show, NULL);
+static DEVICE_ATTR(poison_list, S_IRUGO | S_IWUSR, poison_list_show,
+ poison_list_store);
#ifdef CONFIG_FAIL_MAKE_REQUEST
static struct device_attribute dev_attr_fail =
__ATTR(make-it-fail, S_IRUGO|S_IWUSR, part_fail_show, part_fail_store);
@@ -1009,6 +1510,7 @@ static struct attribute *disk_attrs[] = {
&dev_attr_capability.attr,
&dev_attr_stat.attr,
&dev_attr_inflight.attr,
+ &dev_attr_poison_list.attr,
#ifdef CONFIG_FAIL_MAKE_REQUEST
&dev_attr_fail.attr,
#endif
diff --git a/include/linux/genhd.h b/include/linux/genhd.h
index 2adbfa6..9acfe1b 100644
--- a/include/linux/genhd.h
+++ b/include/linux/genhd.h
@@ -163,6 +163,24 @@ struct disk_part_tbl {
struct disk_events;
+#define DISK_MAX_POISON (PAGE_SIZE/8)
+
+struct disk_poison {
+ int count; /* count of bad blocks */
+ int unacked_exist; /* there probably are unacknowledged
+ * bad blocks. This is only cleared
+ * when a read discovers none
+ */
+ int shift; /* shift from sectors to block size
+ * a -ve shift means badblocks are
+ * disabled.*/
+ u64 *page; /* badblock list */
+ int changed;
+ seqlock_t lock;
+ sector_t sector;
+ sector_t size; /* in sectors */
+};
+
struct gendisk {
/* major, first_minor and minors are input parameters only,
* don't use directly. Use disk_devt() and disk_max_parts().
@@ -201,6 +219,7 @@ struct gendisk {
struct blk_integrity *integrity;
#endif
int node_id;
+ struct disk_poison *plist;
};
static inline struct gendisk *part_to_disk(struct hd_struct *part)
@@ -434,6 +453,13 @@ extern void disk_block_events(struct gendisk *disk);
extern void disk_unblock_events(struct gendisk *disk);
extern void disk_flush_events(struct gendisk *disk, unsigned int mask);
extern unsigned int disk_clear_events(struct gendisk *disk, unsigned int mask);
+extern int disk_poison_list_init(struct gendisk *disk);
+extern int disk_check_poison(struct gendisk *disk, sector_t s, int sectors,
+ sector_t *first_bad, int *bad_sectors);
+extern int disk_add_poison(struct gendisk *disk, sector_t s, int sectors,
+ int acknowledged);
+extern int disk_clear_poison(struct gendisk *disk, sector_t s, int sectors);
+extern void disk_ack_all_poison(struct gendisk *disk);
/* drivers/char/random.c */
extern void add_disk_randomness(struct gendisk *disk);
--
2.5.0
^ permalink raw reply related
* Re: [PATCH v3 0/7] User namespace mount updates
From: Seth Forshee @ 2015-11-19 16:31 UTC (permalink / raw)
To: Octavian Purdila
Cc: Richard Weinberger, Al Viro, Eric W. Biederman, linux-bcache,
device-mapper development, linux-raid@vger.kernel.org,
linux-mtd@lists.infradead.org, linux-fsdevel, LSM, selinux,
Serge Hallyn, Andy Lutomirski, LKML, Theodore Ts'o
In-Reply-To: <CAE1zot+G=SohXzuV6Gs7APdCo5pu9xLwE38kZKNUuWRnSh++BA@mail.gmail.com>
On Thu, Nov 19, 2015 at 06:19:10PM +0200, Octavian Purdila wrote:
> On Thu, Nov 19, 2015 at 5:23 PM, Seth Forshee
> <seth.forshee@canonical.com> wrote:
> > On Wed, Nov 18, 2015 at 12:00:17AM +0200, Octavian Purdila wrote:
> >> On Tue, Nov 17, 2015 at 10:12 PM, Richard Weinberger <richard@nod.at> wrote:
> >> > Am 17.11.2015 um 20:25 schrieb Octavian Purdila:
> >> >> On Tue, Nov 17, 2015 at 9:21 PM, Seth Forshee
> >> >> <seth.forshee@canonical.com> wrote:
> >> >>>
> >> >>> On Tue, Nov 17, 2015 at 08:12:31PM +0100, Richard Weinberger wrote:
> >> >>>> On Tue, Nov 17, 2015 at 7:34 PM, Seth Forshee
> >> >>>> <seth.forshee@canonical.com> wrote:
> >> >>>>> On Tue, Nov 17, 2015 at 05:55:06PM +0000, Al Viro wrote:
> >> >>>>>> On Tue, Nov 17, 2015 at 11:25:51AM -0600, Seth Forshee wrote:
> >> >>>>>>
> >> >>>>>>> Shortly after that I plan to follow with support for ext4. I've been
> >> >>>>>>> fuzzing ext4 for a while now and it has held up well, and I'm currently
> >> >>>>>>> working on hand-crafted attacks. Ted has commented privately (to others,
> >> >>>>>>> not to me personally) that he will fix bugs for such attacks, though I
> >> >>>>>>> haven't seen any public comments to that effect.
> >> >>>>>>
> >> >>>>>> _Static_ attacks, or change-image-under-mounted-fs attacks?
> >> >>>>>
> >> >>>>> Right now only static attacks, change-image-under-mounted-fs attacks
> >> >>>>> will be next.
> >> >>>>
> >> >>>> Do we *really* need to enable unprivileged mounting of kernel filesystems?
> >> >>>> What about just enabling fuse and implement ext4 and friends as fuse
> >> >>>> filesystems?
> >> >>>> Using the approaching Linux Kernel Libary[1] this is easy.
> >> >>>
> >> >>> I haven't looked at this project, but I'm guessing that programs must be
> >> >>> written specifically to make use of it? I.e. you can't just use the
> >> >>> mount syscall, and thus all existing software still doesn't work?
> >> >>>
> >> >>
> >> >> The projects includes a lklfuse program that uses fuse to mount a
> >> >> fileystem image.
> >> >
> >> > Cool. I gave it a try.
> >> > It seems to work fine, but only if I run it in foreground (using -d)
> >> > otherwise fuse blocks every filesystem request.
> >> >
> >>
> >> Now it should work in the background as well, thanks for reporting the issue.
> >
>
> Hi Seth,
>
> > I'm playing with lklfuse now, it's surprisingly easy to get up and
> > running. I did have a few problems though that I thought you'd like to
> > know about.
> >
>
> Great, thanks for giving it a try and reporting the issues.
No problem, looks like a promising project.
> > Unfortunately I still can't run it in background mode, I get a segfault.
>
> I got it to reproduce as well now. Not sure why how it worked before,
> probably a race condition between lkl initialization and fuse calls.
>
> > It's working fine on light workloads, but I'm having issues when I start
> > trying to stress it. In a couple runs of the stress-ng filesystem
> > stressors I saw both stress-ng and lklfuse get stuck in uninterruptible
> > sleep during the first run, and during the second I got some OOM errors
> > in lklfuse followed by I/O errors and eventually a journal error that
> > cause the filesystem to go read-only.
> >
> > The command I used for the first run was:
> >
> > stress-ng --class filesystem --all 0
> >
>
> I will reproduce it and take a look.
>
> > And for the second:
> >
> > stress-ng --class filesystem --seq 0 -v -t 60
> >
> > There really wasn't anything interesting in the lklfuse output for the
> > first run, but for the second run I pasted the output here:
> > http://paste.ubuntu.com/13346993/
>
> lklfuse allocates a fixed 100MB to the kernel and this is probably not
> enough. For the short term I can add a parameter to lklfuse that
> allows the user to specify the amount of memory to allocate to lkl. A
> better fix would probably be to dynamically adjust the memory size of
> lkl. I am thinking of using the ballon virtio driver or the memory
> hotplug infrastructure. Any other suggestions?
>
> I created a couple of issues in github [1] that you can track if you
> want - I want to avoid spamming the list with reporting progress on
> them.
Makes sense, I'm watching those issues now and will direct any furure
discussion there. Thanks!
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Octavian Purdila @ 2015-11-19 16:19 UTC (permalink / raw)
To: Seth Forshee
Cc: Richard Weinberger, Al Viro, Eric W. Biederman, linux-bcache,
device-mapper development, linux-raid@vger.kernel.org,
linux-mtd@lists.infradead.org, linux-fsdevel, LSM, selinux,
Serge Hallyn, Andy Lutomirski, LKML, Theodore Ts'o
In-Reply-To: <20151119152347.GA45540@ubuntu-hedt>
On Thu, Nov 19, 2015 at 5:23 PM, Seth Forshee
<seth.forshee@canonical.com> wrote:
> On Wed, Nov 18, 2015 at 12:00:17AM +0200, Octavian Purdila wrote:
>> On Tue, Nov 17, 2015 at 10:12 PM, Richard Weinberger <richard@nod.at> wrote:
>> > Am 17.11.2015 um 20:25 schrieb Octavian Purdila:
>> >> On Tue, Nov 17, 2015 at 9:21 PM, Seth Forshee
>> >> <seth.forshee@canonical.com> wrote:
>> >>>
>> >>> On Tue, Nov 17, 2015 at 08:12:31PM +0100, Richard Weinberger wrote:
>> >>>> On Tue, Nov 17, 2015 at 7:34 PM, Seth Forshee
>> >>>> <seth.forshee@canonical.com> wrote:
>> >>>>> On Tue, Nov 17, 2015 at 05:55:06PM +0000, Al Viro wrote:
>> >>>>>> On Tue, Nov 17, 2015 at 11:25:51AM -0600, Seth Forshee wrote:
>> >>>>>>
>> >>>>>>> Shortly after that I plan to follow with support for ext4. I've been
>> >>>>>>> fuzzing ext4 for a while now and it has held up well, and I'm currently
>> >>>>>>> working on hand-crafted attacks. Ted has commented privately (to others,
>> >>>>>>> not to me personally) that he will fix bugs for such attacks, though I
>> >>>>>>> haven't seen any public comments to that effect.
>> >>>>>>
>> >>>>>> _Static_ attacks, or change-image-under-mounted-fs attacks?
>> >>>>>
>> >>>>> Right now only static attacks, change-image-under-mounted-fs attacks
>> >>>>> will be next.
>> >>>>
>> >>>> Do we *really* need to enable unprivileged mounting of kernel filesystems?
>> >>>> What about just enabling fuse and implement ext4 and friends as fuse
>> >>>> filesystems?
>> >>>> Using the approaching Linux Kernel Libary[1] this is easy.
>> >>>
>> >>> I haven't looked at this project, but I'm guessing that programs must be
>> >>> written specifically to make use of it? I.e. you can't just use the
>> >>> mount syscall, and thus all existing software still doesn't work?
>> >>>
>> >>
>> >> The projects includes a lklfuse program that uses fuse to mount a
>> >> fileystem image.
>> >
>> > Cool. I gave it a try.
>> > It seems to work fine, but only if I run it in foreground (using -d)
>> > otherwise fuse blocks every filesystem request.
>> >
>>
>> Now it should work in the background as well, thanks for reporting the issue.
>
Hi Seth,
> I'm playing with lklfuse now, it's surprisingly easy to get up and
> running. I did have a few problems though that I thought you'd like to
> know about.
>
Great, thanks for giving it a try and reporting the issues.
> Unfortunately I still can't run it in background mode, I get a segfault.
I got it to reproduce as well now. Not sure why how it worked before,
probably a race condition between lkl initialization and fuse calls.
> It's working fine on light workloads, but I'm having issues when I start
> trying to stress it. In a couple runs of the stress-ng filesystem
> stressors I saw both stress-ng and lklfuse get stuck in uninterruptible
> sleep during the first run, and during the second I got some OOM errors
> in lklfuse followed by I/O errors and eventually a journal error that
> cause the filesystem to go read-only.
>
> The command I used for the first run was:
>
> stress-ng --class filesystem --all 0
>
I will reproduce it and take a look.
> And for the second:
>
> stress-ng --class filesystem --seq 0 -v -t 60
>
> There really wasn't anything interesting in the lklfuse output for the
> first run, but for the second run I pasted the output here:
> http://paste.ubuntu.com/13346993/
lklfuse allocates a fixed 100MB to the kernel and this is probably not
enough. For the short term I can add a parameter to lklfuse that
allows the user to specify the amount of memory to allocate to lkl. A
better fix would probably be to dynamically adjust the memory size of
lkl. I am thinking of using the ballon virtio driver or the memory
hotplug infrastructure. Any other suggestions?
I created a couple of issues in github [1] that you can track if you
want - I want to avoid spamming the list with reporting progress on
them.
[1] https://github.com/lkl/linux/issues
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Seth Forshee @ 2015-11-19 15:23 UTC (permalink / raw)
To: Octavian Purdila
Cc: Richard Weinberger, Al Viro, Eric W. Biederman, linux-bcache,
device-mapper development, linux-raid@vger.kernel.org,
linux-mtd@lists.infradead.org, linux-fsdevel, LSM, selinux,
Serge Hallyn, Andy Lutomirski, LKML, Theodore Ts'o
In-Reply-To: <CAE1zotJRp3=G2Vdud7kVbkToKe2YsTF3gXUnLCNzKV6F0oumdg@mail.gmail.com>
On Wed, Nov 18, 2015 at 12:00:17AM +0200, Octavian Purdila wrote:
> On Tue, Nov 17, 2015 at 10:12 PM, Richard Weinberger <richard@nod.at> wrote:
> > Am 17.11.2015 um 20:25 schrieb Octavian Purdila:
> >> On Tue, Nov 17, 2015 at 9:21 PM, Seth Forshee
> >> <seth.forshee@canonical.com> wrote:
> >>>
> >>> On Tue, Nov 17, 2015 at 08:12:31PM +0100, Richard Weinberger wrote:
> >>>> On Tue, Nov 17, 2015 at 7:34 PM, Seth Forshee
> >>>> <seth.forshee@canonical.com> wrote:
> >>>>> On Tue, Nov 17, 2015 at 05:55:06PM +0000, Al Viro wrote:
> >>>>>> On Tue, Nov 17, 2015 at 11:25:51AM -0600, Seth Forshee wrote:
> >>>>>>
> >>>>>>> Shortly after that I plan to follow with support for ext4. I've been
> >>>>>>> fuzzing ext4 for a while now and it has held up well, and I'm currently
> >>>>>>> working on hand-crafted attacks. Ted has commented privately (to others,
> >>>>>>> not to me personally) that he will fix bugs for such attacks, though I
> >>>>>>> haven't seen any public comments to that effect.
> >>>>>>
> >>>>>> _Static_ attacks, or change-image-under-mounted-fs attacks?
> >>>>>
> >>>>> Right now only static attacks, change-image-under-mounted-fs attacks
> >>>>> will be next.
> >>>>
> >>>> Do we *really* need to enable unprivileged mounting of kernel filesystems?
> >>>> What about just enabling fuse and implement ext4 and friends as fuse
> >>>> filesystems?
> >>>> Using the approaching Linux Kernel Libary[1] this is easy.
> >>>
> >>> I haven't looked at this project, but I'm guessing that programs must be
> >>> written specifically to make use of it? I.e. you can't just use the
> >>> mount syscall, and thus all existing software still doesn't work?
> >>>
> >>
> >> The projects includes a lklfuse program that uses fuse to mount a
> >> fileystem image.
> >
> > Cool. I gave it a try.
> > It seems to work fine, but only if I run it in foreground (using -d)
> > otherwise fuse blocks every filesystem request.
> >
>
> Now it should work in the background as well, thanks for reporting the issue.
I'm playing with lklfuse now, it's surprisingly easy to get up and
running. I did have a few problems though that I thought you'd like to
know about.
Unfortunately I still can't run it in background mode, I get a segfault.
It's working fine on light workloads, but I'm having issues when I start
trying to stress it. In a couple runs of the stress-ng filesystem
stressors I saw both stress-ng and lklfuse get stuck in uninterruptible
sleep during the first run, and during the second I got some OOM errors
in lklfuse followed by I/O errors and eventually a journal error that
cause the filesystem to go read-only.
The command I used for the first run was:
stress-ng --class filesystem --all 0
And for the second:
stress-ng --class filesystem --seq 0 -v -t 60
There really wasn't anything interesting in the lklfuse output for the
first run, but for the second run I pasted the output here:
http://paste.ubuntu.com/13346993/
I still need to compare this to other fuse filesystems since I haven't
tried this kind of stress test on any others.
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Richard W.M. Jones @ 2015-11-19 15:17 UTC (permalink / raw)
To: Richard Weinberger
Cc: Colin Walters, James Morris, Al Viro, Seth Forshee,
Austin S Hemmelgarn, Eric W. Biederman, linux-bcache,
device-mapper development, linux-raid, linux-mtd, linux-fsdevel,
LSM, selinux, Serge Hallyn, Andy Lutomirski, LKML,
Theodore Ts'o
In-Reply-To: <564DE15C.3050207@nod.at>
On Thu, Nov 19, 2015 at 03:49:00PM +0100, Richard Weinberger wrote:
> Am 19.11.2015 um 15:37 schrieb Colin Walters:
> > On Thu, Nov 19, 2015, at 02:53 AM, Richard Weinberger wrote:
> >
> >> Erm, I don't want this in the kernel. That's why I've proposed the lklfuse approach.
> >
> > I already said this before but just to repeat, since I'm confused:
> >
> > How would "lklfuse" be different from http://libguestfs.org/
> > which we at Red Hat (and a number of other organizations)
> > use quite widely now for build systems, debugging etc.
>
> Currently libguestfs has a rather huge overhead because it
> boots a full virtual machine and hence a lot of communication
> is needed.
> With LKL you can use Linux as Library and link it to fuse.
> AFAIK Richard added already a LKL backend to libguestfs. :-)
Right. For the longer story, see:
https://rwmj.wordpress.com/2015/11/07/linux-kernel-library-backend-for-libguestfs/#content
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines. Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Richard Weinberger @ 2015-11-19 15:04 UTC (permalink / raw)
To: Serge E. Hallyn
Cc: James Morris, Al Viro, Seth Forshee, Austin S Hemmelgarn,
Eric W. Biederman, linux-bcache, device-mapper development,
linux-raid@vger.kernel.org, linux-mtd@lists.infradead.org,
linux-fsdevel, LSM, selinux, Serge Hallyn, Andy Lutomirski, LKML,
Theodore Ts'o
In-Reply-To: <20151119142113.GA5607@mail.hallyn.com>
Am 19.11.2015 um 15:21 schrieb Serge E. Hallyn:
> On Thu, Nov 19, 2015 at 08:53:26AM +0100, Richard Weinberger wrote:
>> Am 19.11.2015 um 08:47 schrieb James Morris:
>>> On Wed, 18 Nov 2015, Richard Weinberger wrote:
>>>
>>>> On Wed, Nov 18, 2015 at 4:13 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
>>>>> On Wed, Nov 18, 2015 at 09:05:12AM -0600, Seth Forshee wrote:
>>>>>
>>>>>> Yes, the host admin. I'm not talking about trusting the admin inside the
>>>>>> container at all.
>>>>>
>>>>> Then why not have the same host admin just plain mount it when setting the
>>>>> container up and be done with that? From the host namespace, before spawning
>>>>> the docker instance or whatever framework you are using. IDGI...
>>>>
>>>> Because hosting companies sell containers as "full virtual machines"
>>>> and customers expect to be able mount stuff like disk images they upload.
>>>
>>> I don't think this is a valid reason for merging functionality into the
>>> kernel.
>>
>> Erm, I don't want this in the kernel. That's why I've proposed the lklfuse approach.
>
> That would require the fuse-in-containers functionality, right?
Correct.
Still a less large attack surface. :-)
Thanks,
//richard
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Serge E. Hallyn @ 2015-11-19 14:58 UTC (permalink / raw)
To: Al Viro
Cc: Seth Forshee, Austin S Hemmelgarn, Eric W. Biederman,
linux-bcache, dm-devel, linux-raid, linux-mtd, linux-fsdevel,
linux-security-module, selinux, Serge Hallyn, Andy Lutomirski,
linux-kernel, Theodore Ts'o, James Bottomley
In-Reply-To: <20151118151335.GD22011@ZenIV.linux.org.uk>
On Wed, Nov 18, 2015 at 03:13:35PM +0000, Al Viro wrote:
> On Wed, Nov 18, 2015 at 09:05:12AM -0600, Seth Forshee wrote:
>
> > Yes, the host admin. I'm not talking about trusting the admin inside the
> > container at all.
>
> Then why not have the same host admin just plain mount it when setting the
> container up and be done with that? From the host namespace, before spawning
> the docker instance or whatever framework you are using. IDGI...
fwiw one example use case is building a livecd or vm image from inside
container, on a system where each piece of functionality is
compartmentalized in a separate container.
For specific programs we can engineer them to call out to a helper
in the init user_ns, but that doesn't work for existing tools.
James Bottomley a year or two ago had mentioned the idea of using
seccomp to have the mount syscall in a container trap into a
init_user_ns helper, but we'd have to ptrace the whole container
for its whole lifetime to do that, or use SECCOMP_RET_TRAP with
a LD_PRELOAD that does goes through a proxy to do the remote
request (which becomes infeasible bc LD_PRELOAD won't stick).
If we did enable anything but FUSE I would hope each fs would have
a separate sysctl to enable non-init-userns mounts. Then the admin
could enable them just as they do the automount of whatever garbage
is on a usb stick found lying on the trade floor.
-serge
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Richard Weinberger @ 2015-11-19 14:49 UTC (permalink / raw)
To: Colin Walters, James Morris
Cc: Al Viro, Seth Forshee, Austin S Hemmelgarn, Eric W. Biederman,
linux-bcache, device-mapper development, linux-raid, linux-mtd,
linux-fsdevel, LSM, selinux, Serge Hallyn, Andy Lutomirski, LKML,
Theodore Ts'o, Richard W.M. Jones
In-Reply-To: <1447943873.2240617.444348369.7F817E6C@webmail.messagingengine.com>
Am 19.11.2015 um 15:37 schrieb Colin Walters:
> On Thu, Nov 19, 2015, at 02:53 AM, Richard Weinberger wrote:
>
>> Erm, I don't want this in the kernel. That's why I've proposed the lklfuse approach.
>
> I already said this before but just to repeat, since I'm confused:
>
> How would "lklfuse" be different from http://libguestfs.org/
> which we at Red Hat (and a number of other organizations)
> use quite widely now for build systems, debugging etc.
Currently libguestfs has a rather huge overhead because it
boots a full virtual machine and hence a lot of communication
is needed.
With LKL you can use Linux as Library and link it to fuse.
AFAIK Richard added already a LKL backend to libguestfs. :-)
> In the end it's just running the kernel in KVM with a custom protocol,
> with support for non-filesystem things like "install a bootloader",
> and it already supports FUSE.
>
> I'm pretty firmly with Al here - the attack surface increase here
> is too great, and we'd likely turn this off if it even did make it
> into the kernel.
Agreed. This is why I'm promoting the fuse solution.
Thanks,
//richard
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Colin Walters @ 2015-11-19 14:37 UTC (permalink / raw)
To: Richard Weinberger, James Morris
Cc: Al Viro, Seth Forshee, Austin S Hemmelgarn, Eric W. Biederman,
linux-bcache, device-mapper development, linux-raid, linux-mtd,
linux-fsdevel, LSM, selinux, Serge Hallyn, Andy Lutomirski, LKML,
Theodore Ts'o
In-Reply-To: <564D7FF6.50408@nod.at>
On Thu, Nov 19, 2015, at 02:53 AM, Richard Weinberger wrote:
> Erm, I don't want this in the kernel. That's why I've proposed the lklfuse approach.
I already said this before but just to repeat, since I'm confused:
How would "lklfuse" be different from http://libguestfs.org/
which we at Red Hat (and a number of other organizations)
use quite widely now for build systems, debugging etc.
In the end it's just running the kernel in KVM with a custom protocol,
with support for non-filesystem things like "install a bootloader",
and it already supports FUSE.
I'm pretty firmly with Al here - the attack surface increase here
is too great, and we'd likely turn this off if it even did make it
into the kernel.
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Serge E. Hallyn @ 2015-11-19 14:21 UTC (permalink / raw)
To: Richard Weinberger
Cc: James Morris, Al Viro, Seth Forshee, Austin S Hemmelgarn,
Eric W. Biederman, linux-bcache, device-mapper development,
linux-raid@vger.kernel.org, linux-mtd@lists.infradead.org,
linux-fsdevel, LSM, selinux, Serge Hallyn, Andy Lutomirski, LKML,
Theodore Ts'o
In-Reply-To: <564D7FF6.50408@nod.at>
On Thu, Nov 19, 2015 at 08:53:26AM +0100, Richard Weinberger wrote:
> Am 19.11.2015 um 08:47 schrieb James Morris:
> > On Wed, 18 Nov 2015, Richard Weinberger wrote:
> >
> >> On Wed, Nov 18, 2015 at 4:13 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> >>> On Wed, Nov 18, 2015 at 09:05:12AM -0600, Seth Forshee wrote:
> >>>
> >>>> Yes, the host admin. I'm not talking about trusting the admin inside the
> >>>> container at all.
> >>>
> >>> Then why not have the same host admin just plain mount it when setting the
> >>> container up and be done with that? From the host namespace, before spawning
> >>> the docker instance or whatever framework you are using. IDGI...
> >>
> >> Because hosting companies sell containers as "full virtual machines"
> >> and customers expect to be able mount stuff like disk images they upload.
> >
> > I don't think this is a valid reason for merging functionality into the
> > kernel.
>
> Erm, I don't want this in the kernel. That's why I've proposed the lklfuse approach.
That would require the fuse-in-containers functionality, right?
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Richard Weinberger @ 2015-11-19 7:53 UTC (permalink / raw)
To: James Morris
Cc: Al Viro, Seth Forshee, Austin S Hemmelgarn, Eric W. Biederman,
linux-bcache, device-mapper development,
linux-raid@vger.kernel.org, linux-mtd@lists.infradead.org,
linux-fsdevel, LSM, selinux, Serge Hallyn, Andy Lutomirski, LKML,
Theodore Ts'o
In-Reply-To: <alpine.LRH.2.20.1511191845510.10804@namei.org>
Am 19.11.2015 um 08:47 schrieb James Morris:
> On Wed, 18 Nov 2015, Richard Weinberger wrote:
>
>> On Wed, Nov 18, 2015 at 4:13 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
>>> On Wed, Nov 18, 2015 at 09:05:12AM -0600, Seth Forshee wrote:
>>>
>>>> Yes, the host admin. I'm not talking about trusting the admin inside the
>>>> container at all.
>>>
>>> Then why not have the same host admin just plain mount it when setting the
>>> container up and be done with that? From the host namespace, before spawning
>>> the docker instance or whatever framework you are using. IDGI...
>>
>> Because hosting companies sell containers as "full virtual machines"
>> and customers expect to be able mount stuff like disk images they upload.
>
> I don't think this is a valid reason for merging functionality into the
> kernel.
Erm, I don't want this in the kernel. That's why I've proposed the lklfuse approach.
Thanks,
//richard
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: James Morris @ 2015-11-19 7:47 UTC (permalink / raw)
To: Richard Weinberger
Cc: Al Viro, Seth Forshee, Austin S Hemmelgarn, Eric W. Biederman,
linux-bcache, device-mapper development,
linux-raid@vger.kernel.org, linux-mtd@lists.infradead.org,
linux-fsdevel, LSM, selinux, Serge Hallyn, Andy Lutomirski, LKML,
Theodore Ts'o
In-Reply-To: <CAFLxGvwwhXqVM71m1kEDTZk_YTKGtbJ3gWoXw3wRgcwDiC38Bg@mail.gmail.com>
On Wed, 18 Nov 2015, Richard Weinberger wrote:
> On Wed, Nov 18, 2015 at 4:13 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> > On Wed, Nov 18, 2015 at 09:05:12AM -0600, Seth Forshee wrote:
> >
> >> Yes, the host admin. I'm not talking about trusting the admin inside the
> >> container at all.
> >
> > Then why not have the same host admin just plain mount it when setting the
> > container up and be done with that? From the host namespace, before spawning
> > the docker instance or whatever framework you are using. IDGI...
>
> Because hosting companies sell containers as "full virtual machines"
> and customers expect to be able mount stuff like disk images they upload.
I don't think this is a valid reason for merging functionality into the
kernel.
--
James Morris
<jmorris@namei.org>
^ permalink raw reply
* [PATCH 2/2] mdadm: output info more precisely when change bitmap
From: Guoqing Jiang @ 2015-11-19 5:59 UTC (permalink / raw)
To: neilb; +Cc: rgoldwyn, linux-raid, Guoqing Jiang
In-Reply-To: <1447912759-6239-1-git-send-email-gqjiang@suse.com>
When change bitmap to none, the infos could be more accurate
based on existed state type.
And s->bitmap_file is passed from cmd "--bitmap=TYPE", so remove
s->bitmap_file from err info since it should means change the
bitmap to one type failed rather than the type is already present.
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Grow.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/Grow.c b/Grow.c
index 80d7b22..30ff7f3 100644
--- a/Grow.c
+++ b/Grow.c
@@ -328,12 +328,15 @@ int Grow_addbitmap(char *devname, int fd, struct context *c, struct shape *s)
if (strcmp(s->bitmap_file, "none")==0) {
array.state &= ~(1<<MD_SB_BITMAP_PRESENT);
if (ioctl(fd, SET_ARRAY_INFO, &array)!= 0) {
- pr_err("failed to remove internal bitmap.\n");
+ if (array.state & (1<<MD_SB_CLUSTERED))
+ pr_err("failed to remove clustered bitmap.\n");
+ else
+ pr_err("failed to remove internal bitmap.\n");
return 1;
}
return 0;
}
- pr_err("%s bitmap already present on %s\n", s->bitmap_file, devname);
+ pr_err("bitmap already present on %s\n", devname);
return 1;
}
--
2.1.4
^ permalink raw reply related
* [PATCH 1/2] mdadm: let cluster raid could also add disk within incremental mode
From: Guoqing Jiang @ 2015-11-19 5:59 UTC (permalink / raw)
To: neilb; +Cc: rgoldwyn, linux-raid, Guoqing Jiang
For cluster raid, the disc.state need to be changed accordingly under
incremental mode.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Guoqing Jiang <gqjiang@suse.com>
---
Incremental.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/Incremental.c b/Incremental.c
index 781d27d..7afa7d7 100644
--- a/Incremental.c
+++ b/Incremental.c
@@ -444,6 +444,10 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
/* add disk needs to know about containers */
if (st->ss->external)
sra->array.level = LEVEL_CONTAINER;
+
+ if (info.array.state & (1 << MD_SB_CLUSTERED))
+ info.disk.state |= (1 << MD_DISK_CLUSTER_ADD);
+
err = add_disk(mdfd, st, sra, &info);
if (err < 0 && errno == EBUSY) {
/* could be another device present with the same
--
2.1.4
^ permalink raw reply related
* Re: RAID 6 Reshape Woes
From: Adam Goryachev @ 2015-11-19 4:23 UTC (permalink / raw)
To: Francisco Parada; +Cc: linux-raid
In-Reply-To: <6D622406-5CAB-4096-987B-AA45AF9F448A@abitofthisabitofthat.com>
On 18/11/2015 21:45, Francisco Parada wrote:
>
> What do you guys think?
>> New enclosures & controllers so you can ditch the port multipliers?
> Yeah, I'm thinking of just building a 4U and using an LSI RAID controller I have laying around, use it as JBOD letting mdadm do the lifting, and stop with these damn cases already. I was just trying to do the best with what I had, but going this Port Multiplier route is what's getting me into this mess. I have a couple of SunFire x2270 1U systems fully spec'ed out that I wanted to take advantage of, and by using these external enclosures, I was getting around the 4 hard drive bay limitation. I guess I'll build something that's cheap just to mount the drives and share via Samba and NFS onto the SunFire systems, then do the heavy CPU lifting from there instead.
I've used nbd before to export a drive from one machine to use as a
RAID1 member, added with a local block device (mdadm) and that worked
really reliably for me (the remote nbd was also set as write-mostly).
Since then I've also used DRBD to manage both parts of that.
However, in your case, I think there is a more modern version of nbd
which basically exports a block device as a SATA device over the
network. Could you utilise your "couple" of 4 bay 1RU servers, and most
of them use a single drive (or partition from a couple of drives in
RAID1), and export the other partition (95% disk space) to the one
"server" which will then combine them all with mdadm RAID6 or whatever
is needed ?
One problem you have here is that you will lose too many "members"
whenever one of your machines reboots.
Other options come to mind, but they certainly become more complex, and
further away from where you are now.
Not sure if any of the above is useful, or if anyone else can comment on
how well those options work, especially the SATA export for use with
mdadm would be interesting (better if each server was exporting only 1
disk, then you can lose a machine (or disk) without data loss.
Regards,
Adam
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox