* Re: Fwd: (user) Help needed: mdadm seems to constantly touch my disks
From: Jure Erznožnik @ 2016-12-19 7:12 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <87twa0tvma.fsf@notabene.neil.brown.name>
I have made two blktraces at the same time: one for md0 and one for a
member of the array. I hope they will show something sensible.
I dropped them here:
http://expirebox.com/download/ee081fa4f85ffbd0bfad68e4ee257e11.html
The file will be available for 48 hours or so they say.
LP,
Jure
On Mon, Dec 19, 2016 at 5:01 AM, NeilBrown <neilb@suse.com> wrote:
> [please remember to keep linux-raid cc:ed]
>
> On Mon, Dec 19 2016, Jure Erznožnik wrote:
>
>> I wrote this in OP: iosnoop shows COMM=[idle, md0_raid5, kworker] as
>> processes working on the disk. Blocks reported are 8, 16 (data size a
>> few KB) and 18446744073709500000 (data size 0). That last one must be
>> some virtual thingie as the disks are nowhere near that large.
>>
>> Does this answer the question or did you mean something else?
>
> Maybe if you just make the blktrace logs available somewhere and I will
> look at them myself.
>
> NeilBrown
^ permalink raw reply
* Re: [PATCH 5/8] linux: drop __bitwise__ everywhere
From: Luca Coelho @ 2016-12-19 9:09 UTC (permalink / raw)
To: Michael S. Tsirkin, linux-kernel
Cc: Kukjin Kim, Krzysztof Kozlowski, Javier Martinez Canillas,
Russell King, Alasdair Kergon, Mike Snitzer, dm-devel, Shaohua Li,
Johannes Berg, Emmanuel Grumbach, Intel Linux Wireless,
Kalle Valo, Greg Kroah-Hartman, Jiri Slaby, Lee Duncan,
Chris Leech, James E.J. Bottomley, Martin K. Petersen,
Nicholas A. Bellinger, Jason Wang, Alexander Aring,
Stefan Schmidt, David S. Miller
In-Reply-To: <1481778865-27667-6-git-send-email-mst@redhat.com>
On Thu, 2016-12-15 at 07:15 +0200, Michael S. Tsirkin wrote:
> __bitwise__ used to mean "yes, please enable sparse checks
> unconditionally", but now that we dropped __CHECK_ENDIAN__
> __bitwise is exactly the same.
> There aren't many users, replace it by __bitwise everywhere.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
> arch/arm/plat-samsung/include/plat/gpio-cfg.h | 2 +-
> drivers/md/dm-cache-block-types.h | 6 +++---
> drivers/net/ethernet/sun/sunhme.h | 2 +-
> drivers/net/wireless/intel/iwlwifi/iwl-fw-file.h | 4 ++--
For drivers/net/wireless/intel/iwlwifi/iwl-fw-file.h:
Acked-by: Luca Coelho <luciano.coelho@intel.com>
--
Luca.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply
* Re: [BUG] MD/RAID1 hung forever on freeze_array
From: Jinpu Wang @ 2016-12-19 14:56 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, Shaohua Li, Nate Dailey
In-Reply-To: <CAMGffEkufeaDytaHxtLR02iiQifZDhcwkLdzMj3X8_yaitSoFQ@mail.gmail.com>
On Mon, Dec 19, 2016 at 12:16 PM, Jinpu Wang
<jinpu.wang@profitbricks.com> wrote:
> Hi Neil,
>
> After apply the patch below, it paniced during boot in
> generic_make_request-> bio_list_pop.
> Looks related to you do bio_list_init(&bio_list_on_stack); again.
>> diff --git a/block/blk-core.c b/block/blk-core.c
>> index 14d7c0740dc0..3436b6fc3ef8 100644
>> --- a/block/blk-core.c
>> +++ b/block/blk-core.c
>> @@ -2036,10 +2036,31 @@ blk_qc_t generic_make_request(struct bio *bio)
>> struct request_queue *q = bdev_get_queue(bio->bi_bdev);
>>
>> if (likely(blk_queue_enter(q, false) == 0)) {
>> + struct bio_list hold;
>> + struct bio_list lower, same;
>> +
>> + /* Create a fresh bio_list for all subordinate requests */
>> + bio_list_merge(&hold, &bio_list_on_stack);
>> + bio_list_init(&bio_list_on_stack); ??? maybe init hold, and then merge bio_list_on_stack?
>> ret = q->make_request_fn(q, bio);
>>
>> blk_queue_exit(q);
>>
>> + /* sort new bios into those for a lower level
>> + * and those for the same level
>> + */
>> + bio_list_init(&lower);
>> + bio_list_init(&same);
>> + while ((bio = bio_list_pop(&bio_list_on_stack)) != NULL)
>> + if (q == bdev_get_queue(bio->bi_bdev))
>> + bio_list_add(&same, bio);
>> + else
>> + bio_list_add(&lower, bio);
>> + /* now assemble so we handle the lowest level first */
>> + bio_list_merge(&bio_list_on_stack, &lower);
>> + bio_list_merge(&bio_list_on_stack, &same);
>> + bio_list_merge(&bio_list_on_stack, &hold);
>> +
>> bio = bio_list_pop(current->bio_list);
>> } else {
>> struct bio *bio_next = bio_list_pop(current->bio_list);
>
>
Seems we don't need bio_list hold at all, lower and seem merge back to
bio_list_on_stack?
I'm currently testing patch below, so far so good.
diff --git a/block/blk-core.c b/block/blk-core.c
index e8d15d8..9c56cf7 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -2064,10 +2064,24 @@ blk_qc_t generic_make_request(struct bio *bio)
struct request_queue *q = bdev_get_queue(bio->bi_bdev);
if (likely(blk_queue_enter(q, __GFP_DIRECT_RECLAIM) == 0)) {
+ struct bio_list lower, same;
ret = q->make_request_fn(q, bio);
blk_queue_exit(q);
+ /* sort new bios into those for a lower level
+ * and those for the same level
+ */
+ bio_list_init(&lower);
+ bio_list_init(&same);
+ while ((bio = bio_list_pop(&bio_list_on_stack)) != NULL)
+ if (q == bdev_get_queue(bio->bi_bdev))
+ bio_list_add(&same, bio);
+ else
+ bio_list_add(&lower, bio);
+ /* now assemble so we handle the lowest level first */
+ bio_list_merge(&bio_list_on_stack, &lower);
+ bio_list_merge(&bio_list_on_stack, &same);
bio = bio_list_pop(current->bio_list);
} else {
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang@profitbricks.com
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
^ permalink raw reply related
* Re: [PATCH] mdadm: add test case for raid5 write back cache
From: Jes Sorensen @ 2016-12-19 15:08 UTC (permalink / raw)
To: Song Liu; +Cc: linux-raid, neilb, shli, liuzhengyuan, liuyun01
In-Reply-To: <20161216000016.3840135-1-songliubraving@fb.com>
Song Liu <songliubraving@fb.com> writes:
> This test cases checks data integrity of raid5 write back cache
> under various scenarios:
>
> degraded mode, non-overwrite, raid-5/6.
>
> Signed-off-by: Song Liu <songliubraving@fb.com>
> ---
> tests/21raid5cache | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 87 insertions(+)
> create mode 100644 tests/21raid5cache
Applied!
Thanks,
Jes
^ permalink raw reply
* Re: [BUG] MD/RAID1 hung forever on freeze_array
From: NeilBrown @ 2016-12-19 22:45 UTC (permalink / raw)
To: Jinpu Wang; +Cc: linux-raid, Shaohua Li, Nate Dailey
In-Reply-To: <CAMGffEkufeaDytaHxtLR02iiQifZDhcwkLdzMj3X8_yaitSoFQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]
On Mon, Dec 19 2016, Jinpu Wang wrote:
> Hi Neil,
>
> After apply the patch below, it paniced during boot in
> generic_make_request-> bio_list_pop.
> Looks related to you do bio_list_init(&bio_list_on_stack); again.
>> diff --git a/block/blk-core.c b/block/blk-core.c
>> index 14d7c0740dc0..3436b6fc3ef8 100644
>> --- a/block/blk-core.c
>> +++ b/block/blk-core.c
>> @@ -2036,10 +2036,31 @@ blk_qc_t generic_make_request(struct bio *bio)
>> struct request_queue *q = bdev_get_queue(bio->bi_bdev);
>>
>> if (likely(blk_queue_enter(q, false) == 0)) {
>> + struct bio_list hold;
>> + struct bio_list lower, same;
>> +
>> + /* Create a fresh bio_list for all subordinate requests */
>> + bio_list_merge(&hold, &bio_list_on_stack);
This is the problem. 'hold' hasn't been initialised.
We could either do:
bio_list_init(&hold);
bio_list_merge(&hold, &bio_list_on_stack);
or just
hold = bio_list_on_stack;
You didn't find 'hold' to be necessary in your testing, but I think that
is more complex arrangements it could make an important difference.
Thanks,
NeilBrown
>> + bio_list_init(&bio_list_on_stack); ??? maybe init hold, and then merge bio_list_on_stack?
>> ret = q->make_request_fn(q, bio);
>>
>> blk_queue_exit(q);
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Fwd: (user) Help needed: mdadm seems to constantly touch my disks
From: NeilBrown @ 2016-12-19 23:39 UTC (permalink / raw)
To: Jure Erznožnik; +Cc: linux-raid
In-Reply-To: <CAJ=9zidt_25tCF8QBy0To9oY+QNVD1ufF+CQ-pSjD97zVaLoGg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1115 bytes --]
On Mon, Dec 19 2016, Jure Erznožnik wrote:
> I have made two blktraces at the same time: one for md0 and one for a
> member of the array. I hope they will show something sensible.
>
> I dropped them here:
> http://expirebox.com/download/ee081fa4f85ffbd0bfad68e4ee257e11.html
>
> The file will be available for 48 hours or so they say.
Thanks.
The blktrace_md0.txt file shows several writes that originate from
iscsi_trx
so it looks like 'md0' (or some other device that is built on it) is
being exported as an iscsi target. Is that correct?
There are also some large (1024 sector) writes being generated about
once per second. That is very clearly something being written to md0,
not something that md0 is doing itself.
They are generated by a kworker. It might be iscsi related, but it
might be something else.
Try disabling the iscsi export and if the strange traffic still appears,
collect another blktrace (maybe for a bit longer - 30 seconds?).
Also please report output of
ls -l /sys/block/md0/holders/
fuser /dev/md0
when the problem is occurring.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* RAID without dev size prevents adding a new member
From: Robert LeBlanc @ 2016-12-20 1:01 UTC (permalink / raw)
To: linux-raid
I have an array that failed to readd a member saying that the new
member was too small. After trying a few things including "--add",
"--force", zeroing out the superblock, clearing out the first 100 MB
to get rid of the partition table, I found that mdadm thinks the Used
Dev Size is unknown. I looked at the superblock and it appears to have
a valid used dev size. Both disks are exactly the same size. I'm not
sure how fix this.
# mdadm --version
mdadm - v3.3.2 - 21st August 2014
# mdadm --detail /dev/md_root
/dev/md_root:
Version : 1.0
Creation Time : Wed Sep 7 15:58:30 2016
Raid Level : raid1
Array Size : 5368708928 (5120.00 GiB 5497.56 GB)
Used Dev Size : unknown
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Dec 19 17:05:29 2016
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : 10.91.0.11-volume
UUID : 2ea91e32:59b7806d:956803c6:00c410eb
Events : 3691225
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
2 0 0 2 removed
# mdadm --detail /dev/md_root
/dev/md_root:
Version : 1.0
Creation Time : Wed Sep 7 15:58:30 2016
Raid Level : raid1
Array Size : 5368708928 (5120.00 GiB 5497.56 GB)
Used Dev Size : unknown
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Mon Dec 19 17:05:29 2016
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : 10.91.0.11-volume
UUID : 2ea91e32:59b7806d:956803c6:00c410eb
Events : 3691225
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
2 0 0 2 removed
# hexdump -C -s 5497558130688 /dev/sdb
4ffffffe000 fc 4e 2b a9 01 00 00 00 01 00 00 00 00 00 00 00
|.N+.............|
4ffffffe010 2e a9 1e 32 59 b7 80 6d 95 68 03 c6 00 c4 10 eb
|...2Y..m.h......|
4ffffffe020 31 30 2e 39 31 2e 30 2e 31 31 2d 76 6f 6c 75 6d
|10.91.0.11-volum|
4ffffffe030 65 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|e...............|
4ffffffe040 86 8d d0 57 00 00 00 00 01 00 00 00 00 00 00 00
|...W............|
4ffffffe050 80 fe ff 7f 02 00 00 00 00 00 00 00 02 00 00 00
|................|
4ffffffe060 a0 ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00
|................|
4ffffffe070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
4ffffffe080 00 00 00 00 00 00 00 00 e8 fe ff 7f 02 00 00 00
|................|
4ffffffe090 f0 ff ff 7f 02 00 00 00 00 00 00 00 00 00 00 00
|................|
4ffffffe0a0 00 00 00 00 00 00 00 00 4c 6c 99 5a bd 1a 4c 7d
|........Ll.Z..L}|
4ffffffe0b0 32 29 7a bc e2 45 89 dd 00 00 08 00 f8 ff ff ff
|2)z..E..........|
4ffffffe0c0 4f 66 58 58 00 00 00 00 7b 33 38 00 00 00 00 00
|OfXX....{38.....|
4ffffffe0d0 ff ff ff ff ff ff ff ff 33 f3 f0 e6 80 00 00 00
|........3.......|
4ffffffe0e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
4ffffffe100 00 00 fe ff fe ff fe ff fe ff fe ff fe ff fe ff
|................|
4ffffffe110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff
|................|
*
4ffffffe200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
50000000000
# blockdev --report /dev/sd[a,b]
RO RA SSZ BSZ StartSec Size Device
rw 256 512 4096 0 5497558138880 /dev/sda
rw 256 512 4096 0 5497558138880 /dev/sdb
# mdadm /dev/md_root --add /dev/sda
mdadm: /dev/sda not large enough to join array
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
^ permalink raw reply
* (unknown),
From: Jinpu Wang @ 2016-12-20 8:38 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, Shaohua Li, Nate Dailey
Hi Neil,
On Mon, Dec 19, 2016 at 11:45 PM, NeilBrown <neilb@suse.com> wrote:
> On Mon, Dec 19 2016, Jinpu Wang wrote:
>
>> Hi Neil,
>>
>> After apply the patch below, it paniced during boot in
>> generic_make_request-> bio_list_pop.
>> Looks related to you do bio_list_init(&bio_list_on_stack); again.
>>> diff --git a/block/blk-core.c b/block/blk-core.c
>>> index 14d7c0740dc0..3436b6fc3ef8 100644
>>> --- a/block/blk-core.c
>>> +++ b/block/blk-core.c
>>> @@ -2036,10 +2036,31 @@ blk_qc_t generic_make_request(struct bio *bio)
>>> struct request_queue *q = bdev_get_queue(bio->bi_bdev);
>>>
>>> if (likely(blk_queue_enter(q, false) == 0)) {
>>> + struct bio_list hold;
>>> + struct bio_list lower, same;
>>> +
>>> + /* Create a fresh bio_list for all subordinate requests */
>>> + bio_list_merge(&hold, &bio_list_on_stack);
>
> This is the problem. 'hold' hasn't been initialised.
> We could either do:
> bio_list_init(&hold);
> bio_list_merge(&hold, &bio_list_on_stack);
I did try first variant, it leads to panic in bio_check_pages_dirty
PID: 4004 TASK: ffff8802337f3400 CPU: 1 COMMAND: "fio"
#0 [ffff88023ec838d0] machine_kexec at ffffffff8104075a
#1 [ffff88023ec83918] crash_kexec at ffffffff810d54c3
#2 [ffff88023ec839e0] oops_end at ffffffff81008784
#3 [ffff88023ec83a08] no_context at ffffffff8104a8f6
#4 [ffff88023ec83a60] __bad_area_nosemaphore at ffffffff8104abcf
#5 [ffff88023ec83aa8] bad_area_nosemaphore at ffffffff8104ad3e
#6 [ffff88023ec83ab8] __do_page_fault at ffffffff8104afd7
#7 [ffff88023ec83b10] do_page_fault at ffffffff8104b33c
#8 [ffff88023ec83b20] page_fault at ffffffff818173a2
[exception RIP: bio_check_pages_dirty+65]
RIP: ffffffff813f6221 RSP: ffff88023ec83bd8 RFLAGS: 00010212
RAX: 0000000000000020 RBX: ffff880232d75010 RCX: 0000000000000001
RDX: ffff880232d74000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff88023ec83bf8 R8: 0000000000000001 R9: 0000000000000000
R10: ffffffff81f25ac0 R11: ffff8802348acef0 R12: 0000000000000001
R13: 0000000000000000 R14: ffff8800b53b7d00 R15: ffff88009704d180
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff88023ec83c00] dio_bio_complete at ffffffff811d010e
#10 [ffff88023ec83c38] dio_bio_end_aio at ffffffff811d0367
#11 [ffff88023ec83c68] bio_endio at ffffffff813f637a
#12 [ffff88023ec83c80] call_bio_endio at ffffffffa0868220 [raid1]
#13 [ffff88023ec83cc8] raid_end_bio_io at ffffffffa086885b [raid1]
#14 [ffff88023ec83cf8] raid1_end_read_request at ffffffffa086a184 [raid1]
#15 [ffff88023ec83d50] bio_endio at ffffffff813f637a
#16 [ffff88023ec83d68] blk_update_request at ffffffff813fdab6
#17 [ffff88023ec83da8] blk_mq_end_request at ffffffff81406dfe
> or just
> hold = bio_list_on_stack;
>
>
> You didn't find 'hold' to be necessary in your testing, but I think that
> is more complex arrangements it could make an important difference.
Could you elaborate a bit more, from my understanding, in later
logic, we traverse the whole bio_list_on_stack,
and sort it into either lower or same bio_list, merge again the
initial bio_list_on_stack will lead to duplicated bio, isn't it?
>
> Thanks,
> NeilBrown
Thanks
Jinpu
>
>
>>> + bio_list_init(&bio_list_on_stack); ??? maybe init hold, and then merge bio_list_on_stack?
>>> ret = q->make_request_fn(q, bio);
>>>
>>> blk_queue_exit(q);
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang@profitbricks.com
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
^ permalink raw reply
* dm-crypt optimization
From: Binoy Jayan @ 2016-12-20 9:41 UTC (permalink / raw)
To: Milan Broz, Oded, Ofir, Herbert Xu, Arnd Bergmann, Mark Brown,
Alasdair Kergon, David S. Miller, private-kwg
Cc: dm-devel, linux-crypto, Rajendra, Linux kernel mailing list,
linux-raid, Shaohua Li, Mike Snitzer
At a high level the goal is to maximize the size of data blocks that get passed
to hardware accelerators, minimizing the overhead from setting up and tearing
down operations in the hardware. Currently dm-crypt itself is a big blocker as
it manually implements ESSIV and similar algorithms which allow per-block
encryption of the data so the low level operations from the crypto API can
only operate on a single block. This is done because currently the crypto API
doesn't have software implementations of these algorithms itself so dm-crypt
can't rely on it being able to provide the functionality. The plan to address
this was to provide some software implementations in the crypto API, then
update dm-crypt to rely on those. Even for a pure software implementation
with no hardware acceleration that should hopefully provide a small
optimization as we need to call into the crypto API less often but it's likely
to be marginal given the overhead of crypto, the real win would be on a system
that has an accelerator that can replace the software implementation.
Currently dm-crypt handles data only in single blocks. This means that it can't
make good use of hardware cryptography engines since there is an overhead to
each transaction with the engine but transfers must be split into block sized
chunks. Allowing the transfer of larger blocks e.g. 'struct bio', could
mitigate against these costs and could improve performance in operating systems
with encrypted filesystems. Although qualcomm chipsets support another variant
of the device-mapper dm-req-crypt, it is not something generic and in
mainline-able state. Also, it only supports 'XTS-AES' mode of encryption and
is not compatible with other modes supported by dm-crypt.
However, there are some challenges and a few possibilities to address this. I
request you to provide your suggestions on whether the points mentioned below
makes sense and if it could be done differently.
1. Move the 'real' IV generation algorithms to crypto layer (e.g. essiv)
2. Increase the 'length' of the scatterlist nodes used in the crypto api. It
can be made equal to the size of a main memory segment (as defined in
'struct bio') as they are physcially contiguous.
3. Multiple segments in 'struct bio' can be represented as scatterlist of all
segments in a 'struct bio'.
4. Move algorithms 'lmk' and 'tcw' (which are IV combined with hacks to the
cbc mode) to create a customized cbc algorithm, implemented in a seperate
file (e.g. cbc_lmk.c/cbc_tcw.c). As Milan suggested, these can't be treated
as real IVs as these include hacks to the cbc mode (and directly manipulate
encrypted data).
5. Move key selection logic to user space or always assume keycount as '1'
(as mentioned in the dm-crypt param format below) so that the key selection
logic does not have to be dependent on the sector number. This is necessary
as the key is selected otherwise based on sector number:
key_index = sector & (key_count - 1)
If block size for scatterlist nodes are increased beyond sector boundary
(which is what we plan to achieve, for performance), the key set for every
cipher operation cannot be changed at the sector level.
dm-crypt param format : cipher[:keycount]-mode-iv:ivopts
Example : aes:2-cbc-essiv:sha256
Also as Milan suggested, it is not wise to move the key selection logic to
the crypto layer as it will prevent any changes to the key structure later.
The following is a reference to an earlier patchset. It had the cipher mode
'cbc' mixed up with the IV algorithms and is usually not the preferred way.
Reference:
https://lkml.org/lkml/2016/12/13/65
https://lkml.org/lkml/2016/12/13/66
^ permalink raw reply
* Re: [BUG] MD/RAID1 hung forever on freeze_array
From: Jinpu Wang @ 2016-12-20 10:34 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, Shaohua Li, Nate Dailey
In-Reply-To: <87oa07tu6q.fsf@notabene.neil.brown.name>
Hi Neil,
> This is the problem. 'hold' hasn't been initialised.
> We could either do:
> bio_list_init(&hold);
> bio_list_merge(&hold, &bio_list_on_stack);
I tried above variant first and it lead to panic in endio path:
PID: 4004 TASK: ffff8802337f3400 CPU: 1 COMMAND: "fio"
#0 [ffff88023ec838d0] machine_kexec at ffffffff8104075a
#1 [ffff88023ec83918] crash_kexec at ffffffff810d54c3
#2 [ffff88023ec839e0] oops_end at ffffffff81008784
#3 [ffff88023ec83a08] no_context at ffffffff8104a8f6
#4 [ffff88023ec83a60] __bad_area_nosemaphore at ffffffff8104abcf
#5 [ffff88023ec83aa8] bad_area_nosemaphore at ffffffff8104ad3e
#6 [ffff88023ec83ab8] __do_page_fault at ffffffff8104afd7
#7 [ffff88023ec83b10] do_page_fault at ffffffff8104b33c
#8 [ffff88023ec83b20] page_fault at ffffffff818173a2
[exception RIP: bio_check_pages_dirty+65]
RIP: ffffffff813f6221 RSP: ffff88023ec83bd8 RFLAGS: 00010212
RAX: 0000000000000020 RBX: ffff880232d75010 RCX: 0000000000000001
RDX: ffff880232d74000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffff88023ec83bf8 R8: 0000000000000001 R9: 0000000000000000
R10: ffffffff81f25ac0 R11: ffff8802348acef0 R12: 0000000000000001
R13: 0000000000000000 R14: ffff8800b53b7d00 R15: ffff88009704d180
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff88023ec83c00] dio_bio_complete at ffffffff811d010e
#10 [ffff88023ec83c38] dio_bio_end_aio at ffffffff811d0367
#11 [ffff88023ec83c68] bio_endio at ffffffff813f637a
#12 [ffff88023ec83c80] call_bio_endio at ffffffffa0868220 [raid1]
#13 [ffff88023ec83cc8] raid_end_bio_io at ffffffffa086885b [raid1]
#14 [ffff88023ec83cf8] raid1_end_read_request at ffffffffa086a184 [raid1]
#15 [ffff88023ec83d50] bio_endio at ffffffff813f637a
#16 [ffff88023ec83d68] blk_update_request at ffffffff813fdab6
#17 [ffff88023ec83da8] blk_mq_end_request at ffffffff81406dfe
> or just
> hold = bio_list_on_stack;
>
>
> You didn't find 'hold' to be necessary in your testing, but I think that
> is more complex arrangements it could make an important difference.
Could you elaborate a bit more, from my understanding, in later code,
we pop all bio from bio_list_on_stack,
add it to either "lower" or "same" bio_list, so merge both will have
the whole list again, right?
>
> Thanks,
> NeilBrown
>
>
>>> + bio_list_init(&bio_list_on_stack); ??? maybe init hold, and then merge bio_list_on_stack?
>>> ret = q->make_request_fn(q, bio);
>>>
>>> blk_queue_exit(q);
Thanks
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang@profitbricks.com
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
^ permalink raw reply
* [PATCH] dm bufio: use rb_entry()
From: Geliang Tang @ 2016-12-20 14:02 UTC (permalink / raw)
To: Alasdair Kergon, Mike Snitzer, Shaohua Li
Cc: Geliang Tang, dm-devel, linux-raid, linux-kernel
To make the code clearer, use rb_entry() instead of container_of() to
deal with rbtree.
Signed-off-by: Geliang Tang <geliangtang@gmail.com>
---
drivers/md/dm-bufio.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c
index 84d2f0e..ae3c396 100644
--- a/drivers/md/dm-bufio.c
+++ b/drivers/md/dm-bufio.c
@@ -266,7 +266,7 @@ static struct dm_buffer *__find(struct dm_bufio_client *c, sector_t block)
struct dm_buffer *b;
while (n) {
- b = container_of(n, struct dm_buffer, node);
+ b = rb_entry(n, struct dm_buffer, node);
if (b->block == block)
return b;
@@ -283,7 +283,7 @@ static void __insert(struct dm_bufio_client *c, struct dm_buffer *b)
struct dm_buffer *found;
while (*new) {
- found = container_of(*new, struct dm_buffer, node);
+ found = rb_entry(*new, struct dm_buffer, node);
if (found->block == b->block) {
BUG_ON(found != b);
--
2.9.3
^ permalink raw reply related
* Re: [BUG] MD/RAID1 hung forever on freeze_array
From: NeilBrown @ 2016-12-20 21:23 UTC (permalink / raw)
To: Jinpu Wang; +Cc: linux-raid, Shaohua Li, Nate Dailey
In-Reply-To: <CAMGffEmEUg2Xrg-v_z+cMxzsH-Uk6bDCqTSA7vGoPw4f2dvu0A@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1783 bytes --]
On Tue, Dec 20 2016, Jinpu Wang wrote:
> Hi Neil,
>> This is the problem. 'hold' hasn't been initialised.
>> We could either do:
>> bio_list_init(&hold);
>> bio_list_merge(&hold, &bio_list_on_stack);
> I tried above variant first and it lead to panic in endio path:
>
...
>
> [exception RIP: bio_check_pages_dirty+65]
I can't explain that at all. Maybe if I saw the complete patch I would
be able to see what went wrong (or maybe there is a separate cause).
>>
>> You didn't find 'hold' to be necessary in your testing, but I think that
>> is more complex arrangements it could make an important difference.
>
> Could you elaborate a bit more, from my understanding, in later code,
> we pop all bio from bio_list_on_stack,
> add it to either "lower" or "same" bio_list, so merge both will have
> the whole list again, right?
If there are several bios on bio_list_on_stack, and if processing the
first one causes several calls to generic_make_request(), then we want
all the bios passed to generic_make_request() to be handled *before* the
remaining bios that were originally on bio_list_on_stack. Of these new
bios, we want to handle those for a lower level device first, and those
for the same device after that. Only when all of those have been
completely submitted (together with any subordinate bios submitted to
generic_make_request()) do we move on to the rest of bio_list_on_stack
(which are at the same level as the first bio, or higher in the device
stack).
The usage of 'hold' would become important when you have multiple
levels. e.g. dm on md on scsi. If the first bio send to dm is split,
then the first half needs to be processed all the way down to all the
scsi devices, before the second half of the split is handled.
Hope that helps,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Inconsistent use of sectors vs 1k-blocks in sysfs and other places
From: John Brooks @ 2016-12-20 22:14 UTC (permalink / raw)
To: NeilBrown, linux-raid
In-Reply-To: <87fulkvpgf.fsf@notabene.neil.brown.name>
On Mon, Dec 19, 2016 at 09:32:00AM +1100, NeilBrown wrote:
> The code is right by definition. As you say, mdadm uses this interface
> so we cannot change it.
Yep, I expected as much and fully agree.
> >
> > Somewhat related, suspend_{lo|hi}, resync_{min|max} attributes specify ranges
> > in sectors, but the documentation does not specify if they are ranges on the
> > array size or device size. And RAID10 may even handle resync_max differently
> > from the rest; I didn't look deeply into that but see commit c805cdecea.
>
> suspend_{lo|hi} are are array addresses
> resync_{min|max} are array addresses for RAID1 and RAID10, and they are
> device-addresses-offset-from-data_offset for RAID1 and RAID456.
Good to know. One question, is it possible for the offset or size to be
different across disks in the same array? The documentation says that the size
is "normally" the same as component_size, implying that perhaps it could be
different.
> It is bad that the documentation is incorrect and incomplete. If you we
> to post a patch which fixed some of it, I'm sure that would be
> thankfully accepted.
I will see what I can do.
> In theory you could have a RAID1 with an odd number of sectors used in
> each component, but I doubt that happens in practice. So you can get
> the component size in sectors by reading component_size and multiplying
> by two.
Is that within a realm of possibility worth mentioning in the docs?
^ permalink raw reply
* Re: Inconsistent use of sectors vs 1k-blocks in sysfs and other places
From: NeilBrown @ 2016-12-21 0:06 UTC (permalink / raw)
To: John Brooks, linux-raid
In-Reply-To: <20161220221430.GA19146@oldkitsune.fastquake.com>
[-- Attachment #1: Type: text/plain, Size: 2241 bytes --]
On Wed, Dec 21 2016, John Brooks wrote:
> On Mon, Dec 19, 2016 at 09:32:00AM +1100, NeilBrown wrote:
>> The code is right by definition. As you say, mdadm uses this interface
>> so we cannot change it.
>
> Yep, I expected as much and fully agree.
>
>> >
>> > Somewhat related, suspend_{lo|hi}, resync_{min|max} attributes specify ranges
>> > in sectors, but the documentation does not specify if they are ranges on the
>> > array size or device size. And RAID10 may even handle resync_max differently
>> > from the rest; I didn't look deeply into that but see commit c805cdecea.
>>
>> suspend_{lo|hi} are are array addresses
>> resync_{min|max} are array addresses for RAID1 and RAID10, and they are
>> device-addresses-offset-from-data_offset for RAID1 and RAID456.
>
> Good to know. One question, is it possible for the offset or size to be
> different across disks in the same array? The documentation says that the size
> is "normally" the same as component_size, implying that perhaps it could be
> different.
raid0 and linear are special.
For linear, it is obvious that differently sized devices can be
combined.
For raid0, it is not quite so obvious. The array is divided into
region. The first region is striped over all devices. Then next region
is striped over all devices that are bigger than the smallest, etc.
The offset can be different on all devices. The size can be only for
raid0 and linear.
>
>> It is bad that the documentation is incorrect and incomplete. If you we
>> to post a patch which fixed some of it, I'm sure that would be
>> thankfully accepted.
>
> I will see what I can do.
>
>> In theory you could have a RAID1 with an odd number of sectors used in
>> each component, but I doubt that happens in practice. So you can get
>> the component size in sectors by reading component_size and multiplying
>> by two.
>
> Is that within a realm of possibility worth mentioning in the docs?
Probably not. I'd have to check the code to see what it actually
allows.
If the kernel allows odd-sized RAID1, we would have to leave it that
way. But I'd support changing mdadm (if needed) to refuse to create
a RAID1 with an odd number of sectors (or even a size that is not a
multiple of 4K).
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: RAID without dev size prevents adding a new member
From: NeilBrown @ 2016-12-21 4:58 UTC (permalink / raw)
To: Robert LeBlanc, linux-raid
In-Reply-To: <CAANLjFo09x0VMj2KbPZzWwpgXebS0d1ZGNTwDqi7fiVsKvFJ+g@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 4506 bytes --]
On Tue, Dec 20 2016, Robert LeBlanc wrote:
> I have an array that failed to readd a member saying that the new
> member was too small. After trying a few things including "--add",
> "--force", zeroing out the superblock, clearing out the first 100 MB
> to get rid of the partition table, I found that mdadm thinks the Used
> Dev Size is unknown. I looked at the superblock and it appears to have
> a valid used dev size. Both disks are exactly the same size. I'm not
> sure how fix this.
>
> # mdadm --version
> mdadm - v3.3.2 - 21st August 2014
Try a newer mdadm.
git clone git://git.kernel.org/pub/scm/utils/mdadm/mdadm.git
cd mdadm
make
your bug sounds familiar. I think it has been fixed.
NeilBrown
>
> # mdadm --detail /dev/md_root
> /dev/md_root:
> Version : 1.0
> Creation Time : Wed Sep 7 15:58:30 2016
> Raid Level : raid1
> Array Size : 5368708928 (5120.00 GiB 5497.56 GB)
> Used Dev Size : unknown
> Raid Devices : 2
> Total Devices : 1
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Mon Dec 19 17:05:29 2016
> State : clean, degraded
> Active Devices : 1
> Working Devices : 1
> Failed Devices : 0
> Spare Devices : 0
>
> Name : 10.91.0.11-volume
> UUID : 2ea91e32:59b7806d:956803c6:00c410eb
> Events : 3691225
>
> Number Major Minor RaidDevice State
> 0 8 16 0 active sync /dev/sdb
> 2 0 0 2 removed
>
> # mdadm --detail /dev/md_root
> /dev/md_root:
> Version : 1.0
> Creation Time : Wed Sep 7 15:58:30 2016
> Raid Level : raid1
> Array Size : 5368708928 (5120.00 GiB 5497.56 GB)
> Used Dev Size : unknown
> Raid Devices : 2
> Total Devices : 1
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Mon Dec 19 17:05:29 2016
> State : clean, degraded
> Active Devices : 1
> Working Devices : 1
> Failed Devices : 0
> Spare Devices : 0
>
> Name : 10.91.0.11-volume
> UUID : 2ea91e32:59b7806d:956803c6:00c410eb
> Events : 3691225
>
> Number Major Minor RaidDevice State
> 0 8 16 0 active sync /dev/sdb
> 2 0 0 2 removed
>
> # hexdump -C -s 5497558130688 /dev/sdb
> 4ffffffe000 fc 4e 2b a9 01 00 00 00 01 00 00 00 00 00 00 00
> |.N+.............|
> 4ffffffe010 2e a9 1e 32 59 b7 80 6d 95 68 03 c6 00 c4 10 eb
> |...2Y..m.h......|
> 4ffffffe020 31 30 2e 39 31 2e 30 2e 31 31 2d 76 6f 6c 75 6d
> |10.91.0.11-volum|
> 4ffffffe030 65 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> |e...............|
> 4ffffffe040 86 8d d0 57 00 00 00 00 01 00 00 00 00 00 00 00
> |...W............|
> 4ffffffe050 80 fe ff 7f 02 00 00 00 00 00 00 00 02 00 00 00
> |................|
> 4ffffffe060 a0 ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00
> |................|
> 4ffffffe070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> |................|
> 4ffffffe080 00 00 00 00 00 00 00 00 e8 fe ff 7f 02 00 00 00
> |................|
> 4ffffffe090 f0 ff ff 7f 02 00 00 00 00 00 00 00 00 00 00 00
> |................|
> 4ffffffe0a0 00 00 00 00 00 00 00 00 4c 6c 99 5a bd 1a 4c 7d
> |........Ll.Z..L}|
> 4ffffffe0b0 32 29 7a bc e2 45 89 dd 00 00 08 00 f8 ff ff ff
> |2)z..E..........|
> 4ffffffe0c0 4f 66 58 58 00 00 00 00 7b 33 38 00 00 00 00 00
> |OfXX....{38.....|
> 4ffffffe0d0 ff ff ff ff ff ff ff ff 33 f3 f0 e6 80 00 00 00
> |........3.......|
> 4ffffffe0e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> |................|
> *
> 4ffffffe100 00 00 fe ff fe ff fe ff fe ff fe ff fe ff fe ff
> |................|
> 4ffffffe110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff
> |................|
> *
> 4ffffffe200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> |................|
> *
> 50000000000
>
> # blockdev --report /dev/sd[a,b]
> RO RA SSZ BSZ StartSec Size Device
> rw 256 512 4096 0 5497558138880 /dev/sda
> rw 256 512 4096 0 5497558138880 /dev/sdb
>
> # mdadm /dev/md_root --add /dev/sda
> mdadm: /dev/sda not large enough to join array
>
> ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Linux software RAID becomes unresponsive after removing a disk from server
From: NeilBrown @ 2016-12-21 5:12 UTC (permalink / raw)
To: PHP-Friends GmbH, linux-raid
In-Reply-To: <7b8b1130-e79b-efcd-0579-c4a349ec5814@php-friends.de>
[-- Attachment #1: Type: text/plain, Size: 2258 bytes --]
On Sat, Dec 17 2016, PHP-Friends GmbH wrote:
> Hello everyone,
>
> first of all: This is in fact a crossposting from serverfault
> (http://serverfault.com/questions/821195/linux-software-raid-becomes-unresponsive-after-removing-a-disk-from-server),
> as the user shodanshok recommended contacting this mailing list because
> to him this seems like a possible bug in the Linux RAID software. I want
> to add that I can provide more logs and information if they are needed,
> but as the text is already quite long I thought that would be enough for
> the moment.
>
> I am running a CentOS 7 machine (standard kernel:
> 3.10.0-327.36.3.el7.x86_64) with a software RAID-10 over 16x 1 TB SSDs
> (to be more precise, there are two RAID arrays on the disks; one of the
> arrays is providing the host's swap partition). Last week, a SSD failed:
>
...
> 11:48:00 kvm7 kernel: INFO: task md3_raid10:1293 blocked for more than
> 120 seconds.
> 11:48:00 kvm7 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> 11:48:00 kvm7 kernel: md3_raid10 D ffff883f26e55c00 0 1293
> 2 0x00000000
> 11:48:00 kvm7 kernel: ffff887f24bd7c58 0000000000000046 ffff887f212eb980
> ffff887f24bd7fd8
> 11:48:00 kvm7 kernel: ffff887f24bd7fd8 ffff887f24bd7fd8 ffff887f212eb980
> ffff887f23514400
> 11:48:00 kvm7 kernel: ffff887f235144dc 0000000000000001 ffff887f23514500
> ffff8807fa4c4300
> 11:48:00 kvm7 kernel: Call Trace:
> 11:48:00 kvm7 kernel: [<ffffffff8163bb39>] schedule+0x29/0x70
> 11:48:00 kvm7 kernel: [<ffffffffa0104ef7>] freeze_array+0xb7/0x180 [raid10]
Might be a known bug, maybe the one fixed by
Commit: ccfc7bf1f09d ("raid1: include bio_end_io_list in nr_queued to prevent freeze_array hang")
I have no idea what patches are included in your centos kernel.
In general, we only provide support for mainline kernels here. Not
because we don't want to support others, but because digging around
inside a non-mainline kernel is much more work.
BTW, to remove a device from an md array after it has been physically
removed from the system, you can use
mdadm /dev/mdXX --remove detached
That wouldn't have helped you here, but for future reference.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: Fwd: (user) Help needed: mdadm seems to constantly touch my disks
From: Jure Erznožnik @ 2016-12-21 11:33 UTC (permalink / raw)
To: NeilBrown, linux-raid, tkonto
In-Reply-To: <87inqftro1.fsf@notabene.neil.brown.name>
Mr Brown,
Let me begin with: please give me your paypal address or something so
that I can at least buy you a beer or something
Your analysis and discovery that iSCSI is the origin of writes got me
thinking: how can he see that on md0 device if that device has two
more layers (bcache + LVM) before iSCSI even comes into play. Maybe
the system propagates the origin down the block devices or something,
totally not relevant here. So I embarked on a journey of total data
destruction by disabling one layer at a time. I started by simply
detaching bcache as that was the first thing on the list - and was
non-destructive to boot :)
I have found the culprit:
It is bcache that does the one second writes. I have yet to find the
exact parameters that influence this behaviour, but the output of
writeback_rate_debug is EXTREMELY clear: it's writing a bit of data
each second, reducing the dirty cache by that tiny amount. This is
what causes the write "amplification" resulting in clicks long after a
write has been done - because bcache only writes tiny amounts each
second instead of flushing the entire cache at once when the time
comes.
Thank you for your time and please consider the first sentence of this mail
LP,
Jure
On Tue, Dec 20, 2016 at 12:39 AM, NeilBrown <neilb@suse.com> wrote:
> On Mon, Dec 19 2016, Jure Erznožnik wrote:
>
>> I have made two blktraces at the same time: one for md0 and one for a
>> member of the array. I hope they will show something sensible.
>>
>> I dropped them here:
>> http://expirebox.com/download/ee081fa4f85ffbd0bfad68e4ee257e11.html
>>
>> The file will be available for 48 hours or so they say.
>
> Thanks.
> The blktrace_md0.txt file shows several writes that originate from
> iscsi_trx
>
> so it looks like 'md0' (or some other device that is built on it) is
> being exported as an iscsi target. Is that correct?
>
> There are also some large (1024 sector) writes being generated about
> once per second. That is very clearly something being written to md0,
> not something that md0 is doing itself.
> They are generated by a kworker. It might be iscsi related, but it
> might be something else.
>
> Try disabling the iscsi export and if the strange traffic still appears,
> collect another blktrace (maybe for a bit longer - 30 seconds?).
> Also please report output of
> ls -l /sys/block/md0/holders/
> fuser /dev/md0
>
> when the problem is occurring.
>
> NeilBrown
^ permalink raw reply
* Re: dm-crypt optimization
From: Milan Broz @ 2016-12-21 12:47 UTC (permalink / raw)
To: Binoy Jayan, Oded, Ofir, Herbert Xu, Arnd Bergmann, Mark Brown,
Alasdair Kergon, David S. Miller, private-kwg
Cc: dm-devel, linux-crypto, Rajendra, Linux kernel mailing list,
linux-raid, Shaohua Li, Mike Snitzer
In-Reply-To: <CAHv-k_98w-9whFiCLAnTGL2QAirExsix_S8ZWc8tM7wj=gzD6Q@mail.gmail.com>
On 12/20/2016 10:41 AM, Binoy Jayan wrote:
> At a high level the goal is to maximize the size of data blocks that get passed
> to hardware accelerators, minimizing the overhead from setting up and tearing
> down operations in the hardware. Currently dm-crypt itself is a big blocker as
> it manually implements ESSIV and similar algorithms which allow per-block
> encryption of the data so the low level operations from the crypto API can
> only operate on a single block. This is done because currently the crypto API
> doesn't have software implementations of these algorithms itself so dm-crypt
> can't rely on it being able to provide the functionality. The plan to address
> this was to provide some software implementations in the crypto API, then
> update dm-crypt to rely on those. Even for a pure software implementation
> with no hardware acceleration that should hopefully provide a small
> optimization as we need to call into the crypto API less often but it's likely
> to be marginal given the overhead of crypto, the real win would be on a system
> that has an accelerator that can replace the software implementation.
>
> Currently dm-crypt handles data only in single blocks. This means that it can't
> make good use of hardware cryptography engines since there is an overhead to
> each transaction with the engine but transfers must be split into block sized
> chunks. Allowing the transfer of larger blocks e.g. 'struct bio', could
> mitigate against these costs and could improve performance in operating systems
> with encrypted filesystems. Although qualcomm chipsets support another variant
> of the device-mapper dm-req-crypt, it is not something generic and in
> mainline-able state. Also, it only supports 'XTS-AES' mode of encryption and
> is not compatible with other modes supported by dm-crypt.
So the core problem is that your crypto accelerator can operate efficiently only
with bigger batch sizes.
How big blocks your crypto hw need to be able to operate more efficiently?
What about 4k blocks (no batches), could it be usable trade-off?
With some (backward incompatible) changes in LUKS format I would like to see support
for encryption blocks equivalent to sectors size, so it basically means for 4k drive 4k
encryption block.
(This should decrease overhead, now is everything processed on 512 blocks only.)
Support of bigger block sizes would be unsafe without additional mechanism that provides
atomic writes of multiple sectors. Maybe it applies to 4k as well on some devices though...)
The above is not going against your proposal, I am just curious if this is enough
to provide better performance on your hw accelerator or not.
Milan
> However, there are some challenges and a few possibilities to address this. I
> request you to provide your suggestions on whether the points mentioned below
> makes sense and if it could be done differently.
>
> 1. Move the 'real' IV generation algorithms to crypto layer (e.g. essiv)
> 2. Increase the 'length' of the scatterlist nodes used in the crypto api. It
> can be made equal to the size of a main memory segment (as defined in
> 'struct bio') as they are physcially contiguous.
> 3. Multiple segments in 'struct bio' can be represented as scatterlist of all
> segments in a 'struct bio'.
>
> 4. Move algorithms 'lmk' and 'tcw' (which are IV combined with hacks to the
> cbc mode) to create a customized cbc algorithm, implemented in a seperate
> file (e.g. cbc_lmk.c/cbc_tcw.c). As Milan suggested, these can't be treated
> as real IVs as these include hacks to the cbc mode (and directly manipulate
> encrypted data).
>
> 5. Move key selection logic to user space or always assume keycount as '1'
> (as mentioned in the dm-crypt param format below) so that the key selection
> logic does not have to be dependent on the sector number. This is necessary
> as the key is selected otherwise based on sector number:
>
> key_index = sector & (key_count - 1)
>
> If block size for scatterlist nodes are increased beyond sector boundary
> (which is what we plan to achieve, for performance), the key set for every
> cipher operation cannot be changed at the sector level.
>
> dm-crypt param format : cipher[:keycount]-mode-iv:ivopts
> Example : aes:2-cbc-essiv:sha256
>
> Also as Milan suggested, it is not wise to move the key selection logic to
> the crypto layer as it will prevent any changes to the key structure later.
>
> The following is a reference to an earlier patchset. It had the cipher mode
> 'cbc' mixed up with the IV algorithms and is usually not the preferred way.
>
> Reference:
> https://lkml.org/lkml/2016/12/13/65
> https://lkml.org/lkml/2016/12/13/66
>
^ permalink raw reply
* Re: [BUG] MD/RAID1 hung forever on freeze_array
From: Jinpu Wang @ 2016-12-21 12:48 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid, Shaohua Li, Nate Dailey
In-Reply-To: <871sx2thvt.fsf@notabene.neil.brown.name>
[-- Attachment #1: Type: text/plain, Size: 2806 bytes --]
Thanks Neil,
On Tue, Dec 20, 2016 at 10:23 PM, NeilBrown <neilb@suse.com> wrote:
> On Tue, Dec 20 2016, Jinpu Wang wrote:
>
>> Hi Neil,
>>> This is the problem. 'hold' hasn't been initialised.
>>> We could either do:
>>> bio_list_init(&hold);
>>> bio_list_merge(&hold, &bio_list_on_stack);
>> I tried above variant first and it lead to panic in endio path:
>>
> ...
>>
>> [exception RIP: bio_check_pages_dirty+65]
>
>
> I can't explain that at all. Maybe if I saw the complete patch I would
> be able to see what went wrong (or maybe there is a separate cause).
I think I know the cause now, I did
+bio_list_init(&hold);
bio_list_merge(&hold, &bio_list_on_stack);
- bio_list_init(&bio_list_on_stack)
I didn't read your comment clear enough, just thought we maybe don't
need to reinit &bio_list_on_stack twice.
But your idea is same the copy before we can q->make_request_fn.
>
>>>
>>> You didn't find 'hold' to be necessary in your testing, but I think that
>>> is more complex arrangements it could make an important difference.
>>
>> Could you elaborate a bit more, from my understanding, in later code,
>> we pop all bio from bio_list_on_stack,
>> add it to either "lower" or "same" bio_list, so merge both will have
>> the whole list again, right?
>
> If there are several bios on bio_list_on_stack, and if processing the
> first one causes several calls to generic_make_request(), then we want
> all the bios passed to generic_make_request() to be handled *before* the
> remaining bios that were originally on bio_list_on_stack. Of these new
> bios, we want to handle those for a lower level device first, and those
> for the same device after that. Only when all of those have been
> completely submitted (together with any subordinate bios submitted to
> generic_make_request()) do we move on to the rest of bio_list_on_stack
> (which are at the same level as the first bio, or higher in the device
> stack).
>
> The usage of 'hold' would become important when you have multiple
> levels. e.g. dm on md on scsi. If the first bio send to dm is split,
> then the first half needs to be processed all the way down to all the
> scsi devices, before the second half of the split is handled.
>
> Hope that helps,
> NeilBrown
>
Thanks, it does help a lot, I attached the patch I'm still testing,
but so far so good.
Could you check if I got it right?
Cheers,
--
Jinpu Wang
Linux Kernel Developer
ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 577 008 042
Fax: +49 30 577 008 299
Email: jinpu.wang@profitbricks.com
URL: https://www.profitbricks.de
Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss
[-- Attachment #2: 0001-block-fix-deadlock-between-freeze_array-and-wait_bar.patch --]
[-- Type: text/x-patch, Size: 2300 bytes --]
From 9500d5e9a7ccd1f4397f927ffdc8d378955d477c Mon Sep 17 00:00:00 2001
From: Jack Wang <jinpu.wang@profitbricks.com>
Date: Wed, 14 Dec 2016 16:55:52 +0100
Subject: [PATCH] block: fix deadlock between freeze_array() and wait_barrier()
When we call wait_barrier, we might have some bios waiting
in current->bio_list, which prevents the array_freeze call to
complete. Those can only be internal READs, which have already
passed the wait_barrier call (thus incrementing nr_pending), but
still were not submitted to the lower level, due to generic_make_request
logic to avoid recursive calls. In such case, we have a deadlock:
- array_frozen is already set to 1, so wait_barrier unconditionally waits, so
- internal READ bios will not be submitted, thus freeze_array will
never completes.
To fix this, modify generic_make_request to always sort bio_list_on_stack
first with lowest level, then higher, until same level.
Suggested-by: Neil Brown <neil.brown.suse.com>
Signed-off-by: Jack Wang <jinpu.wang@profitbricks.com>
---
block/blk-core.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/block/blk-core.c b/block/blk-core.c
index e8d15d8..5a6390a 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -2064,10 +2064,30 @@ blk_qc_t generic_make_request(struct bio *bio)
struct request_queue *q = bdev_get_queue(bio->bi_bdev);
if (likely(blk_queue_enter(q, __GFP_DIRECT_RECLAIM) == 0)) {
+ struct bio_list lower, same, hold;
+
+ /* Create a fresh bio_list for all subordinate requests */
+ bio_list_init(&hold);
+ bio_list_merge(&hold, &bio_list_on_stack);
+ bio_list_init(&bio_list_on_stack);
ret = q->make_request_fn(q, bio);
blk_queue_exit(q);
+ /* sort new bios into those for a lower level
+ * and those for the same level
+ */
+ bio_list_init(&lower);
+ bio_list_init(&same);
+ while ((bio = bio_list_pop(&bio_list_on_stack)) != NULL)
+ if (q == bdev_get_queue(bio->bi_bdev))
+ bio_list_add(&same, bio);
+ else
+ bio_list_add(&lower, bio);
+ /* now assemble so we handle the lowest level first */
+ bio_list_merge(&bio_list_on_stack, &lower);
+ bio_list_merge(&bio_list_on_stack, &same);
+ bio_list_merge(&bio_list_on_stack, &hold);
bio = bio_list_pop(current->bio_list);
} else {
--
2.7.4
^ permalink raw reply related
* Re: RAID without dev size prevents adding a new member
From: Robert LeBlanc @ 2016-12-21 16:44 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <8760mdswsn.fsf@notabene.neil.brown.name>
It looks like it is still there on the latest master.
# ./mdadm --version
mdadm - v3.4-88-gfa219dd - 22nd September 2016
# ./mdadm --detail /dev/md_root
/dev/md_root:
Version : 1.0
Creation Time : Wed Sep 7 15:58:30 2016
Raid Level : raid1
Array Size : 5368708928 (5120.00 GiB 5497.56 GB)
Used Dev Size : unknown
Raid Devices : 2
Total Devices : 1
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Dec 21 09:40:21 2016
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0
Name : 10.91.0.11-volume
UUID : 2ea91e32:59b7806d:956803c6:00c410eb
Events : 3989416
Number Major Minor RaidDevice State
0 8 16 0 active sync /dev/sdb
- 0 0 1 removed
# ./mdadm /dev/md_root --add /dev/sda
mdadm: /dev/sda not large enough to join array
Any other ideas or information that I can provide that would be helpful?
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Tue, Dec 20, 2016 at 9:58 PM, NeilBrown <neilb@suse.com> wrote:
> On Tue, Dec 20 2016, Robert LeBlanc wrote:
>
>> I have an array that failed to readd a member saying that the new
>> member was too small. After trying a few things including "--add",
>> "--force", zeroing out the superblock, clearing out the first 100 MB
>> to get rid of the partition table, I found that mdadm thinks the Used
>> Dev Size is unknown. I looked at the superblock and it appears to have
>> a valid used dev size. Both disks are exactly the same size. I'm not
>> sure how fix this.
>>
>> # mdadm --version
>> mdadm - v3.3.2 - 21st August 2014
>
> Try a newer mdadm.
> git clone git://git.kernel.org/pub/scm/utils/mdadm/mdadm.git
> cd mdadm
> make
>
> your bug sounds familiar. I think it has been fixed.
>
> NeilBrown
>
>
>>
>> # mdadm --detail /dev/md_root
>> /dev/md_root:
>> Version : 1.0
>> Creation Time : Wed Sep 7 15:58:30 2016
>> Raid Level : raid1
>> Array Size : 5368708928 (5120.00 GiB 5497.56 GB)
>> Used Dev Size : unknown
>> Raid Devices : 2
>> Total Devices : 1
>> Persistence : Superblock is persistent
>>
>> Intent Bitmap : Internal
>>
>> Update Time : Mon Dec 19 17:05:29 2016
>> State : clean, degraded
>> Active Devices : 1
>> Working Devices : 1
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Name : 10.91.0.11-volume
>> UUID : 2ea91e32:59b7806d:956803c6:00c410eb
>> Events : 3691225
>>
>> Number Major Minor RaidDevice State
>> 0 8 16 0 active sync /dev/sdb
>> 2 0 0 2 removed
>>
>> # mdadm --detail /dev/md_root
>> /dev/md_root:
>> Version : 1.0
>> Creation Time : Wed Sep 7 15:58:30 2016
>> Raid Level : raid1
>> Array Size : 5368708928 (5120.00 GiB 5497.56 GB)
>> Used Dev Size : unknown
>> Raid Devices : 2
>> Total Devices : 1
>> Persistence : Superblock is persistent
>>
>> Intent Bitmap : Internal
>>
>> Update Time : Mon Dec 19 17:05:29 2016
>> State : clean, degraded
>> Active Devices : 1
>> Working Devices : 1
>> Failed Devices : 0
>> Spare Devices : 0
>>
>> Name : 10.91.0.11-volume
>> UUID : 2ea91e32:59b7806d:956803c6:00c410eb
>> Events : 3691225
>>
>> Number Major Minor RaidDevice State
>> 0 8 16 0 active sync /dev/sdb
>> 2 0 0 2 removed
>>
>> # hexdump -C -s 5497558130688 /dev/sdb
>> 4ffffffe000 fc 4e 2b a9 01 00 00 00 01 00 00 00 00 00 00 00
>> |.N+.............|
>> 4ffffffe010 2e a9 1e 32 59 b7 80 6d 95 68 03 c6 00 c4 10 eb
>> |...2Y..m.h......|
>> 4ffffffe020 31 30 2e 39 31 2e 30 2e 31 31 2d 76 6f 6c 75 6d
>> |10.91.0.11-volum|
>> 4ffffffe030 65 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> |e...............|
>> 4ffffffe040 86 8d d0 57 00 00 00 00 01 00 00 00 00 00 00 00
>> |...W............|
>> 4ffffffe050 80 fe ff 7f 02 00 00 00 00 00 00 00 02 00 00 00
>> |................|
>> 4ffffffe060 a0 ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00
>> |................|
>> 4ffffffe070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> |................|
>> 4ffffffe080 00 00 00 00 00 00 00 00 e8 fe ff 7f 02 00 00 00
>> |................|
>> 4ffffffe090 f0 ff ff 7f 02 00 00 00 00 00 00 00 00 00 00 00
>> |................|
>> 4ffffffe0a0 00 00 00 00 00 00 00 00 4c 6c 99 5a bd 1a 4c 7d
>> |........Ll.Z..L}|
>> 4ffffffe0b0 32 29 7a bc e2 45 89 dd 00 00 08 00 f8 ff ff ff
>> |2)z..E..........|
>> 4ffffffe0c0 4f 66 58 58 00 00 00 00 7b 33 38 00 00 00 00 00
>> |OfXX....{38.....|
>> 4ffffffe0d0 ff ff ff ff ff ff ff ff 33 f3 f0 e6 80 00 00 00
>> |........3.......|
>> 4ffffffe0e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> |................|
>> *
>> 4ffffffe100 00 00 fe ff fe ff fe ff fe ff fe ff fe ff fe ff
>> |................|
>> 4ffffffe110 fe ff fe ff fe ff fe ff fe ff fe ff fe ff fe ff
>> |................|
>> *
>> 4ffffffe200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> |................|
>> *
>> 50000000000
>>
>> # blockdev --report /dev/sd[a,b]
>> RO RA SSZ BSZ StartSec Size Device
>> rw 256 512 4096 0 5497558138880 /dev/sda
>> rw 256 512 4096 0 5497558138880 /dev/sdb
>>
>> # mdadm /dev/md_root --add /dev/sda
>> mdadm: /dev/sda not large enough to join array
>>
>> ----------------
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Fwd: (user) Help needed: mdadm seems to constantly touch my disks
From: NeilBrown @ 2016-12-21 22:52 UTC (permalink / raw)
To: Jure Erznožnik, linux-raid, tkonto
In-Reply-To: <CAJ=9zicozhGYUy_iB7EPSRxMVDJBG0ArhKNqyKQYCVwA0zEkew@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2296 bytes --]
On Wed, Dec 21 2016, Jure Erznožnik wrote:
> Mr Brown,
>
> Let me begin with: please give me your paypal address or something so
> that I can at least buy you a beer or something
Thanks :-)
My first inclination is to say "no thanks" as I am very adequately
compensated by SUSE, and it is part of my role at SUSE to ensure the
upstream kernel remains healthy. Encouraging a health community is part
of that (and I often learn something while helping people fix things).
But my second inclination is to recognize that gratitude is an important
part of human interactions, and that a community is strong when
gratitude is appropriately given and received. It is not my place to
direct others how they should show gratitude.
So I'll tell you my paypal address is neil@brown.name and that I'm more
likely to enjoy hot chocolate than beer, but I'll also emphasize that
there is no expectation attached to this information. :-)
>
>
> Your analysis and discovery that iSCSI is the origin of writes got me
> thinking: how can he see that on md0 device if that device has two
> more layers (bcache + LVM) before iSCSI even comes into play. Maybe
> the system propagates the origin down the block devices or something,
> totally not relevant here. So I embarked on a journey of total data
> destruction by disabling one layer at a time. I started by simply
> detaching bcache as that was the first thing on the list - and was
> non-destructive to boot :)
>
> I have found the culprit:
> It is bcache that does the one second writes. I have yet to find the
> exact parameters that influence this behaviour, but the output of
> writeback_rate_debug is EXTREMELY clear: it's writing a bit of data
> each second, reducing the dirty cache by that tiny amount. This is
> what causes the write "amplification" resulting in clicks long after a
> write has been done - because bcache only writes tiny amounts each
> second instead of flushing the entire cache at once when the time
> comes.
Now that we have an understanding of what is happening, I can recommend
that you increase /sys/block/md0/md/safe_mode_delay. It is measured in
seconds. If you make it larger than the period of the bcache writes, it
should stop the 'ticking' you mentioned.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: RAID without dev size prevents adding a new member
From: NeilBrown @ 2016-12-21 23:27 UTC (permalink / raw)
To: Robert LeBlanc; +Cc: linux-raid
In-Reply-To: <CAANLjFpYKFNG8mmBfVmXuOGwcs5BvMwiqjsy9nLw3LpkhGtLyg@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1436 bytes --]
On Thu, Dec 22 2016, Robert LeBlanc wrote:
> It looks like it is still there on the latest master.
Thanks for checking!
>
> # ./mdadm --version
> mdadm - v3.4-88-gfa219dd - 22nd September 2016
>
> # ./mdadm --detail /dev/md_root
> /dev/md_root:
> Version : 1.0
> Creation Time : Wed Sep 7 15:58:30 2016
> Raid Level : raid1
> Array Size : 5368708928 (5120.00 GiB 5497.56 GB)
> Used Dev Size : unknown
This implies that mdadm is opening /dev/md_root/md/component_size ....
Ah-ha. get_component_size() hasn't been updated to work with named
arrays. it will only open
"/sys/block/md%d/md/component_size",
or
"/sys/block/md_d%d/md/component_size",
Please try with the following patch. I'll send a proper version to Jes.
Thanks,
NeilBrown
diff --git a/sysfs.c b/sysfs.c
index 84c7348526c9..b0657a04b3a3 100644
--- a/sysfs.c
+++ b/sysfs.c
@@ -400,14 +400,8 @@ unsigned long long get_component_size(int fd)
int n;
if (fstat(fd, &stb))
return 0;
- if (major(stb.st_rdev) != (unsigned)get_mdp_major())
- snprintf(fname, MAX_SYSFS_PATH_LEN,
- "/sys/block/md%d/md/component_size",
- (int)minor(stb.st_rdev));
- else
- snprintf(fname, MAX_SYSFS_PATH_LEN,
- "/sys/block/md_d%d/md/component_size",
- (int)minor(stb.st_rdev)>>MdpMinorShift);
+ snprintf(fname, MAX_SYSFS_PATH_LEN,
+ "/sys/block/%s/md/component_size", stat2devnm(&stb));
fd = open(fname, O_RDONLY);
if (fd < 0)
return 0;
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply related
* Re: [BUG] MD/RAID1 hung forever on freeze_array
From: NeilBrown @ 2016-12-21 23:51 UTC (permalink / raw)
To: Jinpu Wang; +Cc: linux-raid, Shaohua Li, Nate Dailey
In-Reply-To: <CAMGffE=OVX-jp+-SY8A4pmJHyVAkVQVXSCa_3VgndFf4tYyG9w@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 295 bytes --]
On Wed, Dec 21 2016, Jinpu Wang wrote:
>>
> Thanks, it does help a lot, I attached the patch I'm still testing,
> but so far so good.
> Could you check if I got it right?
Yes, that looks exactly right.
I guess one of us should try to push it upstream... maybe next year :-)
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply
* Re: RAID without dev size prevents adding a new member
From: Robert LeBlanc @ 2016-12-21 23:58 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
In-Reply-To: <87h95wrhh6.fsf@notabene.neil.brown.name>
On Wed, Dec 21, 2016 at 4:27 PM, NeilBrown <neilb@suse.com> wrote:
> snprintf(fname, MAX_SYSFS_PATH_LEN,
Bingo! The size is now reported and I could add the drive. Hopefully
this can get added to Red Hat... Thanks for the fix.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
^ permalink raw reply
* [mdadm PATCH] Make get_component_size() work with named array.
From: NeilBrown @ 2016-12-22 2:14 UTC (permalink / raw)
To: Jes Sorensen; +Cc: Robert LeBlanc, linux-raid
[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]
get_component_size() still assumes that all array are
/sys/block/md%d or /sys/block/md_d%d
and so doesn't work with e.g. /sys/block/md_foo.
This cause "mdadm --detail" to report
Used Dev Size : unknown
and causes problems when added spares and in other circumstances.
So change it to use stat2devnm() which does the right thing with all
types of array names.
Reported-and-tested-by: Robert LeBlanc <robert@leblancnet.us>
Signed-off-by: NeilBrown <neilb@suse.com>
---
sysfs.c | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
diff --git a/sysfs.c b/sysfs.c
index 84c7348526c9..b0657a04b3a3 100644
--- a/sysfs.c
+++ b/sysfs.c
@@ -400,14 +400,8 @@ unsigned long long get_component_size(int fd)
int n;
if (fstat(fd, &stb))
return 0;
- if (major(stb.st_rdev) != (unsigned)get_mdp_major())
- snprintf(fname, MAX_SYSFS_PATH_LEN,
- "/sys/block/md%d/md/component_size",
- (int)minor(stb.st_rdev));
- else
- snprintf(fname, MAX_SYSFS_PATH_LEN,
- "/sys/block/md_d%d/md/component_size",
- (int)minor(stb.st_rdev)>>MdpMinorShift);
+ snprintf(fname, MAX_SYSFS_PATH_LEN,
+ "/sys/block/%s/md/component_size", stat2devnm(&stb));
fd = open(fname, O_RDONLY);
if (fd < 0)
return 0;
--
2.11.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox