* [PATCH 1/7] Show device as cache in --detail
2015-05-14 6:43 [PATCH 0/7] mdadm support for caching layer in raid 5/6 Song Liu
@ 2015-05-14 6:43 ` Song Liu
2015-05-14 6:43 ` [PATCH 2/7] Enable create array with write cache (--write-cache DEVICE) Song Liu
` (6 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
Example output:
./mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Wed May 13 17:01:12 2015
Raid Level : raid5
Array Size : 11720662464 (11177.69 GiB 12001.96 GB)
Used Dev Size : 3906887488 (3725.90 GiB 4000.65 GB)
Raid Devices : 4
Total Devices : 5
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed May 13 17:01:12 2015
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 32K
Name : 0
UUID : 8fb9ee05:3831d52f:e5c23825:28cd6881
Events : 0
Number Major Minor RaidDevice State
0 8 32 0 active sync /dev/sdc
1 8 48 1 active sync /dev/sdd
2 8 64 2 active sync /dev/sde
3 8 80 3 active sync /dev/sdf
4 8 17 - cache /dev/sdb1
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Detail.c | 3 ++-
md_p.h | 2 ++
super1.c | 2 ++
3 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/Detail.c b/Detail.c
index dd72ede..c93a962 100644
--- a/Detail.c
+++ b/Detail.c
@@ -650,9 +650,10 @@ This is pretty boring
}
if (disk.state & (1<<MD_DISK_REMOVED)) printf(" removed");
if (disk.state & (1<<MD_DISK_WRITEMOSTLY)) printf(" writemostly");
+ if (disk.state & (1<<MD_DISK_WRITECACHE)) printf(" cache");
if ((disk.state &
((1<<MD_DISK_ACTIVE)|(1<<MD_DISK_SYNC)
- |(1<<MD_DISK_REMOVED)|(1<<MD_DISK_FAULTY)))
+ |(1<<MD_DISK_REMOVED)|(1<<MD_DISK_FAULTY)|(1<<MD_DISK_WRITECACHE)))
== 0) {
printf(" spare");
if (is_26) {
diff --git a/md_p.h b/md_p.h
index c4846ba..f45c4e0 100644
--- a/md_p.h
+++ b/md_p.h
@@ -85,6 +85,8 @@
*/
#define MD_DISK_REPLACEMENT 17
+#define MD_DISK_WRITECACHE 18 /* disk is used as the write cache in RAID-5/6 */
+
typedef struct mdp_device_descriptor_s {
__u32 number; /* 0 Device number in the entire set */
diff --git a/super1.c b/super1.c
index f0508fe..7af0fd5 100644
--- a/super1.c
+++ b/super1.c
@@ -465,6 +465,8 @@ static void examine_super1(struct supertype *st, char *homehost)
role = 0xFFFF;
if (role >= 0xFFFE)
printf("spare\n");
+ else if (role == 0xFFFD)
+ printf("cache\n");
else if (sb->feature_map & __cpu_to_le32(MD_FEATURE_REPLACEMENT))
printf("Replacement device %d\n", role);
else
--
1.8.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 2/7] Enable create array with write cache (--write-cache DEVICE).
2015-05-14 6:43 [PATCH 0/7] mdadm support for caching layer in raid 5/6 Song Liu
2015-05-14 6:43 ` [PATCH 1/7] Show device as cache in --detail Song Liu
@ 2015-05-14 6:43 ` Song Liu
2015-05-14 6:43 ` [PATCH 3/7] Create write-cache superblock in mdadm --create Song Liu
` (5 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
Specify the write cache device with --write-cache DEVICE
./mdadm --create -f /dev/md0 --assume-clean -c 32 --raid-devices=4 --level=5 /dev/sd[c-f] --write-cache /dev/sdb1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
Only one cache device is allowed. If multiple --write-cache
are given, mdadm will use the first and ignore others
./mdadm --create -f /dev/md0 --assume-clean -c 32 --raid-devices=4 --level=5 /dev/sd[c-f] --write-cache /dev/sdb1 --write-cache /dev/sdx
mdadm: Please specify only one cache device for the array.
mdadm: Ignoring --write-cache /dev/sdx...
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Create.c | 21 +++++++++++++++------
ReadMe.c | 1 +
mdadm.c | 19 +++++++++++++++++++
mdadm.h | 2 ++
super1.c | 8 ++++++++
5 files changed, 45 insertions(+), 6 deletions(-)
diff --git a/Create.c b/Create.c
index ef28da0..0d2a591 100644
--- a/Create.c
+++ b/Create.c
@@ -87,7 +87,7 @@ int Create(struct supertype *st, char *mddev,
unsigned long long minsize=0, maxsize=0;
char *mindisc = NULL;
char *maxdisc = NULL;
- int dnum;
+ int dnum, raid_disk_num;
struct mddev_dev *dv;
int fail=0, warn=0;
struct stat stb;
@@ -180,11 +180,11 @@ int Create(struct supertype *st, char *mddev,
pr_err("This metadata type does not support spare disks at create time\n");
return 1;
}
- if (subdevs > s->raiddisks+s->sparedisks) {
+ if (subdevs > s->raiddisks+s->sparedisks+s->cachedisks) {
pr_err("You have listed more devices (%d) than are in the array(%d)!\n", subdevs, s->raiddisks+s->sparedisks);
return 1;
}
- if (!have_container && subdevs < s->raiddisks+s->sparedisks) {
+ if (!have_container && subdevs < s->raiddisks+s->sparedisks+s->cachedisks) {
pr_err("You haven't given enough devices (real or missing) to create this array\n");
return 1;
}
@@ -397,6 +397,9 @@ int Create(struct supertype *st, char *mddev,
}
}
+ if (dv->disposition == 'c')
+ continue; /* skip write cache for size check */
+
freesize /= 2; /* convert to K */
if (s->chunk && s->chunk != UnSet) {
/* round to chunk size */
@@ -834,7 +837,7 @@ int Create(struct supertype *st, char *mddev,
for (pass=1; pass <=2 ; pass++) {
struct mddev_dev *moved_disk = NULL; /* the disk that was moved out of the insert point */
- for (dnum=0, dv = devlist ; dv ;
+ for (dnum=0, raid_disk_num=0, dv = devlist ; dv ;
dv=(dv->next)?(dv->next):moved_disk, dnum++) {
int fd;
struct stat stb;
@@ -859,8 +862,13 @@ int Create(struct supertype *st, char *mddev,
*inf = info;
inf->disk.number = dnum;
- inf->disk.raid_disk = dnum;
- if (inf->disk.raid_disk < s->raiddisks)
+ inf->disk.raid_disk = raid_disk_num++;
+
+ if (dv->disposition == 'c') {
+ inf->disk.raid_disk = 0xfffd;
+ inf->disk.state = (1<<MD_DISK_WRITECACHE);
+ raid_disk_num--;
+ } else if (inf->disk.raid_disk < s->raiddisks)
inf->disk.state = (1<<MD_DISK_ACTIVE) |
(1<<MD_DISK_SYNC);
else
@@ -909,6 +917,7 @@ int Create(struct supertype *st, char *mddev,
inf->disk.major = major(stb.st_rdev);
inf->disk.minor = minor(stb.st_rdev);
}
+
break;
case 2:
inf->errors = 0;
diff --git a/ReadMe.c b/ReadMe.c
index 87a4916..74f7162 100644
--- a/ReadMe.c
+++ b/ReadMe.c
@@ -140,6 +140,7 @@ struct option long_options[] = {
{"homehost", 1, 0, HomeHost},
{"symlinks", 1, 0, Symlinks},
{"data-offset",1, 0, DataOffset},
+ {"write-cache",1, 0, WriteCache},
/* For assemble */
{"uuid", 1, 0, 'u'},
diff --git a/mdadm.c b/mdadm.c
index 3e8c49b..9a7048b 100644
--- a/mdadm.c
+++ b/mdadm.c
@@ -74,6 +74,7 @@ int main(int argc, char *argv[])
.require_homehost = 1,
};
struct shape s = {
+ .cachedisks = 0,
.level = UnSet,
.layout = UnSet,
.bitmap_chunk = UnSet,
@@ -1137,6 +1138,24 @@ int main(int argc, char *argv[])
case O(INCREMENTAL, IncrementalPath):
remove_path = optarg;
continue;
+ case O(CREATE, WriteCache):
+ if (s.cachedisks) {
+ pr_err("Please specify only one cache device for the array.\n");
+ pr_err("Ignoring --write-cache %s...\n", optarg);
+ continue;
+ }
+ dv = xmalloc(sizeof(*dv));
+ dv->devname = optarg;
+ dv->disposition = 'c'; /* WriteCache */
+ dv->writemostly = writemostly;
+ dv->used = 0;
+ dv->next = NULL;
+ *devlistend = dv;
+ devlistend = &dv->next;
+ devs_found++;
+
+ s.cachedisks = 1;
+ continue;
}
/* We have now processed all the valid options. Anything else is
* an error
diff --git a/mdadm.h b/mdadm.h
index 141f963..d28caa0 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -344,6 +344,7 @@ enum special_options {
Dump,
Restore,
Action,
+ WriteCache,
};
enum prefix_standard {
@@ -423,6 +424,7 @@ struct context {
struct shape {
int raiddisks;
int sparedisks;
+ int cachedisks;
int level;
int layout;
char *layout_str;
diff --git a/super1.c b/super1.c
index 7af0fd5..f8a55c6 100644
--- a/super1.c
+++ b/super1.c
@@ -977,6 +977,10 @@ static void getinfo_super1(struct supertype *st, struct mdinfo *info, char *map)
case 0xFFFE:
info->disk.state = 1; /* faulty */
break;
+ case 0xFFFD:
+ info->disk.state = (1 << MD_DISK_WRITECACHE); /* faulty */
+ info->disk.raid_disk = role;
+ break;
default:
info->disk.state = 6; /* active and in sync */
info->disk.raid_disk = role;
@@ -1096,6 +1100,8 @@ static int update_super1(struct supertype *st, struct mdinfo *info,
int want;
if (info->disk.state & (1<<MD_DISK_ACTIVE))
want = info->disk.raid_disk;
+ else if (info->disk.state & (1<<MD_DISK_WRITECACHE))
+ want = 0xFFFD;
else
want = 0xFFFF;
if (sb->dev_roles[d] != __cpu_to_le16(want)) {
@@ -1422,6 +1428,8 @@ static int add_to_super1(struct supertype *st, mdu_disk_info_t *dk,
*rp = __cpu_to_le16(dk->raid_disk);
else if ((dk->state & ~2) == 0) /* active or idle -> spare */
*rp = 0xffff;
+ else if (dk->state & (1<<MD_DISK_WRITECACHE))
+ *rp = 0xfffd;
else
*rp = 0xfffe;
--
1.8.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 3/7] Create write-cache superblock in mdadm --create
2015-05-14 6:43 [PATCH 0/7] mdadm support for caching layer in raid 5/6 Song Liu
2015-05-14 6:43 ` [PATCH 1/7] Show device as cache in --detail Song Liu
2015-05-14 6:43 ` [PATCH 2/7] Enable create array with write cache (--write-cache DEVICE) Song Liu
@ 2015-05-14 6:43 ` Song Liu
2015-05-14 6:43 ` [PATCH 4/7] Assemble array with writecache Song Liu
` (4 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
Signed-off-by: Song Liu <songliubraving@fb.com>
---
md_p.h | 72 +++++++++++++++++++++++++++++++
mdadm.h | 7 ++-
super1.c | 148 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
3 files changed, 221 insertions(+), 6 deletions(-)
diff --git a/md_p.h b/md_p.h
index f45c4e0..8a30f7b 100644
--- a/md_p.h
+++ b/md_p.h
@@ -197,4 +197,76 @@ static inline __u64 md_event(mdp_super_t *sb) {
return (ev<<32)| sb->events_lo;
}
+/* all disk position of below struct start from rdev->start_offset */
+struct r5l_meta_header {
+ __u32 magic;
+ __u32 type;
+ __u32 checksum; /* checksum(metadata block + uuid) */
+ __u32 meta_size;
+ __u64 seq;
+ __u64 position; /* block number the meta is written */
+} __attribute__ ((__packed__));
+
+#define R5LOG_VERSION 0x1
+#define R5LOG_MAGIC 0x6433c509
+
+enum {
+ R5LOG_TYPE_META = 0,
+ R5LOG_TYPE_SUPER = 1,
+ R5LOG_TYPE_FLUSH_START = 2,
+ R5LOG_TYPE_FLUSH_END = 3,
+};
+
+struct r5l_super_block {
+ struct r5l_meta_header header;
+ __u32 version;
+ __u32 stripe_cache_size; /* bytes */
+ __u32 block_size; /* bytes */
+ __u32 stripe_data_size; /* bytes */
+ __u32 chunk_size; /* bytes */
+ __u32 stripe_size; /* bytes */
+ __u32 parity_disks;
+ __u32 zero_padding;
+ __u64 total_blocks; /* block */
+ __u64 first_block; /* block */
+ __u64 last_checkpoint; /* block */
+ __u64 update_time_sec;
+ __u64 update_time_nsec;
+ __u8 meta_checksum_type;
+ __u8 data_checksum_type;
+ __u8 uuid[16];
+ /* fill with 0 */
+} __attribute__ ((__packed__));
+
+enum {
+ R5LOG_CHECKSUM_CRC32 = 0,
+ R5LOG_CHECKSUM_NR = 1,
+};
+
+struct r5l_meta_payload {
+ __u16 payload_type;
+ __u16 payload_flags;
+ __u32 blocks; /* block. For parity, should be 1 or 2 pages */
+ __u64 location; /* sector. For data, it's raid sector.
+ For parity, it's stripe sector */
+ __u32 data_checksum[]; /* checksum(data + uuid) */
+} __attribute__ ((__packed__));
+
+enum {
+ /* type */
+ R5LOG_PAYLOAD_DATA = 0,
+ R5LOG_PAYLOAD_PARITY = 1,
+ /* flags */
+ R5LOG_PAYLOAD_DISCARD = 1,
+};
+
+struct r5l_meta_block {
+ struct r5l_meta_header header;
+ struct r5l_meta_payload payloads[];
+} __attribute__ ((__packed__));
+
+struct r5l_flush_block {
+ struct r5l_meta_header header;
+ __u64 flush_stripes[]; /* stripe sector */
+} __attribute__ ((__packed__));
#endif
diff --git a/mdadm.h b/mdadm.h
index d28caa0..d7a205c 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -969,6 +969,8 @@ extern struct superswitch {
/* validate container after assemble */
int (*validate_container)(struct mdinfo *info);
+ /* write super block of raid5-cache*/
+ int (*write_r5l_super)(struct supertype *st, int fd);
int swapuuid; /* true if uuid is bigending rather than hostendian */
int external;
const char *name; /* canonical metadata name */
@@ -1033,7 +1035,7 @@ struct supertype {
int retry_soon;
struct mdinfo *devs;
-
+ struct r5l_super_block *r5l_sb;
};
extern struct supertype *super_by_fd(int fd, char **subarray);
@@ -1585,3 +1587,6 @@ char *xstrdup(const char *str);
#define INVALID_SECTORS 1
/* And another special number needed for --data_offset=variable */
#define VARIABLE_OFFSET 3
+
+#define LOG_BLOCK_SIZE 4096
+extern int ExamineR5LSuper(struct r5l_super_block *sb_blk);
diff --git a/super1.c b/super1.c
index f8a55c6..f2697a8 100644
--- a/super1.c
+++ b/super1.c
@@ -133,6 +133,20 @@ struct misc_dev_info {
|MD_FEATURE_RESHAPE_BACKWARDS \
|MD_FEATURE_NEW_OFFSET \
)
+static int write_r5l_super1(struct supertype *st, int fd);
+
+static int role_from_sb(struct mdp_superblock_1 *sb)
+{
+ unsigned int d;
+ int role;
+
+ d = __le32_to_cpu(sb->dev_number);
+ if (d < __le32_to_cpu(sb->max_dev))
+ role = __le16_to_cpu(sb->dev_roles[d]);
+ else
+ role = 0xFFFF;
+ return role;
+}
static unsigned int calc_sb_1_csum(struct mdp_superblock_1 * sb)
{
@@ -458,11 +472,7 @@ static void examine_super1(struct supertype *st, char *homehost)
printf(")\n");
#endif
printf(" Device Role : ");
- d = __le32_to_cpu(sb->dev_number);
- if (d < __le32_to_cpu(sb->max_dev))
- role = __le16_to_cpu(sb->dev_roles[d]);
- else
- role = 0xFFFF;
+ role = role_from_sb(sb);
if (role >= 0xFFFE)
printf("spare\n");
else if (role == 0xFFFD)
@@ -1559,8 +1569,11 @@ static int write_init_super1(struct supertype *st)
unsigned long long dsize, array_size;
unsigned long long sb_offset;
unsigned long long data_offset;
+ struct devinfo *cache_di = NULL;
for (di = st->info; di; di = di->next) {
+ if (di->disk.state & (1 << MD_DISK_WRITECACHE))
+ cache_di = di;
if (di->disk.state & (1 << MD_DISK_FAULTY))
continue;
if (di->fd < 0)
@@ -1700,11 +1713,17 @@ static int write_init_super1(struct supertype *st)
rv = store_super1(st, di->fd);
if (rv == 0 && (__le32_to_cpu(sb->feature_map) & 1))
rv = st->ss->write_bitmap(st, di->fd);
+
+ if (cache_di == di) {
+ rv = write_r5l_super1(st, cache_di->fd);
+ }
+
close(di->fd);
di->fd = -1;
if (rv)
goto error_out;
}
+
error_out:
if (rv)
pr_err("Failed to write metadata to %s\n",
@@ -1758,6 +1777,7 @@ static int load_super1(struct supertype *st, int fd, char *devname)
unsigned long long dsize;
unsigned long long sb_offset;
struct mdp_superblock_1 *super;
+ struct r5l_super_block *r5l_super;
int uuid[4];
struct bitmap_super_s *bsb;
struct misc_dev_info *misc;
@@ -1848,11 +1868,18 @@ static int load_super1(struct supertype *st, int fd, char *devname)
return 1;
}
+ if (posix_memalign((void**)&r5l_super, 4096, LOG_BLOCK_SIZE) != 0) {
+ pr_err("could not allocate superblock\n");
+ free(super);
+ return 1;
+ }
+
if (aread(&afd, super, MAX_SB_SIZE) != MAX_SB_SIZE) {
if (devname)
pr_err("Cannot read superblock on %s\n",
devname);
free(super);
+ free(r5l_super);
return 1;
}
@@ -1861,6 +1888,7 @@ static int load_super1(struct supertype *st, int fd, char *devname)
pr_err("No super block found on %s (Expected magic %08x, got %08x)\n",
devname, MD_SB_MAGIC, __le32_to_cpu(super->magic));
free(super);
+ free(r5l_super);
return 2;
}
@@ -1869,6 +1897,7 @@ static int load_super1(struct supertype *st, int fd, char *devname)
pr_err("Cannot interpret superblock on %s - version is %d\n",
devname, __le32_to_cpu(super->major_version));
free(super);
+ free(r5l_super);
return 2;
}
if (__le64_to_cpu(super->super_offset) != sb_offset) {
@@ -1876,9 +1905,27 @@ static int load_super1(struct supertype *st, int fd, char *devname)
pr_err("No superblock found on %s (super_offset is wrong)\n",
devname);
free(super);
+ free(r5l_super);
return 2;
}
st->sb = super;
+ if (0xFFFD == role_from_sb(super)) {
+ if (lseek64(fd, 512 * (super->data_offset), 0) < 0LL) {
+ if (devname)
+ pr_err("Cannot seek to r5l superblock on %s: %s\n",
+ devname, strerror(errno));
+ free(r5l_super);
+ } else {
+ if (aread(&afd, r5l_super, LOG_BLOCK_SIZE) != LOG_BLOCK_SIZE) {
+ if (devname)
+ pr_err("Cannot read r5l superblock on %s: %s\n",
+ devname, strerror(errno));
+ free(r5l_super);
+ } else
+ st->r5l_sb = r5l_super;
+ }
+ } else
+ free(r5l_super);
bsb = (struct bitmap_super_s *)(((char*)super)+MAX_SB_SIZE);
@@ -2223,6 +2270,8 @@ static int write_bitmap1(struct supertype *st, int fd)
static void free_super1(struct supertype *st)
{
+ if (st->r5l_sb)
+ free(st->r5l_sb);
if (st->sb)
free(st->sb);
while (st->info) {
@@ -2233,6 +2282,7 @@ static void free_super1(struct supertype *st)
free(di);
}
st->sb = NULL;
+ st->r5l_sb = NULL;
}
#ifndef MDASSEMBLE
@@ -2389,6 +2439,93 @@ void *super1_make_v0(struct supertype *st, struct mdinfo *info, mdp_super_t *sb0
return ret;
}
+unsigned long crc32(
+ unsigned long crc,
+ const unsigned char *buf,
+ unsigned len);
+
+static int write_r5l_super1(struct supertype *st, int fd)
+{
+ struct r5l_super_block *sb_blk;
+ struct mdp_superblock_1 *sb = st->sb;
+ unsigned long long dsize;
+ int parity_disks;
+ struct align_fd afd;
+ __u32 crc;
+
+ init_afd(&afd, fd);
+ if (sb->level == 5 || sb->level == 4)
+ parity_disks = 1;
+ else if (sb->level == 6)
+ parity_disks = 2;
+ else {
+ pr_err("Write cache is only applicable to RAID 4/5/6.\n");
+ return 1;
+ }
+
+ if (!get_dev_size(fd, NULL, &dsize)) {
+ pr_err("Cannot get size of cache device.\n");
+ return 1;
+ }
+
+ if (st->r5l_sb)
+ free(st->r5l_sb);
+
+ if (posix_memalign((void**)&sb_blk, 4096, LOG_BLOCK_SIZE) != 0) {
+ pr_err("Could not allocate memory for writecache superblock.\n");
+ return 1;
+ }
+
+ memset(sb_blk, 0, LOG_BLOCK_SIZE);
+
+ sb_blk->header.magic = __cpu_to_le32(R5LOG_MAGIC);
+ sb_blk->header.type = __cpu_to_le32(R5LOG_TYPE_SUPER);
+ sb_blk->header.seq = __cpu_to_le64(random32());
+ sb_blk->header.meta_size = __cpu_to_le32(sizeof(*sb_blk));
+ sb_blk->version = __cpu_to_le32(R5LOG_VERSION);
+ sb_blk->stripe_cache_size = __cpu_to_le32(LOG_BLOCK_SIZE);
+ sb_blk->block_size = __cpu_to_le32(LOG_BLOCK_SIZE);
+ sb_blk->total_blocks = __cpu_to_le64(((dsize - (sb->data_offset << 9)) / LOG_BLOCK_SIZE) - 1);
+ sb_blk->stripe_data_size = __cpu_to_le32((sb->chunksize << 9) *
+ (sb->raid_disks - parity_disks));
+ sb_blk->chunk_size = __cpu_to_le32(sb->chunksize << 9);
+
+
+ sb_blk->stripe_size = __cpu_to_le32(sb_blk->chunk_size * sb->raid_disks);
+ sb_blk->parity_disks = __cpu_to_le32(parity_disks);
+
+ sb_blk->first_block = __cpu_to_le64(1);
+ sb_blk->last_checkpoint = __cpu_to_le64(1);
+ sb_blk->update_time_sec = __cpu_to_le64((unsigned long long)time(0));
+ sb_blk->update_time_nsec = 0;
+ sb_blk->meta_checksum_type = R5LOG_CHECKSUM_CRC32;
+ sb_blk->data_checksum_type = R5LOG_CHECKSUM_CRC32;
+ memcpy(sb_blk->uuid, sb->set_uuid, 16);
+
+ crc = __cpu_to_le32(crc32(0xffffffff, (unsigned char *)(sb_blk->uuid), sizeof(sb_blk->uuid)));
+ crc = __cpu_to_le32(crc32(crc, (unsigned char *)sb_blk, LOG_BLOCK_SIZE));
+ sb_blk->header.checksum = crc;
+
+ if (lseek64(fd, (sb->data_offset) * 512, 0) < 0LL) {
+ pr_err("cannot seek to offset of write cache superblock\n");
+ goto fail_to_write;
+ }
+ if (awrite(&afd, sb_blk, sizeof(*sb_blk)) != sizeof(*sb_blk)) {
+ pr_err("failed to store write cache superblock \n");
+ goto fail_to_write;
+ }
+ fsync(fd);
+
+ st->r5l_sb = sb_blk;
+ return 0;
+
+fail_to_write:
+ free(sb_blk);
+ st->r5l_sb = NULL;
+ return 1;
+}
+
+
struct superswitch super1 = {
#ifndef MDASSEMBLE
.examine_super = examine_super1,
@@ -2418,6 +2555,7 @@ struct superswitch super1 = {
.locate_bitmap = locate_bitmap1,
.write_bitmap = write_bitmap1,
.free_super = free_super1,
+ .write_r5l_super = write_r5l_super1,
#if __BYTE_ORDER == BIG_ENDIAN
.swapuuid = 0,
#else
--
1.8.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 4/7] Assemble array with writecache
2015-05-14 6:43 [PATCH 0/7] mdadm support for caching layer in raid 5/6 Song Liu
` (2 preceding siblings ...)
2015-05-14 6:43 ` [PATCH 3/7] Create write-cache superblock in mdadm --create Song Liu
@ 2015-05-14 6:43 ` Song Liu
2015-05-14 6:43 ` [PATCH 5/7] Check write cache in incremental Song Liu
` (3 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
Example output:
./mdadm --assemble /dev/md0 /dev/sd[c-f] /dev/sdb1
mdadm: /dev/md0 has been started with 4 drives and 1 cache.
mdadm checks superblock for cache devices. If the
array appears to have a cache device, but it is not given,
it will complain as
./mdadm --assemble /dev/md0 /dev/sd[c-f]
mdadm: Not safe to assemble with cache device missing, consider --force.
This can be overwritten with --force
./mdadm --assemble /dev/md0 /dev/sd[c-f] --force
mdadm: Force start with missing cache device...
mdadm: /dev/md0 has been started with 4 drives.
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Assemble.c | 52 ++++++++++++++++++++++++++++++++++++++++++----------
mdadm.h | 2 ++
super1.c | 22 ++++++++++++++++++++++
util.c | 3 ++-
4 files changed, 68 insertions(+), 11 deletions(-)
diff --git a/Assemble.c b/Assemble.c
index 25a103d..ccf142e 100644
--- a/Assemble.c
+++ b/Assemble.c
@@ -718,7 +718,8 @@ static int load_devices(struct devs *devices, char *devmap,
i = devcnt;
else
i = devices[devcnt].i.disk.raid_disk;
- if (i+1 == 0) {
+
+ if (i+1 == 0 || i == 0xfffd) {
if (nextspare < content->array.raid_disks*2)
nextspare = content->array.raid_disks*2;
i = nextspare++;
@@ -772,8 +773,9 @@ static int load_devices(struct devs *devices, char *devmap,
}
if (best[i] == -1
|| (devices[best[i]].i.events
- < devices[devcnt].i.events))
+ < devices[devcnt].i.events)) {
best[i] = devcnt;
+ }
}
devcnt++;
}
@@ -927,6 +929,7 @@ static int start_array(int mdfd,
unsigned int okcnt,
unsigned int sparecnt,
unsigned int rebuilding_cnt,
+ unsigned int cachecnt,
struct context *c,
int clean, char *avail,
int start_partial_ok,
@@ -938,6 +941,22 @@ static int start_array(int mdfd,
int i;
unsigned int req_cnt;
+ if (st->ss->require_cache) {
+ rv = st->ss->require_cache(st);
+ if (rv == 2) {
+ pr_err("BUG: Superblock not loaded in Assemble.c:start_array\n");
+ return 1;
+ }
+
+ if (cachecnt == 0 && rv == 1) {
+ if (!(c->force)) {
+ pr_err("Not safe to assemble with cache device missing, consider --force.\n");
+ return 1;
+ } else
+ pr_err("Force start with missing cache device...\n");
+ }
+ }
+
rv = set_array_info(mdfd, st, content);
if (rv && !err_ok) {
pr_err("failed to set array info for %s: %s\n",
@@ -1015,7 +1034,8 @@ static int start_array(int mdfd,
if (content->array.level == LEVEL_CONTAINER) {
if (c->verbose >= 0) {
pr_err("Container %s has been assembled with %d drive%s",
- mddev, okcnt+sparecnt, okcnt+sparecnt==1?"":"s");
+ mddev, okcnt+sparecnt+cachecnt,
+ okcnt+sparecnt+cachecnt==1?"":"s");
if (okcnt < (unsigned)content->array.raid_disks)
fprintf(stderr, " (out of %d)",
content->array.raid_disks);
@@ -1101,6 +1121,8 @@ static int start_array(int mdfd,
fprintf(stderr, "%s %d rebuilding", sparecnt?",":" and", rebuilding_cnt);
if (sparecnt)
fprintf(stderr, " and %d spare%s", sparecnt, sparecnt==1?"":"s");
+ if (cachecnt == 1)
+ fprintf(stderr, " and 1 cache");
fprintf(stderr, ".\n");
}
if (content->reshape_active &&
@@ -1268,7 +1290,7 @@ int Assemble(struct supertype *st, char *mddev,
int *best = NULL; /* indexed by raid_disk */
int bestcnt = 0;
int devcnt;
- unsigned int okcnt, sparecnt, rebuilding_cnt, replcnt;
+ unsigned int okcnt, sparecnt, rebuilding_cnt, replcnt, cachecnt;
int i;
int was_forced = 0;
int most_recent = 0;
@@ -1479,6 +1501,7 @@ try_again:
devcnt = load_devices(devices, devmap, ident, &st, devlist,
c, content, mdfd, mddev,
&most_recent, &bestcnt, &best, inargv);
+
if (devcnt < 0)
return 1;
@@ -1506,7 +1529,9 @@ try_again:
okcnt = 0;
replcnt = 0;
sparecnt=0;
+ cachecnt=0;
rebuilding_cnt=0;
+
for (i=0; i< bestcnt; i++) {
int j = best[i];
int event_margin = 1; /* always allow a difference of '1'
@@ -1516,8 +1541,10 @@ try_again:
/* note: we ignore error flags in multipath arrays
* as they don't make sense
*/
- if (content->array.level != LEVEL_MULTIPATH)
- if (!(devices[j].i.disk.state & (1<<MD_DISK_ACTIVE))) {
+ if (content->array.level != LEVEL_MULTIPATH) {
+ if (devices[j].i.disk.state & (1<<MD_DISK_WRITECACHE)) {
+ cachecnt++;
+ } else if (!(devices[j].i.disk.state & (1<<MD_DISK_ACTIVE))) {
if (!(devices[j].i.disk.state
& (1<<MD_DISK_FAULTY))) {
devices[j].uptodate = 1;
@@ -1525,6 +1552,7 @@ try_again:
}
continue;
}
+ }
/* If this device thinks that 'most_recent' has failed, then
* we must reject this device.
*/
@@ -1559,10 +1587,11 @@ try_again:
replcnt++;
} else
rebuilding_cnt++;
- } else
+ } else if (devices[j].i.disk.raid_disk != 0xfffd)
sparecnt++;
}
}
+
free(devmap);
if (c->force) {
int force_ok = force_array(content, devices, best, bestcnt,
@@ -1583,8 +1612,9 @@ try_again:
int j = best[i];
int fd;
- if (j<0)
+ if (j<0) {
continue;
+ }
if (!devices[j].uptodate)
continue;
if (devices[j].i.events < devices[most_recent].i.events)
@@ -1623,7 +1653,9 @@ try_again:
int j = best[i];
unsigned int desired_state;
- if (i >= content->array.raid_disks * 2)
+ if (devices[j].i.disk.raid_disk == 0xfffd)
+ desired_state = (1<<MD_DISK_WRITECACHE);
+ else if (i >= content->array.raid_disks * 2)
desired_state = 0;
else if (i & 1)
desired_state = (1<<MD_DISK_ACTIVE) | (1<<MD_DISK_REPLACEMENT);
@@ -1770,7 +1802,7 @@ try_again:
rv = start_array(mdfd, mddev, content,
st, ident, best, bestcnt,
chosen_drive, devices, okcnt, sparecnt,
- rebuilding_cnt,
+ rebuilding_cnt, cachecnt,
c,
clean, avail, start_partial_ok,
pre_exist != NULL,
diff --git a/mdadm.h b/mdadm.h
index d7a205c..62a0293 100644
--- a/mdadm.h
+++ b/mdadm.h
@@ -971,6 +971,8 @@ extern struct superswitch {
/* write super block of raid5-cache*/
int (*write_r5l_super)(struct supertype *st, int fd);
+ /* whether the array require a cache device */
+ int (*require_cache)(struct supertype *st);
int swapuuid; /* true if uuid is bigending rather than hostendian */
int external;
const char *name; /* canonical metadata name */
diff --git a/super1.c b/super1.c
index f2697a8..c345a40 100644
--- a/super1.c
+++ b/super1.c
@@ -135,6 +135,27 @@ struct misc_dev_info {
)
static int write_r5l_super1(struct supertype *st, int fd);
+/* return value:
+ * 0, cache not required
+ * 1, cache required
+ * 2, no superblock loated (st->sb == NULL)
+ */
+static int require_cache1(struct supertype *st)
+{
+ struct mdp_superblock_1 *sb = st->sb;
+ int i;
+ if (sb)
+ for (i=0; i<MAX_DEVS; i++) {
+ if (0xFFFD == sb->dev_roles[i])
+ return 1;
+ }
+ else
+ return 2; /* no sb loaded */
+ return 0;
+}
+
+static int write_r5l_super1(struct supertype *st, int fd);
+
static int role_from_sb(struct mdp_superblock_1 *sb)
{
unsigned int d;
@@ -2556,6 +2577,7 @@ struct superswitch super1 = {
.write_bitmap = write_bitmap1,
.free_super = free_super1,
.write_r5l_super = write_r5l_super1,
+ .require_cache = require_cache1,
#if __BYTE_ORDER == BIG_ENDIAN
.swapuuid = 0,
#else
diff --git a/util.c b/util.c
index cc98d3b..f314748 100644
--- a/util.c
+++ b/util.c
@@ -334,8 +334,9 @@ int enough(int level, int raid_disks, int layout, int clean, char *avail)
int i;
int avail_disks = 0;
- for (i = 0; i < raid_disks; i++)
+ for (i = 0; i < raid_disks; i++) {
avail_disks += !!avail[i];
+ }
switch (level) {
case 10:
--
1.8.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 5/7] Check write cache in incremental
2015-05-14 6:43 [PATCH 0/7] mdadm support for caching layer in raid 5/6 Song Liu
` (3 preceding siblings ...)
2015-05-14 6:43 ` [PATCH 4/7] Assemble array with writecache Song Liu
@ 2015-05-14 6:43 ` Song Liu
2015-05-14 6:43 ` [PATCH 6/7] Zero write-cache superblock in --zero-super Song Liu
` (2 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
If cache device is missing, do not start the array, and shows:
./mdadm -I /dev/sdf
mdadm: Cache device is missing, not safe to start yet.
The array will be started when the cache device is attached with -I
./mdadm -I /dev/sdb1
mdadm: /dev/sdb1 attached to /dev/md/0_0, which has been started.
To force start without cache device:
./mdadm -I /dev/sdf --run
mdadm: Trying to run with missing cache device
mdadm: /dev/sdf attached to /dev/md/0_0, which has been started.
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Incremental.c | 37 +++++++++++++++++++++++++++++++++----
super1.c | 2 ++
2 files changed, 35 insertions(+), 4 deletions(-)
diff --git a/Incremental.c b/Incremental.c
index 0c9a9a4..500fd9e 100644
--- a/Incremental.c
+++ b/Incremental.c
@@ -35,7 +35,7 @@
static int count_active(struct supertype *st, struct mdinfo *sra,
int mdfd, char **availp,
- struct mdinfo *info);
+ struct mdinfo *info, int *cache_device_missing);
static void find_reject(int mdfd, struct supertype *st, struct mdinfo *sra,
int number, __u64 events, int verbose,
char *array_name);
@@ -104,6 +104,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
struct map_ent target_array;
int have_target;
char *devname = devlist->devname;
+ int cache_device_missing = 0;
struct createinfo *ci = conf_get_create_info();
@@ -216,6 +217,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
free(st);
goto out;
}
+
close (dfd); dfd = -1;
st->ss->getinfo_super(st, &info, NULL);
@@ -470,6 +472,7 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
info.array.working_disks ++;
}
+
if (strncmp(chosen_name, "/dev/md/", 8) == 0)
md_devname = chosen_name+8;
else
@@ -511,10 +514,13 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
* state. Eventually this state should be kept up-to-date as
* things change.
*/
+
sysfs_free(sra);
sra = sysfs_read(mdfd, NULL, (GET_DEVS | GET_STATE |
GET_OFFSET | GET_SIZE));
- active_disks = count_active(st, sra, mdfd, &avail, &info);
+
+ active_disks = count_active(st, sra, mdfd, &avail, &info, &cache_device_missing);
+
if (enough(info.array.level, info.array.raid_disks,
info.array.layout, info.array.state & 1,
avail) == 0) {
@@ -544,10 +550,13 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
}
map_unlock(&map);
- if (c->runstop > 0 || active_disks >= info.array.working_disks) {
+ if (c->runstop > 0 || (!cache_device_missing && active_disks >= info.array.working_disks)) {
struct mdinfo *dsk;
/* Let's try to start it */
+ if (cache_device_missing)
+ pr_err("Trying to run with missing cache device\n");
+
if (info.reshape_active && !(info.reshape_active & RESHAPE_NO_BACKUP)) {
pr_err("%s: This array is being reshaped and cannot be started\n",
chosen_name);
@@ -614,6 +623,8 @@ int Incremental(struct mddev_dev *devlist, struct context *c,
} else {
if (c->export) {
printf("MD_STARTED=unsafe\n");
+ } else if (cache_device_missing) {
+ pr_err("Cache device is missing, not safe to start yet.\n");
} else if (c->verbose >= 0)
pr_err("%s attached to %s, not enough to start safely.\n",
devname, chosen_name);
@@ -680,7 +691,8 @@ static void find_reject(int mdfd, struct supertype *st, struct mdinfo *sra,
static int count_active(struct supertype *st, struct mdinfo *sra,
int mdfd, char **availp,
- struct mdinfo *bestinfo)
+ struct mdinfo *bestinfo,
+ int *cache_device_missing)
{
/* count how many devices in sra think they are active */
struct mdinfo *d;
@@ -694,6 +706,8 @@ static int count_active(struct supertype *st, struct mdinfo *sra,
int devnum;
int b, i;
int raid_disks = 0;
+ int require_cache_dev = 0;
+ int has_cache_dev = 0;
if (!sra)
return 0;
@@ -714,8 +728,19 @@ static int count_active(struct supertype *st, struct mdinfo *sra,
close(dfd);
if (ok != 0)
continue;
+
+ if (st->ss->require_cache) {
+ require_cache_dev = st->ss->require_cache(st);
+ if (require_cache_dev == 2) {
+ pr_err("BUG: Superblock not loaded in Incremental.c:count_active\n");
+ return 0;
+ }
+ }
+
info.array.raid_disks = raid_disks;
st->ss->getinfo_super(st, &info, devmap + raid_disks * devnum);
+ if (info.disk.raid_disk == 0xFFFD)
+ has_cache_dev = 1;
if (!avail) {
raid_disks = info.array.raid_disks;
avail = xcalloc(raid_disks, 1);
@@ -765,6 +790,10 @@ static int count_active(struct supertype *st, struct mdinfo *sra,
replcnt++;
st->ss->free_super(st);
}
+
+ if (require_cache_dev && !has_cache_dev)
+ *cache_device_missing = 1;
+
if (!avail)
return 0;
/* We need to reject any device that thinks the best device is
diff --git a/super1.c b/super1.c
index c345a40..e229efe 100644
--- a/super1.c
+++ b/super1.c
@@ -144,6 +144,7 @@ static int require_cache1(struct supertype *st)
{
struct mdp_superblock_1 *sb = st->sb;
int i;
+
if (sb)
for (i=0; i<MAX_DEVS; i++) {
if (0xFFFD == sb->dev_roles[i])
@@ -151,6 +152,7 @@ static int require_cache1(struct supertype *st)
}
else
return 2; /* no sb loaded */
+
return 0;
}
--
1.8.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 6/7] Zero write-cache superblock in --zero-super
2015-05-14 6:43 [PATCH 0/7] mdadm support for caching layer in raid 5/6 Song Liu
` (4 preceding siblings ...)
2015-05-14 6:43 ` [PATCH 5/7] Check write cache in incremental Song Liu
@ 2015-05-14 6:43 ` Song Liu
2015-05-14 6:43 ` [PATCH 7/7] Add information about write-cache superblock to --examine Song Liu
[not found] ` <14d521c83b0.276a.cf392d89e21755b689a9b33faa59c92f@cse.yorku.ca>
7 siblings, 0 replies; 10+ messages in thread
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Kill.c | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/Kill.c b/Kill.c
index f2fdb85..9dc566b 100644
--- a/Kill.c
+++ b/Kill.c
@@ -29,6 +29,18 @@
#include "md_u.h"
#include "md_p.h"
+int KillR5LSuper(struct supertype *st, int fd)
+{
+ memset(st->r5l_sb, 0, LOG_BLOCK_SIZE);
+
+ if (lseek64(fd, 512 * (st->data_offset), 0) < 0LL)
+ return 1;
+ if (write(fd, st->r5l_sb, LOG_BLOCK_SIZE))
+ return 1;
+ fsync(fd);
+ return 0;
+}
+
int Kill(char *dev, struct supertype *st, int force, int verbose, int noexcl)
{
/*
@@ -60,8 +72,15 @@ int Kill(char *dev, struct supertype *st, int force, int verbose, int noexcl)
return 2;
}
st->ignore_hw_compat = 1;
+ st->data_offset = INVALID_SECTORS; /* load st->data_offset from sb for KillR5LSuper */
rv = st->ss->load_super(st, fd, dev);
if (rv == 0 || (force && rv >= 2)) {
+ if (st->r5l_sb) {
+ if (verbose > 0)
+ pr_err("zero write_cache superblock\n");
+ KillR5LSuper(st, fd);
+ }
+
st->ss->free_super(st);
st->ss->init_super(st, NULL, 0, "", NULL, NULL,
INVALID_SECTORS);
--
1.8.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 7/7] Add information about write-cache superblock to --examine
2015-05-14 6:43 [PATCH 0/7] mdadm support for caching layer in raid 5/6 Song Liu
` (5 preceding siblings ...)
2015-05-14 6:43 ` [PATCH 6/7] Zero write-cache superblock in --zero-super Song Liu
@ 2015-05-14 6:43 ` Song Liu
[not found] ` <14d521c83b0.276a.cf392d89e21755b689a9b33faa59c92f@cse.yorku.ca>
7 siblings, 0 replies; 10+ messages in thread
From: Song Liu @ 2015-05-14 6:43 UTC (permalink / raw)
To: linux-raid; +Cc: shli, dan.j.williams, neilb, hch, Song Liu
Example output:
./mdadm -E /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 261d25c2:7264d13e:670f0307:71441559
Name : 0
Creation Time : Wed May 13 11:04:07 2015
Raid Level : raid5
Raid Devices : 4
Avail Dev Size : 706560 (345.00 MiB 361.76 MB)
Array Size : 11720662464 (11177.69 GiB 12001.96 GB)
Used Dev Size : 7813774976 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=18446744065896483200 sectors
State : clean
Device UUID : f89a0182:10bc963c:3e6a2735:16d306fa
Internal Bitmap : 8 sectors from superblock
Update Time : Wed May 13 11:54:37 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 74417070 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 32K
Device Role : cache
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
Write Cache Info:
Magic : 6433c509
Superblock Version : 1
Stripe Cache Size : 4096
Block Size : 8 sectors
Stripe Data Size : 192 sectors
Chunk Size : 64 sectors
Stripe Size : 256 sectors
Total Blocks : 88319 blocks
First Block : 1
Last Checkpoint : 58641
Update Time : Wed May 13 11:54:37 2015
Signed-off-by: Song Liu <songliubraving@fb.com>
---
Examine.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/Examine.c b/Examine.c
index 953b8ee..f6086a5 100644
--- a/Examine.c
+++ b/Examine.c
@@ -141,7 +141,8 @@ int Examine(struct mddev_dev *devlist,
} else
st->ss->getinfo_super(st, &ap->info, NULL);
if (!have_container &&
- !(ap->info.disk.state & (1<<MD_DISK_SYNC)))
+ !(ap->info.disk.state & (1<<MD_DISK_SYNC)) &&
+ !(ap->info.disk.state & (1<<MD_DISK_WRITECACHE)))
ap->spares++;
d = dl_strdup(devlist->devname);
dl_add(ap->devs, d);
@@ -152,6 +153,9 @@ int Examine(struct mddev_dev *devlist,
} else {
printf("%s:\n",devlist->devname);
st->ss->examine_super(st, c->homehost);
+ if (st->r5l_sb) {
+ ExamineR5LSuper(st->r5l_sb);
+ }
st->ss->free_super(st);
}
}
@@ -223,3 +227,26 @@ out:
}
return err;
}
+
+int ExamineR5LSuper(struct r5l_super_block *sb_blk)
+{
+ time_t atime;
+
+ if (sb_blk == NULL)
+ return 1;
+ printf("\n\nWrite Cache Info:\n");
+ printf(" Magic : %08x\n", __le32_to_cpu(sb_blk->header.magic));
+ printf("Superblock Version : %d\n", sb_blk->version);
+ printf(" Stripe Cache Size : %d\n", sb_blk->stripe_cache_size);
+ printf(" Block Size : %d sectors\n", sb_blk->block_size >> 9);
+ printf(" Stripe Data Size : %d sectors\n", sb_blk->stripe_data_size >> 9);
+ printf(" Chunk Size : %d sectors\n", sb_blk->chunk_size >> 9);
+ printf(" Stripe Size : %d sectors\n", sb_blk->stripe_size >> 9);
+ printf(" Total Blocks : %lld blocks\n", sb_blk->total_blocks);
+ printf(" Last Checkpoint : %lld\n", sb_blk->last_checkpoint);
+
+ atime = __le64_to_cpu(sb_blk->update_time_sec) & 0xFFFFFFFFFFULL;
+ printf(" Update Time : %.24s\n", ctime(&atime));
+
+ return 0;
+}
--
1.8.1
^ permalink raw reply related [flat|nested] 10+ messages in thread[parent not found: <14d521c83b0.276a.cf392d89e21755b689a9b33faa59c92f@cse.yorku.ca>]
* Re: [PATCH 0/7] mdadm support for caching layer in raid 5/6
[not found] ` <14d521c83b0.276a.cf392d89e21755b689a9b33faa59c92f@cse.yorku.ca>
@ 2015-05-14 11:30 ` Jason Keltz
2015-05-14 18:01 ` Song Liu
0 siblings, 1 reply; 10+ messages in thread
From: Jason Keltz @ 2015-05-14 11:30 UTC (permalink / raw)
To: Song Liu, linux-raid
Hi..
I'm curious - why not support write cache for other raid levels? If the
write cache becomes a fast SSD (with high endurance) for a pool that is all
standard HD then wouldn't the write cache benefit there as well? (Eg.
Raid10) . Can the write cache also be an MD device? I would think you
could use that to improve performance and redundancy of the cache?
I'm using RHEL 7.1 on an NFS server for a virtualization back end. Because
of sync writes the write performance is poor. Its really no surprise.. I've
got a 22 disk raid10. At the moment I could try dm-cache in rhel7 in an
effort to improve write performance but its a technology preview and not
sure I'd want to rely on that for production. It's not really just a
simple write cache either which is all that I'm really after. I like
simple! I could also enable async writes on my NFS export and not sleep at
night (only 3 virtualization nodes.. UPS protection everywhere protected..
But what if the file server crashes? You want eerie? I just got an email
that power is off on one node of my test setup....UPS failed... Aie! That
could just as easily have been the FS.) ..I could stop using MD for NFS
backend storage altogether and rely on a raid card with battery backed
cache as well and then when there's some kind of card firmware issue wait
for the hardware vendor to close my support ticket due to inactivity before
they even respond.... which has happened several times with a common 3
letter storage company that has more than 3 letters in their name now... I
could wait for someone to create a magical hardware device that would do
for MD software raid what the battery backed cache does for hardware raid.
To this day I really don't understand why that hasn't been done. It would
be a whole lot cheaper than SSD. . it would have higher longevity.... I'd
buy it... And I suspect a billion other people as well... But I guess
simple write caching will help as well....
Since I don't have any of the above goodies I've decided to try a switch
from ext4 to xfs. Then test with HD based external log... Then Ill try
SSD based external log ... And finally potentially external log to an MD
raid1 SSD log device.... Though I can't find too many details, I've got a
hunch that if the log fails I'd be in trouble...
J.
Sent with AquaMail for Android
http://www.aqua-mail.com
On May 14, 2015 2:44:04 AM Song Liu <songliubraving@fb.com> wrote:
> Hi,
>
> These are mdadm patches to support cache layer in raid 5/6. Shaohua
> has sent the kernel patch earlier with subject "a caching layer
> for raid 5/6".
>
> These patches add write cache support for the following commands:
>
> mdadm --detail
> mdadm --create
> mdadm --assemble
> mdadm --incremental
> mdadm --examine
> mdadm --zero-super
>
> Cache device is assigned with dev_role 0xFFFD (where 0xFFFF is for
> spare and 0xFFFE is for failed). Note that there is compatibility
> issue that older mdadm will show cache device as spare in --detail:
>
> Number Major Minor RaidDevice State
> 0 8 32 0 active sync /dev/sdc
> 1 8 48 1 active sync /dev/sdd
> 2 8 64 2 active sync /dev/sde
> 3 8 80 3 active sync /dev/sdf
>
> 4 8 17 - spare /dev/sdb1
>
> Also, older mdadm will show cache device as "Active device 65533"
> in --examine:
>
> Device Role : Active device 65533
> Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
>
>
> Song Liu (7):
> Show device as cache in --detail
> Enable create array with write cache (--write-cache DEVICE).
> Create write-cache superblock in mdadm --create
> Assemble array with writecache
> Check write cache in incremental
> Zero write-cache superblock in --zero-super
> Add information about write-cache superblock to --examine
>
> Assemble.c | 52 +++++++++++++----
> Create.c | 21 +++++--
> Detail.c | 3 +-
> Examine.c | 29 +++++++++-
> Incremental.c | 37 ++++++++++--
> Kill.c | 19 ++++++
> ReadMe.c | 1 +
> md_p.h | 74 ++++++++++++++++++++++++
> mdadm.c | 19 ++++++
> mdadm.h | 11 +++-
> super1.c | 182 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
> util.c | 3 +-
> 12 files changed, 422 insertions(+), 29 deletions(-)
>
> --
> 1.8.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 10+ messages in thread* RE: [PATCH 0/7] mdadm support for caching layer in raid 5/6
2015-05-14 11:30 ` [PATCH 0/7] mdadm support for caching layer in raid 5/6 Jason Keltz
@ 2015-05-14 18:01 ` Song Liu
0 siblings, 0 replies; 10+ messages in thread
From: Song Liu @ 2015-05-14 18:01 UTC (permalink / raw)
To: Jason Keltz, linux-raid@vger.kernel.org
Hi Jason,
As Shaohua discussed earlier in another thread, in this write cache approach,
we are trying to solve two problems together:
1. RAID-5/6 write hole;
2. Slow sync writes to RAID-5/6 (due to read-modify-write).
For other RAID types, there is no write hole issue. To get better sync write
performance on RAID-10, you can use other caching solutions, like bcache
or flashcache.
The write cache can also be a MD device.
Thanks,
Song
> -----Original Message-----
> From: Jason Keltz [mailto:jas@cse.yorku.ca]
> Sent: Thursday, May 14, 2015 4:31 AM
> To: Song Liu; linux-raid@vger.kernel.org
> Subject: Re: [PATCH 0/7] mdadm support for caching layer in raid 5/6
>
> Hi..
> I'm curious - why not support write cache for other raid levels? If the write
> cache becomes a fast SSD (with high endurance) for a pool that is all standard
> HD then wouldn't the write cache benefit there as well? (Eg.
> Raid10) . Can the write cache also be an MD device? I would think you could
> use that to improve performance and redundancy of the cache?
>
> I'm using RHEL 7.1 on an NFS server for a virtualization back end. Because of
> sync writes the write performance is poor. Its really no surprise.. I've got a 22
> disk raid10. At the moment I could try dm-cache in rhel7 in an effort to
> improve write performance but its a technology preview and not sure I'd want
> to rely on that for production. It's not really just a simple write cache either
> which is all that I'm really after. I like simple! I could also enable async writes
> on my NFS export and not sleep at night (only 3 virtualization nodes.. UPS
> protection everywhere protected..
> But what if the file server crashes? You want eerie? I just got an email that
> power is off on one node of my test setup....UPS failed... Aie! That could just as
> easily have been the FS.) ..I could stop using MD for NFS backend storage
> altogether and rely on a raid card with battery backed cache as well and then
> when there's some kind of card firmware issue wait for the hardware vendor to
> close my support ticket due to inactivity before they even respond.... which has
> happened several times with a common 3 letter storage company that has
> more than 3 letters in their name now... I could wait for someone to create a
> magical hardware device that would do for MD software raid what the battery
> backed cache does for hardware raid.
> To this day I really don't understand why that hasn't been done. It would be a
> whole lot cheaper than SSD. . it would have higher longevity.... I'd buy it... And I
> suspect a billion other people as well... But I guess simple write caching will
> help as well....
>
> Since I don't have any of the above goodies I've decided to try a switch
> from ext4 to xfs. Then test with HD based external log... Then Ill try
> SSD based external log ... And finally potentially external log to an MD
> raid1 SSD log device.... Though I can't find too many details, I've got a hunch
> that if the log fails I'd be in trouble...
>
> J.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
>
> On May 14, 2015 2:44:04 AM Song Liu <songliubraving@fb.com> wrote:
>
> > Hi,
> >
> > These are mdadm patches to support cache layer in raid 5/6. Shaohua
> > has sent the kernel patch earlier with subject "a caching layer for
> > raid 5/6".
> >
> > These patches add write cache support for the following commands:
> >
> > mdadm --detail
> > mdadm --create
> > mdadm --assemble
> > mdadm --incremental
> > mdadm --examine
> > mdadm --zero-super
> >
> > Cache device is assigned with dev_role 0xFFFD (where 0xFFFF is for
> > spare and 0xFFFE is for failed). Note that there is compatibility
> > issue that older mdadm will show cache device as spare in --detail:
> >
> > Number Major Minor RaidDevice State
> > 0 8 32 0 active sync /dev/sdc
> > 1 8 48 1 active sync /dev/sdd
> > 2 8 64 2 active sync /dev/sde
> > 3 8 80 3 active sync /dev/sdf
> >
> > 4 8 17 - spare /dev/sdb1
> >
> > Also, older mdadm will show cache device as "Active device 65533"
> > in --examine:
> >
> > Device Role : Active device 65533
> > Array State : AAAA ('A' == active, '.' == missing, 'R' ==
> > replacing)
> >
> >
> > Song Liu (7):
> > Show device as cache in --detail
> > Enable create array with write cache (--write-cache DEVICE).
> > Create write-cache superblock in mdadm --create
> > Assemble array with writecache
> > Check write cache in incremental
> > Zero write-cache superblock in --zero-super
> > Add information about write-cache superblock to --examine
> >
> > Assemble.c | 52 +++++++++++++----
> > Create.c | 21 +++++--
> > Detail.c | 3 +-
> > Examine.c | 29 +++++++++-
> > Incremental.c | 37 ++++++++++--
> > Kill.c | 19 ++++++
> > ReadMe.c | 1 +
> > md_p.h | 74 ++++++++++++++++++++++++
> > mdadm.c | 19 ++++++
> > mdadm.h | 11 +++-
> > super1.c | 182
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
> > util.c | 3 +-
> > 12 files changed, 422 insertions(+), 29 deletions(-)
> >
> > --
> > 1.8.1
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid"
> > in the body of a message to majordomo@vger.kernel.org More majordomo
> > info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 10+ messages in thread