[RFC] FIEMAP and disk sync issues

[RFC] FIEMAP and disk sync issues

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1915 bytes --]

Hello, all. When someone installs using blocklists (discouraged and 
unreliable), we use GRUB internal fs reader in order to determine 
core.img blocklists. With Linux we call fsync and an ioctl on partition 
in order to flush the cache. I also tried adding "sync" which didn't 
make any difference. Unfortunately it doesn't seem to be good enough on 
some filesystems e.g. ext2 (ext3 and ext4 seem to work).
Does anyone know how to ensure flush of a file AND of all filesystem 
metadata needed to read it?
Alternatively we can ask Linux itself to tell us the blocks the file 
resides in. There are 2 ioctls for this: FIBMAP and FIEMAP. FIBMAP is 
unfortunately limited to 32-bit blocks. Most of FS supported by both 
GRUB and Linux lack FIEMAP. If we exclude the 32-bit fs (which will work 
fine with FIBMAP as long as it uses the same block size as underlying 
FS, which seems reasonable but I haven't verified it) (minix1/2/3, 
reiserfs, hfs, jfs, udf, fat, affs) and read-only filesystems 
(unsuitable for install) (iso9660, romfs, squash4), the remaining 
trouble is NTFS and HFS+. So with this approach we'll be unable to 
install on >2TiB HFS+ or NTFS filesystems. I see following possibilities:
- Someone write fiemap support for HFS+ and NTFS. This would be probably 
the best approach
- We find a way to flush disk reliably. This would eliminate the 
necessity of having different branches for Linux and rest.
- Ask Linux devs if they are willing to provide FIBMAP64 since 
internally 64-bits are used.
Another problem with FIBMAP/FIEMAP is that fuse-mounted BFS lacks them 
altogether. We will need to find some other way of doing BFS install.
Also some COW filesystems may supply a somewhat valid blocklist with 
FIBMAP/FIEMAP but which is unusable to GRUB because of COW nature. So we 
need an FS-whitelist.
Experiment patch attached.
Any thoughts?

-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko


[-- Attachment #2: fiemap.diff --]
[-- Type: text/x-diff, Size: 17098 bytes --]

=== modified file 'grub-core/fs/affs.c'
--- grub-core/fs/affs.c	2012-01-14 14:44:34 +0000
+++ grub-core/fs/affs.c	2012-01-27 11:55:52 +0000
@@ -571,6 +571,7 @@
     .label = grub_affs_label,
 #ifdef GRUB_UTIL
     .reserved_first_sector = 0,
+    .blocklist_install = 1,
 #endif
     .next = 0
   };

=== modified file 'grub-core/fs/bfs.c'
--- grub-core/fs/bfs.c	2011-10-30 15:10:18 +0000
+++ grub-core/fs/bfs.c	2012-01-27 11:56:05 +0000
@@ -1026,6 +1026,7 @@
 #endif
 #ifdef GRUB_UTIL
   .reserved_first_sector = 1,
+  .blocklist_install = 1,
 #endif
 };
 

=== modified file 'grub-core/fs/btrfs.c'
--- grub-core/fs/btrfs.c	2012-01-25 14:10:56 +0000
+++ grub-core/fs/btrfs.c	2012-01-27 11:56:17 +0000
@@ -1653,6 +1653,7 @@
 #ifdef GRUB_UTIL
   .embed = grub_btrfs_embed,
   .reserved_first_sector = 1,
+  .blocklist_install = 0,
 #endif
 };
 

=== modified file 'grub-core/fs/cpio.c'
--- grub-core/fs/cpio.c	2011-12-24 14:09:26 +0000
+++ grub-core/fs/cpio.c	2012-01-27 11:56:31 +0000
@@ -714,6 +714,7 @@
   .close = grub_cpio_close,
 #ifdef GRUB_UTIL
   .reserved_first_sector = 0,
+  .blocklist_install = 0,
 #endif
 };
 

=== modified file 'grub-core/fs/ext2.c'
--- grub-core/fs/ext2.c	2012-01-14 10:55:20 +0000
+++ grub-core/fs/ext2.c	2012-01-27 11:56:43 +0000
@@ -971,6 +971,7 @@
     .mtime = grub_ext2_mtime,
 #ifdef GRUB_UTIL
     .reserved_first_sector = 1,
+    .blocklist_install = 1,
 #endif
     .next = 0
   };

=== modified file 'grub-core/fs/fat.c'
--- grub-core/fs/fat.c	2011-12-13 00:41:16 +0000
+++ grub-core/fs/fat.c	2012-01-27 11:56:56 +0000
@@ -1158,6 +1158,7 @@
 #else
     .reserved_first_sector = 1,
 #endif
+    .blocklist_install = 1,
 #endif
     .next = 0
   };

=== modified file 'grub-core/fs/hfs.c'
--- grub-core/fs/hfs.c	2012-01-14 22:34:33 +0000
+++ grub-core/fs/hfs.c	2012-01-27 11:57:13 +0000
@@ -1356,6 +1356,10 @@
     .label = grub_hfs_label,
     .uuid = grub_hfs_uuid,
     .mtime = grub_hfs_mtime,
+#ifdef GRUB_UTIL
+    .reserved_first_sector = 1,
+    .blocklist_install = 1,
+#endif
     .next = 0
   };
 

=== modified file 'grub-core/fs/hfsplus.c'
--- grub-core/fs/hfsplus.c	2012-01-14 22:34:33 +0000
+++ grub-core/fs/hfsplus.c	2012-01-27 11:57:20 +0000
@@ -1084,6 +1084,7 @@
     .uuid = grub_hfsplus_uuid,
 #ifdef GRUB_UTIL
     .reserved_first_sector = 1,
+    .blocklist_install = 1,
 #endif
     .next = 0
   };

=== modified file 'grub-core/fs/iso9660.c'
--- grub-core/fs/iso9660.c	2011-12-13 15:35:12 +0000
+++ grub-core/fs/iso9660.c	2012-01-27 11:57:51 +0000
@@ -1092,6 +1092,10 @@
     .label = grub_iso9660_label,
     .uuid = grub_iso9660_uuid,
     .mtime = grub_iso9660_mtime,
+#ifdef GRUB_UTIL
+    .reserved_first_sector = 1,
+    .blocklist_install = 1,
+#endif
     .next = 0
   };
 

=== modified file 'grub-core/fs/jfs.c'
--- grub-core/fs/jfs.c	2011-12-13 22:11:48 +0000
+++ grub-core/fs/jfs.c	2012-01-27 11:58:04 +0000
@@ -906,6 +906,10 @@
     .close = grub_jfs_close,
     .label = grub_jfs_label,
     .uuid = grub_jfs_uuid,
+#ifdef GRUB_UTIL
+    .reserved_first_sector = 1,
+    .blocklist_install = 1,
+#endif
     .next = 0
   };
 

=== modified file 'grub-core/fs/minix.c'
--- grub-core/fs/minix.c	2011-12-25 21:44:42 +0000
+++ grub-core/fs/minix.c	2012-01-27 11:58:13 +0000
@@ -653,6 +653,10 @@
     .open = grub_minix_open,
     .read = grub_minix_read,
     .close = grub_minix_close,
+#ifdef GRUB_UTIL
+    .reserved_first_sector = 1,
+    .blocklist_install = 1,
+#endif
     .next = 0
   };
 

=== modified file 'grub-core/fs/nilfs2.c'
--- grub-core/fs/nilfs2.c	2012-01-25 14:09:00 +0000
+++ grub-core/fs/nilfs2.c	2012-01-27 11:58:24 +0000
@@ -1178,6 +1178,7 @@
   .mtime = grub_nilfs2_mtime,
 #ifdef GRUB_UTIL
   .reserved_first_sector = 1,
+  .blocklist_install = 0,
 #endif
   .next = 0
 };

=== modified file 'grub-core/fs/ntfs.c'
--- grub-core/fs/ntfs.c	2012-01-20 14:01:35 +0000
+++ grub-core/fs/ntfs.c	2012-01-27 11:58:35 +0000
@@ -1248,6 +1248,7 @@
     .uuid = grub_ntfs_uuid,
 #ifdef GRUB_UTIL
     .reserved_first_sector = 1,
+    .blocklist_install = 1,
 #endif
     .next = 0
 };

=== modified file 'grub-core/fs/reiserfs.c'
--- grub-core/fs/reiserfs.c	2012-01-14 22:36:21 +0000
+++ grub-core/fs/reiserfs.c	2012-01-27 11:58:46 +0000
@@ -1387,6 +1387,10 @@
     .close = grub_reiserfs_close,
     .label = grub_reiserfs_label,
     .uuid = grub_reiserfs_uuid,
+#ifdef GRUB_UTIL
+    .reserved_first_sector = 1,
+    .blocklist_install = 1,
+#endif
     .next = 0
   };
 

=== modified file 'grub-core/fs/romfs.c'
--- grub-core/fs/romfs.c	2011-12-13 22:15:56 +0000
+++ grub-core/fs/romfs.c	2012-01-27 11:58:54 +0000
@@ -452,6 +452,7 @@
     .label = grub_romfs_label,
 #ifdef GRUB_UTIL
     .reserved_first_sector = 0,
+    .blocklist_install = 0,
 #endif
     .next = 0
   };

=== modified file 'grub-core/fs/sfs.c'
--- grub-core/fs/sfs.c	2011-12-25 21:39:56 +0000
+++ grub-core/fs/sfs.c	2012-01-27 11:59:04 +0000
@@ -616,6 +616,7 @@
     .label = grub_sfs_label,
 #ifdef GRUB_UTIL
     .reserved_first_sector = 0,
+    .blocklist_install = 1,
 #endif
     .next = 0
   };

=== modified file 'grub-core/fs/squash4.c'
--- grub-core/fs/squash4.c	2012-01-27 12:09:57 +0000
+++ grub-core/fs/squash4.c	2012-01-27 12:11:00 +0000
@@ -935,6 +935,7 @@
     .mtime = grub_squash_mtime,
 #ifdef GRUB_UTIL
     .reserved_first_sector = 0,
+    .blocklist_install = 0,
 #endif
     .next = 0
   };

=== modified file 'grub-core/fs/udf.c'
--- grub-core/fs/udf.c	2011-12-13 12:58:33 +0000
+++ grub-core/fs/udf.c	2012-01-27 11:59:22 +0000
@@ -1097,6 +1097,10 @@
   .read = grub_udf_read,
   .close = grub_udf_close,
   .label = grub_udf_label,
+#ifdef GRUB_UTIL
+  .reserved_first_sector = 1,
+  .blocklist_install = 1,
+#endif
   .next = 0
 };
 

=== modified file 'grub-core/fs/ufs.c'
--- grub-core/fs/ufs.c	2011-11-03 14:00:45 +0000
+++ grub-core/fs/ufs.c	2012-01-27 12:00:04 +0000
@@ -792,6 +792,10 @@
     .label = grub_ufs_label,
     .uuid = grub_ufs_uuid,
     .mtime = grub_ufs_mtime,
+    /* FIXME: set reserved_first_sector.  */
+#ifdef GRUB_UTIL
+    .blocklist_install = 1,
+#endif
     .next = 0
   };
 

=== modified file 'grub-core/fs/xfs.c'
--- grub-core/fs/xfs.c	2011-12-13 01:02:38 +0000
+++ grub-core/fs/xfs.c	2012-01-27 12:00:20 +0000
@@ -884,6 +884,7 @@
     .uuid = grub_xfs_uuid,
 #ifdef GRUB_UTIL
     .reserved_first_sector = 0,
+    .blocklist_install = 1,
 #endif
     .next = 0
   };

=== modified file 'grub-core/fs/zfs/zfs.c'
--- grub-core/fs/zfs/zfs.c	2012-01-27 12:50:21 +0000
+++ grub-core/fs/zfs/zfs.c	2012-01-27 12:51:06 +0000
@@ -3931,6 +3931,7 @@
 #ifdef GRUB_UTIL
   .embed = grub_zfs_embed,
   .reserved_first_sector = 1,
+  .blocklist_install = 0,
 #endif
   .next = 0
 };

=== modified file 'include/grub/fs.h'
--- include/grub/fs.h	2012-01-24 12:31:12 +0000
+++ include/grub/fs.h	2012-01-27 11:55:24 +0000
@@ -86,6 +86,9 @@
 
   /* Whether this filesystem reserves first sector for DOS-style boot.  */
   int reserved_first_sector;
+
+  /* Whether blocklist installs have a chance to work.  */
+  int blocklist_install;
 #endif
 };
 typedef struct grub_fs *grub_fs_t;

=== modified file 'util/grub-setup.c'
--- util/grub-setup.c	2012-01-24 13:39:29 +0000
+++ util/grub-setup.c	2012-01-27 12:24:59 +0000
@@ -51,6 +51,12 @@
 #include <grub/msdos_partition.h>
 #include <include/grub/crypto.h>
 
+#ifdef __linux__
+#include <sys/ioctl.h>
+#include <linux/fs.h>
+#include <linux/fiemap.h>
+#endif
+
 #define _GNU_SOURCE	1
 #include <argp.h>
 
@@ -141,7 +147,7 @@
   size_t boot_size, core_size;
   grub_uint16_t core_sectors;
   grub_device_t root_dev, dest_dev;
-  struct grub_boot_blocklist *first_block, *block;
+  struct grub_boot_blocklist *first_block, *block, *last_block;
   char *tmp_img;
   int i;
   grub_disk_addr_t first_sector;
@@ -150,7 +156,6 @@
     = GRUB_BOOT_MACHINE_KERNEL_SEG + (GRUB_DISK_SECTOR_SIZE >> 4);
 #endif
   grub_uint16_t last_length = GRUB_DISK_SECTOR_SIZE;
-  grub_file_t file;
   FILE *fp;
 
   auto void NESTED_FUNC_ATTR save_first_sector (grub_disk_addr_t sector,
@@ -490,11 +495,23 @@
 		       "the root device is on a RAID array or LVM volume"));
 
 #ifdef GRUB_MACHINE_PCBIOS
-  if (dest_dev->disk->id != root_dev->disk->id)
+  if (dest_dev->disk->id != root_dev->disk->id
+      || dest_dev->disk->dev->id != root_dev->disk->dev->id)
     grub_util_error (_("embedding is not possible, but this is required for "
 		       "cross-disk install"));
 #endif
 
+  {
+    grub_fs_t fs;
+    fs = grub_fs_probe (root_dev);
+    if (!fs)
+      grub_util_error (_("can't determine filesystem"));
+
+    if (!fs->blocklist_install)
+      grub_util_error (_("filesystem '%s' doesn't support blocklists"),
+		       fs->name);
+  }
+
   grub_util_warn (_("Embedding is not possible.  GRUB can only be installed in this "
 		    "setup by using blocklists.  However, blocklists are UNRELIABLE and "
 		    "their use is discouraged."));
@@ -512,10 +529,14 @@
 
   grub_util_biosdisk_flush (root_dev->disk);
 
+#ifndef __linux__
+
 #define MAX_TRIES	5
 
   for (i = 0; i < MAX_TRIES; i++)
     {
+      grub_file_t file;
+
       grub_util_info ((i == 0) ? _("attempting to read the core image `%s' from GRUB")
 		      : _("attempting to read the core image `%s' from GRUB again"),
 		      core_path_dev);
@@ -578,6 +599,8 @@
   if (i == MAX_TRIES)
     grub_util_error (_("cannot read `%s' correctly"), core_path_dev);
 
+#endif
+
   /* Clean out the blocklists.  */
   block = first_block;
   while (block->len)
@@ -593,23 +616,130 @@
       if ((char *) block <= core_img)
 	grub_util_error (_("no terminator in the core image"));
     }
-
-  /* Now read the core image to determine where the sectors are.  */
-  grub_file_filter_disable_compression ();
-  file = grub_file_open (core_path_dev);
-  if (! file)
-    grub_util_error ("%s", _(grub_errmsg));
-
-  file->read_hook = save_first_sector;
-  if (grub_file_read (file, tmp_img, GRUB_DISK_SECTOR_SIZE)
-      != GRUB_DISK_SECTOR_SIZE)
-    grub_util_error (_("failed to read the first sector of the core image"));
-
+  last_block = block + 1;
   block = first_block;
-  file->read_hook = save_blocklists;
-  if (grub_file_read (file, tmp_img, core_size - GRUB_DISK_SECTOR_SIZE)
-      != (grub_ssize_t) core_size - GRUB_DISK_SECTOR_SIZE)
-    grub_util_error (_("failed to read the rest sectors of the core image"));
+
+#ifdef __linux__
+  {
+    grub_partition_t container = root_dev->disk->partition;
+    struct fiemap fie1;
+    int fd;
+
+    /* Write the first two sectors of the core image onto the disk.  */
+    grub_util_info ("opening the core image `%s'", core_path);
+    fp = fopen (core_path, "rb");
+    if (! fp)
+      grub_util_error (_("cannot open `%s'"), core_path);
+    fd = fileno (fp);
+
+    grub_memset (&fie1, 0, sizeof (fie1));
+    fie1.fm_length = core_size;
+    fie1.fm_flags = FIEMAP_FLAG_SYNC;
+
+    if (ioctl (fd, FS_IOC_FIEMAP, &fie1) < 0)
+      {
+	int nblocks, i, j;
+	int bsize;
+	int mul;
+
+	grub_util_warn ("FIEMAP failed. Reverting to FIBMAP");
+
+	if (ioctl (fd, FIGETBSZ, &bsize) < 0)
+	  grub_util_error (_("can't get blocklist"));
+	if (bsize & (GRUB_DISK_SECTOR_SIZE - 1))
+	  grub_util_error (_("blocksize not divisible by 512"));
+	mul = bsize >> GRUB_DISK_SECTOR_BITS;
+	nblocks = (core_size + bsize - 1) / bsize;
+	for (i = 0; i < nblocks; i++)
+	  {
+	    unsigned blk = i;
+	    grub_err_t err;
+	    if (ioctl (fd, FIBMAP, &blk) < 0)
+	      grub_util_error (_("can't get blocklist"));
+	    
+	    for (j = 0; j < mul; j++)
+	      {
+		int rest = core_size - ((i * mul + j) << GRUB_DISK_SECTOR_BITS);
+		if (rest <= 0)
+		  break;
+		if (rest > GRUB_DISK_SECTOR_SIZE)
+		  rest = GRUB_DISK_SECTOR_SIZE;
+		if (i == 0 && j == 0)
+		  save_first_sector (((grub_uint64_t) blk) * mul
+				     + grub_partition_get_start (container),
+				     0, rest);
+		else
+		  save_blocklists (((grub_uint64_t) blk) * mul + j
+				   + grub_partition_get_start (container),
+				   0, rest);
+	      }
+	  }
+      }
+    else
+      {
+	struct fiemap *fie2;
+	int i, j;
+	fie2 = xmalloc (sizeof (*fie2)
+			+ fie1.fm_mapped_extents
+			* sizeof (fie1.fm_extents[1]));
+	memset (fie2, 0, sizeof (*fie2)
+		+ fie1.fm_mapped_extents * sizeof (fie2->fm_extents[1]));
+	fie2->fm_length = core_size;
+	fie2->fm_flags = FIEMAP_FLAG_SYNC;
+	fie2->fm_extent_count = fie1.fm_mapped_extents;
+	if (ioctl (fd, FS_IOC_FIEMAP, fie2) < 0)
+	  grub_util_error ("FIEMAP fails");
+	for (i = 0; i < fie2->fm_mapped_extents; i++)
+	  {
+	    for (j = 0;
+		 j < ((fie2->fm_extents[i].fe_length
+		       + GRUB_DISK_SECTOR_SIZE - 1)
+		      >> GRUB_DISK_SECTOR_BITS);
+		 j++)
+	      {
+		size_t len = (fie2->fm_extents[i].fe_length
+			      - j * GRUB_DISK_SECTOR_SIZE);
+		if (len > GRUB_DISK_SECTOR_SIZE)
+		  len = GRUB_DISK_SECTOR_SIZE;
+		if (i == 0 && j == 0)
+		  save_first_sector ((fie2->fm_extents[i].fe_physical
+				      >> GRUB_DISK_SECTOR_BITS)
+				     + j,
+				     fie2->fm_extents[i].fe_physical
+				     & (GRUB_DISK_SECTOR_SIZE - 1), len);
+		else
+		  save_blocklists ((fie2->fm_extents[i].fe_physical
+				    >> GRUB_DISK_SECTOR_BITS)
+				   + j,
+				   fie2->fm_extents[i].fe_physical
+				   & (GRUB_DISK_SECTOR_SIZE - 1), len);
+
+
+	      }
+	  }
+      }
+    fclose (fp);
+  }
+#else
+  {
+    /* Now read the core image to determine where the sectors are.  */
+    grub_file_filter_disable_compression ();
+    file = grub_file_open (core_path_dev);
+    if (! file)
+      grub_util_error ("%s", _(grub_errmsg));
+
+    file->read_hook = save_first_sector;
+    if (grub_file_read (file, tmp_img, GRUB_DISK_SECTOR_SIZE)
+	!= GRUB_DISK_SECTOR_SIZE)
+      grub_util_error (_("failed to read the first sector of the core image"));
+
+    file->read_hook = save_blocklists;
+    if (grub_file_read (file, tmp_img, core_size - GRUB_DISK_SECTOR_SIZE)
+	!= (grub_ssize_t) core_size - GRUB_DISK_SECTOR_SIZE)
+      grub_util_error (_("failed to read the rest sectors of the core image"));
+    grub_file_close (file);
+  }
+#endif
 
 #ifdef GRUB_MACHINE_IEEE1275
   {
@@ -617,11 +747,12 @@
     boot_devpath = (char *) (boot_img
 			     + GRUB_BOOT_AOUT_HEADER_SIZE
 			     + GRUB_BOOT_MACHINE_BOOT_DEVPATH);
-    if (file->device->disk->id != dest_dev->disk->id)
+    if (dest_dev->disk->id != root_dev->disk->id
+	|| dest_dev->disk->dev->id != root_dev->disk->dev->id)
       {
 	const char *dest_ofpath;
 	dest_ofpath
-	  = grub_util_devname_to_ofpath (grub_util_biosdisk_get_osdev (file->device->disk));
+	  = grub_util_devname_to_ofpath (grub_util_biosdisk_get_osdev (root_dev->disk));
 	grub_util_info ("dest_ofpath is `%s'", dest_ofpath);
 	strncpy (boot_devpath, dest_ofpath, GRUB_BOOT_MACHINE_BOOT_DEVPATH_END
 		 - GRUB_BOOT_MACHINE_BOOT_DEVPATH - 1);
@@ -638,8 +769,6 @@
   }
 #endif
 
-  grub_file_close (file);
-
   free (core_path_dev);
   free (tmp_img);
 
@@ -652,8 +781,80 @@
     grub_util_error (_("cannot open `%s'"), core_path);
 
   grub_util_write_image (core_img, GRUB_DISK_SECTOR_SIZE * 2, fp);
+  fsync (fileno (fp));
   fclose (fp);
 
+  grub_disk_cache_invalidate_all ();
+
+  {
+    char *buf, *ptr = core_img;
+    size_t len = core_size;
+    grub_uint64_t blk;
+    grub_partition_t container = root_dev->disk->partition;
+    grub_err_t err;
+
+    root_dev->disk->partition = 0;
+
+    buf = xmalloc (core_size);
+    blk = first_sector;
+    err = grub_disk_read (dest_dev->disk, blk, 0, GRUB_DISK_SECTOR_SIZE, buf);
+    if (err)
+      grub_util_error (_("read failed: %s"), grub_errmsg);
+    if (grub_memcmp (buf, ptr, (char *) last_block - core_img) != 0)
+      {
+	int i;
+	grub_printf ("%ld\n", ptr - core_img);
+	for (i = 0; i < 512; i++)
+	  grub_printf ("%02x%c", (unsigned char) ptr[i], 
+		       ((i & 0xf) == 0xf) ? '\n': ' ');
+	grub_printf ("--\n");
+	for (i = 0; i < 512; i++)
+	  grub_printf ("%02x%c", (unsigned char) buf[i], 
+		       ((i & 0xf) == 0xf) ? '\n': ' ');
+	grub_util_error (_("blocklists are invalid"));
+      }
+
+    ptr += GRUB_DISK_SECTOR_SIZE;
+    len -= GRUB_DISK_SECTOR_SIZE;
+
+    block = first_block;
+    while (block->len)
+      {
+	size_t cur = grub_target_to_host16 (block->len) << GRUB_DISK_SECTOR_BITS;
+	blk = grub_target_to_host64 (block->start);
+
+	if (cur > len)
+	  cur = len;
+
+	err = grub_disk_read (dest_dev->disk, blk, 0, cur, buf);
+	if (err)
+	  grub_util_error (_("read failed: %s"), grub_errmsg);
+
+	if (grub_memcmp (buf, ptr, cur) != 0)
+	  {
+	    int i;
+	    grub_printf ("%ld\n", ptr - core_img);
+	    for (i = 0; i < cur; i++)
+	      grub_printf ("%02x%c", (unsigned char) ptr[i], 
+			   ((i & 0xf) == 0xf) ? '\n': ' ');
+	    grub_printf ("--\n");
+	    for (i = 0; i < cur; i++)
+	      grub_printf ("%02x%c", (unsigned char) buf[i], 
+			   ((i & 0xf) == 0xf) ? '\n': ' ');
+	    grub_util_error (_("blocklists are invalid"));
+	  }
+
+	ptr += cur;
+	len -= cur;
+	block--;
+	
+	if ((char *) block <= core_img)
+	  grub_util_error (_("no terminator in the core image"));
+      }
+    root_dev->disk->partition = container;
+    free (buf);
+  }
+
  finish:
 
   /* Write the boot image onto the disk.  */

Re: [RFC] FIEMAP and disk sync issues

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 1/27/2012 8:28 AM, Vladimir 'φ-coder/phcoder' Serbinenko wrote:
> Hello, all. When someone installs using blocklists (discouraged
> and unreliable), we use GRUB internal fs reader in order to
> determine core.img blocklists. With Linux we call fsync and an
> ioctl on partition in order to flush the cache. I also tried adding
> "sync" which didn't make any difference. Unfortunately it doesn't
> seem to be good enough on

By "sync" you mean running the sync command?  If so, that absolutely,
positively should flush everything to disk.  If it isn't then there's
a major kernel bug.  An fsync() on the core.img file ( not the
partition device ) also should flush everything needed to find that
file, without flushing everything else that isn't.

Doing an fsync() on the partition device will flush buffers from a
process directly opening the block dev and performing IO there, not
the buffers for files within the filesystem on that block device.
IIRC, ext2 uses the block buffers for directory access so that
partition fsync will flush directories, but ext3 and ext4 do not.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPIvUhAAoJEJrBOlT6nu75ZSwH/2vgqHY3L+9xPGAsMDMzelAq
ae6P0aKgqEsZPEiTDx6+sBNE9uvJO6va5mKv9eyvDhpmKeq64mLa3EDJBKUmnSL+
E/b/cAAYPV7RaJ7Rc3tUNa96PRC8rOPyOcGeOVQxqjFS5rmZOa4p+wrO3YitqIGR
IUD7HSN8wt4PtmeS8thRXQPhcSICQwPy8rg4QVq2mFHxFws/WALAr7p5uVu/SH2Q
E8lXARKdvqd6HPjxlo7A/WeDEVvRd25eb84vX33wAhaauAI9VFD2+eCWR5ayyVUe
aymZ+DNyXpenEGK5JDdW38ovmAwIsWI+LbWVCh/gInEfHJkDVqN9XogtmCc9S68=
=3Yxi
-----END PGP SIGNATURE-----

Re: [RFC] FIEMAP and disk sync issues

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It looks like BLKFLSBUF also should do the trick.  That appears to
both tell the fs ( if one is mounted ) to sync, and flush the dirty
block buffers like fsync() on the blockdev does.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPIxxOAAoJEJrBOlT6nu75yMkIAKCDIhGFevEuEDsOxoIXWH88
/i9Txf5fpDiwDRp+ryU194pCfHoZXxZ1/7gkqTVhj9TEVpqGtXlYL6JdwSHbUWeJ
+RCzrxEoVHLlAKxoTFrjZQorM+fKe0rleibAa9vGu98XzBvXCGiD8B1n/Mv2f6LT
HbRsYs1IFEeF/YfH3d+nbjInQZQgTvGu5lnltBFsHr/V892q7Wqs5WuT/oqRMfvw
L2g90Ft9YmfgwItbmAmhfPtFayus4TrOhXHSCCj4JsDEjeZpdNSnEJs/QrlqcYHU
xRjKz9DG//1RncTpXun9C8ikpez66RT1voOL0v8xR5f8QPaJ68w1+1lTA+Hxtj0=
=n8zF
-----END PGP SIGNATURE-----

Re: [RFC] FIEMAP and disk sync issues

[Vladimir 'φ-coder/phcoder' Serbinenko]

On 27.01.2012 20:04, Phillip Susi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 1/27/2012 8:28 AM, Vladimir 'φ-coder/phcoder' Serbinenko wrote:
>> Hello, all. When someone installs using blocklists (discouraged
>> and unreliable), we use GRUB internal fs reader in order to
>> determine core.img blocklists. With Linux we call fsync and an
>> ioctl on partition in order to flush the cache. I also tried adding
>> "sync" which didn't make any difference. Unfortunately it doesn't
>> seem to be good enough on
> By "sync" you mean running the sync command?
Actually the sync () call
>   If so, that absolutely,
> positively should flush everything to disk.  If it isn't then there's
> a major kernel bug.  An fsync() on the core.img file ( not the
> partition device ) also should flush everything needed to find that
> file, without flushing everything else that isn't.
I'll recheck. But considering that I ran into some data loss it's 
possible that I have some kind of hardware or software failure (most 
likely because of new kernel and xen tests). Busy now doing the backups 
in case it gets worse
> Doing an fsync() on the partition device will flush buffers from a
> process directly opening the block dev and performing IO there, not
> the buffers for files within the filesystem on that block device.
> IIRC, ext2 uses the block buffers for directory access so that
> partition fsync will flush directories, but ext3 and ext4 do not.
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJPIvUhAAoJEJrBOlT6nu75ZSwH/2vgqHY3L+9xPGAsMDMzelAq
> ae6P0aKgqEsZPEiTDx6+sBNE9uvJO6va5mKv9eyvDhpmKeq64mLa3EDJBKUmnSL+
> E/b/cAAYPV7RaJ7Rc3tUNa96PRC8rOPyOcGeOVQxqjFS5rmZOa4p+wrO3YitqIGR
> IUD7HSN8wt4PtmeS8thRXQPhcSICQwPy8rg4QVq2mFHxFws/WALAr7p5uVu/SH2Q
> E8lXARKdvqd6HPjxlo7A/WeDEVvRd25eb84vX33wAhaauAI9VFD2+eCWR5ayyVUe
> aymZ+DNyXpenEGK5JDdW38ovmAwIsWI+LbWVCh/gInEfHJkDVqN9XogtmCc9S68=
> =3Yxi
> -----END PGP SIGNATURE-----
>


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko

Re: [RFC] FIEMAP and disk sync issues

[Goswin von Brederlow]

"Vladimir 'φ-coder/phcoder' Serbinenko" <phcoder@gmail.com> writes:

> Another problem with FIBMAP/FIEMAP is that fuse-mounted BFS lacks them
> altogether. We will need to find some other way of doing BFS install.
> Also some COW filesystems may supply a somewhat valid blocklist with
> FIBMAP/FIEMAP but which is unusable to GRUB because of COW nature. So
> we need an FS-whitelist.
> Experiment patch attached.
> Any thoughts?

The fuse/lowlevel.h has:

        /**
         * Map block index within file to block index within device
         *
         * Note: This makes sense only for block device backed filesystems
         * mounted with the 'blkdev' option
         *
         * Introduced in version 2.6
         *
         * Valid replies:
         *   fuse_reply_bmap
         *   fuse_reply_err
         *
         * @param req request handle
         * @param ino the inode number
         * @param blocksize unit of block index
         * @param idx block index within file
         */
        void (*bmap) (fuse_req_t req, fuse_ino_t ino, size_t blocksize,
                      uint64_t idx);

/**
 * Reply with block index
 *
 * Possible requests:
 *   bmap
 *
 * @param req request handle
 * @param idx block index within device
 * @return zero for success, -errno for failure to send reply
 */
int fuse_reply_bmap(fuse_req_t req, uint64_t idx);


So it is a matter of implementing the right callback in BFS to support
FIBMAP/FIEMAP. Same for ntfs3g.

MfG
        Goswin

Re: [RFC] FIEMAP and disk sync issues

[Vladimir 'φ-coder/phcoder' Serbinenko]

>  So it is a matter of implementing the right callback in BFS to support
>  FIBMAP/FIEMAP. Same for ntfs3g.

ntfs3g does implement bmap. Problem is that the only way to access this 
from userspace is FIBMAP which cuts to 32-bits.
FIEMAP is implemented as a different ioctl rather than piggybacking on 
bmap.


-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko