* Re: HPA unlock during partition scan of RAID components [not found] <74AAB12B538EC94087A0D16AFDFC24F4045674@IRSMSX102.ger.corp.intel.com> @ 2011-11-03 15:54 ` Tejun Heo 2011-11-03 19:00 ` Phillip Susi 2011-11-03 20:46 ` NeilBrown 0 siblings, 2 replies; 17+ messages in thread From: Tejun Heo @ 2011-11-03 15:54 UTC (permalink / raw) To: Hawrylewicz Czarnowski, Przemyslaw; +Cc: dm-devel, Bartlomiej Zolnierkiewicz Hello, (cc'ing dm-devel) On Thu, Nov 03, 2011 at 01:02:37AM +0000, Hawrylewicz Czarnowski, Przemyslaw wrote: > Recently we have encountered a problem with unlocking of HPA on disks belonging to RAID array. > The scenario in the simplest form is as follows: > * take HPA resized drives > * use SW Raid solution with metadata located at the end of disk (eg. md v1.0, IMSM) > * create raid0/10/5 using whole devices (eg. /dev/sda) (raid size must extend capacity of single device; raid1 is not the case here) > * create any partition with size exceeding HPA limit of single component > > According to the code of rescan_partitions(), if disk has capacity limit enabled (HPA) and boundary of partition extends beyond that limit, it is bypassed/unlocked (regardless of libata.ignore_hpa state). > As for single block device it is proper action - for raid using whole block device it is not. The partition table despite the fact it is read from single disk belongs to raid array. Values in RAID context are set properly. But from the point of view of single device - they are not. > Problem is that in general rescan_partition() has no knowledge of any raid array using that block device. Moreover, that raid is not assembled yet. > And the result: as HPA limit is unlocked and metadata on this device in not recognizable anymore - in worst case raid is not assembled or failed. > > The simplest resolution is to conditionally forbid HPA unlocking (eg. extending ignore_hpa parameter) but I suppose a better and more 'intelligent' solution can be found. This has come up a couple times and I think the proper solution is to always unlock HPA and expose both sizes - locked and unlocked and let dm, md or whatever do whatever is approriate. Block or driver layer doesn't have any way to determine which one is the right bet - it simply doesn't have enough information. I tried to bounce this idea off people who whould actually be using this (dm/md) but haven't heard back yet. Thanks. -- tejun ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-03 15:54 ` HPA unlock during partition scan of RAID components Tejun Heo @ 2011-11-03 19:00 ` Phillip Susi 2011-11-04 14:48 ` Tejun Heo 2011-11-03 20:46 ` NeilBrown 1 sibling, 1 reply; 17+ messages in thread From: Phillip Susi @ 2011-11-03 19:00 UTC (permalink / raw) To: device-mapper development Cc: Tejun Heo, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw On 11/3/2011 11:54 AM, Tejun Heo wrote: > This has come up a couple times and I think the proper solution is to > always unlock HPA and expose both sizes - locked and unlocked and let > dm, md or whatever do whatever is approriate. Block or driver layer > doesn't have any way to determine which one is the right bet - it > simply doesn't have enough information. I tried to bounce this idea > off people who whould actually be using this (dm/md) but haven't heard > back yet. Simply making the smaller size available does not solve the problem of making the part of the drive that is supposed to remain hidden accessible to user space, and it remains unlocked across a reboot, which usually makes the bios fail to recognize such drives. The only reason I am aware of for unlocking the hpa is to avoid problems caused by upgrading an old system that was installed using the unlock behavior and thus, incorrectly extended its partition into the protected area. It seems the appropriate fix for that it for distribution upgrade scripts to test for this and configure the boot loader to pass the unlock flag ( or maybe fix the problem by shrinking the partition ), rather than have the kernel continue to try unlocking things by default. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-03 19:00 ` Phillip Susi @ 2011-11-04 14:48 ` Tejun Heo 2011-11-04 15:42 ` Phillip Susi 0 siblings, 1 reply; 17+ messages in thread From: Tejun Heo @ 2011-11-04 14:48 UTC (permalink / raw) To: Phillip Susi Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw Hello, On Thu, Nov 3, 2011 at 12:00 PM, Phillip Susi <psusi@cfl.rr.com> wrote: > Simply making the smaller size available does not solve the problem of > making the part of the drive that is supposed to remain hidden accessible to > user space, and it remains unlocked across a reboot, which usually makes the > bios fail to recognize such drives. > > The only reason I am aware of for unlocking the hpa is to avoid problems > caused by upgrading an old system that was installed using the unlock > behavior and thus, incorrectly extended its partition into the protected > area. It seems the appropriate fix for that it for distribution upgrade > scripts to test for this and configure the boot loader to pass the unlock > flag ( or maybe fix the problem by shrinking the partition ), rather than > have the kernel continue to try unlocking things by default. I frankly don't care about BIOSen locking up afterwards and I don't think they are more common than the issues from not unlocking (e.g. moving harddrive to another machine, hot-plugging - partition - reboot). There will be a kernel param to disable unlocking behavior and that would be it. On top of that if bios is that screwed it's probably a good idea to not use bios raid which is a silly thing to begin with. Currently the problem is that the kernel doesn't unlock by default and the heuristics can't be turned off, so we have unhappy customers on both sides. Anyways, if md/dm people are on board, I'm gonna push for that but otherwise there isn't much point. Thanks. -- tejun ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-04 14:48 ` Tejun Heo @ 2011-11-04 15:42 ` Phillip Susi 2011-11-04 15:52 ` Tejun Heo 0 siblings, 1 reply; 17+ messages in thread From: Phillip Susi @ 2011-11-04 15:42 UTC (permalink / raw) To: Tejun Heo Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw On 11/4/2011 10:48 AM, Tejun Heo wrote: > I frankly don't care about BIOSen locking up afterwards and I don't > think they are more common than the issues from not unlocking (e.g. > moving harddrive to another machine, hot-plugging - partition - > reboot). There will be a kernel param to disable unlocking behavior Could you clarify what issues you refer to? Moving a drive to another machine? > and that would be it. On top of that if bios is that screwed it's > probably a good idea to not use bios raid which is a silly thing to > begin with. Currently the problem is that the kernel doesn't unlock > by default and the heuristics can't be turned off, so we have unhappy > customers on both sides. Anyways, if md/dm people are on board, I'm > gonna push for that but otherwise there isn't much point. It isn't only a problem for bios raid, it also affects mdadm raid. Being able to defeat the heuristic would help, but I still don't see why the heuristic should be implemented in the kernel instead of user space? ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-04 15:42 ` Phillip Susi @ 2011-11-04 15:52 ` Tejun Heo 2011-11-04 16:26 ` Phillip Susi 0 siblings, 1 reply; 17+ messages in thread From: Tejun Heo @ 2011-11-04 15:52 UTC (permalink / raw) To: Phillip Susi Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw Hello, On Fri, Nov 04, 2011 at 11:42:20AM -0400, Phillip Susi wrote: > On 11/4/2011 10:48 AM, Tejun Heo wrote: > >I frankly don't care about BIOSen locking up afterwards and I don't > >think they are more common than the issues from not unlocking (e.g. > >moving harddrive to another machine, hot-plugging - partition - > >reboot). There will be a kernel param to disable unlocking behavior > > Could you clarify what issues you refer to? Moving a drive to > another machine? Yeah, that or hotplugging, or changing BIOS config, updating BIOS, and, some (rare) BIOSen even fail to set up things consistently across reboots. > >and that would be it. On top of that if bios is that screwed it's > >probably a good idea to not use bios raid which is a silly thing to > >begin with. Currently the problem is that the kernel doesn't unlock > >by default and the heuristics can't be turned off, so we have unhappy > >customers on both sides. Anyways, if md/dm people are on board, I'm > >gonna push for that but otherwise there isn't much point. > > It isn't only a problem for bios raid, it also affects mdadm raid. Exposing alt size should be enough for md/dm no matter how hpa is setup. > Being able to defeat the heuristic would help, but I still don't see > why the heuristic should be implemented in the kernel instead of > user space? Because sans raids, it was good enough and doesn't recent md's have metadata at the beginning anyway? The thing is that you have your problem, and other people have other issues w/ HPA. At this point, it's a misfeature abused by some silly BIOSen mostly for silly BIOS raids. I think we should expose the most we can to our users and if some BIOSen are screwed on soft reboot, well, just accept it or turn off HPA unlocking manually. Thanks. -- tejun ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-04 15:52 ` Tejun Heo @ 2011-11-04 16:26 ` Phillip Susi 2011-11-04 16:32 ` Tejun Heo 0 siblings, 1 reply; 17+ messages in thread From: Phillip Susi @ 2011-11-04 16:26 UTC (permalink / raw) To: Tejun Heo Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw On 11/4/2011 11:52 AM, Tejun Heo wrote: >> Could you clarify what issues you refer to? Moving a drive to >> another machine? > > Yeah, that or hotplugging, or changing BIOS config, updating BIOS, > and, some (rare) BIOSen even fail to set up things consistently across > reboots. I'm asking for a clarification not just a restatement. What problem occurs when you move a drive to another machine, or hotplug the drive? > Exposing alt size should be enough for md/dm no matter how hpa is > setup. So you are saying that mdadm should store the metadata for 0.9 and 1.0 relative to the locked end of the disk, even if it is currently unlocked? I suppose then grub also will need to be able to detect if the disk has been unlocked and adjust to using the locked size. >> Being able to defeat the heuristic would help, but I still don't see >> why the heuristic should be implemented in the kernel instead of >> user space? > > Because sans raids, it was good enough and doesn't recent md's have > metadata at the beginning anyway? It isn't about whether it is good enough or not, it's about whether the kernel should be using heuristics to make decisions, or leaving it up to user space, where distros can come up with all kinds of complex logic to handle all of the goofy cases. You complain that the block layer does not have enough information to get it right, so why not let user space decide, where it has access to all of the information it needs. > The thing is that you have your problem, and other people have other > issues w/ HPA. At this point, it's a misfeature abused by some silly > BIOSen mostly for silly BIOS raids. I think we should expose the most > we can to our users and if some BIOSen are screwed on soft reboot, > well, just accept it or turn off HPA unlocking manually. It is actually not used FOR the bios raid. The bios is using the HPA for unrelated reasons and it just happens that unlocking it breaks the bios raid ( and mdadm with format 0.9 or 1.0 ). ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-04 16:26 ` Phillip Susi @ 2011-11-04 16:32 ` Tejun Heo 2011-11-04 21:08 ` Phillip Susi 0 siblings, 1 reply; 17+ messages in thread From: Tejun Heo @ 2011-11-04 16:32 UTC (permalink / raw) To: Phillip Susi Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw Hello, On Fri, Nov 04, 2011 at 12:26:02PM -0400, Phillip Susi wrote: > I'm asking for a clarification not just a restatement. What problem > occurs when you move a drive to another machine, or hotplug the > drive? Umm... partioned / mkfs'd while unlocked and bios (on whatever machine) locks it on later boots. > >Exposing alt size should be enough for md/dm no matter how hpa is > >setup. > > So you are saying that mdadm should store the metadata for 0.9 and > 1.0 relative to the locked end of the disk, even if it is currently > unlocked? I suppose then grub also will need to be able to detect > if the disk has been unlocked and adjust to using the locked size. No, with both sizes exposed, mdadm should be able to detect metadata at the end even if it was created w/ hpa locked. This is still fundamentally flawed as if you move it to another machine, it may be locked in completely different way, but that's as far as we can go. > >Because sans raids, it was good enough and doesn't recent md's have > >metadata at the beginning anyway? > > It isn't about whether it is good enough or not, it's about whether > the kernel should be using heuristics to make decisions, or leaving > it up to user space, where distros can come up with all kinds of > complex logic to handle all of the goofy cases. You complain that > the block layer does not have enough information to get it right, so > why not let user space decide, where it has access to all of the > information it needs. Upto userspace how? Unlock it after root is mounted? > >The thing is that you have your problem, and other people have other > >issues w/ HPA. At this point, it's a misfeature abused by some silly > >BIOSen mostly for silly BIOS raids. I think we should expose the most > >we can to our users and if some BIOSen are screwed on soft reboot, > >well, just accept it or turn off HPA unlocking manually. > > It is actually not used FOR the bios raid. The bios is using the > HPA for unrelated reasons and it just happens that unlocking it > breaks the bios raid ( and mdadm with format 0.9 or 1.0 ). It is also used for bios raids. Some BIOSen lock it for no reason. Some lock it if BIOS raid is enabled. Some lock it for recovery partition. Some are just crazy. This already has been discussed to death. Thanks. -- tejun ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-04 16:32 ` Tejun Heo @ 2011-11-04 21:08 ` Phillip Susi 2011-11-04 21:50 ` Tejun Heo 0 siblings, 1 reply; 17+ messages in thread From: Phillip Susi @ 2011-11-04 21:08 UTC (permalink / raw) To: Tejun Heo Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw On 11/4/2011 12:32 PM, Tejun Heo wrote: > Umm... partioned / mkfs'd while unlocked and bios (on whatever > machine) locks it on later boots. So the first machine does not set an HPA at all, you format the drive, then move it to a machine whose bios blindly adds an HPA to the drive? A bios certainly shouldn't be blindly adding an HPA to an already formatted drive that did not have one before. If it does, why is that our problem? > No, with both sizes exposed, mdadm should be able to detect metadata > at the end even if it was created w/ hpa locked. This is still > fundamentally flawed as if you move it to another machine, it may be > locked in completely different way, but that's as far as we can go. So mdadm could check both places after the kernel always unlocks, without any heuristic? And when creating an array, it would write it to the real, unlocked end of the disk, possibly trashing whatever the bios has stored there? > Upto userspace how? Unlock it after root is mounted? Or in the initramfs. Or just deal with it as part of the upgrade process, as I suggested before. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-04 21:08 ` Phillip Susi @ 2011-11-04 21:50 ` Tejun Heo 2011-11-05 1:29 ` Phillip Susi 0 siblings, 1 reply; 17+ messages in thread From: Tejun Heo @ 2011-11-04 21:50 UTC (permalink / raw) To: Phillip Susi Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw On Fri, Nov 04, 2011 at 05:08:55PM -0400, Phillip Susi wrote: > On 11/4/2011 12:32 PM, Tejun Heo wrote: > >Umm... partioned / mkfs'd while unlocked and bios (on whatever > >machine) locks it on later boots. > > So the first machine does not set an HPA at all, you format the > drive, then move it to a machine whose bios blindly adds an HPA to > the drive? A bios certainly shouldn't be blindly adding an HPA to an > already formatted drive that did not have one before. If it does, > why is that our problem? Sigh, yes, it is as much our problem as any other HPA related issues and it can be much more common because it can happen with hot plugging/swapping. -- tejun ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-04 21:50 ` Tejun Heo @ 2011-11-05 1:29 ` Phillip Susi 2011-11-05 1:43 ` Tejun Heo 0 siblings, 1 reply; 17+ messages in thread From: Phillip Susi @ 2011-11-05 1:29 UTC (permalink / raw) To: Tejun Heo Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/04/2011 05:50 PM, Tejun Heo wrote: >> So the first machine does not set an HPA at all, you format the >> drive, then move it to a machine whose bios blindly adds an HPA >> to the drive? A bios certainly shouldn't be blindly adding an HPA >> to an already formatted drive that did not have one before. If >> it does, why is that our problem? > > Sigh, yes, it is as much our problem as any other HPA related > issues and it can be much more common because it can happen with > hot plugging/swapping. That would also break a Windows drive. Are there any such biosen in the wild? They should only add an HPA to a drive that is unformatted. In other words, if you had a drive with Windows on it, and transplanted it into such a broken machine, Windows would no longer be able to mount it. I can't imagine there are such broken biosen in the wild, and if there are, we can at least point and say "see, it breaks when you do that with windows too so it is the fault of the bios, not Linux". -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk60kZEACgkQJ4UciIs+XuITagCfZKTvfglv1jUl3VOsfoZC04/P Ia8AoJAZykXjJvkz1OHY1zqAVetXYBPn =5V5N -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-05 1:29 ` Phillip Susi @ 2011-11-05 1:43 ` Tejun Heo 2011-11-05 2:26 ` Phillip Susi 0 siblings, 1 reply; 17+ messages in thread From: Tejun Heo @ 2011-11-05 1:43 UTC (permalink / raw) To: Phillip Susi Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw Hello, On Fri, Nov 4, 2011 at 6:29 PM, Phillip Susi <psusi@cfl.rr.com> wrote: > That would also break a Windows drive. Are there any such biosen in > the wild? They should only add an HPA to a drive that is unformatted. Yes, it would break under windows and it does. Get a motherboard which always locks HPA, do hotplug, create filesystem filling the full disk and reboot. BIOS doesn't have a reliable way of determining whether the disk is "unformatted" or not. Some try to scan dos partition table but filesystem may be created on the whole device or software raid could be using the whole drive. There is no reliable way to tell. Some BIOSen may try to do HPA locking after hotplug using ACPI _GTF which in theory can work but in practice many motherboards either don't implement them or are horridly broken. So, yeah, it isn't too difficult to break whether the os is windows or linux. The only argument against unlocking by default and providing both sizes is that some BIOSen may act incorrectly after soft reset, which seems acceptable provided there's an override to disable the automatic unlocking. Maybe we should just add yet another param to suppress the block layer heuristics and be done with it. I don't know. Thanks. -- tejun ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-05 1:43 ` Tejun Heo @ 2011-11-05 2:26 ` Phillip Susi 2011-11-05 2:52 ` Tejun Heo 0 siblings, 1 reply; 17+ messages in thread From: Phillip Susi @ 2011-11-05 2:26 UTC (permalink / raw) To: Tejun Heo Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/04/2011 09:43 PM, Tejun Heo wrote: > Yes, it would break under windows and it does. Get a motherboard > which always locks HPA, do hotplug, create filesystem filling the > full disk and reboot. BIOS doesn't have a reliable way of > determining whether the disk is "unformatted" or not. Some try to > scan dos partition table but filesystem may be created on the whole > device or software raid could be using the whole drive. There is no > reliable way to tell. Some BIOSen may try to do HPA locking after > hotplug using ACPI _GTF which in theory can work but in practice > many motherboards either don't implement them or are horridly > broken. So, yeah, it isn't too difficult to break whether the os is > windows or linux. Well if it breaks under Windows, then we can't be faulted for having the same results now can we? > The only argument against unlocking by default and providing both > sizes is that some BIOSen may act incorrectly after soft reset, > which seems acceptable provided there's an override to disable the > automatic unlocking. The other argument is that unlocking by default trashes whatever data the bios was trying to hide in the HPA, and deviates from the behavior of Windows. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk60nt0ACgkQJ4UciIs+XuIdkACfVPbcO/Tl0bJ2g/wEb2HQ6Gpl 0u4AoL67GnVVZZgW98tGElJtYpNENXaU =bxzR -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-05 2:26 ` Phillip Susi @ 2011-11-05 2:52 ` Tejun Heo 2011-11-05 3:50 ` Phillip Susi 0 siblings, 1 reply; 17+ messages in thread From: Tejun Heo @ 2011-11-05 2:52 UTC (permalink / raw) To: Phillip Susi Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw Hello, > Well if it breaks under Windows, then we can't be faulted for having > the same results now can we? Heh, let's just remove all hpa features and screw all the current users which depend on them! > The other argument is that unlocking by default trashes whatever data > the bios was trying to hide in the HPA, and deviates from the behavior > of Windows. On the former point, I don't think it matters. If the user doesn't know what it is, [s]he can leave it alone. Maybe userland can do something smarter if it knows the partition lives in hpa locked area, like not mounting it by default or hiding it from browser. On the point of not deviating from windows, I kinda agree and my previous "let's just remove all hpa stuff" isn't all that sarcastic, but it really isn't that simple. On BIOS raid configs, the vendor controls both bios and driver and those drivers are very self contained. They implement all from bit pushing driver itself to raid stack and high level device management, and actually assumes that they have full control of the whole stack and do good number of crazy things. It would be surprising if no driver is unlocking hpa during driver init. And they do all sorts of crazy and broken stuff. Try to follow windows bios raid troubleshooting threads, it often ends up "umm... I lost everything and had to reinitialized the whole array from BIOS and reinstall". So... um... I don't know. The dual size thing was the best solution that I know of (don't remember who came up with it first) and at least some of the ATA people seem to agree with it. Anyways, I don't think we're making any useful progress in this thread, so this probably will be my last message unless something new comes up. If you have a great new idea to solve them all, please go ahead. I'll be happy to review. Good luck. -- tejun ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-05 2:52 ` Tejun Heo @ 2011-11-05 3:50 ` Phillip Susi 0 siblings, 0 replies; 17+ messages in thread From: Phillip Susi @ 2011-11-05 3:50 UTC (permalink / raw) To: Tejun Heo Cc: device-mapper development, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/04/2011 10:52 PM, Tejun Heo wrote: > Hello, > >> Well if it breaks under Windows, then we can't be faulted for >> having the same results now can we? > > Heh, let's just remove all hpa features and screw all the current > users which depend on them! The only users that depend on them are those who originally installed a version of Linux that did it wrong, and those can be handled with proper distribution upgrade scripts, so yes, why not stop unlocking by default, and leave it up to the distro upgrade scripts to enable the override when backward compatibility requires it? >> The other argument is that unlocking by default trashes whatever >> data the bios was trying to hide in the HPA, and deviates from >> the behavior of Windows. > > On the former point, I don't think it matters. If the user doesn't > know what it is, [s]he can leave it alone. Maybe userland can do > something smarter if it knows the partition lives in hpa locked > area, like not mounting it by default or hiding it from browser. You don't think it matters that we are overwriting data that the bios marked as reserved for its use only, and Windows respects that? And user land certainly can do things smarter, the question is what the kernel default should be. When upgrading, user land scripts can decide that the previous kernel was doing it wring, and force the new kernel to continue doing it wrong for the sake of backward compatibility, but I see no reason why the default on new installs should deviate from Windows and refuse to respect the will of the bios. In other words, if a new install of Windows respects the HPA, then so should a new install of Linux. > On the point of not deviating from windows, I kinda agree and my > previous "let's just remove all hpa stuff" isn't all that > sarcastic, but it really isn't that simple. On BIOS raid configs, > the vendor controls both bios and driver and those drivers are very > self contained. They implement all from bit pushing driver itself > to raid stack and high level device management, and actually > assumes that they have full control of the whole stack and do good > number of crazy things. It would be surprising if no driver is > unlocking hpa during driver init. And they do all sorts of crazy > and broken stuff. Try to follow windows bios raid troubleshooting > threads, it often ends up "umm... I lost everything and had to > reinitialized the whole array from BIOS and reinstall". Again though, it doesn't really have anything to do with fakeraid. If there is an HPA without a fakeraid, and Linux violates that HPA, then Linux is causing damage that Windows does not. The only reason that fakeraid has anything to do with it is that for reasons unknown, there seems to be a general overlap between boards that create an HPA and boards that do fakeraid. If fakeraid is out of the picture, and you have a drive that has an HPA for whatever reason, Windows respects it, and Linux trashes it. That's not a good policy. > If you have a great new idea to solve them all, please go ahead. > I'll be happy to review. I'm still looking for a problem that is not both caused by a broken Linux policy in the past ( installing with an older kernel that automatically unlocked ), and is solved by unlocking now. The fact that the broken Linux policy seems to mostly affect fakeraid users does not justify continuing that broken policy. In other words, Windows respects the protected area, so we should as well, unless we have very good reason to believe that doing so would cause loss of existing user data. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk60sqEACgkQJ4UciIs+XuLtuwCgg2fBXxE+PjW9i38jYvxFUrqa qAMAn0+KBgzMnstTZpaOUKZXZeXQ3oU3 =106j -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-03 15:54 ` HPA unlock during partition scan of RAID components Tejun Heo 2011-11-03 19:00 ` Phillip Susi @ 2011-11-03 20:46 ` NeilBrown 2011-11-03 21:08 ` Tejun Heo 1 sibling, 1 reply; 17+ messages in thread From: NeilBrown @ 2011-11-03 20:46 UTC (permalink / raw) To: Tejun Heo Cc: dm-devel, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw [-- Attachment #1.1: Type: text/plain, Size: 2609 bytes --] On Thu, 3 Nov 2011 08:54:16 -0700 Tejun Heo <tj@kernel.org> wrote: > Hello, > > (cc'ing dm-devel) > > On Thu, Nov 03, 2011 at 01:02:37AM +0000, Hawrylewicz Czarnowski, Przemyslaw wrote: > > Recently we have encountered a problem with unlocking of HPA on disks belonging to RAID array. > > The scenario in the simplest form is as follows: > > * take HPA resized drives > > * use SW Raid solution with metadata located at the end of disk (eg. md v1.0, IMSM) > > * create raid0/10/5 using whole devices (eg. /dev/sda) (raid size must extend capacity of single device; raid1 is not the case here) > > * create any partition with size exceeding HPA limit of single component > > > > According to the code of rescan_partitions(), if disk has capacity limit enabled (HPA) and boundary of partition extends beyond that limit, it is bypassed/unlocked (regardless of libata.ignore_hpa state). > > As for single block device it is proper action - for raid using whole block device it is not. The partition table despite the fact it is read from single disk belongs to raid array. Values in RAID context are set properly. But from the point of view of single device - they are not. > > Problem is that in general rescan_partition() has no knowledge of any raid array using that block device. Moreover, that raid is not assembled yet. > > And the result: as HPA limit is unlocked and metadata on this device in not recognizable anymore - in worst case raid is not assembled or failed. > > > > The simplest resolution is to conditionally forbid HPA unlocking (eg. extending ignore_hpa parameter) but I suppose a better and more 'intelligent' solution can be found. > > This has come up a couple times and I think the proper solution is to > always unlock HPA and expose both sizes - locked and unlocked and let > dm, md or whatever do whatever is approriate. Block or driver layer > doesn't have any way to determine which one is the right bet - it > simply doesn't have enough information. I tried to bounce this idea > off people who whould actually be using this (dm/md) but haven't heard > back yet. > > Thanks. > What exactly do you mean by "expose both sizes" ?? A new ioctl - BLKGETHPASIZE64 ?? That might work, but I think it would be good if there were also an ioctl BLKHBALOCK which changed BLKGETSIZE64 to match BLKGETHPASIZE64. Then some user-space tools could examine the device with a full understanding of md, dm, dmraid, partitions, filesystems etc etc and make a reasonably informed decision. And then put that decision into effect. ?? NeilBrown [-- Attachment #1.2: signature.asc --] [-- Type: application/pgp-signature, Size: 828 bytes --] [-- Attachment #2: Type: text/plain, Size: 0 bytes --] ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-03 20:46 ` NeilBrown @ 2011-11-03 21:08 ` Tejun Heo 2012-05-15 0:50 ` Charles Nordlund 0 siblings, 1 reply; 17+ messages in thread From: Tejun Heo @ 2011-11-03 21:08 UTC (permalink / raw) To: NeilBrown Cc: dm-devel, Bartlomiej Zolnierkiewicz, Hawrylewicz Czarnowski, Przemyslaw On Fri, Nov 04, 2011 at 07:46:52AM +1100, NeilBrown wrote: > What exactly do you mean by "expose both sizes" ?? > A new ioctl - BLKGETHPASIZE64 ?? > > That might work, but I think it would be good if there were also an ioctl > BLKHBALOCK which changed BLKGETSIZE64 to match BLKGETHPASIZE64. > Then some user-space tools could examine the device with a full understanding > of md, dm, dmraid, partitions, filesystems etc etc and make a reasonably > informed decision. And then put that decision into effect. In kernel, just another size field. Out of kernel, I was thinking more along the line of a new /sysfs field, but yeah maybe another ioctl. At this point, I don't really think making unlocking selectable is a good idea. That has to go through device detach / attach cycle and what if someone else is already using first half of the disk? We can try to be sneaky and slip in device size change underneath it but that just sounds too crazy to me. IMHO, we should unlock by default and just let everyone know what the size before unlocking was in case that could be useful. Thank you. -- tejun ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: HPA unlock during partition scan of RAID components 2011-11-03 21:08 ` Tejun Heo @ 2012-05-15 0:50 ` Charles Nordlund 0 siblings, 0 replies; 17+ messages in thread From: Charles Nordlund @ 2012-05-15 0:50 UTC (permalink / raw) To: dm-devel Tejun Heo <tj <at> kernel.org> writes: > > On Fri, Nov 04, 2011 at 07:46:52AM +1100, NeilBrown wrote: > > What exactly do you mean by "expose both sizes" ?? > > A new ioctl - BLKGETHPASIZE64 ?? > > > > That might work, but I think it would be good if there were also an ioctl > > BLKHBALOCK which changed BLKGETSIZE64 to match BLKGETHPASIZE64. > > Then some user-space tools could examine the device with a full understanding > > of md, dm, dmraid, partitions, filesystems etc etc and make a reasonably > > informed decision. And then put that decision into effect. > > In kernel, just another size field. Out of kernel, I was thinking > more along the line of a new /sysfs field, but yeah maybe another > ioctl. At this point, I don't really think making unlocking > selectable is a good idea. That has to go through device detach / > attach cycle and what if someone else is already using first half of > the disk? We can try to be sneaky and slip in device size change > underneath it but that just sounds too crazy to me. IMHO, we should > unlock by default and just let everyone know what the size before > unlocking was in case that could be useful. > > Thank you. > Just wanted to pop in and say that thanks to your "intelligent HPA unlock" code you submitted to the kernel, I am completely unable to mount and use an NTFS RAID0. The way it ignores the ignore_hpa=0 parameter causes the fakeraid to fail; since the first disk in the array reports a partition size that goes beyond EOD, the code you submitted goes ahead and unlocks it to native -- not HPA limited -- capacity, with the expected result that it completely trashes the array as far as Linux is concerned. What was the point of the huge debate over locking/unlocking by default and the associated parameters if the kernel is going to simply ignore all that -- and my explicit parameter to NOT unlock past HPA size -- and unlock it anyway (which was determined to be a very very bad idea way back in 2009)? ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2012-05-15 0:50 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <74AAB12B538EC94087A0D16AFDFC24F4045674@IRSMSX102.ger.corp.intel.com>
2011-11-03 15:54 ` HPA unlock during partition scan of RAID components Tejun Heo
2011-11-03 19:00 ` Phillip Susi
2011-11-04 14:48 ` Tejun Heo
2011-11-04 15:42 ` Phillip Susi
2011-11-04 15:52 ` Tejun Heo
2011-11-04 16:26 ` Phillip Susi
2011-11-04 16:32 ` Tejun Heo
2011-11-04 21:08 ` Phillip Susi
2011-11-04 21:50 ` Tejun Heo
2011-11-05 1:29 ` Phillip Susi
2011-11-05 1:43 ` Tejun Heo
2011-11-05 2:26 ` Phillip Susi
2011-11-05 2:52 ` Tejun Heo
2011-11-05 3:50 ` Phillip Susi
2011-11-03 20:46 ` NeilBrown
2011-11-03 21:08 ` Tejun Heo
2012-05-15 0:50 ` Charles Nordlund
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.