Re: Seagate Flex SMR

["Theodore Tso" <tytso@mit.edu>]

On Sun, Aug 23, 2026 at 06:45:49PM -0500, Noah Bergbauer wrote:
> > "Flex SMR" is not referring to any standard feature. So it is hard to see what
> > you are talking about. In Linux, we support only drives that follow a standard,
> > so for HDDs, that is SPC/SBC/ZBC for SAS drives and ACS/ZAC for SATA.
> 
> The specification can be found in T10/18-007r0. It's not exactly ZD/ZR as it
> seems to predate those standards, but it's close.

Flex SMR refers to an early Seagate proposal for something which we
pushed for at Google called Hybrid SMR.  (See the Disks for Data
Center Paper / Keynote from the FAST 2016 conference).  Western
Digital countered with something called the REALMS API, and this got
hashed out in the standards committee, with a resulting set of ZAC and
SBC commands which include "REPORT REALMS", "REPORT ZONE DOMAINS" and
"ZONE ACTIVATE".  The basic idea is that zones can be activated and
deactivated, and when a set of SMR zones is activated a portion of the
disk platter corresponding to a set of CMR zones might get
deactivated, such that the contents of the deactivated zones would be
lost, but all other parts of the disk would be unaffected.

It is a feature standardized in T10 and T13, but it's not particularly
popular.  HOWEVER, all disks purchased by Google have firmware which
support these commands, and we have multiple HDD vendors who are our
suppliers.  It had been our hope that other hyperscalers would find
this useful.  Unfortunately, the programming model was far too complex
for most HDD users, and so I wasn't aware of anyone other than Google
using it.

This has saved us a huge amount of storage TCO costs at Google, and it
is how I earned my promotion to Senior Staff Engineer.  So I can say
that it is a useful feature, and we're using it to this day.

> > The reason is that properly supporting the zone domains/zone realms feature is
> > *extremely hard*. This is full of gotcha and plenty of things will not be
> > backward compatible with pure SMR support that we have. E.g. SOBR and
> > conventional zones are very different before the SOBR zone is fully written.

Yes.  Zome Domains/Realms is a completely different model from the
traditional SMR, and I *knew* it would be very hard to get it
upstream.  Given that we were using a userspace Cluster Filesystem
(e.g, Colossus), we really didn't need kernel support, since we could
manage the CMR and SMR zones from userspace.

Unfortunately, some of our operational requirements meant that it
could never be fully compatible with SMR, since we wanted disks to be
compatible with traditional CMR disks from the factory, but we then
wanted to be able to dynamically convert portions of the disks back
and forth between SMR and CMR in a non-destructive fashion for the
portion of the platter that was not converted.  This was a key part of
the storage cost savings, and we were able to convince our HDD
suppliers to support us in the T10/T13 standards committees to produce
something that would meet our requirements.

> To be clear, what I have implemented is all about running a filesystem in
> domain 1. And I want to push back a little on your claim that this is
> extremely hard, because in a handful of small patches totaling around 500
> lines of code I have found solutions for all of these challenges. I am
> running btrfs on the sequential zones and it passes every test I have thrown
> at it. I can show you the code if you like.

The hard part is what do you do with the deactived zones in domain 0?
Attempts to read or write into the deactived zones will result in SCSI
errors, so making it work in a way that allows you to have some
percentage of your disk that has the IOPS-optimzied CMR volumes, or
the bytes-optimized SMR volumes is the tricky bit, especially if you
are trying to be backwards compatible with local disk file systems
using partitions.

I suppose if you only allow zone activation/deactivations to be done
only when partitions are created or removed, it could be made simpler,
but that removes a lot of the actual storage TCO advantage of Hybrid
SMR disks.  (We can perform the SMR<->CMR conversions while the disk
is serving as part of the Colossus cluster file system, and that was a
key requirement for how to justify the SWE investment of our Hybrid
SMR project, and my spending a lot of time travelling to T10 standards
meetings.  :-)

> The use case is that these drives exist, they're out there, and with a
> little bit of kernel work we could squeeze a few extra terabytes out of each
> and every one of them by leveraging their SMR capabilities. That's a clear
> win to me. Yes, they don't entirely follow the latest standards, but to me
> this is no different than the many device-specific quirks already supported
> by the kernel. And apart from the Flex-specific discovery procedure, all of
> this work should be usable for supporting other ZR/ZD drives in the future.

Oh, they are standardized, but it's just not a very commercially
successful standard.  (Sort of like Object Based Disks, except there
is a hyperscaler which is actually still using Zone Realm/Domain
disks, and in fact all disks purchased by that hyperscaler have the
feature.  So arguably more successful than OBD, but not as successful
as we had hoped.)

The surprising thing to me is that there are people outside of my
company who are interested.  You have to be running at a pretty large
scale before the Storage TCO benefits dominate the SWE investment
costs, and while I did try to get other hyperscalers interested, they
had already invested in alternative technologies, and so we weren't
able to lower our Hybrid SMR costs by making it a more commonly
demanded feature in the marketplace.

Cheers,

						- Ted