* UBIFS data integrity
@ 2014-01-14 2:51 rcrb rcrb
2014-01-14 9:32 ` Mats Kärrman
0 siblings, 1 reply; 8+ messages in thread
From: rcrb rcrb @ 2014-01-14 2:51 UTC (permalink / raw)
To: linux-mtd
Hello,
I have a question concerning data integrity of UBIFS.
I am considering using a database over UBIFS on my embedded device.
The database file is approximately 1 MiB. I wonder if I can rely on
the filesystem to be sure than this file will never have a corruption,
even in case of power cut during update. Will the filesystem guarantee
that, after reboot, the file will be either in the pre-update state or
be correctly updated ?
I am asking this because I read that UBIFS can have partially updated
files in case of power-cut during creation of a new file, and I am
wondering if it is the same for already existing files.
Best regards,
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: UBIFS data integrity
2014-01-14 2:51 UBIFS data integrity rcrb rcrb
@ 2014-01-14 9:32 ` Mats Kärrman
2014-01-15 18:31 ` rcrb rcrb
0 siblings, 1 reply; 8+ messages in thread
From: Mats Kärrman @ 2014-01-14 9:32 UTC (permalink / raw)
To: rcrb rcrb, linux-mtd@lists.infradead.org
Hi,
On Tuesday, January 14, 2014 3:51 AM, rcrb rcrb [rcrbsw@gmail.com] wrote:
> I am considering using a database over UBIFS on my embedded device.
> The database file is approximately 1 MiB. I wonder if I can rely on
> the filesystem to be sure than this file will never have a corruption,
> even in case of power cut during update. Will the filesystem guarantee
> that, after reboot, the file will be either in the pre-update state or
> be correctly updated ?
First of all, UBIFS relies on the underlying MTD driver also
doing its job so you should also make sure this works on your
board.
If your MTD driver works, UBIFS is supposed to guarantee
that every write operation is atomic, i.e. it is either fully written or
discarded.
If your database is guaranteed correct after a power-cut then
depends on if it is consistent after every write or if it requires
several write operations to produce a new consistent state.
In general this requires that your database also is implemented
in a fault tolerant way.
BR // Mats
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: UBIFS data integrity
2014-01-14 9:32 ` Mats Kärrman
@ 2014-01-15 18:31 ` rcrb rcrb
2014-01-16 8:00 ` Artem Bityutskiy
0 siblings, 1 reply; 8+ messages in thread
From: rcrb rcrb @ 2014-01-15 18:31 UTC (permalink / raw)
To: linux-mtd
On Tue, Jan 14, 2014 at 10:32 AM, Mats Kärrman <Mats.Karrman@tritech.se> wrote:
> Hi,
>
> On Tuesday, January 14, 2014 3:51 AM, rcrb rcrb [rcrbsw@gmail.com] wrote:
>> I am considering using a database over UBIFS on my embedded device.
>> The database file is approximately 1 MiB. I wonder if I can rely on
>> the filesystem to be sure than this file will never have a corruption,
>> even in case of power cut during update. Will the filesystem guarantee
>> that, after reboot, the file will be either in the pre-update state or
>> be correctly updated ?
>
> First of all, UBIFS relies on the underlying MTD driver also
> doing its job so you should also make sure this works on your
> board.
> If your MTD driver works, UBIFS is supposed to guarantee
> that every write operation is atomic, i.e. it is either fully written or
> discarded.
> If your database is guaranteed correct after a power-cut then
> depends on if it is consistent after every write or if it requires
> several write operations to produce a new consistent state.
> In general this requires that your database also is implemented
> in a fault tolerant way.
>
> BR // Mats
Thank you for you answer Mats. I checked the write system calls with
the database I plan to use, it appears that the transactions are
validated with only one write. So I think it will be OK.
rc
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: UBIFS data integrity
2014-01-15 18:31 ` rcrb rcrb
@ 2014-01-16 8:00 ` Artem Bityutskiy
2014-01-16 8:16 ` Artem Bityutskiy
0 siblings, 1 reply; 8+ messages in thread
From: Artem Bityutskiy @ 2014-01-16 8:00 UTC (permalink / raw)
To: rcrb rcrb; +Cc: linux-mtd
On Wed, 2014-01-15 at 19:31 +0100, rcrb rcrb wrote:
> On Tue, Jan 14, 2014 at 10:32 AM, Mats Kärrman <Mats.Karrman@tritech.se> wrote:
> > Hi,
> >
> > On Tuesday, January 14, 2014 3:51 AM, rcrb rcrb [rcrbsw@gmail.com] wrote:
> >> I am considering using a database over UBIFS on my embedded device.
> >> The database file is approximately 1 MiB. I wonder if I can rely on
> >> the filesystem to be sure than this file will never have a corruption,
> >> even in case of power cut during update. Will the filesystem guarantee
> >> that, after reboot, the file will be either in the pre-update state or
> >> be correctly updated ?
> >
> > First of all, UBIFS relies on the underlying MTD driver also
> > doing its job so you should also make sure this works on your
> > board.
> > If your MTD driver works, UBIFS is supposed to guarantee
> > that every write operation is atomic, i.e. it is either fully written or
> > discarded.
> > If your database is guaranteed correct after a power-cut then
> > depends on if it is consistent after every write or if it requires
> > several write operations to produce a new consistent state.
> > In general this requires that your database also is implemented
> > in a fault tolerant way.
> >
> > BR // Mats
>
> Thank you for you answer Mats. I checked the write system calls with
> the database I plan to use, it appears that the transactions are
> validated with only one write. So I think it will be OK.
UBIFS writes in 4KiB units. So any aligned 4KiB piece of your file is
guaranteed to either be fully written, then the read will return new
data, or not be written at all, then the read will return old data.
--
Best Regards,
Artem Bityutskiy
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: UBIFS data integrity
2014-01-16 8:00 ` Artem Bityutskiy
@ 2014-01-16 8:16 ` Artem Bityutskiy
2014-01-16 9:12 ` Mats Kärrman
0 siblings, 1 reply; 8+ messages in thread
From: Artem Bityutskiy @ 2014-01-16 8:16 UTC (permalink / raw)
To: rcrb rcrb; +Cc: linux-mtd
On Thu, 2014-01-16 at 10:00 +0200, Artem Bityutskiy wrote:
> On Wed, 2014-01-15 at 19:31 +0100, rcrb rcrb wrote:
> > On Tue, Jan 14, 2014 at 10:32 AM, Mats Kärrman <Mats.Karrman@tritech.se> wrote:
> > > Hi,
> > >
> > > On Tuesday, January 14, 2014 3:51 AM, rcrb rcrb [rcrbsw@gmail.com] wrote:
> > >> I am considering using a database over UBIFS on my embedded device.
> > >> The database file is approximately 1 MiB. I wonder if I can rely on
> > >> the filesystem to be sure than this file will never have a corruption,
> > >> even in case of power cut during update. Will the filesystem guarantee
> > >> that, after reboot, the file will be either in the pre-update state or
> > >> be correctly updated ?
> > >
> > > First of all, UBIFS relies on the underlying MTD driver also
> > > doing its job so you should also make sure this works on your
> > > board.
> > > If your MTD driver works, UBIFS is supposed to guarantee
> > > that every write operation is atomic, i.e. it is either fully written or
> > > discarded.
> > > If your database is guaranteed correct after a power-cut then
> > > depends on if it is consistent after every write or if it requires
> > > several write operations to produce a new consistent state.
> > > In general this requires that your database also is implemented
> > > in a fault tolerant way.
> > >
> > > BR // Mats
> >
> > Thank you for you answer Mats. I checked the write system calls with
> > the database I plan to use, it appears that the transactions are
> > validated with only one write. So I think it will be OK.
>
> UBIFS writes in 4KiB units.
I guess this is a bit confusing. The low-level flash write operations
can be bigger or smaller. However, the file data "buffers" that UBIFS
writes contain 4KiB of file data (aligned, e.g., regions of the file
would be 0-4K, 4K-8K, etc). Every such "buffer" is protected with CRC32,
and if a power cut results in the buffer not being fully written, the
CRC32 mismatches and UBIFS discards the entire buffer. And the old
contents of this file region is stays on the flash. The old contents
does not go away before the new contents reaches the flash media. So the
read operation for any aligned 4KiB region always returns the old
contents, or the new contents, nothing else.
> So any aligned 4KiB piece of your file is
> guaranteed to either be fully written, then the read will return new
> data, or not be written at all, then the read will return old data.
--
Best Regards,
Artem Bityutskiy
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: UBIFS data integrity
2014-01-16 8:16 ` Artem Bityutskiy
@ 2014-01-16 9:12 ` Mats Kärrman
2014-01-16 9:26 ` Artem Bityutskiy
0 siblings, 1 reply; 8+ messages in thread
From: Mats Kärrman @ 2014-01-16 9:12 UTC (permalink / raw)
To: rcrb rcrb; +Cc: linux-mtd@lists.infradead.org, dedekind1@gmail.com
On Thursday, January 16, 2014 9:16 AM, Artem Bityutskiy wrote:
> > UBIFS writes in 4KiB units.
>
> I guess this is a bit confusing. The low-level flash write operations
> can be bigger or smaller. However, the file data "buffers" that UBIFS
> writes contain 4KiB of file data (aligned, e.g., regions of the file
> would be 0-4K, 4K-8K, etc). Every such "buffer" is protected with CRC32,
> and if a power cut results in the buffer not being fully written, the
> CRC32 mismatches and UBIFS discards the entire buffer. And the old
> contents of this file region is stays on the flash. The old contents
> does not go away before the new contents reaches the flash media. So the
> read operation for any aligned 4KiB region always returns the old
> contents, or the new contents, nothing else.
>
> > So any aligned 4KiB piece of your file is
> > guaranteed to either be fully written, then the read will return new
> > data, or not be written at all, then the read will return old data.
Sorry if I mislead you, I guess the final statement of my original mail is
still valid then:
"In general this requires that your database also is implemented
in a fault tolerant way."
I.e. it must itself make sure it's entire write is completed before
switching to a new "state". Unless the DB implements some kind of
journaling this may be a problem.
BR // Mats
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: UBIFS data integrity
2014-01-16 9:12 ` Mats Kärrman
@ 2014-01-16 9:26 ` Artem Bityutskiy
2014-01-17 9:21 ` Gupta, Pekon
0 siblings, 1 reply; 8+ messages in thread
From: Artem Bityutskiy @ 2014-01-16 9:26 UTC (permalink / raw)
To: Mats Kärrman; +Cc: linux-mtd@lists.infradead.org, rcrb rcrb
On Thu, 2014-01-16 at 09:12 +0000, Mats Kärrman wrote:
> On Thursday, January 16, 2014 9:16 AM, Artem Bityutskiy wrote:
> > > UBIFS writes in 4KiB units.
> >
> > I guess this is a bit confusing. The low-level flash write operations
> > can be bigger or smaller. However, the file data "buffers" that UBIFS
> > writes contain 4KiB of file data (aligned, e.g., regions of the file
> > would be 0-4K, 4K-8K, etc). Every such "buffer" is protected with CRC32,
> > and if a power cut results in the buffer not being fully written, the
> > CRC32 mismatches and UBIFS discards the entire buffer. And the old
> > contents of this file region is stays on the flash. The old contents
> > does not go away before the new contents reaches the flash media. So the
> > read operation for any aligned 4KiB region always returns the old
> > contents, or the new contents, nothing else.
> >
> > > So any aligned 4KiB piece of your file is
> > > guaranteed to either be fully written, then the read will return new
> > > data, or not be written at all, then the read will return old data.
>
> Sorry if I mislead you, I guess the final statement of my original mail is
> still valid then:
> "In general this requires that your database also is implemented
> in a fault tolerant way."
> I.e. it must itself make sure it's entire write is completed before
> switching to a new "state". Unless the DB implements some kind of
> journaling this may be a problem.
Sure. The ordering or the 4K writes is not guaranteed, but I think the
memory management subsystem uses LRU lists so older writes go first. Any
write that crosses the 4096 byte-aligned boundary is not atomic. So the
only thing UBIFS guarantees is something like:
fsync
write(4k-5k)
fsync
will guarantee that the 1K piece of data either gets fully written or
not at all.
--
Best Regards,
Artem Bityutskiy
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: UBIFS data integrity
2014-01-16 9:26 ` Artem Bityutskiy
@ 2014-01-17 9:21 ` Gupta, Pekon
0 siblings, 0 replies; 8+ messages in thread
From: Gupta, Pekon @ 2014-01-17 9:21 UTC (permalink / raw)
To: dedekind1@gmail.com, Mats Kärrman
Cc: rcrb rcrb, linux-mtd@lists.infradead.org
>From: Artem Bityutskiy
>>On Thu, 2014-01-16 at 09:12 +0000, Mats Kärrman wrote:
[...]
>> Sorry if I mislead you, I guess the final statement of my original mail is
>> still valid then:
>> "In general this requires that your database also is implemented
>> in a fault tolerant way."
>> I.e. it must itself make sure it's entire write is completed before
>> switching to a new "state". Unless the DB implements some kind of
>> journaling this may be a problem.
>
>Sure. The ordering or the 4K writes is not guaranteed, but I think the
>memory management subsystem uses LRU lists so older writes go first. Any
>write that crosses the 4096 byte-aligned boundary is not atomic. So the
>only thing UBIFS guarantees is something like:
>
>fsync
>write(4k-5k)
>fsync
>
>will guarantee that the 1K piece of data either gets fully written or
>not at all.
>
Mounting UBIFS with -o sync will also do the same.
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics
Alternatively, if somehow in your application you can guarantee that
every update to database is within 4K boundary (and aligned), then your
database should always be in recoverable state.
with regards, pekon
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-01-17 9:22 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-01-14 2:51 UBIFS data integrity rcrb rcrb
2014-01-14 9:32 ` Mats Kärrman
2014-01-15 18:31 ` rcrb rcrb
2014-01-16 8:00 ` Artem Bityutskiy
2014-01-16 8:16 ` Artem Bityutskiy
2014-01-16 9:12 ` Mats Kärrman
2014-01-16 9:26 ` Artem Bityutskiy
2014-01-17 9:21 ` Gupta, Pekon
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox