* UBIFS data integrity @ 2014-01-14 2:51 rcrb rcrb 2014-01-14 9:32 ` Mats Kärrman 0 siblings, 1 reply; 8+ messages in thread From: rcrb rcrb @ 2014-01-14 2:51 UTC (permalink / raw) To: linux-mtd Hello, I have a question concerning data integrity of UBIFS. I am considering using a database over UBIFS on my embedded device. The database file is approximately 1 MiB. I wonder if I can rely on the filesystem to be sure than this file will never have a corruption, even in case of power cut during update. Will the filesystem guarantee that, after reboot, the file will be either in the pre-update state or be correctly updated ? I am asking this because I read that UBIFS can have partially updated files in case of power-cut during creation of a new file, and I am wondering if it is the same for already existing files. Best regards, ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: UBIFS data integrity 2014-01-14 2:51 UBIFS data integrity rcrb rcrb @ 2014-01-14 9:32 ` Mats Kärrman 2014-01-15 18:31 ` rcrb rcrb 0 siblings, 1 reply; 8+ messages in thread From: Mats Kärrman @ 2014-01-14 9:32 UTC (permalink / raw) To: rcrb rcrb, linux-mtd@lists.infradead.org Hi, On Tuesday, January 14, 2014 3:51 AM, rcrb rcrb [rcrbsw@gmail.com] wrote: > I am considering using a database over UBIFS on my embedded device. > The database file is approximately 1 MiB. I wonder if I can rely on > the filesystem to be sure than this file will never have a corruption, > even in case of power cut during update. Will the filesystem guarantee > that, after reboot, the file will be either in the pre-update state or > be correctly updated ? First of all, UBIFS relies on the underlying MTD driver also doing its job so you should also make sure this works on your board. If your MTD driver works, UBIFS is supposed to guarantee that every write operation is atomic, i.e. it is either fully written or discarded. If your database is guaranteed correct after a power-cut then depends on if it is consistent after every write or if it requires several write operations to produce a new consistent state. In general this requires that your database also is implemented in a fault tolerant way. BR // Mats ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: UBIFS data integrity 2014-01-14 9:32 ` Mats Kärrman @ 2014-01-15 18:31 ` rcrb rcrb 2014-01-16 8:00 ` Artem Bityutskiy 0 siblings, 1 reply; 8+ messages in thread From: rcrb rcrb @ 2014-01-15 18:31 UTC (permalink / raw) To: linux-mtd On Tue, Jan 14, 2014 at 10:32 AM, Mats Kärrman <Mats.Karrman@tritech.se> wrote: > Hi, > > On Tuesday, January 14, 2014 3:51 AM, rcrb rcrb [rcrbsw@gmail.com] wrote: >> I am considering using a database over UBIFS on my embedded device. >> The database file is approximately 1 MiB. I wonder if I can rely on >> the filesystem to be sure than this file will never have a corruption, >> even in case of power cut during update. Will the filesystem guarantee >> that, after reboot, the file will be either in the pre-update state or >> be correctly updated ? > > First of all, UBIFS relies on the underlying MTD driver also > doing its job so you should also make sure this works on your > board. > If your MTD driver works, UBIFS is supposed to guarantee > that every write operation is atomic, i.e. it is either fully written or > discarded. > If your database is guaranteed correct after a power-cut then > depends on if it is consistent after every write or if it requires > several write operations to produce a new consistent state. > In general this requires that your database also is implemented > in a fault tolerant way. > > BR // Mats Thank you for you answer Mats. I checked the write system calls with the database I plan to use, it appears that the transactions are validated with only one write. So I think it will be OK. rc ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: UBIFS data integrity 2014-01-15 18:31 ` rcrb rcrb @ 2014-01-16 8:00 ` Artem Bityutskiy 2014-01-16 8:16 ` Artem Bityutskiy 0 siblings, 1 reply; 8+ messages in thread From: Artem Bityutskiy @ 2014-01-16 8:00 UTC (permalink / raw) To: rcrb rcrb; +Cc: linux-mtd On Wed, 2014-01-15 at 19:31 +0100, rcrb rcrb wrote: > On Tue, Jan 14, 2014 at 10:32 AM, Mats Kärrman <Mats.Karrman@tritech.se> wrote: > > Hi, > > > > On Tuesday, January 14, 2014 3:51 AM, rcrb rcrb [rcrbsw@gmail.com] wrote: > >> I am considering using a database over UBIFS on my embedded device. > >> The database file is approximately 1 MiB. I wonder if I can rely on > >> the filesystem to be sure than this file will never have a corruption, > >> even in case of power cut during update. Will the filesystem guarantee > >> that, after reboot, the file will be either in the pre-update state or > >> be correctly updated ? > > > > First of all, UBIFS relies on the underlying MTD driver also > > doing its job so you should also make sure this works on your > > board. > > If your MTD driver works, UBIFS is supposed to guarantee > > that every write operation is atomic, i.e. it is either fully written or > > discarded. > > If your database is guaranteed correct after a power-cut then > > depends on if it is consistent after every write or if it requires > > several write operations to produce a new consistent state. > > In general this requires that your database also is implemented > > in a fault tolerant way. > > > > BR // Mats > > Thank you for you answer Mats. I checked the write system calls with > the database I plan to use, it appears that the transactions are > validated with only one write. So I think it will be OK. UBIFS writes in 4KiB units. So any aligned 4KiB piece of your file is guaranteed to either be fully written, then the read will return new data, or not be written at all, then the read will return old data. -- Best Regards, Artem Bityutskiy ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: UBIFS data integrity 2014-01-16 8:00 ` Artem Bityutskiy @ 2014-01-16 8:16 ` Artem Bityutskiy 2014-01-16 9:12 ` Mats Kärrman 0 siblings, 1 reply; 8+ messages in thread From: Artem Bityutskiy @ 2014-01-16 8:16 UTC (permalink / raw) To: rcrb rcrb; +Cc: linux-mtd On Thu, 2014-01-16 at 10:00 +0200, Artem Bityutskiy wrote: > On Wed, 2014-01-15 at 19:31 +0100, rcrb rcrb wrote: > > On Tue, Jan 14, 2014 at 10:32 AM, Mats Kärrman <Mats.Karrman@tritech.se> wrote: > > > Hi, > > > > > > On Tuesday, January 14, 2014 3:51 AM, rcrb rcrb [rcrbsw@gmail.com] wrote: > > >> I am considering using a database over UBIFS on my embedded device. > > >> The database file is approximately 1 MiB. I wonder if I can rely on > > >> the filesystem to be sure than this file will never have a corruption, > > >> even in case of power cut during update. Will the filesystem guarantee > > >> that, after reboot, the file will be either in the pre-update state or > > >> be correctly updated ? > > > > > > First of all, UBIFS relies on the underlying MTD driver also > > > doing its job so you should also make sure this works on your > > > board. > > > If your MTD driver works, UBIFS is supposed to guarantee > > > that every write operation is atomic, i.e. it is either fully written or > > > discarded. > > > If your database is guaranteed correct after a power-cut then > > > depends on if it is consistent after every write or if it requires > > > several write operations to produce a new consistent state. > > > In general this requires that your database also is implemented > > > in a fault tolerant way. > > > > > > BR // Mats > > > > Thank you for you answer Mats. I checked the write system calls with > > the database I plan to use, it appears that the transactions are > > validated with only one write. So I think it will be OK. > > UBIFS writes in 4KiB units. I guess this is a bit confusing. The low-level flash write operations can be bigger or smaller. However, the file data "buffers" that UBIFS writes contain 4KiB of file data (aligned, e.g., regions of the file would be 0-4K, 4K-8K, etc). Every such "buffer" is protected with CRC32, and if a power cut results in the buffer not being fully written, the CRC32 mismatches and UBIFS discards the entire buffer. And the old contents of this file region is stays on the flash. The old contents does not go away before the new contents reaches the flash media. So the read operation for any aligned 4KiB region always returns the old contents, or the new contents, nothing else. > So any aligned 4KiB piece of your file is > guaranteed to either be fully written, then the read will return new > data, or not be written at all, then the read will return old data. -- Best Regards, Artem Bityutskiy ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: UBIFS data integrity 2014-01-16 8:16 ` Artem Bityutskiy @ 2014-01-16 9:12 ` Mats Kärrman 2014-01-16 9:26 ` Artem Bityutskiy 0 siblings, 1 reply; 8+ messages in thread From: Mats Kärrman @ 2014-01-16 9:12 UTC (permalink / raw) To: rcrb rcrb; +Cc: linux-mtd@lists.infradead.org, dedekind1@gmail.com On Thursday, January 16, 2014 9:16 AM, Artem Bityutskiy wrote: > > UBIFS writes in 4KiB units. > > I guess this is a bit confusing. The low-level flash write operations > can be bigger or smaller. However, the file data "buffers" that UBIFS > writes contain 4KiB of file data (aligned, e.g., regions of the file > would be 0-4K, 4K-8K, etc). Every such "buffer" is protected with CRC32, > and if a power cut results in the buffer not being fully written, the > CRC32 mismatches and UBIFS discards the entire buffer. And the old > contents of this file region is stays on the flash. The old contents > does not go away before the new contents reaches the flash media. So the > read operation for any aligned 4KiB region always returns the old > contents, or the new contents, nothing else. > > > So any aligned 4KiB piece of your file is > > guaranteed to either be fully written, then the read will return new > > data, or not be written at all, then the read will return old data. Sorry if I mislead you, I guess the final statement of my original mail is still valid then: "In general this requires that your database also is implemented in a fault tolerant way." I.e. it must itself make sure it's entire write is completed before switching to a new "state". Unless the DB implements some kind of journaling this may be a problem. BR // Mats ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: UBIFS data integrity 2014-01-16 9:12 ` Mats Kärrman @ 2014-01-16 9:26 ` Artem Bityutskiy 2014-01-17 9:21 ` Gupta, Pekon 0 siblings, 1 reply; 8+ messages in thread From: Artem Bityutskiy @ 2014-01-16 9:26 UTC (permalink / raw) To: Mats Kärrman; +Cc: linux-mtd@lists.infradead.org, rcrb rcrb On Thu, 2014-01-16 at 09:12 +0000, Mats Kärrman wrote: > On Thursday, January 16, 2014 9:16 AM, Artem Bityutskiy wrote: > > > UBIFS writes in 4KiB units. > > > > I guess this is a bit confusing. The low-level flash write operations > > can be bigger or smaller. However, the file data "buffers" that UBIFS > > writes contain 4KiB of file data (aligned, e.g., regions of the file > > would be 0-4K, 4K-8K, etc). Every such "buffer" is protected with CRC32, > > and if a power cut results in the buffer not being fully written, the > > CRC32 mismatches and UBIFS discards the entire buffer. And the old > > contents of this file region is stays on the flash. The old contents > > does not go away before the new contents reaches the flash media. So the > > read operation for any aligned 4KiB region always returns the old > > contents, or the new contents, nothing else. > > > > > So any aligned 4KiB piece of your file is > > > guaranteed to either be fully written, then the read will return new > > > data, or not be written at all, then the read will return old data. > > Sorry if I mislead you, I guess the final statement of my original mail is > still valid then: > "In general this requires that your database also is implemented > in a fault tolerant way." > I.e. it must itself make sure it's entire write is completed before > switching to a new "state". Unless the DB implements some kind of > journaling this may be a problem. Sure. The ordering or the 4K writes is not guaranteed, but I think the memory management subsystem uses LRU lists so older writes go first. Any write that crosses the 4096 byte-aligned boundary is not atomic. So the only thing UBIFS guarantees is something like: fsync write(4k-5k) fsync will guarantee that the 1K piece of data either gets fully written or not at all. -- Best Regards, Artem Bityutskiy ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: UBIFS data integrity 2014-01-16 9:26 ` Artem Bityutskiy @ 2014-01-17 9:21 ` Gupta, Pekon 0 siblings, 0 replies; 8+ messages in thread From: Gupta, Pekon @ 2014-01-17 9:21 UTC (permalink / raw) To: dedekind1@gmail.com, Mats Kärrman Cc: rcrb rcrb, linux-mtd@lists.infradead.org >From: Artem Bityutskiy >>On Thu, 2014-01-16 at 09:12 +0000, Mats Kärrman wrote: [...] >> Sorry if I mislead you, I guess the final statement of my original mail is >> still valid then: >> "In general this requires that your database also is implemented >> in a fault tolerant way." >> I.e. it must itself make sure it's entire write is completed before >> switching to a new "state". Unless the DB implements some kind of >> journaling this may be a problem. > >Sure. The ordering or the 4K writes is not guaranteed, but I think the >memory management subsystem uses LRU lists so older writes go first. Any >write that crosses the 4096 byte-aligned boundary is not atomic. So the >only thing UBIFS guarantees is something like: > >fsync >write(4k-5k) >fsync > >will guarantee that the 1K piece of data either gets fully written or >not at all. > Mounting UBIFS with -o sync will also do the same. http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_semantics Alternatively, if somehow in your application you can guarantee that every update to database is within 4K boundary (and aligned), then your database should always be in recoverable state. with regards, pekon ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-01-17 9:22 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-01-14 2:51 UBIFS data integrity rcrb rcrb 2014-01-14 9:32 ` Mats Kärrman 2014-01-15 18:31 ` rcrb rcrb 2014-01-16 8:00 ` Artem Bityutskiy 2014-01-16 8:16 ` Artem Bityutskiy 2014-01-16 9:12 ` Mats Kärrman 2014-01-16 9:26 ` Artem Bityutskiy 2014-01-17 9:21 ` Gupta, Pekon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox