Black box flight recorder for Linux

Black box flight recorder for Linux

[James Courtier-Dutton]

Hi,

I have had an idea for a black box flight recorder type feature for 
Linux. Before I try to implement it, I just wish to ask here if anyone 
has already tried it, and whether the idea works or not.

Description for feature:
Stamp the dmesg output on RAM somewhere, so that after a reset (reset 
button pressed, not power off), the RAM can be read and details of 
oopses etc. can be read.

Now, the question I have is, if I write values to RAM, do any of those 
values survive a reset? If any did survive, one could use them to store 
oops output in. I am currently only interested in Intel CPU and AMD CPU 
based motherboards. If only some values survived, one could use some 
sort of redundant encoding so the good values could be recovered.

The main advantage of something like this would be for newer 
motherboards that are around now that don't have a serial port.

If no one has tried this, I will spend some time testing.

James

Re: Black box flight recorder for Linux

[Andi Kleen]

James Courtier-Dutton <James@superbug.co.uk> writes:
> 
> Now, the question I have is, if I write values to RAM, do any of those
> values survive a reset?

They don't generally.

Some people used to write the oopses into video memory, but that
is not portable.

-Andi

Re: Black box flight recorder for Linux

[Guennadi Liakhovetski]

On Sat, 8 Apr 2006, Andi Kleen wrote:

> James Courtier-Dutton <James@superbug.co.uk> writes:
> > 
> > Now, the question I have is, if I write values to RAM, do any of those
> > values survive a reset?
> 
> They don't generally.
> 
> Some people used to write the oopses into video memory, but that
> is not portable.

Some even write them to mtd (NOR flash), but that is even less portable:-)

Thanks
Guennadi
---
Guennadi Liakhovetski

Re: Black box flight recorder for Linux

[Lee Revell]

On Sat, 2006-04-08 at 12:12 +0100, James Courtier-Dutton wrote:
> The main advantage of something like this would be for newer 
> motherboards that are around now that don't have a serial port.

Did you try netconsole and/or the 1394 based remote debug solution that
was recently posted to LKML (that I now can't find in my mail)?

Lee

Re: Black box flight recorder for Linux

[Matti Aarnio]

On Sat, Apr 08, 2006 at 12:12:56PM +0100, James Courtier-Dutton wrote:
> Hi,
> 
> I have had an idea for a black box flight recorder type feature for 
> Linux. Before I try to implement it, I just wish to ask here if anyone 
> has already tried it, and whether the idea works or not.
> 
> Description for feature:
> Stamp the dmesg output on RAM somewhere, so that after a reset (reset 
> button pressed, not power off), the RAM can be read and details of 
> oopses etc. can be read.

The idea of  dmesg  buffer comes to Linux from SunOS 4.x series
on hardware, where system boot code explicitely left aside memory
space which was not _cleared_ during boot (it was parity-regenerated,
though).  The command to display that ring-buffer content was (no
surprise there?) "dmesg".

I do wish so many things from PC hardware, but it has stayed so b***y
inferior to real computers forever.  Lattest AMD CPUs have nice
features making them almost as good as IBM S/370 from early 1970es,
but still BIOSes are rather primitive things keeping things back.
( IOMMUs are things that have been invented since, and are definitely
a good thing.  Otherwise it has been faster and more capacitious
processing and memory at cheaper system cost... )

Like others have noted, display card memory spaces have been used
for this kind of "survives over reset" uses  -- I do also know some
embedded boot codes that created similar ring buffers for similar
reasons.  They don't generally survive over power-cycling, of course.


> The main advantage of something like this would be for newer 
> motherboards that are around now that don't have a serial port.
> 
> If no one has tried this, I will spend some time testing.
> 
> James


/Matti Aarnio

Re: Black box flight recorder for Linux

[Krzysztof Halasa]

James Courtier-Dutton <James@superbug.co.uk> writes:

> Now, the question I have is, if I write values to RAM, do any of those
> values survive a reset? If any did survive, one could use them to
> store oops output in. I am currently only interested in Intel CPU and
> AMD CPU based motherboards. If only some values survived, one could
> use some sort of redundant encoding so the good values could be
> recovered.
>
> The main advantage of something like this would be for newer
> motherboards that are around now that don't have a serial port.

Interesting idea.

I think the most trivial and reliable way would be to solder some
I^2 or similar EEPROM chip to, for example, parallel port connector.

Most motherboards have an internal I^2C bus / SMBus (for reading RAM
types and for other things) and I think it could be used to connect
the EEPROM instead of external port.

There are 512 Kbit (64 KB) and 1 Mbit (128 KB) EEPROMs available -
there is plenty of space not only for crash dump but for whole dmesg.
-- 
Krzysztof Halasa

Re: Black box flight recorder for Linux

[Andy Green]

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

Krzysztof Halasa wrote:

> I think the most trivial and reliable way would be to solder some
> I^2 or similar EEPROM chip to, for example, parallel port connector.
> 
> Most motherboards have an internal I^2C bus / SMBus (for reading RAM
> types and for other things) and I think it could be used to connect
> the EEPROM instead of external port.
> 
> There are 512 Kbit (64 KB) and 1 Mbit (128 KB) EEPROMs available -
> there is plenty of space not only for crash dump but for whole dmesg.

Just an additional thought on this idea... both VGA and DVI connectors 
on modern video cards appear to have DDC-2 connections, which is in fact 
I2C.  This would provide an (inherently bidirectional :-) ) 3-pin 
digital interface out of a mostly dead box even on laptops and so on 
with no serial, parallel or legacy keyboard/mouse, so long as they had 
reasonably modern VGA or DVI out.  You would need to get access to the 
two I2C pins and Gnd somehow in that scenario.   Since I2C has a concept 
of addressing it should be possible to choose I2C addresses for this 
communication that doesn't address whatever may be listening on the same 
bus in the monitor.

I don't know if you bitbang these or there is some kind of 
shifter/management logic in the video chip, in which case some minimal 
case by case support is needed, but since simple IO is otherwise all but 
gone on modern laptops so I thought it worth mentioning.

-Andy

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 4492 bytes --]

Re: Black box flight recorder for Linux

[Krzysztof Halasa]

Andy Green <andy@warmcat.com> writes:

> Just an additional thought on this idea... both VGA and DVI connectors
> on modern video cards appear to have DDC-2 connections, which is in
> fact I2C.

Yes, that could be used, too. It would be a bit more complicated as
different VGA cards use different access methods (i.e., different
I/O port and bit numbers). X11 drivers probably know how to drive it.
-- 
Krzysztof Halasa

Re: Black box flight recorder for Linux

[Krzysztof Halasa]

Hi,

(Cc to lm-sensors list).

> Yes, that could be used, too. It would be a bit more complicated as
> different VGA cards use different access methods (i.e., different
> I/O port and bit numbers). X11 drivers probably know how to drive it.

BTW: I just soldered a 24C64 I^2C EEPROM (the largest I could find
at home but 24C512 (64 KB) and even 24C1024 (128 KB) are available
on the market) to a connector and connected it to SMBus on my Asus
A7V333 mobo. I don't have code to write "printk" messages to it yet
but will look at it sometime.

Interesting: i2cdetect and friends can only find a custom Asus sensors
chip on this bus (ASB100 chip at addresses 0x2D, 0x48 and 0x49) - and
now my 24C64 at 0x57 (config address 7 = all ones). But the BIOS POST
searches and finds more devices: there is something at 0x2F, 0x69, and
there are (I think) DDR SDRAM EEPROMs and 0x51 and 0x52 (0x50 is an
empty DIMM slot). Got this info with my "DIY" logic analyzer. I think
the BIOS POST disconnects somehow the devices before loading the OS
(in order to prevent data damage?).

No wonder my first attempt with 24C16 which occupies all 0x50 - 0x57
addresses had to fail.

I think VGA monitors respond (at least?) at 0x50 address so
a 16-bit-addressable EEPROM (at least not larger than 24C2048 which
IMHO aren't yet available) with all-ones I^2C address selected
should do as well if connected to VGA/DVI I^2C/ACCESS.bus.
-- 
Krzysztof Halasa

Re: [lm-sensors] Black box flight recorder for Linux

[Rudolf Marek]

Hello,
> 
> Interesting: i2cdetect and friends can only find a custom Asus sensors
> chip on this bus (ASB100 chip at addresses 0x2D, 0x48 and 0x49) - and
> now my 24C64 at 0x57 (config address 7 = all ones). But the BIOS POST
> searches and finds more devices: there is something at 0x2F, 0x69, and
> there are (I think) DDR SDRAM EEPROMs and 0x51 and 0x52 (0x50 is an
> empty DIMM slot). Got this info with my "DIY" logic analyzer. I think
> the BIOS POST disconnects somehow the devices before loading the OS
> (in order to prevent data damage?).

Well Asus likes to play hide and seek...

> No wonder my first attempt with 24C16 which occupies all 0x50 - 0x57
> addresses had to fail.

Hm that should work, because Asus most likely multiplexes physical lines
instead of devices (using 74HC4052 IIRC)

> I think VGA monitors respond (at least?) at 0x50 address so
> a 16-bit-addressable EEPROM (at least not larger than 24C2048 which
> IMHO aren't yet available) with all-ones I^2C address selected
> should do as well if connected to VGA/DVI I^2C/ACCESS.bus.

What about to connect the device to parallel port, there are some adapter
schematics in kernel docs.

Simplest one needs just one 7405 two diodes and four resistors.

There is also a i2c-pport driver that may work without additional parts, just 3
pins of parport are used. Please note that you should set the parport to SPP
mode and not load any linux drivers. The driver is for 2.4. No 2.6 but I have
used it so you may find it on http://assembler.cz

Regards
Rudolf

Re: [lm-sensors] Black box flight recorder for Linux

[Krzysztof Halasa]

Rudolf Marek <r.marek@sh.cvut.cz> writes:

>> No wonder my first attempt with 24C16 which occupies all 0x50 - 0x57
>> addresses had to fail.
>
> Hm that should work, because Asus most likely multiplexes physical lines
> instead of devices (using 74HC4052 IIRC)

No, the motherboard thought that my 24C16 was a set of 3 DIMMs
(answering at 0x50 - 0x57, and DIMMs seem to be at 0x50, 0x51 and 0x52).
Of course the data read back was some product of both DIMM EEPROM and
my 24C16 but the machine didn't pass POST.

I think anyone trying to connect an EEPROM to SMBus has to make sure
there is nothing there, and nothing is expected by things like BIOS.
Not a very plug and play.

> What about to connect the device to parallel port, there are some adapter
> schematics in kernel docs.

Sure, that's one of the options.
-- 
Krzysztof Halasa

Re: Black box flight recorder for Linux

[Krzysztof Halasa]

Andy Green <andy@warmcat.com> writes:

> Just an additional thought on this idea... both VGA and DVI connectors
> on modern video cards appear to have DDC-2 connections, which is in
> fact I2C.  This would provide an (inherently bidirectional :-) ) 3-pin
> digital interface out of a mostly dead box even on laptops and so on
> with no serial, parallel or legacy keyboard/mouse, so long as they had
> reasonably modern VGA or DVI out.  You would need to get access to the
> two I2C pins and Gnd somehow in that scenario.   Since I2C has a
> concept of addressing it should be possible to choose I2C addresses
> for this communication that doesn't address whatever may be listening
> on the same bus in the monitor.

I think I like the idea and have some (not yet finished but working)
code. Any comments?

The first part is the "console" driver (obvious parts removed). It works
with both my Asus A7V333 (VIA KT333, VIA SMBUS driver) and with VGA DDC
interface on a Cirrus Logic GD 5446 VGA chip (simplified source attached
as well). Using respectively 2464 and 24512 set to ID 0x57.

This is, of course, not yet intended for inclusion.

#define DRV "eelogger"
#define EEPROM_MAX_READ_SIZE 128
#define EEPROM_MAX_WRITE_SIZE 128

int bus = -1;
unsigned char device;
unsigned int page_size, size /* in bytes */;

MODULE_PARM_DESC(bus, "I2C bus number");
module_param(bus, int, 0);
MODULE_PARM_DESC(device, "I2C device address");
module_param(device, byte, 0);

MODULE_PARM_DESC(size, "Size of EEPROM in bytes");
module_param(size, uint, 0);
MODULE_PARM_DESC(page_size, "Size of EEPROM write page in bytes");
module_param(page_size, uint, 0);

struct drv_data {
	struct i2c_client client;
	atomic_t idle;
	int (*read_func)(struct i2c_client *client, u8 *address, char *data,
			 int count);
	int (*write_func)(struct i2c_client *client, const char *data,
			  int count);
	u16 max_write_size, write_address; /* bytes */
	u8 buffer[EEPROM_MAX_WRITE_SIZE + 2 /* address */];
};

static int drv_read_i2c(struct i2c_client *client, u8 *address, char *data,
			int count)
{
	struct i2c_msg msgs[2] = {{
			.addr = client->addr,
			.flags = 0,
			.len = 2,
			.buf = address,
		}, {
			.addr = client->addr,
			.flags = I2C_M_RD,
			.len = count,
			.buf = data,
		}};

	if (i2c_transfer(client->adapter, msgs, 2) < 0)
		return -EIO;

	return count;
}


static int drv_read_smbus(struct i2c_client *client, u8 *address, char *data,
			  int count)
{
	int i;

	if (i2c_smbus_write_byte_data(client, address[0], address[1])) {
		dev_dbg(&client->dev, "EEPROM read has failed\n");
		return -EIO;
	}

	for (i = 0; i < count; i++) {
		int val = i2c_smbus_read_byte(client);
		if (val < 0)
			return i ? i : -EIO;
		*(data++) = (u8)val;
	}
	return count;
}


static int drv_write_smbus(struct i2c_client *client, const char *data,
			   int count)
{
	return i2c_smbus_write_i2c_block_data(client, data[0], count - 1,
					      data + 1) >= 0 ? count : -1;
}


static ssize_t drv_read(struct kobject *kobj, char *buffer, loff_t offset,
			size_t count)
{
	struct i2c_client *client;
	struct drv_data *data;
	u8 address[2] = { offset >> 8, offset & 0xFF};

	client = to_i2c_client(container_of(kobj, struct device, kobj));
	data = i2c_get_clientdata(client);

	if (offset > size || count == 0)
		return 0;
	if (offset + count > size)
		count = size - offset;

/* I don't like/need eeprom.c buffering and prefer direct EEPROM read.
   Reads are limited to, say, 128 bytes */

	if (count > EEPROM_MAX_READ_SIZE)
		count = EEPROM_MAX_READ_SIZE;

	dev_dbg(&client->dev, "Reading EEPROM, offset 0x%llX, length %u\n",
		offset, count);

	return data->read_func(client, address, buffer, count);
}


static void console_write(struct console *co, const char *buffer,
			  unsigned count)
{
	struct i2c_client *client = co->data;
	struct drv_data *data = i2c_get_clientdata(client);
	int put_zero = 0, err;

	if (!count)
		return;

/* This atomic_* things are temporary, I thing I should protect
   console_write() and drv_read() from conflicting but it's not yet here.
*/
	if (!atomic_dec_and_test(&data->idle)) { /* would nest, debug only */
		printk(KERN_DEBUG "NESTED access!\n");
		atomic_inc(&data->idle);
		return;
	}

	do {
		unsigned long t;
		u16 len = page_size - (data->write_address & (page_size - 1));
		if (len > data->max_write_size - 2 /* address */)
			len = data->max_write_size - 2;

		if (len >= count + 1 /* zero */) {
			put_zero = 1; /* last part of the string */
			len = count + 1;
		}

		memcpy(data->buffer + 2 /* address */, buffer, len - put_zero);
		if (put_zero)
			data->buffer[2 /* address */ + len - 1] = '\x0';

		data->buffer[0] = data->write_address >> 8;
		data->buffer[1] = data->write_address & 0xFF;

/* not sure if a busy loop is best here but it avoids locking issues and
   console writes are not frequent. EEPROMs write a "row" (128 bytes with
   24512) within, say, 10 ms so I think no big deal here.
*/

		t = jiffies;
		do {
			err = data->write_func(client, data->buffer, len + 2);
			if (err >= 0)
				break;
		} while (time_before(jiffies, t + HZ / 20)); /* 50 ms */

		if (err < 0) {
			dev_dbg(&client->dev, "EEPROM write has failed\n");
			break;
		}

		count -= len - put_zero; /* overwrite trailing zero */
		buffer += len - put_zero;
		data->write_address += len - put_zero;
		data->write_address &= ~size;
	} while (!put_zero);
	atomic_inc(&data->idle);
}


static struct console cons = {
	.name	= "EEcons",
	.write	= console_write,
	.flags	= CON_PRINTBUFFER | CON_ENABLED,
};

static struct i2c_driver drv = {
	.driver = {
		.name	= "eelogger",
	},
	.id		= I2C_DRIVERID_EEPROM,
	.attach_adapter	= drv_attach_adapter,
	.detach_client	= drv_detach_client,
};

static struct bin_attribute drv_attr = {
	.attr = {
		.name = "eelogger",
		.mode = S_IRUGO,
		.owner = THIS_MODULE,
	},
	.read = drv_read,
};

/* I don't like searching for devices in this case, thus blind fixed
   param-based addressing.
*/

static int drv_attach_adapter(struct i2c_adapter *adapter)
{
	struct i2c_client *client;
	struct drv_data *data;
	int err, does_i2c, write_only = 0;

	if (i2c_check_addr(adapter, device)) /* Skip if already in use */
		return -EBUSY;

	if (i2c_adapter_id(adapter) != bus)
		return -ENOSYS;

	does_i2c = i2c_check_functionality(adapter, I2C_FUNC_I2C);

	if (!does_i2c &&
	    !i2c_check_functionality(adapter,
				     I2C_FUNC_SMBUS_WRITE_I2C_BLOCK)) {
		printk(KERN_WARNING DRV ": Adapter %u has no required"
		       " functionality\n", bus);
		return -ENOSYS;
	}

	if (!does_i2c &&
	    !i2c_check_functionality(adapter, I2C_FUNC_SMBUS_READ_BYTE |
				     I2C_FUNC_SMBUS_WRITE_BYTE_DATA)) {
		printk(KERN_WARNING DRV ": Adapter %u is write-only\n",
		       bus);
		write_only = 1;
	}

	if (!(data = kzalloc(sizeof(struct drv_data), GFP_KERNEL)))
		return -ENOMEM;

	client = &data->client;
	i2c_set_clientdata(client, data);
	client->addr = device;
	client->adapter = adapter;
	client->driver = &drv;
	client->flags = 0;

	atomic_set(&data->idle, 1);
	if (does_i2c) {
		data->read_func = drv_read_i2c;
		data->write_func = i2c_master_send;
	} else {
		data->read_func = drv_read_smbus;
		data->write_func = drv_write_smbus;
	}
	data->max_write_size = EEPROM_MAX_WRITE_SIZE + 2 /* address */;
	if (!does_i2c && I2C_SMBUS_BLOCK_MAX + 1 < EEPROM_MAX_WRITE_SIZE + 2)
		data->max_write_size = I2C_SMBUS_BLOCK_MAX + 1 /* MSB */;

	/* Fill in the remaining client fields */
	strlcpy(client->name, "eelogger", I2C_NAME_SIZE);

	/* Tell the I2C layer a new client has arrived */
	if ((err = i2c_attach_client(client))) {
		kfree(data);
		return err;
	}

	cons.data = client;

	/* create the sysfs file */
	if (!write_only) {
		drv_attr.size = size;
		sysfs_create_bin_file(&client->dev.kobj, &drv_attr);
	}

	register_console(&cons);
	printk(KERN_ERR "EEPROM logger installed\n");
	return 0;
}

static int drv_detach_client(struct i2c_client *client)
{
	int err = i2c_detach_client(client);
	if (err)
		return err;

	unregister_console(&cons);

	kfree(i2c_get_clientdata(client));
	return 0;
}


static int __init bits(unsigned int value)
{
	int v = 0;
	while (!(value & 1) && v < 32) {
		value >>= 1;
		v++;
	}
	if (value == 1)
		return v;
	return -1;
}

static int __init drv_init(void)
{
	int b;

	/* sanity checks */

	if (bus < 0) {
		printk(KERN_WARNING DRV ": Invalid I^2C bus ID\n");
		return -EINVAL;
	}

	if (device < 0x50 || device > 0x57) {
		printk(KERN_WARNING DRV ": Invalid EEPROM device ID, "
		       "supported: 0x50 to 0x57\n");
		return -EINVAL;
	}

	/* EEPROMs smaller than 2432 have 1-byte address format and
	   are not handled here.
	   EEPROMs larger than 24512 will have to occupy more than
	   one I^2C device address and are not (yet) handled.
	   Multiple EEPROM chips are not (yet) handled either. */
	b = bits(size);
	if (b < 12 /* 2432 */ || b > 16 /* 24512 */) {
		printk(KERN_WARNING DRV ": Invalid EEPROM size, supported "
		       "sizes: 4096 to 65536 bytes\n");
		return -EINVAL;
	}

	b = bits(page_size);
	if (b < 0 || page_size > size) {
		printk(KERN_WARNING DRV ": Invalid EEPROM page size\n");
		return -EINVAL;
	}

	return i2c_add_driver(&drv);
}

static void __exit drv_exit(void)
{
	i2c_del_driver(&drv);
}


************************************************************************

The following is an adapter for Cirrus Logic 5446 VGA on my old R440LX
test machine:

There is a locking problem - the VGA is (can be) shared between VT console,
X11 and the driver. I'll look at CL FB driver to see how/if it's done.

static void *regbase = NULL; /* This CL VGA is legacy PIO-driven only */
static int scl, sda;

static void alpine_switch_i2c_bus(int bus /* 0 = internal or 1 = DDC */)
{
	u8 val = vga_rgfx(regbase, 0x17);

	if ((val & 0x60) != (bus * 0x60)) {
#ifdef DEBUG
		printk(KERN_DEBUG "Alpine: setting bus#\n");
#endif
		vga_wgfx(regbase, 0x17, (val & ~0x60) | (bus * 0x60));
	}
}

static void alpine_i2c_in(int b, int *clock, int *data)
{
        u8 reg;

        alpine_switch_i2c_bus(b);

        reg = vga_rseq(regbase, 0x08);
        if (clock)
		*clock = !!(reg & 0x04);
        if (data)
		*data  = !!(reg & 0x80);
}

static void alpine_i2c_out(int b, int clock, int data)
{
        u8 reg = 0xfc;
        alpine_switch_i2c_bus(b);

        if (clock)
		reg |= 1;
        if (data)
		reg |= 2;
        vga_wseq(regbase, 0x08, reg);
}

static void alpine_setsda(void *ptr, int state)
{
	alpine_i2c_out(1, scl, sda = state);
}

static void alpine_setscl(void *ptr, int state)
{
	alpine_i2c_out(1, scl = state, sda);
}

static int alpine_getsda(void *ptr)
{
	int val;
	alpine_i2c_in(1, NULL, &val);
	return val;
}

static int alpine_getscl(void *ptr)
{
	int val;
	alpine_i2c_in(1, &val, NULL);
	return val;
}

static struct i2c_algo_bit_data bit_alpine_data = {
	.setsda		= alpine_setsda,
	.setscl		= alpine_setscl,
	.getsda		= alpine_getsda,
	.getscl		= alpine_getscl,
	.udelay		= 5, /* 400 kHz access */
	.mdelay		= 1,
	.timeout	= HZ
};

static struct i2c_adapter alpine_ops = {
	.owner		= THIS_MODULE,
	.id		= I2C_HW_B_LP, /* test only */
	.algo_data	= &bit_alpine_data,
	.name		= "Cirrus Logic Alpine DDC I2C adapter driver",
};

static int __init dev_init_module(void)
{
/* don't change the signals unless requested */
	alpine_i2c_in(1, &scl, &sda);
	return i2c_bit_add_bus(&alpine_ops);
}

static void __exit dev_cleanup_module(void)
{
	i2c_bit_del_bus(&alpine_ops);
}

-- 
Krzysztof Halasa

Re: Black box flight recorder for Linux

[Jean Delvare]

Hi Krzysztof, Andy,

Andy Green writes:
> > Just an additional thought on this idea... both VGA and DVI connectors
> > on modern video cards appear to have DDC-2 connections, which is in
> > fact I2C.  This would provide an (inherently bidirectional :-) ) 3-pin
> > digital interface out of a mostly dead box even on laptops and so on
> > with no serial, parallel or legacy keyboard/mouse, so long as they had
> > reasonably modern VGA or DVI out.  You would need to get access to the
> > two I2C pins and Gnd somehow in that scenario.   Since I2C has a
> > concept of addressing it should be possible to choose I2C addresses
> > for this communication that doesn't address whatever may be listening
> > on the same bus in the monitor.

Krzysztof Halasa  answers:
> I think I like the idea and have some (not yet finished but working)
> code. Any comments?
> 
> The first part is the "console" driver (obvious parts removed). It works
> with both my Asus A7V333 (VIA KT333, VIA SMBUS driver) and with VGA DDC
> interface on a Cirrus Logic GD 5446 VGA chip (simplified source attached
> as well). Using respectively 2464 and 24512 set to ID 0x57.

How do you intend to connect your device to the DDC channel if there's
already a monitor connected to the VGA or DVI port?

Beware that many monitors have EDID EEPROMs responding to all I2C
addresses within the 0x50 - 0x57 range, so it'll be hard to add an
EEPROM on the bus for your own purpose without hitting an address
conflict.

> The following is an adapter for Cirrus Logic 5446 VGA on my old R440LX
> test machine:
> 
> There is a locking problem - the VGA is (can be) shared between VT console,
> X11 and the driver. I'll look at CL FB driver to see how/if it's done.

The current trend is to merge the DDC access driver into the
framebuffer driver. This solves one of the conflicts, and also makes
sense because the EDID data can be used to automatically setup the
framebuffer. We still have a few standalone DDC access drivers
(i2c-i810, i2c-savage4...) but they are considered deprecated and will
probably be deleted in a near feature.

This will be a second problem for you though. Most distributions don't
make use of hardware-specific framebuffer drivers by default, but use
the VESA framebuffer driver. This driver doesn't have DDC support.

Note that I am not trying to dicourage you, Andy's idea has merit for
sure. I just mean that it won't be usable by everyone out of the box.
People will have to use the right driver and pay attention to address
conflicts.

-- 
Jean Delvare

Re: Black box flight recorder for Linux

[Andy Green]

[-- Attachment #1: Type: text/plain, Size: 4594 bytes --]

Jean Delvare wrote:

Hi Folks -

Impressed with Krzysztof doing the work!

>> The first part is the "console" driver (obvious parts removed). It works
>> with both my Asus A7V333 (VIA KT333, VIA SMBUS driver) and with VGA DDC
>> interface on a Cirrus Logic GD 5446 VGA chip (simplified source attached
>> as well). Using respectively 2464 and 24512 set to ID 0x57.
> 
> How do you intend to connect your device to the DDC channel if there's
> already a monitor connected to the VGA or DVI port?

For both VGA and DVI I think the optimal answer is to build a small 
back-back breakout adapter.  The parts seem to be quite reasonably 
priced and reasonably available, eg from Farnell (www.farnell.co.uk):

VGA: Plug partcode 1071807 GBP 1.76   Socket partcode 1071811 GBP 2.10

DVI: Plug partcode 9965190 GBP 1.38   Socket partcode 1012227 GBP 2.01


If the original cabling looks like this:

   <Video card Socket>
     |
   <Original Cable Plug>
     |
     |
     |
   <Original Cable Plug>
     |
   <Monitor Socket>


then the breakout concept looks like this:

   <Video card Socket>
     |
   <Adapter Plug>
     |--------------------- I2C signal wires broken out
     |                      All wires passed to corresponding pins
   <Adapter Socket>
     |
   <Original Cable Plug>
     |
     |
     |
   <Original Cable Plug>
     |
   <Monitor Socket>

> Beware that many monitors have EDID EEPROMs responding to all I2C
> addresses within the 0x50 - 0x57 range, so it'll be hard to add an
> EEPROM on the bus for your own purpose without hitting an address
> conflict.

The adapter does give an opportunity to interrupt the connectivity of 
the I2C to the monitor if worst comes to worst, I guess no traffic is 
typically present after X init or Linux touching it.  Pretty ugly that 
things ignore b2-b0 of the I2C address.

A whole other way forward is to consider to replace the EEPROM from the 
original proposal (which does provide its own advantages such as 
simplicity, I accept) with something else that ends up on another PC. 
In this concept some logic presents a fake I2C peripheral to the DDC 
interface at an I2C address of our choosing.  This logic acts as a 
bidirectional "UART" type of thing, allowing transfer of data in both 
directions between the Linux box being debugged and another PC.

I designed something conceptually similar for the effort to hack the 
original Xbox, which was very effective (and is GPL'd):

http://warmcat.com/milksop/filtror.html

However this would be much simpler, not even needing RAM.  It can hook 
to the second PC by the same I2C method, parallel printer port, RS232 or 
USB depending on the level of complexity of the design.

I guess the link will feel quite like a 9600 or 19200 baud serial port 
in terms of throughput.

>> The following is an adapter for Cirrus Logic 5446 VGA on my old R440LX
>> test machine:
>>
>> There is a locking problem - the VGA is (can be) shared between VT console,
>> X11 and the driver. I'll look at CL FB driver to see how/if it's done.
> 
> The current trend is to merge the DDC access driver into the
> framebuffer driver. This solves one of the conflicts, and also makes
> sense because the EDID data can be used to automatically setup the
> framebuffer. We still have a few standalone DDC access drivers
> (i2c-i810, i2c-savage4...) but they are considered deprecated and will
> probably be deleted in a near feature.
>
> This will be a second problem for you though. Most distributions don't
> make use of hardware-specific framebuffer drivers by default, but use
> the VESA framebuffer driver. This driver doesn't have DDC support.

Maybe this effort is considered too esoteric, but it seems to me to be a 
reason to keep the DDC access drivers standalone, the hardware-specific 
framebuffer drivers can call through to them and we can use them in a 
clean way.  I realize this is a bit of a late objection and that there 
was not previously much point to keeping them as separate things in the 
world.

> Note that I am not trying to dicourage you, Andy's idea has merit for
> sure. I just mean that it won't be usable by everyone out of the box.
> People will have to use the right driver and pay attention to address
> conflicts.

Address conflicts we can solve by widening the scope and powers of the 
interface into a generic communications link by adding a little 
hardware, removing the address clash.  If basically magic-ing a 
cheap-to-drive comms link out of nowhere, on most all PC-type hardware, 
has value then maybe it can sway people to keep the separated DDC drivers.

-Andy

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 4492 bytes --]

Re: Black box flight recorder for Linux

[Krzysztof Halasa]

Hi,

Andy Green <andy@warmcat.com> writes:

> A whole other way forward is to consider to replace the EEPROM from
> the original proposal (which does provide its own advantages such as
> simplicity, I accept) with something else that ends up on another
> PC. In this concept some logic presents a fake I2C peripheral to the
> DDC interface at an I2C address of our choosing.  This logic acts as a
> bidirectional "UART" type of thing, allowing transfer of data in both
> directions between the Linux box being debugged and another PC.

Right. I think one could use something like ATMEL 89F2051 (20-pin 8051
non-SMD clones with flash and hardware UART) or something similar.

Client I2C is difficult in Linux (using general purpose I/O port) but
with a dedicated CPU it's not a problem (not sure about 400 kHz access).

> http://warmcat.com/milksop/filtror.html

Well, that's a bit more complicated. I'm not going to try that on
an experimental PCB :-)

> However this would be much simpler, not even needing RAM.  It can hook
> to the second PC by the same I2C method, parallel printer port, RS232
> or USB depending on the level of complexity of the design.
>
> I guess the link will feel quite like a 9600 or 19200 baud serial port
> in terms of throughput.

Depends on I2C. With something like 400 kHz it should be faster,
probably like 115200.

> Maybe this effort is considered too esoteric, but it seems to me to be
> a reason to keep the DDC access drivers standalone, the
> hardware-specific framebuffer drivers can call through to them and we
> can use them in a clean way.  I realize this is a bit of a late
> objection and that there was not previously much point to keeping them
> as separate things in the world.

Actually we have:
- the Xserver "hardware access" issue
- DRI/DRM
- now the I2C bus driver
- frame buffer

To avoid conflicts we really need them managed by a single driver.
Probably the GGI (KGI?) should be revisited?

Long-term project, unfortunatelly, but I think we'll have to do that
eventually.

The I2C, graphics subsystem and DRI/DRM could be sub-modules, with
the master module only keeping track of hardware access, mode
settings etc.
-- 
Krzysztof Halasa

Re: Black box flight recorder for Linux

[Jan Engelhardt]

>> Now, the question I have is, if I write values to RAM, do any of those
>> values survive a reset? If any did survive, one could use them to
>> store oops output in. [...]
>
>Interesting idea.
>
>I think the most trivial and reliable way would be to solder some
>I^2 or similar EEPROM chip to, for example, parallel port connector.
>

My turn. If the NVRAM was not so small on x86, you could easily put an Oops 
in there. Somewhat portable, I guess.


Jan Engelhardt
-- 

Re: Black box flight recorder for Linux

[Russell Senior]

>>>>> "James" == James Courtier-Dutton <James@superbug.co.uk> writes:

James> The main advantage of something like this would be for newer
James> motherboards that are around now that don't have a serial port.

If it has a PCI slot, you could _add_ a serial port, of course.


-- 
Russell Senior         ``I have nine fingers; you have ten.''
seniorr@aracnet.com

Re: Black box flight recorder for Linux

[Robert Hancock]

Andi Kleen wrote:
> James Courtier-Dutton <James@superbug.co.uk> writes:
>> Now, the question I have is, if I write values to RAM, do any of those
>> values survive a reset?
> 
> They don't generally.
> 
> Some people used to write the oopses into video memory, but that
> is not portable.

I wouldn't think most BIOSes these days would bother to clear system RAM 
on a reboot. Certainly Microsoft was encouraging vendors not to do this 
because it slowed down system boot time.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: Black box flight recorder for Linux

[Andi Kleen]

On Saturday 08 April 2006 16:05, Robert Hancock wrote:
> Andi Kleen wrote:
> > James Courtier-Dutton <James@superbug.co.uk> writes:
> >> Now, the question I have is, if I write values to RAM, do any of those
> >> values survive a reset?
> >
> > They don't generally.
> >
> > Some people used to write the oopses into video memory, but that
> > is not portable.
>
> I wouldn't think most BIOSes these days would bother to clear system RAM
> on a reboot. Certainly Microsoft was encouraging vendors not to do this
> because it slowed down system boot time.to

Reset button is like a cold boot and it generally ends up with cleared 
RAM.

-Andi

Re: Black box flight recorder for Linux

[James Courtier-Dutton]

Andi Kleen wrote:
> On Saturday 08 April 2006 16:05, Robert Hancock wrote:
>> Andi Kleen wrote:
>>> James Courtier-Dutton <James@superbug.co.uk> writes:
>>>> Now, the question I have is, if I write values to RAM, do any of those
>>>> values survive a reset?
>>> They don't generally.
>>>
>>> Some people used to write the oopses into video memory, but that
>>> is not portable.
>> I wouldn't think most BIOSes these days would bother to clear system RAM
>> on a reboot. Certainly Microsoft was encouraging vendors not to do this
>> because it slowed down system boot time.to
> 
> Reset button is like a cold boot and it generally ends up with cleared 
> RAM.
> 
> -Andi

Thank you. That saved me 30mins hacking. :-)

Re: Black box flight recorder for Linux

[JustFillBug]

On 2006-04-08, James Courtier-Dutton <James@superbug.co.uk> wrote:
> Andi Kleen wrote:
>> Reset button is like a cold boot and it generally ends up with cleared 
>> RAM.
>> 
> Thank you. That saved me 30mins hacking. :-)
>

How about Magic sysRq reboot? 

Re: Black box flight recorder for Linux

[James Courtier-Dutton]

JustFillBug wrote:
> On 2006-04-08, James Courtier-Dutton <James@superbug.co.uk> wrote:
>> Andi Kleen wrote:
>>> Reset button is like a cold boot and it generally ends up with cleared 
>>> RAM.
>>>
>> Thank you. That saved me 30mins hacking. :-)
>>
> 
> How about Magic sysRq reboot? 
> 

Another alternative is that the motherboard in question has a PS2 mouse 
port, but not RS232 serial port.
Can I connect the PS2 ports of two PCs together and get the console 
working through that?

James

Re: Black box flight recorder for Linux

[Ville Herva]

James Courtier-Dutton wrote:
> Now, the question I have is, if I write values to RAM, do any of
> those values survive a reset?

I suppose this hasn't been mentioned in the thread yet -
apparently Philip Gladstone did such patch 7 years ago:

http://www.ussg.iu.edu/hypermail/linux/kernel/9906.3/1377.html
http://groups.google.com/groups?q=%22Utility+module+to+capture+OOPS+output+over+reboot%22&hl=en&selm=fa.fbd0l7v.14hau3n%40ifi.uio.no&rnum=1)


-- v -- 

v@iki.fi

Re: Black box flight recorder for Linux

[Andi Kleen]

On Saturday 08 April 2006 18:28, James Courtier-Dutton wrote:
> Andi Kleen wrote:
> > On Saturday 08 April 2006 16:05, Robert Hancock wrote:
> >> Andi Kleen wrote:
> >>> James Courtier-Dutton <James@superbug.co.uk> writes:
> >>>> Now, the question I have is, if I write values to RAM, do any of those
> >>>> values survive a reset?
> >>> They don't generally.
> >>>
> >>> Some people used to write the oopses into video memory, but that
> >>> is not portable.
> >> I wouldn't think most BIOSes these days would bother to clear system RAM
> >> on a reboot. Certainly Microsoft was encouraging vendors not to do this
> >> because it slowed down system boot time.to
> > 
> > Reset button is like a cold boot and it generally ends up with cleared 
> > RAM.
> > 
> > -Andi
> 
> Thank you. That saved me 30mins hacking. :-)

Sorry for having discouraged you.

Actually there is a rare special case - triple fault - where you
might be ok if the BIOS correctly supports the ACPI "bootflag" standard,
but triple faults are relatively rare. They happen when the 
kernel screws up so badly that the CPU cannot even run exception
handlers anymore. But I suspect it's too special for this.

First if you're not aware of this - the "official" way right now
to solve this problem is kexec + kdump + a preloaded crash kernel. But in 
practice it still has many problems because a lot of drivers cannot 
reinitialize the hardware properly. And of course it will users need
to load the crash kernel in advance and lose about 64MB of RAM.

My personal solution to the problem is firescope, but it also has its
drawbacks (needs ohci1394 loaded first, requires a firewire cable)

What I would do for this if you want to hack.- define a generic interface that allows
drivers to register memory storage handlers. Add a entry into the oops die 
and panic notifiers that saves the kernel log into these backends.

Then write some Documentation file for it and add a proof of comcept e.g. to the  
Nvidia/ATI frame buffer drivers. Then driver writers could expose this functionality 
if their hardware supports it or if someone has an embedded platform that
guarantees it they could also use it.

For Nvidia/ATI it might be tricky to get the
X server to keep its hands off the memory, but I assume most graphic cards
these days have more memory than the X server uses at least without 3d (?).
If you're unlucky it will fill up everything with mozilla pixmaps over time though.
In the worst case you would need to define a new interface between X server
and kernel to tell the X server to leave some memory alone.

The generic driver could also do the high level work, like adding proper
checksums and magic values to make sure the data is sane after reboot.
You would also need another driver that allows the boot process to read that
data.

Hope this helps,

-Andi

Re: Black box flight recorder for Linux

[Eric W. Biederman]

Andi Kleen <ak@suse.de> writes:

> On Saturday 08 April 2006 18:28, James Courtier-Dutton wrote:
>> Andi Kleen wrote:
>> > On Saturday 08 April 2006 16:05, Robert Hancock wrote:
>> >> Andi Kleen wrote:
>> >>> James Courtier-Dutton <James@superbug.co.uk> writes:
>> >>>> Now, the question I have is, if I write values to RAM, do any of those
>> >>>> values survive a reset?
>> >>> They don't generally.
>> >>>
>> >>> Some people used to write the oopses into video memory, but that
>> >>> is not portable.
>> >> I wouldn't think most BIOSes these days would bother to clear system RAM
>> >> on a reboot. Certainly Microsoft was encouraging vendors not to do this
>> >> because it slowed down system boot time.to
>> > 
>> > Reset button is like a cold boot and it generally ends up with cleared 
>> > RAM.
>> > 
>> > -Andi
>> 
>> Thank you. That saved me 30mins hacking. :-)

Actually clearing the memory can be done at full memory bandwidth
which can happen in seconds.  On systems with ECC you need initialize
all of the check bits so some kind of write to memory needs to happen.

In practice a reset does not clear the memory and only a few bits
tend to get flipped.  Unless you can get support from the
BIOS/firmware developers it isn't a practical approach though.

> First if you're not aware of this - the "official" way right now
> to solve this problem is kexec + kdump + a preloaded crash kernel. But in 
> practice it still has many problems because a lot of drivers cannot 
> reinitialize the hardware properly. And of course it will users need
> to load the crash kernel in advance and lose about 64MB of RAM.

Does a kernel really need 64M?  That number seems insanely large
to me.  8M should be more than sufficient if someone actually
tried to be small.  If you are aiming for a dedicated hardware
solution you don't need even need a kernel in there and the
amount reserved could be reduced to less than a meg.

The size etc becomes a trade off between what is expedient
and maintainable.

Eric

Re: Black box flight recorder for Linux

[linux-os (Dick Johnson)]

On Sat, 8 Apr 2006, Andi Kleen wrote:

> On Saturday 08 April 2006 16:05, Robert Hancock wrote:
>> Andi Kleen wrote:
>>> James Courtier-Dutton <James@superbug.co.uk> writes:
>>>> Now, the question I have is, if I write values to RAM, do any of those
>>>> values survive a reset?
>>>
>>> They don't generally.
>>>
>>> Some people used to write the oopses into video memory, but that
>>> is not portable.
>>
>> I wouldn't think most BIOSes these days would bother to clear system RAM
>> on a reboot. Certainly Microsoft was encouraging vendors not to do this
>> because it slowed down system boot time.to
>
> Reset button is like a cold boot and it generally ends up with cleared
> RAM.
>
> -Andi

Further, in a boot where the BIOS needs to initialize hardware,
It will write all RAM before enabling NMI. This makes sure that
the parity bit(s) are set properly. Most BIOS will attempt to
preserve RAM on a 'warm' boot as a throw-back to the '286 days
with their above-1MB-memory-manager paged RAM because the
only way to get back from protected mode to 16-bit real mode
was a hardware reset. When using a memory-manager like DOS's
HIMEM.SYS, you might actually be rebooting the machine hundreds
of times per second!

If you want to make a flight-data recorder, you need to use
FAA specs and, as such, you can't rely on second-order effects.
You will need write all the parameters to flash (or equivalent)
at least 10 times per second. I would advise against putting
a file-system on the flash because the file-system might get
corrupt because of a bad write during a crash. Instead, I would
write a large number of identical groups of parameters (a structure
image) at a large number of raw offsets, each with the required
time-stamp. This, done the required 10 times per second.

Note that the NTSB, in investigating some light airplane accidents,
has been successful extracting data from hand-held GPS receivers
and FADEC controllers, so using flash RAM for accident investigation
has some successful history (useful for obtaining certification).

I once proposed a FDR for light aircraft that would cost the end-user
under $2,000. This was to use a small embedded CPU, raw NAV inputs
from the panel, a cheap accelerometer, and some pressure sensors
for altimetry. This would be mounted in the tail, requiring no
user (pilot) input at all. In the event of a crash, the last
hour of flight could be retrieved.

Typical response; "There is no market for this...."

Good luck!

Cheers,
Dick Johnson
Penguin : Linux version 2.6.15.4 on an i686 machine (5589.42 BogoMips).
Warning : 98.36% of all statistics are fiction, book release in April.
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Black box flight recorder for Linux

[Krzysztof Halasa]

"linux-os \(Dick Johnson\)" <linux-os@analogic.com> writes:

> Further, in a boot where the BIOS needs to initialize hardware,
> It will write all RAM before enabling NMI. This makes sure that
> the parity bit(s) are set properly. Most BIOS will attempt to
> preserve RAM on a 'warm' boot as a throw-back to the '286 days
> with their above-1MB-memory-manager paged RAM because the
> only way to get back from protected mode to 16-bit real mode
> was a hardware reset.

I think there is no distinction WRT RAM test between cold and warm
boot anymore. If the BIOS clears the RAM is, I think, determined by
the "fast POST" option in BIOS setup (it always checks the size
so some bytes will be changed anyway).

> When using a memory-manager like DOS's
> HIMEM.SYS, you might actually be rebooting the machine hundreds
> of times per second!

Yes but it uses (or, rather, used) a CMOS flag to skip POST (not only
the RAM test) and to go directly to the entry point in real mode.

IIRC (I may be wrong, that was 15+ years ago) only 286 required
KBC reset to return to real mode (did LOADALL matter?), 386s have
no such problem.


BTW I understand the idea have nothing to do with actual aircraft,
so it would be the admin rather than NTSB looking at the data(?).
-- 
Krzysztof Halasa

Re: Black box flight recorder for Linux

[linux-os (Dick Johnson)]

On Mon, 10 Apr 2006, Krzysztof Halasa wrote:

> "linux-os \(Dick Johnson\)" <linux-os@analogic.com> writes:
>
>> Further, in a boot where the BIOS needs to initialize hardware,
>> It will write all RAM before enabling NMI. This makes sure that
>> the parity bit(s) are set properly. Most BIOS will attempt to
>> preserve RAM on a 'warm' boot as a throw-back to the '286 days
>> with their above-1MB-memory-manager paged RAM because the
>> only way to get back from protected mode to 16-bit real mode
>> was a hardware reset.
>
> I think there is no distinction WRT RAM test between cold and warm
> boot anymore. If the BIOS clears the RAM is, I think, determined by
> the "fast POST" option in BIOS setup (it always checks the size
> so some bytes will be changed anyway).
>
>> When using a memory-manager like DOS's
>> HIMEM.SYS, you might actually be rebooting the machine hundreds
>> of times per second!
>
> Yes but it uses (or, rather, used) a CMOS flag to skip POST (not only
> the RAM test) and to go directly to the entry point in real mode.
>
> IIRC (I may be wrong, that was 15+ years ago) only 286 required
> KBC reset to return to real mode (did LOADALL matter?), 386s have
> no such problem.
>
>
> BTW I understand the idea have nothing to do with actual aircraft,
> so it would be the admin rather than NTSB looking at the data(?).
> --
> Krzysztof Halasa

Yes. After I responded and after reading other responses I noted that
the idea was to review the reason for a PC crash. There are no
good places to store things that can be counted upon to remain after
a crash. Even though CMOS RAM goes to 0x7f and not all of it is
needed for BIOS setup, many/(maybe all) BIOS checksum the whole
thing with some hair-brained checksum routines so it's not a good
place to store things. Many BIOS use 128k or more, which can
be re-written in 64k pages. Certainly there is space available
...but... the BIOS would be trashed if a power-failure occurred
during the write, so that's not a good place either. I did, at
one time, use a "spare" page of screen memory (one that wasn't
displayed) during the boot process (buffering for a bootable NVRAM
disk). This is "available" RAM, but it isn't going to survive
even as screen-card reset!


Cheers,
Dick Johnson
Penguin : Linux version 2.6.15.4 on an i686 machine (5589.42 BogoMips).
Warning : 98.36% of all statistics are fiction, book release in April.
_
\x1a\x04

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: Black box flight recorder for Linux

[linux]

> I wouldn't think most BIOSes these days would bother to clear system RAM 
> on a reboot. Certainly Microsoft was encouraging vendors not to do this 
> because it slowed down system boot time.

I don't think they explicitly clear it all, but they do write to it to
test how much RAM is installed and don't bother to put back what they
scribbled on.


Sufficient ECC techniques sould probably recover from the damage.  For a
first attempt, I'd take 4096-byte pages, not use the first and last 8
bytes at all, and divide the remaining 4080 bytes into 16 interleaved
255-byte ECC segments, each using a byte-wide Reed-Solomon code.
(The fraction of that 255 devoted to ECC is up to you; n-bit-wide
Reed-Solomon just requires that data + ECC <= (2^n - 1) bytes of n
bits each.)

For extra hack value, you could detect at boot what parts of your
log got corrupted and avoid using those parts when logging new data.
(There are complications...)

It is possible to update RS ECC incrementally, or perhaps it would be
better to store the tail of the log in some less efficient form (like
multiple replication) and then pack it into ECC when full.


The other thing that might be a problem is that I don't know how long
refresh stops during reset.  Again, ECC can be your friend.
(And code for it already exists in lib/reed_solomon/)