[uml-devel] Simple db with separable io layer for humfs metadata.

[uml-devel] Simple db with separable io layer for humfs metadata.

[Piotr Neuman]

As Jeff Dike had requested that humfs db should allow UML to do actuall IO, I 
have found 2 simple db-es suitable for such task. I was however unable to 
find one that would have clear distinction of IO operations, like for example 
using another lib to do it. My findings:

TDB: http://sourceforge.net/projects/tdb/ (it's used in samba project)

Need only reimplement 2 pretty simple functions:
static int tdb_write(TDB_CONTEXT *tdb, tdb_off off, void *buf, tdb_len len)
static int tdb_read(TDB_CONTEXT *tdb,tdb_off off,void *buf,tdb_len len,int cv)

The whole db is just two files: tdb.c and tdb.h which weight around 49KB, so 
shipping it along with UML would be no problem imo.

For example (to show difficulty of reimplementing):

static int tdb_write(TDB_CONTEXT *tdb, tdb_off off, void *buf, tdb_len len)
{
	if (tdb_oob(tdb, off + len, 0) != 0)
		return -1;

	if (tdb->map_ptr)
		memcpy(off + (char *)tdb->map_ptr, buf, len);
#ifdef HAVE_PWRITE
	else if (pwrite(tdb->fd, buf, len, off) != (ssize_t)len) {
#else
	else if (lseek(tdb->fd, off, SEEK_SET) != off
		 || write(tdb->fd, buf, len) != (ssize_t)len) {
#endif
		TDB_LOG((tdb, 0,"tdb_write failed at %d len=%d (%s)\n",
			   off, len, strerror(errno)));
		return TDB_ERRCODE(TDB_ERR_IO, -1);
	}
	return 0;
}

QDBM: http://qdbm.sourceforge.net/

In this case need to reimplement 4 functions (also simple):
static int dpwrite(int fd, const void *buf, int size);
static int dpseekwrite(int fd, int off, const void *buf, int size);
static int dpread(int fd, void *buf, int size);
static int dpseekread(int fd, int off, void *buf, int size);

Basic setup is: depot.c, depot.h, myconf.c and myconf.h - ~86KB. But 
additional b+tree support allows for gracious recovery after crash, and 
smaller file size than tdb at the cost of increasing code size to ~308KB. 
qdbm is generally much more advanced and featurefull than tdb. It also has 
plenty of interfaces to other languages, like perl (not counted in that size 
approximation).

Example:

static int dpseekwrite(int fd, int off, const void *buf, int size){
  char *lbuf;
  assert(fd >= 0 && buf && size >= 0);
  if(size < 1) return TRUE;
  lbuf = (char *)buf;
  if(off < 0){
    if(lseek(fd, 0, SEEK_END) == -1){
      dpecode = DP_ESEEK;
      return FALSE;
    }
  } else {
    if(lseek(fd, off, SEEK_SET) != off){
      dpecode = DP_ESEEK;
      return FALSE;
    }
  }
  if(dpwrite(fd, lbuf, size) != size){
    dpecode = DP_EWRITE;
    return FALSE;
  }
  return TRUE;
}

Regards to UML developement team (Jeff Dike that is ;)


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Simple db with separable io layer for humfs metadata.

[Jeff Dike]

sikkh@wp.pl said:
> TDB: http://sourceforge.net/projects/tdb/ (it's used in samba project)
> 
> Need only reimplement 2 pretty simple functions:
> static int tdb_write(TDB_CONTEXT *tdb, tdb_off off, void *buf, tdb_len len)
> static int tdb_read(TDB_CONTEXT *tdb,tdb_off off,void *buf,tdb_len len,int cv)

> QDBM: http://qdbm.sourceforge.net/
> 
> In this case need to reimplement 4 functions (also simple):
> static int dpwrite(int fd, const void *buf, int size);
> static int dpseekwrite(int fd, int off, const void *buf, int size);
> static int dpread(int fd, void *buf, int size);
> static int dpseekread(int fd, int off, void *buf, int size);

That's exactly the sort of thing I was looking for.

The reason I want to do IO on behalf of any DB that gets linked into UML is
that I have a long-term goal of UML never blocking for any reason, except in
its own idle loop.  This mostly includes waiting for disk IO, i.e. for normal
IO, swapping or faulting in pages, populating mmap areas, etc.  So, I especially
want no blocking IO.  I want UML to be able to schedule other things while
it's waiting for IO.

So, when it comes to databases, I want to be able to do AIO on their behalf,
do other things while the IO is happening, and let them know when it has 
finished.

				Jeff



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel