* [PATCH 0/3] Pseudo performance changes...
@ 2013-02-17 2:23 Peter Seebach
2013-02-17 2:23 ` [PATCH 1/3] Use in-memory database for files Peter Seebach
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Peter Seebach @ 2013-02-17 2:23 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org
Unlike most of my submissions, this isn't patches against oe-core; rather,
it's patches against pseudo, and if I can get some confirmation that they do
what I think they do, and some review, I'm planning to make this into
pseudo 1.5, and send a patch "soonish" to merge that into oe-core.
What this does: Fix a number of build performance issues. By far the
largest change is actually not so much a problem with pseudo as a problem
that pseudo can solve by brute force. Packaging systems (at least RPM and
SMART) do a lot of fsync() and fdatasync() calls. That usually implies
flushing EVERYTHING that's been written, not just one specific file. And
that, in turn, results in a severe performance hit.
So, for instance, on one of my test workstations, this moves a do_rootfs
with about 1200 RPMs from about 22 minutes to about 4.5. Yeah.
The other changes aren't as dramatic for that case, but have very significant
performance impact for at least some workloads. The first is switching to
using an in-memory database for the files database, dumping it to disk only
when the pseudo daemon is idle or shutting down. This doesn't produce huge
benefits in all cases, but for workloads with a lot of parallelism, it can
produce a very noticeable reduction in how much pseudo slows things down.
The second is a fairly major protocol change. In short, with this patch,
pseudo clients only wait for a server response when they need information
from the server in order to continue. That's OP_FSTAT, OP_STAT,
OP_MAY_UNLINK, and OP_MKNOD. Everything else just silently assumes that
it probably succeeded.
How much does this matter? Between the protocol change and the memory
DB, a trivial unpack of a tarball (lots of writes to the database, very
few reads) can be about 4x faster. Removing stuff isn't much faster, but
it might be a bit faster.
This is most noticeable, by far, when running more than one build, or
when running builds while doing other things. It has a much smaller effect
on builds with no shared state (compile time still dominates that), but
even there I'm seeing decreases from ~83 minutes to ~64 from just the
fsync and memory changes. Still waiting on my real test case (multiple
simultaneous builds which need compiles) completing.
Peter Seebach (3):
Use in-memory database for files
allow pseudo to force asynchronous behavior
If you don't want the answer, don't ask the question.
ChangeLog.txt | 10 ++++
Makefile.in | 6 +-
configure | 51 ++++++++++++++++
enums/msg_type.in | 1 +
enums/op.in | 46 +++++++-------
enums/query_type.in | 18 +++---
guts/COPYRIGHT | 2 +-
maketables | 41 +++++++------
makewrappers | 13 +++-
ports/darwin/guts/open.c | 5 +-
ports/linux/guts/openat.c | 21 ++++++-
ports/unix/guts/fchmod.c | 16 ++---
ports/unix/guts/fchmodat.c | 31 ++++++----
ports/unix/guts/fchown.c | 16 ++---
ports/unix/guts/fchownat.c | 16 ++---
ports/unix/guts/fdatasync.c | 16 +++++
ports/unix/guts/fsync.c | 16 +++++
ports/unix/guts/mknodat.c | 4 +-
ports/unix/guts/msync.c | 16 +++++
ports/unix/guts/sync.c | 16 +++++
ports/unix/guts/sync_file_range.c | 13 ++++
ports/unix/guts/syncfs.c | 13 ++++
ports/unix/wrapfuncs.in | 9 +++
pseudo.c | 3 +-
pseudo_client.c | 22 ++++---
pseudo_db.c | 119 ++++++++++++++++++++++++++++++++++++-
pseudo_db.h | 1 +
pseudo_server.c | 32 ++++++----
templates/wrapfuncs.c | 2 +
29 files changed, 453 insertions(+), 122 deletions(-)
create mode 100644 ports/unix/guts/fdatasync.c
create mode 100644 ports/unix/guts/fsync.c
create mode 100644 ports/unix/guts/msync.c
create mode 100644 ports/unix/guts/sync.c
create mode 100644 ports/unix/guts/sync_file_range.c
create mode 100644 ports/unix/guts/syncfs.c
--
1.7.9.5
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/3] Use in-memory database for files
2013-02-17 2:23 [PATCH 0/3] Pseudo performance changes Peter Seebach
@ 2013-02-17 2:23 ` Peter Seebach
2013-02-17 2:23 ` [PATCH 2/3] allow pseudo to force asynchronous behavior Peter Seebach
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Peter Seebach @ 2013-02-17 2:23 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org
It turns out that file databases don't get very large, and that
sqlite3 can be quite fast with an in-memory database. It also turns
out that dumping the database to disk on exit (or during idle times)
is pretty cheap compared to constant updates.
So: We add "--enable-memory-db", which defaults to on if you have
sqlite 3.7 or later, and off for 3.6 (because 3.6 has horrible
performance with in-memory db on some hosts we tried).
---
ChangeLog.txt | 5 +++
Makefile.in | 5 ++-
configure | 32 +++++++++++++++
pseudo_db.c | 119 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
pseudo_db.h | 1 +
pseudo_server.c | 2 +
6 files changed, 159 insertions(+), 5 deletions(-)
diff --git a/ChangeLog.txt b/ChangeLog.txt
index 8f52cc9..ed9e5fa 100644
--- a/ChangeLog.txt
+++ b/ChangeLog.txt
@@ -1,3 +1,8 @@
+2013-02-15:
+ * (seebs) Add support for in-memory DB. This, plus upcoming
+ fsync-related changes, are expected to be big enough to justify
+ calling this 1.5.
+
2013-02-13:
* (seebs) calling link while chrooted could in some cases result
in the root path not being prepended at all. One more try!
diff --git a/Makefile.in b/Makefile.in
index e0cd7a9..094deba 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -23,11 +23,12 @@ LIBDIR=@LIBDIR@
SUFFIX=@SUFFIX@
SQLITE=@SQLITE@
SQLITE_LIB=@SQLITE_LIB@
+SQLITE_MEMORY=@SQLITE_MEMORY@
BITS=@BITS@
ARCH_FLAGS=@ARCH_FLAGS@
MARK64=@MARK64@
RPATH=@RPATH@
-VERSION=1.4.5
+VERSION=1.5
LIB=@LIB@
BIN=bin
@@ -37,7 +38,7 @@ LOCALSTATEDIR=$(PREFIX)/$(LOCALSTATE)
CFLAGS_BASE=-pipe -std=gnu99 -Wall -W -Wextra
CFLAGS_CODE=-fPIC -D_LARGEFILE64_SOURCE -D_ATFILE_SOURCE $(ARCH_FLAGS)
-CFLAGS_DEFS=-DPSEUDO_PREFIX='"$(PREFIX)"' -DPSEUDO_SUFFIX='"$(SUFFIX)"' -DPSEUDO_BINDIR='"$(BIN)"' -DPSEUDO_LIBDIR='"$(LIB)"' -DPSEUDO_LOCALSTATEDIR='"$(LOCALSTATE)"' -DPSEUDO_VERSION='"$(VERSION)"'
+CFLAGS_DEFS=-DPSEUDO_PREFIX='"$(PREFIX)"' -DPSEUDO_SUFFIX='"$(SUFFIX)"' -DPSEUDO_BINDIR='"$(BIN)"' -DPSEUDO_LIBDIR='"$(LIB)"' -DPSEUDO_LOCALSTATEDIR='"$(LOCALSTATE)"' -DPSEUDO_VERSION='"$(VERSION)"' $(SQLITE_MEMORY)
CFLAGS_DEBUG=-O2 -g
CFLAGS_SQL=-L$(SQLITE)/$(SQLITE_LIB) -I$(SQLITE)/include $(RPATH)
CFLAGS_PSEUDO=$(CFLAGS_BASE) $(CFLAGS_CODE) $(CFLAGS_DEFS) \
diff --git a/configure b/configure
index 9cb7804..44bee74 100755
--- a/configure
+++ b/configure
@@ -25,6 +25,7 @@ opt_arch=x86
opt_bits=
opt_sqlite=/usr
opt_rpath=
+opt_memory=
compile_x86_32=-m32
compile_x86_64=-m64
@@ -35,6 +36,7 @@ usage()
echo >&2 " configure --prefix=..."
echo >&2 " [--libdir=...]"
echo >&2 " [--suffix=...]"
+ echo >&2 " [--enable-memory-db]"
echo >&2 " [--with-sqlite=...]"
echo >&2 " [--with-sqlite-lib=...]"
echo >&2 " [--enable-static-sqlite]"
@@ -69,6 +71,12 @@ do
sqlite_ldarg='$(SQLITE)/$(SQLITE_LIB)/libsqlite3.a'
use_maybe_rpath=false
;;
+ --enable-memory-db=no)
+ opt_memory=false
+ ;;
+ --enable-memory-db=yes | --enable-memory-db)
+ opt_memory=true
+ ;;
--with-sqlite=*)
opt_sqlite=${arg#--with-sqlite=}
# assign new value if unset
@@ -170,6 +178,29 @@ if [ "${SQLITE3_VERSION}" -lt "03006000" ]; then
exit 1
fi
+if [ -z "$opt_memory" ]; then
+ if [ "${SQLITE3_VERSION}" -lt "03007000" ]; then
+ echo "Disabling in-memory database by default (sqlite too old)."
+ opt_memory=false
+ else
+ echo "Enabling in-memory database by default."
+ opt_memory=true
+ fi
+fi
+
+if $opt_memory; then
+ if [ "${SQLITE3_VERSION}" -lt "03007000" ]; then
+ cat >&2 <<EOF
+WARNING: sqlite prior to 3.7 has been known to perform exceedingly poorly
+with the in-memory database option. You asked for it, you get it, but if
+you get horrible performance, try turning it off.
+EOF
+ fi
+ SQLITE_MEMORY="-DUSE_MEMORY_DB"
+else
+ SQLITE_MEMORY=""
+fi
+
sed -e '
s,@PREFIX@,'"$opt_prefix"',g
s,@LIBDIR@,'"$opt_libdir"',g
@@ -179,6 +210,7 @@ sed -e '
s,@ARCH_FLAGS@,'"$arch_flags"',g
s,@SQLITE_LDARG@,'"$sqlite_ldarg"',g
s,@SQLITE_LIB@,'"$opt_sqlite_lib"',g
+ s,@SQLITE_MEMORY@,'"$SQLITE_MEMORY"',g
s!@RPATH@!'"$opt_rpath"'!g
s,@MARK64@,'"$opt_mark64"',g
s,@ARCH@,'"$opt_arch"',g
diff --git a/pseudo_db.c b/pseudo_db.c
index 540a3c2..21f2f53 100644
--- a/pseudo_db.c
+++ b/pseudo_db.c
@@ -1,7 +1,7 @@
/*
* pseudo_db.c, sqlite3 interface
*
- * Copyright (c) 2008-2010 Wind River Systems, Inc.
+ * Copyright (c) 2008-2010,2013 Wind River Systems, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the Lesser GNU General Public License version 2.1 as
@@ -50,6 +50,10 @@ struct pdb_file_list {
sqlite3_stmt *stmt;
};
+static int file_db_dirty = 0;
+#ifdef USE_MEMORY_DB
+static sqlite3 *real_file_db = 0;
+#endif
static sqlite3 *file_db = 0;
static sqlite3 *log_db = 0;
@@ -118,6 +122,10 @@ static struct sql_index {
static char *file_pragmas[] = {
"PRAGMA legacy_file_format = OFF;",
"PRAGMA journal_mode = OFF;",
+ /* the default page size produces painfully bad behavior
+ * for memory databases with some versions of sqlite.
+ */
+ "PRAGMA page_size = 8192;",
"PRAGMA locking_mode = EXCLUSIVE;",
/* Setting this to NORMAL makes pseudo noticably slower
* than fakeroot, but is perhaps more secure. However,
@@ -139,7 +147,6 @@ static char *log_pragmas[] = {
NULL
};
-
/* table migrations: */
/* If there is no migration table, we assume "version -1" -- the
* version shipped with wrlinux 3.0, which had no version
@@ -239,6 +246,17 @@ static struct database_info db_infos[] = {
file_migrations,
file_pragmas,
file_setups,
+#ifdef USE_MEMORY_DB
+ &real_file_db
+ },
+ {
+ ":memory:",
+ file_indexes,
+ file_tables,
+ file_migrations,
+ file_pragmas,
+ file_setups,
+#endif
&file_db
},
{ 0, 0, 0, 0, 0, 0, 0 }
@@ -263,6 +281,67 @@ dberr(sqlite3 *db, char *fmt, ...) {
}
}
+#ifdef USE_MEMORY_DB
+
+static void
+pdb_backup() {
+ sqlite3_backup *pBackup;
+ /* no point in doing this if we don't have a database to back up,
+ * or nothing's changed.
+ */
+ if (!file_db || !real_file_db || !file_db_dirty)
+ return;
+
+ pBackup = sqlite3_backup_init(real_file_db, "main", file_db, "main");
+ if (pBackup) {
+ int rc;
+ (void)sqlite3_backup_step(pBackup, -1);
+ rc = sqlite3_backup_finish(pBackup);
+ if (rc != SQLITE_OK) {
+ dberr(real_file_db, "error during flush to disk");
+ }
+ }
+ file_db_dirty = 0;
+}
+
+static void
+pdb_restore() {
+ sqlite3_backup *pBackup;
+ /* no point in doing this if we don't have a database to back up */
+ if (!file_db || !real_file_db)
+ return;
+
+ pBackup = sqlite3_backup_init(file_db, "main", real_file_db, "main");
+ if (pBackup) {
+ int rc;
+ (void)sqlite3_backup_step(pBackup, -1);
+ rc = sqlite3_backup_finish(pBackup);
+ if (rc != SQLITE_OK) {
+ dberr(file_db, "error during load from disk");
+ }
+ }
+ file_db_dirty = 0;
+}
+
+int
+pdb_maybe_backup(void) {
+ static int occasional = 0;
+ if (file_db && real_file_db) {
+ occasional = (occasional + 1) % 10;
+ if (occasional == 0) {
+ pdb_backup();
+ return 1;
+ }
+ }
+ return 0;
+}
+#else /* USE_MEMORY_DB */
+int
+pdb_maybe_backup(void) {
+ return 0;
+}
+#endif
+
/* those who enjoy children, sausages, and databases, should not watch
* them being made.
*/
@@ -448,6 +527,12 @@ make_tables(sqlite3 *db,
static void
cleanup_db(void) {
pseudo_debug(1, "server exiting\n");
+#ifdef USE_MEMORY_DB
+ if (real_file_db) {
+ pdb_backup();
+ sqlite3_close(real_file_db);
+ }
+#endif
if (file_db)
sqlite3_close(file_db);
if (log_db)
@@ -478,7 +563,12 @@ get_db(struct database_info *dbinfo) {
return 0;
dbfile = pseudo_localstatedir_path(dbinfo->pathname);
- rc = sqlite3_open(dbfile, &db);
+#ifdef USE_MEMORY_DB
+ if (!strcmp(dbinfo->pathname, ":memory:")) {
+ rc = sqlite3_open(dbinfo->pathname, &db);
+ } else
+#endif
+ rc = sqlite3_open(dbfile, &db);
free(dbfile);
if (rc) {
pseudo_diag("Failed: %s\n", sqlite3_errmsg(db));
@@ -528,6 +618,11 @@ static int
get_dbs(void) {
int err = 0;
int i;
+#ifdef USE_MEMORY_DB
+ int already_loaded = 0;
+ if (file_db)
+ already_loaded = 1;
+#endif
for (i = 0; db_infos[i].db; ++i) {
if (get_db(&db_infos[i])) {
pseudo_diag("Error getting '%s' database.\n",
@@ -535,6 +630,10 @@ get_dbs(void) {
err = 1;
}
}
+#ifdef USE_MEMORY_DB
+ if (!already_loaded && file_db)
+ pdb_restore();
+#endif
return err;
}
@@ -1082,6 +1181,7 @@ pdb_delete(pseudo_query_t *traits, unsigned long fields) {
/* no need to return it, so... */
if (stmt) {
+ file_db_dirty = 1;
int rc = sqlite3_step(stmt);
if (rc != SQLITE_DONE) {
dberr(log_db, "deletion failed");
@@ -1292,6 +1392,7 @@ pdb_link_file(pseudo_msg_t *msg) {
(msg->pathlen ? msg->path : "<nil> (as NAMELESS FILE)"),
(unsigned long long) msg->dev, (unsigned long long) msg->ino,
(int) msg->mode, msg->uid);
+ file_db_dirty = 1;
rc = sqlite3_step(insert);
if (rc != SQLITE_DONE) {
dberr(file_db, "insert may have failed (rc %d)", rc);
@@ -1324,6 +1425,7 @@ pdb_unlink_file_dev(pseudo_msg_t *msg) {
}
sqlite3_bind_int(sql_delete, 1, msg->dev);
sqlite3_bind_int(sql_delete, 2, msg->ino);
+ file_db_dirty = 1;
rc = sqlite3_step(sql_delete);
if (rc != SQLITE_DONE) {
dberr(file_db, "delete by inode may have failed");
@@ -1359,6 +1461,7 @@ pdb_update_file_path(pseudo_msg_t *msg) {
sqlite3_bind_text(update, 1, msg->path, -1, SQLITE_STATIC);
sqlite3_bind_int(update, 2, msg->dev);
sqlite3_bind_int(update, 3, msg->ino);
+ file_db_dirty = 1;
rc = sqlite3_step(update);
if (rc != SQLITE_DONE) {
dberr(file_db, "update path by inode may have failed");
@@ -1396,6 +1499,7 @@ pdb_may_unlink_file(pseudo_msg_t *msg, int deleting) {
pseudo_debug(1, "cannot mark a file for pending deletion without a path.");
return 1;
}
+ file_db_dirty = 1;
rc = sqlite3_step(mark_file);
if (rc != SQLITE_DONE) {
dberr(file_db, "mark for deletion may have failed");
@@ -1439,6 +1543,7 @@ pdb_cancel_unlink_file(pseudo_msg_t *msg) {
pseudo_debug(1, "cannot unmark a file for pending deletion without a path.");
return 1;
}
+ file_db_dirty = 1;
rc = sqlite3_step(mark_file);
if (rc != SQLITE_DONE) {
dberr(file_db, "unmark for deletion may have failed");
@@ -1475,6 +1580,7 @@ pdb_did_unlink_files(int deleting) {
return 0;
}
sqlite3_bind_int(delete_exact, 1, deleting);
+ file_db_dirty = 1;
rc = sqlite3_step(delete_exact);
if (rc != SQLITE_DONE) {
dberr(file_db, "cleanup of files marked for deletion may have failed");
@@ -1510,6 +1616,7 @@ pdb_did_unlink_file(char *path, int deleting) {
}
sqlite3_bind_text(delete_exact, 1, path, -1, SQLITE_STATIC);
sqlite3_bind_int(delete_exact, 2, deleting);
+ file_db_dirty = 1;
rc = sqlite3_step(delete_exact);
if (rc != SQLITE_DONE) {
dberr(file_db, "cleanup of file marked for deletion may have failed");
@@ -1548,6 +1655,7 @@ pdb_unlink_file(pseudo_msg_t *msg) {
pseudo_debug(1, "cannot unlink a file without a path.");
return 1;
}
+ file_db_dirty = 1;
rc = sqlite3_step(delete_exact);
if (rc != SQLITE_DONE) {
dberr(file_db, "delete exact by path may have failed");
@@ -1595,6 +1703,7 @@ pdb_unlink_contents(pseudo_msg_t *msg) {
pseudo_debug(1, "cannot unlink a file without a path.");
return 1;
}
+ file_db_dirty = 1;
rc = sqlite3_step(delete_sub);
if (rc != SQLITE_DONE) {
dberr(file_db, "delete sub by path may have failed");
@@ -1662,6 +1771,7 @@ pdb_rename_file(const char *oldpath, pseudo_msg_t *msg) {
rc = sqlite3_exec(file_db, "BEGIN;", NULL, NULL, NULL);
+ file_db_dirty = 1;
rc = sqlite3_step(update_exact);
if (rc != SQLITE_DONE) {
dberr(file_db, "update exact may have failed: rc %d", rc);
@@ -1712,6 +1822,7 @@ pdb_renumber_all(dev_t from, dev_t to) {
dberr(file_db, "error binding device numbers to update");
}
+ file_db_dirty = 1;
rc = sqlite3_step(update);
if (rc != SQLITE_DONE) {
dberr(file_db, "update may have failed: rc %d", rc);
@@ -1762,6 +1873,7 @@ pdb_update_inode(pseudo_msg_t *msg) {
dberr(file_db, "error binding %s to select", msg->path);
}
+ file_db_dirty = 1;
rc = sqlite3_step(update);
if (rc != SQLITE_DONE) {
dberr(file_db, "update may have failed: rc %d", rc);
@@ -1806,6 +1918,7 @@ pdb_update_file(pseudo_msg_t *msg) {
sqlite3_bind_int(update, 5, msg->dev);
sqlite3_bind_int(update, 6, msg->ino);
+ file_db_dirty = 1;
rc = sqlite3_step(update);
if (rc != SQLITE_DONE) {
dberr(file_db, "update may have failed: rc %d", rc);
diff --git a/pseudo_db.h b/pseudo_db.h
index fe2fb12..9e0382a 100644
--- a/pseudo_db.h
+++ b/pseudo_db.h
@@ -37,6 +37,7 @@ typedef struct {
char *program;
} log_entry;
+extern int pdb_maybe_backup(void);
extern int pdb_cancel_unlink_file(pseudo_msg_t *msg);
extern int pdb_did_unlink_file(char *path, int deleting);
extern int pdb_did_unlink_files(int deleting);
diff --git a/pseudo_server.c b/pseudo_server.c
index f241242..4af5265 100644
--- a/pseudo_server.c
+++ b/pseudo_server.c
@@ -416,6 +416,8 @@ pseudo_server_loop(void) {
*/
if (active_clients == 1) {
loop_timeout -= LOOP_DELAY;
+ /* maybe flush database to disk */
+ pdb_maybe_backup();
if (loop_timeout <= 0) {
pseudo_debug(1, "no more clients, got bored.\n");
die_peacefully = 1;
--
1.7.9.5
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/3] allow pseudo to force asynchronous behavior
2013-02-17 2:23 [PATCH 0/3] Pseudo performance changes Peter Seebach
2013-02-17 2:23 ` [PATCH 1/3] Use in-memory database for files Peter Seebach
@ 2013-02-17 2:23 ` Peter Seebach
2013-02-17 2:23 ` [PATCH 3/3] If you don't want the answer, don't ask the question Peter Seebach
2013-02-17 10:27 ` [PATCH 0/3] Pseudo performance changes Richard Purdie
3 siblings, 0 replies; 5+ messages in thread
From: Peter Seebach @ 2013-02-17 2:23 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org
The openembedded build, at least with RPM or SMART, is heavily affected
by the cost of calling fsync or fdatasync on package databases all the
time. Gosh, wouldn't it be nice if we could suppress that without making
dozens of highly intrusive and risky changes into RPM, various database
packages, and so on?
Yes, yes it would. If only there were a program which could intercept
system calls and change their behavior!
Enter --enable-force-async. There are now wrappers for fsync, fdatasync,
and a few related functions. If --enable-force-async is set, these wrappers
instantly return 0, even if PSEUDO_DISABLED is set. And with any luck,
bitbake will now perform a bit better.
Credit for this insight goes to Richard Purdie. I've reimplemented
this to add the configure option, and make the fsync suppression work
even when PSEUDO_DISABLED is set.
---
ChangeLog.txt | 5 +++++
Makefile.in | 3 ++-
configure | 19 +++++++++++++++++++
guts/COPYRIGHT | 2 +-
makewrappers | 13 ++++++++++++-
ports/darwin/guts/open.c | 5 ++++-
ports/linux/guts/openat.c | 21 ++++++++++++++++++++-
ports/unix/guts/fdatasync.c | 16 ++++++++++++++++
ports/unix/guts/fsync.c | 16 ++++++++++++++++
ports/unix/guts/msync.c | 16 ++++++++++++++++
ports/unix/guts/sync.c | 16 ++++++++++++++++
ports/unix/guts/sync_file_range.c | 13 +++++++++++++
ports/unix/guts/syncfs.c | 13 +++++++++++++
ports/unix/wrapfuncs.in | 9 +++++++++
templates/wrapfuncs.c | 2 ++
15 files changed, 164 insertions(+), 5 deletions(-)
create mode 100644 ports/unix/guts/fdatasync.c
create mode 100644 ports/unix/guts/fsync.c
create mode 100644 ports/unix/guts/msync.c
create mode 100644 ports/unix/guts/sync.c
create mode 100644 ports/unix/guts/sync_file_range.c
create mode 100644 ports/unix/guts/syncfs.c
diff --git a/ChangeLog.txt b/ChangeLog.txt
index ed9e5fa..3dbf183 100644
--- a/ChangeLog.txt
+++ b/ChangeLog.txt
@@ -1,3 +1,8 @@
+2013-02-16:
+ * (seebs) Add wrapper setting which automatically returns a fixed
+ value from some wrappers. Add fixed-value wrappers for all the
+ *sync() functions. These are all contingent on --enable-force-async.
+
2013-02-15:
* (seebs) Add support for in-memory DB. This, plus upcoming
fsync-related changes, are expected to be big enough to justify
diff --git a/Makefile.in b/Makefile.in
index 094deba..8046bda 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -24,6 +24,7 @@ SUFFIX=@SUFFIX@
SQLITE=@SQLITE@
SQLITE_LIB=@SQLITE_LIB@
SQLITE_MEMORY=@SQLITE_MEMORY@
+FORCE_ASYNC=@FORCE_ASYNC@
BITS=@BITS@
ARCH_FLAGS=@ARCH_FLAGS@
MARK64=@MARK64@
@@ -38,7 +39,7 @@ LOCALSTATEDIR=$(PREFIX)/$(LOCALSTATE)
CFLAGS_BASE=-pipe -std=gnu99 -Wall -W -Wextra
CFLAGS_CODE=-fPIC -D_LARGEFILE64_SOURCE -D_ATFILE_SOURCE $(ARCH_FLAGS)
-CFLAGS_DEFS=-DPSEUDO_PREFIX='"$(PREFIX)"' -DPSEUDO_SUFFIX='"$(SUFFIX)"' -DPSEUDO_BINDIR='"$(BIN)"' -DPSEUDO_LIBDIR='"$(LIB)"' -DPSEUDO_LOCALSTATEDIR='"$(LOCALSTATE)"' -DPSEUDO_VERSION='"$(VERSION)"' $(SQLITE_MEMORY)
+CFLAGS_DEFS=-DPSEUDO_PREFIX='"$(PREFIX)"' -DPSEUDO_SUFFIX='"$(SUFFIX)"' -DPSEUDO_BINDIR='"$(BIN)"' -DPSEUDO_LIBDIR='"$(LIB)"' -DPSEUDO_LOCALSTATEDIR='"$(LOCALSTATE)"' -DPSEUDO_VERSION='"$(VERSION)"' $(SQLITE_MEMORY) $(FORCE_ASYNC)
CFLAGS_DEBUG=-O2 -g
CFLAGS_SQL=-L$(SQLITE)/$(SQLITE_LIB) -I$(SQLITE)/include $(RPATH)
CFLAGS_PSEUDO=$(CFLAGS_BASE) $(CFLAGS_CODE) $(CFLAGS_DEFS) \
diff --git a/configure b/configure
index 44bee74..d88a930 100755
--- a/configure
+++ b/configure
@@ -26,6 +26,7 @@ opt_bits=
opt_sqlite=/usr
opt_rpath=
opt_memory=
+opt_async=
compile_x86_32=-m32
compile_x86_64=-m64
@@ -37,6 +38,7 @@ usage()
echo >&2 " [--libdir=...]"
echo >&2 " [--suffix=...]"
echo >&2 " [--enable-memory-db]"
+ echo >&2 " [--enable-force-async]"
echo >&2 " [--with-sqlite=...]"
echo >&2 " [--with-sqlite-lib=...]"
echo >&2 " [--enable-static-sqlite]"
@@ -71,6 +73,12 @@ do
sqlite_ldarg='$(SQLITE)/$(SQLITE_LIB)/libsqlite3.a'
use_maybe_rpath=false
;;
+ --enable-force-async=no | --disable-force-async)
+ opt_async=false
+ ;;
+ --enable-force-async=yes | --enable-force-async)
+ opt_async=true
+ ;;
--enable-memory-db=no)
opt_memory=false
;;
@@ -178,6 +186,16 @@ if [ "${SQLITE3_VERSION}" -lt "03006000" ]; then
exit 1
fi
+if [ -z "$opt_async" ]; then
+ opt_async=false
+fi
+
+if $opt_async; then
+ FORCE_ASYNC="-DPSEUDO_FORCE_ASYNC"
+else
+ FORCE_ASYNC=""
+fi
+
if [ -z "$opt_memory" ]; then
if [ "${SQLITE3_VERSION}" -lt "03007000" ]; then
echo "Disabling in-memory database by default (sqlite too old)."
@@ -211,6 +229,7 @@ sed -e '
s,@SQLITE_LDARG@,'"$sqlite_ldarg"',g
s,@SQLITE_LIB@,'"$opt_sqlite_lib"',g
s,@SQLITE_MEMORY@,'"$SQLITE_MEMORY"',g
+ s,@FORCE_ASYNC@,'"$FORCE_ASYNC"',g
s!@RPATH@!'"$opt_rpath"'!g
s,@MARK64@,'"$opt_mark64"',g
s,@ARCH@,'"$opt_arch"',g
diff --git a/guts/COPYRIGHT b/guts/COPYRIGHT
index c96e1b1..85c47c7 100644
--- a/guts/COPYRIGHT
+++ b/guts/COPYRIGHT
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2008-2010 Wind River Systems, Inc.
+ * Copyright (c) 2008-2013 Wind River Systems, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the Lesser GNU General Public License version 2.1 as
diff --git a/makewrappers b/makewrappers
index 168fd81..fb919b5 100755
--- a/makewrappers
+++ b/makewrappers
@@ -1,6 +1,6 @@
#!/usr/bin/env python
#
-# Copyright (c) 2008-2011 Wind River Systems, Inc.
+# Copyright (c) 2008-2011,2013 Wind River Systems, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the Lesser GNU General Public License version 2.1 as
@@ -219,6 +219,7 @@ class Function:
self.paths_to_munge = []
self.specific_dirfds = {}
self.hand_wrapped = None
+ self.async_skip = None
# used for the copyright date when creating stub functions
self.date = datetime.date.today().year
@@ -300,6 +301,16 @@ class Function:
else:
return ""
+ def maybe_async_skip(self):
+ if self.async_skip:
+ return """/* This function is not called if pseudo is configured --enable-force-async */
+#ifdef PSEUDO_FORCE_ASYNC
+ return %s;
+#endif
+""" % self.async_skip
+ else:
+ return ""
+
def comment(self):
"""declare self (in a comment)"""
return self.decl(comment = True)
diff --git a/ports/darwin/guts/open.c b/ports/darwin/guts/open.c
index 1ed5882..c66cc15 100644
--- a/ports/darwin/guts/open.c
+++ b/ports/darwin/guts/open.c
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2011, 2012 Wind River Systems; see
+ * Copyright (c) 2011-2013 Wind River Systems; see
* guts/COPYRIGHT for information.
*
* int open(const char *path, int flags, ... { int mode })
@@ -9,6 +9,9 @@
struct stat buf = { };
int existed = 1;
int save_errno;
+#ifdef PSEUDO_FORCE_ASYNCH
+ flags &= ~O_SYNC;
+#endif
/* if a creation has been requested, check whether file exists */
if (flags & O_CREAT) {
diff --git a/ports/linux/guts/openat.c b/ports/linux/guts/openat.c
index 14d6c2a..8460073 100644
--- a/ports/linux/guts/openat.c
+++ b/ports/linux/guts/openat.c
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2008-2010 Wind River Systems; see
+ * Copyright (c) 2008-2010, 2013 Wind River Systems; see
* guts/COPYRIGHT for information.
*
* static int
@@ -16,6 +16,25 @@
return -1;
}
#endif
+
+#ifdef PSEUDO_FORCE_ASYNCH
+ /* Yes, I'm aware that every Linux system I've seen has
+ * DSYNC and RSYNC being the same value as SYNC.
+ */
+
+ flags &= ~(O_SYNC
+#ifdef O_DIRECT
+ | O_DIRECT
+#endif
+#ifdef O_DSYNC
+ | O_DSYNC
+#endif
+#ifdef O_RSYNC
+ | O_RSYNC
+#endif
+ );
+#endif
+
/* if a creation has been requested, check whether file exists */
if (flags & O_CREAT) {
save_errno = errno;
diff --git a/ports/unix/guts/fdatasync.c b/ports/unix/guts/fdatasync.c
new file mode 100644
index 0000000..4aa77a8
--- /dev/null
+++ b/ports/unix/guts/fdatasync.c
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) 2013 Wind River Systems; see
+ * guts/COPYRIGHT for information.
+ *
+ * int fdatasync(int fd)
+ * int rc = -1;
+ */
+
+ /* note: wrapper will never call this if PSEUDO_FORCE_ASYNC
+ * is defined.
+ */
+ rc = real_fdatasync(fd);
+
+/* return rc;
+ * }
+ */
diff --git a/ports/unix/guts/fsync.c b/ports/unix/guts/fsync.c
new file mode 100644
index 0000000..6c87a56
--- /dev/null
+++ b/ports/unix/guts/fsync.c
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) 2013 Wind River Systems; see
+ * guts/COPYRIGHT for information.
+ *
+ * int fsync(int fd)
+ * int rc = -1;
+ */
+
+ /* note: wrapper will never call this if PSEUDO_FORCE_ASYNC
+ * is defined.
+ */
+ rc = real_fsync(fd);
+
+/* return rc;
+ * }
+ */
diff --git a/ports/unix/guts/msync.c b/ports/unix/guts/msync.c
new file mode 100644
index 0000000..fbc5e26
--- /dev/null
+++ b/ports/unix/guts/msync.c
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) 2013 Wind River Systems; see
+ * guts/COPYRIGHT for information.
+ *
+ * int msync(void *addr, size_t length, int flags)
+ * int rc = -1;
+ */
+
+ /* note: wrapper will never call this if PSEUDO_FORCE_ASYNC
+ * is defined.
+ */
+ rc = real_msync(addr, length, flags);
+
+/* return rc;
+ * }
+ */
diff --git a/ports/unix/guts/sync.c b/ports/unix/guts/sync.c
new file mode 100644
index 0000000..c5d9554
--- /dev/null
+++ b/ports/unix/guts/sync.c
@@ -0,0 +1,16 @@
+/*
+ * Copyright (c) 2013 Wind River Systems; see
+ * guts/COPYRIGHT for information.
+ *
+ * void sync(void)
+ *
+ */
+
+ /* note: wrapper will never call this if PSEUDO_FORCE_ASYNC
+ * is defined.
+ */
+ (void) real_sync();
+
+/* return;
+ * }
+ */
diff --git a/ports/unix/guts/sync_file_range.c b/ports/unix/guts/sync_file_range.c
new file mode 100644
index 0000000..03cfc6c
--- /dev/null
+++ b/ports/unix/guts/sync_file_range.c
@@ -0,0 +1,13 @@
+/*
+ * Copyright (c) 2013 Wind River Systems; see
+ * guts/COPYRIGHT for information.
+ *
+ * int sync_file_range(int fd, off64_t offset, off64_t nbytes, unsigned int flags)
+ * int rc = -1;
+ */
+
+ rc = real_sync_file_range(fd, offset, nbytes, flags);
+
+/* return rc;
+ * }
+ */
diff --git a/ports/unix/guts/syncfs.c b/ports/unix/guts/syncfs.c
new file mode 100644
index 0000000..2c9a685
--- /dev/null
+++ b/ports/unix/guts/syncfs.c
@@ -0,0 +1,13 @@
+/*
+ * Copyright (c) 2013 Wind River Systems; see
+ * guts/COPYRIGHT for information.
+ *
+ * int syncfs(int fd)
+ * int rc = -1;
+ */
+
+ rc = real_syncfs(fd);
+
+/* return rc;
+ * }
+ */
diff --git a/ports/unix/wrapfuncs.in b/ports/unix/wrapfuncs.in
index 7e205f0..a0f191c 100644
--- a/ports/unix/wrapfuncs.in
+++ b/ports/unix/wrapfuncs.in
@@ -58,3 +58,12 @@ ssize_t readlink(const char *path, char *buf, size_t bufsiz); /* flags=AT_SYMLIN
ssize_t readlinkat(int dirfd, const char *path, char *buf, size_t bufsiz); /* flags=AT_SYMLINK_NOFOLLOW */
int system(const char *command);
FILE *popen(const char *command, const char *mode); /* hand_wrapped=1 */
+# Based on experiments by Richard Purdie: Allow pseudo to eliminate
+# sync-type operations globally, mostly relevant for performance reasons
+# during filesystem assembly.
+int fsync(int fd); /* async_skip=0 */
+int fdatasync(int fd); /* async_skip=0 */
+void sync(void); /* async_skip=0 */
+int syncfs(int fd); /* async_skip=0 */
+int sync_file_range(int fd, off64_t offset, off64_t nbytes, unsigned int flags); /* async_skip=0 */
+int msync(void *addr, size_t length, int flags); /* async_skip=0 */
diff --git a/templates/wrapfuncs.c b/templates/wrapfuncs.c
index 0d0fb52..86fd557 100644
--- a/templates/wrapfuncs.c
+++ b/templates/wrapfuncs.c
@@ -23,6 +23,8 @@ ${name}(${decl_args}) {
${variadic_decl}
${rc_decl}
+${maybe_async_skip}
+
if (!pseudo_check_wrappers() || !real_$name) {
/* rc was initialized to the "failure" value */
pseudo_enosys("${name}");
--
1.7.9.5
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 3/3] If you don't want the answer, don't ask the question.
2013-02-17 2:23 [PATCH 0/3] Pseudo performance changes Peter Seebach
2013-02-17 2:23 ` [PATCH 1/3] Use in-memory database for files Peter Seebach
2013-02-17 2:23 ` [PATCH 2/3] allow pseudo to force asynchronous behavior Peter Seebach
@ 2013-02-17 2:23 ` Peter Seebach
2013-02-17 10:27 ` [PATCH 0/3] Pseudo performance changes Richard Purdie
3 siblings, 0 replies; 5+ messages in thread
From: Peter Seebach @ 2013-02-17 2:23 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org
Most pseudo operations don't actually USE the server's response. So
why wait for a response?
This patch introduces a new message type, PSEUDO_MSG_FASTOP. It
also tags pseudo operation types with whether or not they need to
give a response. This requires updates to maketables to allow non-string
types for additional columns, and the addition of some quotes to the
SQL query enums/query_type.in table.
A few routines are altered to change their behavior and whether or not
they perform a stat operation. The only operations that do wait are
OP_FSTAT and OP_STAT, OP_MKNOD, and OP_MAY_UNLINK. Rationale:
You can't query the server for replacement information and not wait for
it. Makes no sense.
There's extra checking in mknod, because we really do want to fail out
if we couldn't do that -- that implies that we haven't created a thing
that will look like a node.
The result from OP_MAY_UNLINK is checked because it's used to determine
whether we need to send a DID_UNLINK or CANCEL_UNLINK. It might be cheaper
to send two messages without waiting than to send one, wait, and maybe
send another, but I don't want to send invalid messages.
This is highly experimental.
---
enums/msg_type.in | 1 +
enums/op.in | 46 ++++++++++++++++++++++----------------------
enums/query_type.in | 18 ++++++++---------
maketables | 41 +++++++++++++++++++++++----------------
ports/unix/guts/fchmod.c | 16 +++++----------
ports/unix/guts/fchmodat.c | 31 ++++++++++++++++-------------
ports/unix/guts/fchown.c | 16 ++++++---------
ports/unix/guts/fchownat.c | 16 ++++++---------
ports/unix/guts/mknodat.c | 4 +++-
ports/unix/wrapfuncs.in | 2 +-
pseudo.c | 3 ++-
pseudo_client.c | 22 +++++++++++++--------
pseudo_server.c | 30 +++++++++++++++++++----------
13 files changed, 132 insertions(+), 114 deletions(-)
diff --git a/enums/msg_type.in b/enums/msg_type.in
index 0313073..578d571 100644
--- a/enums/msg_type.in
+++ b/enums/msg_type.in
@@ -4,3 +4,4 @@ shutdown
op
ack
nak
+fastop
diff --git a/enums/op.in b/enums/op.in
index 65eb73c..3b8e23e 100644
--- a/enums/op.in
+++ b/enums/op.in
@@ -1,23 +1,23 @@
-op: OP
-chdir
-chmod
-chown
-chroot
-close
-creat
-dup
-fchmod
-fchown
-fstat
-link
-mkdir
-mknod
-open
-rename
-stat
-unlink
-symlink
-exec
-may-unlink
-did-unlink
-cancel-unlink
+op: OP; int wait = 0
+chdir, 0
+chmod, 0
+chown, 0
+chroot, 0
+close, 0
+creat, 0
+dup, 0
+fchmod, 0
+fchown, 0
+fstat, 1
+link, 0
+mkdir, 0
+mknod, 1
+open, 0
+rename, 0
+stat, 1
+unlink, 0
+symlink, 0
+exec, 0
+may-unlink, 1
+did-unlink, 0
+cancel-unlink, 0
diff --git a/enums/query_type.in b/enums/query_type.in
index 5bfc741..974e24f 100644
--- a/enums/query_type.in
+++ b/enums/query_type.in
@@ -1,9 +1,9 @@
-query_type: PSQT; sql = LITTLE BOBBY TABLES
-exact, =
-less, <
-greater, >
-bitand, &
-notequal, !=
-like, LIKE
-notlike, NOT LIKE
-sqlpat, LIKE
+query_type: PSQT; const char * sql = "LITTLE BOBBY TABLES"
+exact, "="
+less, "<"
+greater, ">"
+bitand, "&"
+notequal, "!="
+like, "LIKE"
+notlike, "NOT LIKE"
+sqlpat, "LIKE"
diff --git a/maketables b/maketables
index fe4e62c..7c54c6a 100755
--- a/maketables
+++ b/maketables
@@ -1,6 +1,6 @@
#!/usr/bin/env python
#
-# Copyright (c) 2008-2010 Wind River Systems, Inc.
+# Copyright (c) 2008-2010, 2013 Wind River Systems, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the Lesser GNU General Public License version 2.1 as
@@ -23,9 +23,9 @@ The names are used to create enums and a table of strings, as well as
to/from lookups between the ids and names. If additional columns are
defined, each column (separated by ", ") is used to create an additional
table of the given name, and a lookup function from ids. Example:
- foo: FFF; bar = GOZINTA
- hello, yah
- world, nope
+ foo: FFF; const char *bar = "GOZINTA"
+ hello, "yah"
+ world, "nope"
produces:
typedef enum {
FFF_UNKNOWN = -1,
@@ -74,7 +74,14 @@ class DataType:
name, default = string.split(col, ' = ')
else:
name, default = col, ""
- self.columns.append({"name":name, "value":default})
+ if " " in name:
+ words = string.split(name, ' ')
+ name = words[-1]
+ del words[-1]
+ type = ' '.join(words)
+ else:
+ type = "char *"
+ self.columns.append({"type":type, "name":name, "value":default})
else:
self.columns = []
self.data = []
@@ -94,7 +101,7 @@ class DataType:
if len(cols) > 0:
column_list.append({"name":col["name"], "value":cols.pop(0)})
else:
- column_list.append({"name":col["name"], "value":col["default"]})
+ column_list.append({"name":col["name"], "value":col["value"]})
item["cols"] = column_list
self.data.append(item)
@@ -113,15 +120,15 @@ class DataType:
out += "type: %s_t" % self.name
out += " (prefix '%s_ENUM')\n" % self.prefix
for col in self.columns:
- out += " extra column: %s (default '%s')\n" % (col["name"], col["value"])
+ out += " extra column: %s %s (default %s)\n" % (col["type"], col["name"], col["value"])
out += " "
for item in self.data:
column = column + 1
if column > 4 and column % 4 == 1:
out += "\n "
out += "%-19s" % item["name"]
-# for col in item["cols"]:
-# out += "\t%s(%s)\n" % (col["name"], col["value"])
+# for col in item["cols"]:
+# out += "\t%s(%s)\n" % (col["name"], col["value"])
return out
def comment(self):
@@ -140,11 +147,11 @@ class DataType:
decl_lines = []
column = 0
for col in self.columns:
- decl_lines.append("static const char *%s_id_to_%s[] = {" % (self.name, col["name"]))
- decl_lines.append('\t"%s",' % col["value"])
+ decl_lines.append("static %s %s_id_to_%s[] = {" % (col["type"], self.name, col["name"]))
+ decl_lines.append('\t%s,' % col["value"])
for item in self.data:
- decl_lines.append('\t"%s",' % item["cols"][column]["value"])
- decl_lines.append('\tNULL')
+ decl_lines.append('\t%s,' % item["cols"][column]["value"])
+ decl_lines.append('\t0')
decl_lines.append("};")
column = column + 1
return '\n'.join(decl_lines)
@@ -152,11 +159,11 @@ class DataType:
def column_funcs(self):
decl_lines = []
for col in self.columns:
- decl_lines.append('extern const char *')
+ decl_lines.append('extern %s' % col["type"])
decl_lines.append('pseudo_%s_%s(pseudo_%s_t id) {' %
(self.name, col["name"], self.name))
decl_lines.append('\tif (id < 0 || id >= %s_MAX)' % (self.prefix))
- decl_lines.append('\t\treturn "%s";' % col["value"])
+ decl_lines.append('\t\treturn %s;' % col["value"])
decl_lines.append('\treturn %s_id_to_%s[id];' %
(self.name, col["name"]))
decl_lines.append('}')
@@ -165,8 +172,8 @@ class DataType:
def column_protos(self):
decl_lines = []
for col in self.columns:
- decl_lines.append('extern const char *pseudo_%s_%s(pseudo_%s_t id);' %
- (self.name, col["name"], self.name))
+ decl_lines.append('extern %s pseudo_%s_%s(pseudo_%s_t id);' %
+ (col["type"], self.name, col["name"], self.name))
return '\n'.join(decl_lines)
def main():
diff --git a/ports/unix/guts/fchmod.c b/ports/unix/guts/fchmod.c
index 1d4fae8..e2301e3 100644
--- a/ports/unix/guts/fchmod.c
+++ b/ports/unix/guts/fchmod.c
@@ -1,12 +1,11 @@
/*
- * Copyright (c) 2008-2010, 2012 Wind River Systems; see
+ * Copyright (c) 2008-2010, 2012, 2013 Wind River Systems; see
* guts/COPYRIGHT for information.
*
* static int
* wrap_fchmod(int fd, mode_t mode) {
* int rc = -1;
*/
- pseudo_msg_t *msg;
PSEUDO_STATBUF buf;
int save_errno = errno;
@@ -15,16 +14,11 @@
return -1;
}
buf.st_mode = (buf.st_mode & ~07777) | (mode & 07777);
- msg = pseudo_client_op(OP_FCHMOD, 0, fd, -1, 0, &buf);
+ pseudo_client_op(OP_FCHMOD, 0, fd, -1, 0, &buf);
real_fchmod(fd, PSEUDO_FS_MODE(mode, S_ISDIR(buf.st_mode)));
- if (msg && msg->result != RESULT_SUCCEED) {
- errno = EPERM;
- rc = -1;
- } else {
- /* just pretend we worked */
- errno = save_errno;
- rc = 0;
- }
+ /* just pretend we worked */
+ errno = save_errno;
+ rc = 0;
/* return rc;
* }
diff --git a/ports/unix/guts/fchmodat.c b/ports/unix/guts/fchmodat.c
index a14d1b1..59a92ce 100644
--- a/ports/unix/guts/fchmodat.c
+++ b/ports/unix/guts/fchmodat.c
@@ -1,14 +1,13 @@
/*
- * Copyright (c) 2008-2010, 2012 Wind River Systems; see
+ * Copyright (c) 2008-2010, 2012, 2013 Wind River Systems; see
* guts/COPYRIGHT for information.
*
* static int
* wrap_fchmodat(int dirfd, const char *path, mode_t mode, int flags) {
* int rc = -1;
*/
- pseudo_msg_t *msg;
PSEUDO_STATBUF buf;
- int save_errno;
+ int save_errno = errno;
#ifdef PSEUDO_NO_REAL_AT_FUNCTIONS
if (dirfd != AT_FDCWD) {
@@ -33,8 +32,13 @@
}
save_errno = errno;
+#if 0
+ pseudo_msg_t *msg;
/* purely for debugging purposes: check whether file
- * is already in database.
+ * is already in database. We don't need the resulting
+ * information for anything. This is currently ifdefed
+ * out because it's only useful when trying to track where
+ * files are coming from.
*/
msg = pseudo_client_op(OP_STAT, 0, -1, -1, path, &buf);
if (!msg || msg->result != RESULT_SUCCEED) {
@@ -42,6 +46,7 @@
mode, dirfd, path, (unsigned long long) buf.st_ino);
}
+#endif
/* user bits added so "root" can always access files. */
#ifdef PSEUDO_NO_REAL_AT_FUNCTIONS
@@ -53,18 +58,18 @@
#endif
/* we ignore a failure from underlying fchmod, because pseudo
* may believe you are permitted to change modes that the filesystem
- * doesn't.
+ * doesn't. Note that we also don't need to know whether the
+ * file might be a (pseudo) block device or some such; pseudo
+ * will only modify permission bits based on an OP_CHMOD, and does
+ * not care about device/file type mismatches, only directory/file
+ * or symlink/file.
*/
buf.st_mode = (buf.st_mode & ~07777) | (mode & 07777);
- msg = pseudo_client_op(OP_CHMOD, 0, -1, dirfd, path, &buf);
- if (msg && msg->result != RESULT_SUCCEED) {
- errno = EPERM;
- rc = -1;
- } else {
- /* if server is down, just pretend we worked */
- rc = 0;
- }
+ pseudo_client_op(OP_CHMOD, 0, -1, dirfd, path, &buf);
+ /* don't change errno from what it was originally */
+ errno = save_errno;
+ rc = 0;
/* return rc;
* }
diff --git a/ports/unix/guts/fchown.c b/ports/unix/guts/fchown.c
index 72a1e46..1514e35 100644
--- a/ports/unix/guts/fchown.c
+++ b/ports/unix/guts/fchown.c
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2008-2010, 2012 Wind River Systems; see
+ * Copyright (c) 2008-2010, 2012, 2013 Wind River Systems; see
* guts/COPYRIGHT for information.
*
* static int
@@ -8,7 +8,7 @@
*/
pseudo_msg_t *msg;
PSEUDO_STATBUF buf;
- int save_errno;
+ int save_errno = errno;
if (base_fstat(fd, &buf) == -1) {
save_errno = errno;
@@ -41,14 +41,10 @@
}
pseudo_debug(2, "fchown, fd %d: %d:%d -> %d:%d\n",
fd, owner, group, buf.st_uid, buf.st_gid);
- msg = pseudo_client_op(OP_FCHOWN, 0, fd, -1, 0, &buf);
- if (msg && msg->result != RESULT_SUCCEED) {
- errno = EPERM;
- rc = -1;
- } else {
- /* just pretend we worked */
- rc = 0;
- }
+ pseudo_client_op(OP_FCHOWN, 0, fd, -1, 0, &buf);
+ /* pretend we worked, errno should be unchanged */
+ errno = save_errno;
+ rc = 0;
/* return rc;
* }
diff --git a/ports/unix/guts/fchownat.c b/ports/unix/guts/fchownat.c
index 8d3ee05..60ccfa5 100644
--- a/ports/unix/guts/fchownat.c
+++ b/ports/unix/guts/fchownat.c
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2008-2010, 2012 Wind River Systems; see
+ * Copyright (c) 2008-2010, 2012, 2013 Wind River Systems; see
* guts/COPYRIGHT for information.
*
* static int
@@ -8,7 +8,7 @@
*/
pseudo_msg_t *msg;
PSEUDO_STATBUF buf;
- int save_errno;
+ int save_errno = errno;
int doing_link = 0;
#ifdef PSEUDO_NO_REAL_AT_FUNCTIONS
@@ -53,14 +53,10 @@
if (group != (gid_t) -1) {
buf.st_gid = group;
}
- msg = pseudo_client_op(OP_CHOWN, 0, -1, dirfd, path, &buf);
- if (msg && msg->result != RESULT_SUCCEED) {
- errno = EPERM;
- rc = -1;
- } else {
- /* just pretend we worked */
- rc = 0;
- }
+ pseudo_client_op(OP_CHOWN, 0, -1, dirfd, path, &buf);
+ /* just pretend we worked */
+ errno = save_errno;
+ rc = 0;
/* return rc;
* }
diff --git a/ports/unix/guts/mknodat.c b/ports/unix/guts/mknodat.c
index 32a4d5b..6fd5b42 100644
--- a/ports/unix/guts/mknodat.c
+++ b/ports/unix/guts/mknodat.c
@@ -8,6 +8,7 @@
pseudo_msg_t *msg;
PSEUDO_STATBUF buf;
+ int save_errno = errno;
#ifdef PSEUDO_NO_REAL_AT_FUNCTIONS
if (dirfd != AT_FDCWD) {
@@ -50,10 +51,11 @@
rc = -1;
} else {
/* just pretend we worked */
+ errno = save_errno;
rc = 0;
}
if (rc == -1) {
- int save_errno = errno;
+ save_errno = errno;
#ifdef PSEUDO_NO_REAL_AT_FUNCTIONS
real_unlink(path);
#else
diff --git a/ports/unix/wrapfuncs.in b/ports/unix/wrapfuncs.in
index a0f191c..8460a65 100644
--- a/ports/unix/wrapfuncs.in
+++ b/ports/unix/wrapfuncs.in
@@ -63,7 +63,7 @@ FILE *popen(const char *command, const char *mode); /* hand_wrapped=1 */
# during filesystem assembly.
int fsync(int fd); /* async_skip=0 */
int fdatasync(int fd); /* async_skip=0 */
-void sync(void); /* async_skip=0 */
+void sync(void); /* async_skip= */
int syncfs(int fd); /* async_skip=0 */
int sync_file_range(int fd, off64_t offset, off64_t nbytes, unsigned int flags); /* async_skip=0 */
int msync(void *addr, size_t length, int flags); /* async_skip=0 */
diff --git a/pseudo.c b/pseudo.c
index 4ea79c5..ad71062 100644
--- a/pseudo.c
+++ b/pseudo.c
@@ -1,7 +1,7 @@
/*
* pseudo.c, main pseudo utility program
*
- * Copyright (c) 2008-2010 Wind River Systems, Inc.
+ * Copyright (c) 2008-2013 Wind River Systems, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the Lesser GNU General Public License version 2.1 as
@@ -933,6 +933,7 @@ pseudo_server_response(pseudo_msg_t *msg, const char *program, const char *tag)
return 0;
break;
case PSEUDO_MSG_OP:
+ case PSEUDO_MSG_FASTOP:
return pseudo_op(msg, program, tag);
break;
case PSEUDO_MSG_ACK: /* FALLTHROUGH */
diff --git a/pseudo_client.c b/pseudo_client.c
index a7ba100..f58ce4c 100644
--- a/pseudo_client.c
+++ b/pseudo_client.c
@@ -876,14 +876,18 @@ pseudo_client_request(pseudo_msg_t *msg, size_t len, const char *path) {
}
} while (rc != 0);
pseudo_debug(5, "sent!\n");
- response = pseudo_msg_receive(connect_fd);
- if (!response) {
- ++tries;
- if (tries > 3) {
- pseudo_debug(1, "can't get responses.\n");
- return 0;
- }
- }
+ if (msg->type != PSEUDO_MSG_FASTOP) {
+ response = pseudo_msg_receive(connect_fd);
+ if (!response) {
+ ++tries;
+ if (tries > 3) {
+ pseudo_debug(1, "can't get responses.\n");
+ return 0;
+ }
+ }
+ } else {
+ return 0;
+ }
} while (response == 0);
if (response->type != PSEUDO_MSG_ACK) {
pseudo_debug(2, "got non-ack response %d\n", response->type);
@@ -1225,6 +1229,8 @@ pseudo_client_op(pseudo_op_t op, int access, int fd, int dirfd, const char *path
}
if (do_request) {
struct timeval tv1, tv2;
+ if (!pseudo_op_wait(msg.op))
+ msg.type = PSEUDO_MSG_FASTOP;
pseudo_debug(4, "sending request [ino %llu]\n", (unsigned long long) msg.ino);
gettimeofday(&tv1, NULL);
if (pseudo_local_only) {
diff --git a/pseudo_server.c b/pseudo_server.c
index 4af5265..2597def 100644
--- a/pseudo_server.c
+++ b/pseudo_server.c
@@ -1,7 +1,7 @@
/*
* pseudo_server.c, pseudo's server-side logic and message handling
*
- * Copyright (c) 2008-2010 Wind River Systems, Inc.
+ * Copyright (c) 2008-2010, 2013 Wind River Systems, Inc.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the Lesser GNU General Public License version 2.1 as
@@ -76,7 +76,7 @@ quit_now(int signal) {
die_forcefully = 1;
}
-static int messages = 0;
+static int messages = 0, responses = 0;
static struct timeval message_time = { .tv_sec = 0 };
static void pseudo_server_loop(void);
@@ -268,6 +268,7 @@ serve_client(int i) {
in = pseudo_msg_receive(clients[i].fd);
if (in) {
char *response_path = 0;
+ int send_response = 1;
pseudo_debug(4, "got a message (%d): %s\n", in->type, (in->pathlen ? in->path : "<no path>"));
/* handle incoming ping */
if (in->type == PSEUDO_MSG_PING && !clients[i].pid) {
@@ -303,6 +304,8 @@ serve_client(int i) {
* pseudo_server_response.
*/
if (in->type != PSEUDO_MSG_SHUTDOWN) {
+ if (in->type == PSEUDO_MSG_FASTOP)
+ send_response = 0;
if (pseudo_server_response(in, clients[i].program, clients[i].tag)) {
in->type = PSEUDO_MSG_NAK;
} else {
@@ -343,10 +346,14 @@ serve_client(int i) {
die_peacefully = 1;
}
}
- if ((rc = pseudo_msg_send(clients[i].fd, in, -1, response_path)) != 0)
- pseudo_debug(1, "failed to send response to client %d [%d]: %d (%s)\n",
- i, (int) clients[i].pid, rc, strerror(errno));
- rc = in->op;
+ if (send_response) {
+ if ((rc = pseudo_msg_send(clients[i].fd, in, -1, response_path)) != 0) {
+ pseudo_debug(1, "failed to send response to client %d [%d]: %d (%s)\n",
+ i, (int) clients[i].pid, rc, strerror(errno));
+ }
+ } else {
+ rc = 1;
+ }
free(response_path);
return rc;
} else {
@@ -423,10 +430,11 @@ pseudo_server_loop(void) {
die_peacefully = 1;
} else {
/* display this if not exiting */
- pseudo_debug(1, "%d messages handled in %.4f seconds\n",
+ pseudo_debug(1, "%d messages handled in %.4f seconds, %d responses\n",
messages,
(double) message_time.tv_sec +
- (double) message_time.tv_usec / 1000000.0);
+ (double) message_time.tv_usec / 1000000.0,
+ responses);
}
}
} else if (rc > 0) {
@@ -439,11 +447,13 @@ pseudo_server_loop(void) {
close_client(i);
} else if (FD_ISSET(clients[i].fd, &reads)) {
struct timeval tv1, tv2;
- int op;
+ int rc;
gettimeofday(&tv1, NULL);
- op = serve_client(i);
+ rc = serve_client(i);
gettimeofday(&tv2, NULL);
++messages;
+ if (rc == 0)
+ ++responses;
message_time.tv_sec += (tv2.tv_sec - tv1.tv_sec);
message_time.tv_usec += (tv2.tv_usec - tv1.tv_usec);
if (message_time.tv_usec < 0) {
--
1.7.9.5
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3] Pseudo performance changes...
2013-02-17 2:23 [PATCH 0/3] Pseudo performance changes Peter Seebach
` (2 preceding siblings ...)
2013-02-17 2:23 ` [PATCH 3/3] If you don't want the answer, don't ask the question Peter Seebach
@ 2013-02-17 10:27 ` Richard Purdie
3 siblings, 0 replies; 5+ messages in thread
From: Richard Purdie @ 2013-02-17 10:27 UTC (permalink / raw)
To: Peter Seebach; +Cc: openembedded-core@lists.openembedded.org
On Sat, 2013-02-16 at 20:23 -0600, Peter Seebach wrote:
> Unlike most of my submissions, this isn't patches against oe-core; rather,
> it's patches against pseudo, and if I can get some confirmation that they do
> what I think they do, and some review, I'm planning to make this into
> pseudo 1.5, and send a patch "soonish" to merge that into oe-core.
>
> What this does: Fix a number of build performance issues. By far the
> largest change is actually not so much a problem with pseudo as a problem
> that pseudo can solve by brute force. Packaging systems (at least RPM and
> SMART) do a lot of fsync() and fdatasync() calls. That usually implies
> flushing EVERYTHING that's been written, not just one specific file. And
> that, in turn, results in a severe performance hit.
>
> So, for instance, on one of my test workstations, this moves a do_rootfs
> with about 1200 RPMs from about 22 minutes to about 4.5. Yeah.
>
> The other changes aren't as dramatic for that case, but have very significant
> performance impact for at least some workloads. The first is switching to
> using an in-memory database for the files database, dumping it to disk only
> when the pseudo daemon is idle or shutting down. This doesn't produce huge
> benefits in all cases, but for workloads with a lot of parallelism, it can
> produce a very noticeable reduction in how much pseudo slows things down.
>
> The second is a fairly major protocol change. In short, with this patch,
> pseudo clients only wait for a server response when they need information
> from the server in order to continue. That's OP_FSTAT, OP_STAT,
> OP_MAY_UNLINK, and OP_MKNOD. Everything else just silently assumes that
> it probably succeeded.
>
> How much does this matter? Between the protocol change and the memory
> DB, a trivial unpack of a tarball (lots of writes to the database, very
> few reads) can be about 4x faster. Removing stuff isn't much faster, but
> it might be a bit faster.
>
> This is most noticeable, by far, when running more than one build, or
> when running builds while doing other things. It has a much smaller effect
> on builds with no shared state (compile time still dominates that), but
> even there I'm seeing decreases from ~83 minutes to ~64 from just the
> fsync and memory changes. Still waiting on my real test case (multiple
> simultaneous builds which need compiles) completing.
All sounds good to me. I've set the autobuilder away to test this set of
changes.
Cheers,
Richard
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2013-02-17 10:43 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-02-17 2:23 [PATCH 0/3] Pseudo performance changes Peter Seebach
2013-02-17 2:23 ` [PATCH 1/3] Use in-memory database for files Peter Seebach
2013-02-17 2:23 ` [PATCH 2/3] allow pseudo to force asynchronous behavior Peter Seebach
2013-02-17 2:23 ` [PATCH 3/3] If you don't want the answer, don't ask the question Peter Seebach
2013-02-17 10:27 ` [PATCH 0/3] Pseudo performance changes Richard Purdie
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox