Re: RT File Logging

[Sabar Siddhartha Dasgupta <sabard@stanford.edu>]

Thank you for the help!

It ended up being a problem with sqlite3's journaling system waiting
for the writer to unlock the OS buffers before creating a rollback
file. Changing the journal configuration to stay in memory rather than
disk made the journaling way faster and fixed the problem.

I will keep the potential priority problems in mind for the future!

Thanks again,
Sabar

On Mon, Oct 3, 2016 at 11:44 AM, Brian Silverman <brian@peloton-tech.com> wrote:
> Have you looked for any internal locks sqlite3 is using? Keeping in
> mind I have no experience with that library whatsoever, it sounds like
> you may be running into sqlite3async_run holding a lock while it does
> IO, which causes sqlite3_step to block waiting for it to finish. Even
> if the disk IO thread doesn't do much with the lock held, you could be
> running into priority inversion
> (https://en.wikipedia.org/wiki/Priority_inversion has a description
> and some solutions).
>
> You might have less problems if you split anything realtime (like
> reading your data from shared memory) and all the sqlite3 things out
> into separate processes. Pipes and POSIX shared memory queues should
> work.
>
> If you're still having problems, it might be worth asking sqlite3
> people. Specifically, how they do locking etc sounds relevant.
>
> Best of luck,
> Brian