Re: Storing/extracting embedded db within .odb

On Wed, 2013-07-10 at 10:25 +0200, Lionel Elie Mamane wrote:
> On Wed, Jul 10, 2013 at 08:47:49AM +0100, Andrzej J. R. Hunt wrote:
> > On Tue, 2013-07-09 at 15:59 +0200, Lionel Elie Mamane wrote:
> > > On Mon, Jul 08, 2013 at 03:19:49PM +0100, Andrzej J. R. Hunt wrote:
>
> > > > W.r.t. to the location of the extracted firebird db: would an
> > > > appropriate naming scheme be to have e.g. ~foo.odb.fdb# for a
> > > > foo.odb (in the same directory) similarly to the lock files that
> > > > are created? (This could also allow for better recovery should we
> > > > end up with similar corruption as is apparently happening with
> > > > HSQLDB if there is a complete separate db.)
>
> > > It is a temporary file. I'd stick it in $TMPDIR (or the platform
> > > equivalent). Surely LibreOffice already has a platform abstraction
> > > utility function to find where to put temporary files.
>
> > That was my original plan, however that would require writing the
> > whole db back to the .odb every time there is a change.
>
> Or every time the user presses the "save" button for the odb... OO.org
> moved away from requiring "save" for embedded database in the past,
> but frankly to me it seems to bring more problems than it is worth.
Makes sense. I still need to figure out how to do this given I'm dealing
with the storage within the driver, and the driver currently doesn't get
told when the .odb is being saved.

(Currently the driver can create new dbs in $TMPDIR, or extract them
from the .odb as appropriate -- there's no method of saving them
yet though.)

> > If there were a file in the same directory instead we could
> > potentially only have to extract the db at startup and write it back
> > into the .odb at shutdown (which means we can ensure that the .fdb
> > is in a good state by shutting down FB first) -- if we crash (lose
> > power/etc.) in the meantime the local file could then be used to
> > restore the database (alternatively the location of the temp file
> > could be stored, but that would be more complicated to do I think).
>
> You make a good point. I wouldn't go into the "alternatively", because
> temp files are typically deleted on computer startup / shutdown and /
> or on user logout / user login, so thinking of crash / power loss
> recovery, temp dir is not such a good idea.
>
> OTOH, I'd like to see an analysis (and plan of implementation) of how
> this way of crash recovery would interact with the recovery of the
> autosave copy of .odb; which one will override the other? Will we
> "intelligently" take the newest data, or if one of them is newer but
> corrupted, then take the older clean copy? The plan of implementation
> can be done at a later stage, not necessarily this week, obviously :)
>
> I'm a bit uncomfortable that the database file is committed to the odb
> only when closing the file. I'd definitely do it at least at each odb
> save operation (including the background auto-save, but then commit in
> the auto-save copy, *not* in the main odb file). With some luck,
> Firebird has a way of making that safe without closing the connection
> (which would not be acceptable); for some databases all one has to do
> is take a write lock on the whole database / all tables, flush
> buffers, copy the file(s) and release the write lock. That's how
> mysqlhotcopy does database backups:
I haven't looked at any of this in detail yet, but I do agree that the db
should be saved more often than shutdown. On the other hand I assume that the odb
recovery deals with the recovery of non db parts (since LO is probably completely
unaware of what the db is doing) -- the db driver doesn't get loaded until the
odb is opened and the db is actually needed -- and it would then be able to
decide between the internal db and the external ("recovery") db if it finds both,
i.e. that recovery could be completely independent of the other recovery -- in
any case this is more complex so probably not a good solution.


> Whatever is most convenient / safe / clean / ... to implement. The
> firebird driver could also directly handle the different sdbc URLs:
>
>  sdbc:embedded:firebird
>  sdbc:firebird:XXXXXXXXXXXXXXXXX
>
> and set an internal flag "m_bIsEmbedded" based on that, and then do
> different stuff in the few places where different stuff is needed in
> the two cases. My guess would be that this would be best, but since
> you are writing the code, you have much better information than me on
> that :-) As far as I am concerned, you can make that decision.
This is what I've been doing so far (although I've not actually implemented
the external file stuff yet -- the only extra code there will be to check
the path, and use the correct call to create the db if the file doesn't
exist yet I think).


Cheers,

        Andrzej