Fwd: The Future of the Simple API (and ODF Toolkit)

Dear LibreOffice developers,

allow me to cross-post you an update from our TDF ODF Toolkit project
<https://tdf.github.io/odftoolkit/>!

There will be a documentation update end of next month with following
releases for 0.9.0 and 1.0.0, the latest documentation can be found here
<https://tdf.github.io/odftoolkit/docs>.
Shortly after I aim to release a release supporting ODF 1.3.

If you are even vaguely curious, our email dev list
<https://tdf.github.io/odftoolkit/docs/mailing-lists.html> is very low
traffic.
Happy to answer any questions or receive feedback at the ODF dev list
<https://tdf.github.io/odftoolkit/docs/mailing-lists.html>.

Hope you are doing well!
Svante

---------- Forwarded message ---------
Von: Svante Schubert <svante.schubert@gmail.com>
Date: Do., 30. Apr. 2020 um 10:03 Uhr
Subject: The Future of the Simple API (and ODF Toolkit)
To: <dev@odftoolkit.org>


Hi everyone,

The future of the Simple API is that it will be discontinued after 0.9.0
due to copied source code parts and the resulting inability to add the new
change support.
Still, the very idea of a high-level API will be continued.

In the future, we should have some architecture as the following

[image: archi.png]
Slide 16 from last year's LO presentation:
https://libocon.org/assets/Conference/Almeria/Svante-Interoperable-Office-CollaborationwithNotes.pdf


Currently, we have unfortunately up to two to three places where the
functionality of this future high-level API resides. This has to be
consolidated.

   1. In the Simple API (deprecated)
   <https://github.com/tdf/odftoolkit/tree/master/simple/src/main/java/org/odftoolkit/simple>
   2. In the ODFDOM doc package
   
<https://github.com/tdf/odftoolkit/tree/1.0.0_SNAPSHOT/odfdom/src/main/java/org/odftoolkit/odfdom/doc>
   (the origin of Simple API fork, deprecated afterwards in favour of the
   Simple API and now likely being enabled again)
   3. In the ODFDOM changes packages
   
<https://github.com/tdf/odftoolkit/tree/1.0.0_SNAPSHOT/odfdom/src/main/java/org/odftoolkit/odfdom/changes>,
   which embraces the functionality to 'compile' an OpenDocument text file
   
<https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/docs/docs/presentations/character-styles.odt>
   into an equivalent sequence of (JSON) changes
   
<https://github.com/tdf/odftoolkit/blob/1.0.0_SNAPSHOT/docs/docs/presentations/character-styles.json>
   and allows merging of new changes back. Why are compiling an ODT into
   changes? Because sending documents around for collaboration is as clever as
   if software developer would be zipping their source code repositories and
   sending those instead of change commits!

There is already some change API documentation
<https://tdf.github.io/odftoolkit/docs/odfdom/operations/operations.html>
and it will be improved for the 1.0.0 release.
The transforming concept is as follows: Transforming an ODT into the user
changes creating it:
[image: changes-to-2020-04-30_0926.png]
Slide 13 (above) and 14 (below) from last year's LO presentation:
https://libocon.org/assets/Conference/Almeria/Svante-Interoperable-Office-CollaborationwithNotes.pdf

Thanks again to the PrototypeFund project who had supported this
<https://prototypefund.de/project/documents-for-democracy/>!

And the other way around, it is a merging of new user changes back into the
ODT:
[image: changes-back-2020-04-30_0925.png]
These changes are currently serialized as JSON, for instance, to be sent to
a web editor such as OX Documents
<https://www.open-xchange.com/portfolio/ox-documents/>.
(FYI - I implemented several years for Open-XChange this implementation in
AL2 license on their fork of ODFDOM).
But changes could be serialized for other reasons, for instance in XML and
being added to the ODT zip for Office change-tracking. I have suggested
this approach to the OASIS TC and my approach succeded against DeltaXML and
Microsoft to be implemented in ODF for change-tracking, but currently,
aside of us/me nobody is implementing it yet, therefore the Subcommittee of
"Advanced Document Collaboration" of OASIS - which I am chairing - is
hibernating atm.

What's next?
I have applied/looking for funding to refactor the implementation of the
existing changes
<https://github.com/tdf/odftoolkit/tree/1.0.0_SNAPSHOT/odfdom/src/main/java/org/odftoolkit/odfdom/changes>
.
In the nutshell, to describe declaratively this bidirectional
transformation (ODT <=> changes) and generate source code (and the OASIS
changes specification) from it.
Basically, the ODT grammar will be mapped to an API (a way to change the
XML, like explaining what happens during an insertRow).
Naturally, aside from the underlying DOM alterations, the higher-level API
would be generated as well, but this work is not done at once, but step by
step exchanging existing manual written code.
By this, we could work on one logic creating also C++ code for LibreOffice
or a TypeScript implementation for the browser or testing it in RUST.

For more information, I have tried to explain all the above (and the larger
picture) within two talks at the XML Prague this year:

   1. https://www.xmlprague.cz/day2-2020/#xmlpp (slides)
   https://www.youtube.com/watch?v=pLwH4q-R55M (video)
   2. https://www.xmlprague.cz/day3-2020/#nmdf  (slides)
   https://www.youtube.com/watch?v=dkTflH3yQZE (video)

and earlier written the final vision down on 3-4 pages for a poster session
and short presentation at the ACM symposium end of last year here in Berlin.
It is a summary of all the pain points I met in the past 20 years working
on document formats and standardization: "The Next Millennium Document
Format
<https://github.com/tdf/odftoolkit/blob/master/docs/docs/presentations/2019%20-%20ACM%20DocEng%20-%20The_Next_Millennium_Document_Format.pdf>
"

Looking forward to questions and feedback,
Svante


PS: I plan to add in the future the pictures from the slides directly to
our project online documentation.
Currently aiming for the end of May to focus my work on the ODF Toolkit
releases (and documentation).