Re: Fw: benchmark of Excel, Calc, Google Docs

Wols,

Thanks for sharing your concerns, replete with quotes from Einstein :-) !

I believe I share those concerns... but there are of course some liberties
we can take as an academic group that you folks managing a popular
spreadsheet tool cannot take (e.g., doing a proof of concept as opposed to
a robust implementation.)   From a research/academic standpoint it is
valuable to note that something is possible, even if the solution is not
ideal given other pragmatic considerations.  BTW, I don't believe that
anything we're doing *requires* a relational database -- a NoSQL setup
would work just fine.

I'd be happy to discuss more.  Our goal is to understand the stumbling
blocks in translating our work into spreadsheet tools such as Calc and see
how we can best help with what we've learned.

Cheers,
Aditya

On Mon, Dec 9, 2019 at 3:37 PM Wols Lists <antlists@youngman.org.uk> wrote:

> On 09/12/19 19:14, Aditya Parameswaran wrote:
> >            The idea of converting to SQL queries is an interesting one
> >     but I find
> >     it very hard to believe it would provide any performance advantage at
> >     the same memory footprint. Furthermore - I'd be interested to know
> how
> >     you do other spreadsheet operations: row & column insertion,
> addressing,
> >     and dependency work on top of a SQL database with any efficiency.
> >
> >
> > We started by having the relational database be a simple persistent
> > storage layer, when coupled with an index to retrieve data by position,
> > can allow us to scroll through large datasets of billions of rows at
> > ease. We developed a new positional index to handle insertions and
> > deletions in O(log(n)) -- https://arxiv.org/pdf/1708.06712.pdf. I agree
> > that pushing the computation to the relational database does have
> > overheads; but at the same time, it allows for scaling to arbitrarily
> > large datasets.
>
> "the quickest way to optimise database access is to ditch first normal
> form".
>
> A provocative statement I know, but I'm very much in the NoSQL camp. I
> can hunt up the details of a face-off between Oracle and Cache, where
> Oracle had to "cheat" to achieve 100K tpm (things like deferring index
> updates) whereas Cache blasted through 250K barely breaking a sweat ...
> (or it might well have been tps)
>
> The maths supports this ...
>
> That said, a spreadsheet is inherently first normal formal, so tying a
> spreadsheet and a relational database together MAY make sense.
>
> In general though, Einstein said "make things as simple as possible BUT
> NO SIMPLER". Relational oversimplifies the database side, which means
> the application side is over-complex in order to compensate. (Which is
> why Cache blew Oracle out of the water.)
>
> I'm quite happy to wax lyrical, but I'd rather not preach to an audience
> who aren't at least interested. Feel free to ask me to continue on list,
> or contact me privately, and I'll try to prove everything as
> mathematically as I can :-)
>
> > but at the same time, it allows for scaling to arbitrarily
> > large datasets.
>
> At the price of massive unnecessary complexity.
>
> Cheers,
> Wol