Anti-Virus vendors & warnings

Dear Nicholas,

On Tue, 2014-09-30 at 17:19 -0400, nicholas ferguson wrote:
> I duplicated their directory structure.  And my build still failed.

        Grief; we should certainly document turning off AV more prominently.
Ideally we could find a reproducer that we could check during configure
and print out:

        "You have a (typically) rubbish AV product installed -
         please un-install and or disable it" ;-)

        It'd be great to isolate exactly what is causing the problem, so we can
save other people this suffering; I'd love to invest in that.

> Wow.  So I did a forensic on the env.  And I discovered that Norton
> Antivirus was isolating state files and some executables being built by the
> LibreOffice build system.

        Great - is any of these small enough that we can build a reproducer out
of it ?

> So that alone took two to three weeks.  I even had to resort to buying a new
> machine...devoted to libreoffice.  $300 machine.  Trying to solve why my
> builds were failing on windows.  

        Sorry it bit you so hard - we aim to be easy to build =) that's mostly
achieved by people iterating and helping to fix problems they find.

> If an antivirus was turned on when LibreOffice staffers do builds..then they
> would have had to correct something..so that Norton Antivirus would not
> decide that a virus had been generated.

        So - my opinion of anti-virus' is that they are appallingly poorly
performing, superstition-ridden, scare-ware products. They are also
mostly proprietary. Each time we build LibreOffice - there is some other
co-incidence that triggers some AV fingerprinting with 200Mb of 'stuff'
on disk, what is the chance that something frightens an AV ?

        It has got -so- bad that some of our plain-text SVG files were
triggering one AV or other - because they contained co-ordinates lists
that looked like "credit card numbers" ;-) That takes the biscuit.

> This is probably why Michael and Tor rememeber me for too many emails. What
> the heck is going on here? I would email them.  how can you claim your stuff
> builds?

        This noisy mail exchange by itself is sufficient proof of verbosity and
a feeling of entitlement that doesn't, at least to my mind match a
reasonable expectation of what you can get for free from a Free Software
project =) I'd love to help you get over that. Collapsing some other
bits here:

On Tue, 2014-09-30 at 17:50 -0400, nicholas ferguson wrote:
> I think that is a bad idea.  A good idea is to turn on anti virus
> where work is done.  you can't tell developers to turn off their
> anti virus when working on windows.  That’s  crazy talk

        Did you read the recent interview where a prominent AV vendor said
their (debilitatingly slow and expensive) solution was only about 50%
effective ? [ IIRC ].

        It is easy to be full of good ideas of the form: "someone else should
do a lot of work to make my life easier" ;-) I have a lot of those kind
of good ideas too - they are mostly focused on encouraging -you- to do
something to improve things. Along those lines I loved your idea of
working on a different VS project file target - that was a positive
direction. In general in a volunteer project - if something is not done
-you- are the default solution to your own problem / need =)

        So - if you genuinely want to start this new "Anti-Virus clean"
initiative - then I suggest that you get a set of tinderboxes setup to
build with X, Y, and Z AV solutions enabled. Then when they fail -
you'll need to try to remediate the failure. In the SVG case above -
that might mean working out a different way to represent co-ordinates
(changing the SVG standard is perhaps hard), and/or compressing /
crypting the files with some non-standard header/magic so the AV doesn't
de-compress it to peek inside. That we could obscure the co-ordinates
that look like credit card numbers ;-) [ you'd also need to do some work
to persuade people to accept piece-meal changes like this into
LibreOffice ].

        In the more common / general case - you will need to work out why a
random 50Mb DLL triggers some arbitrary signature (the AV reports are
-very- spartan on details around this - they often won't tell you byte
offsets or - well anything much), and then when you've worked out what
the binary signature, you can then try to either:

        a) report it to the AV vendors (who will just white-list
           an md5sum or moral equivalent of that DLL you compiled just
           once leaving it to break again next commit / compile; and
           they'll white-list without any real understanding or analysis
           of the code too FWIW ;-)

        b) encourage Microsoft to 'fix' their compiler to generate
           (perhaps less optimal) code that doesn't co-incidentally
           include this particular fingerprint. or

        c) write an x86 binary re-writer that munges the generated
           code to do the same thing or

        d) find and tweak the random piece of source code to make
           it less optimal (eg. add a few volatiles around the place)
           to (hopefully) not trigger the issue; perhaps renaming some
           functions might help too ;-)

        Then repeat - for each AV product (each with their own distinct and
acute lamenesses) and for each of many false-positives they flag.

        You are -more- than welcome to do this of course. It'd be amusing to
write a paper on your progress as you go; you'd learn a -lot- about the
appalling lameness of AV solutions, end up wiser, and have some well
attended comic presentations at various conferences ;-) I know I'd come
to listen.

        In the meantime, our current approach is to turn off AV while building;
we should recommend that emphatically in the wiki.

        If we can - we should add a configure test to catch this madness
earlier - I wonder if we can look in the registry to see if XYZ AV is
enabled or even just installed somehow / easily ? That would really help
others like you Nicholas.

On Tue, 2014-09-30 at 17:46 -0400, nicholas ferguson wrote:
> I would think..that having to deal with this single issue, outlined
> below, that Michael and Tor would send me a sample of sc unit tests
> migrated over to a console application or at least a linux
> application, built as a standalone app, with a main in it.

        So - lets say that takes (finger in the air) one+ man days to do for
you; plus I and others already spent a considerable time answering your
questions, and trying to help you to help yourself [ which is a far more
scalable approach in the end BTW ;-].

> That would be a good gesture.

        An expensive gesture - for sure. It's not entirely clear why we should
do that for you, when you could do it yourself ? and in doing it
yourself learn a lot of useful things and avoid some moral hazard.

        Just so it's clear - I don't feel at all responsible for your inability
to build LibreOffice for some weeks. When I was first involved in OO.o
development it took a man-month [ full time ] to get my first build ;-)
I (and many others here) worked over many things to improve things, and
they are incredibly better today than then - ie. you're lucky ;-)

        All the best,

                Michael.

-- 
 michael.meeks@collabora.com  <><, Pseudo Engineer, itinerant idiot