Re: Invoking gla11y [was: core.git: bin/gla11y config_host.mk.in configure.ac solenv/gbuild]

On 22.02.2018 17:08, Samuel Thibault wrote:
> Stephan Bergmann, on jeu. 22 févr. 2018 16:43:37 +0100, wrote:
> > On 22.02.2018 13:54, Samuel Thibault wrote:
> > > Miklos Vajna, on jeu. 22 févr. 2018 09:53:43 +0100, wrote:
> > > > On Wed, Feb 21, 2018 at 10:11:18AM +0100, Samuel Thibault <sthibault@hypra.fr> wrote:
> > > > > Rene Engelhard, on mer. 21 févr. 2018 10:02:08 +0100, wrote:
> > > > > > OK, thanks - and then I wonder why this is ran in "normal" UIConfig targets and
> > > > > > not only in make check...
> > > > > For the "error" cases (-W none), it makes sense:
> > > > >
> > > > > « Add to the build process error checking (only the hard errors such as
> > > > > bogus target names). »
> > > > >
> > > > > That's really the kind of issues one should get as soon as compilation
> > > > > time.
> > > > >
> > > > > Later on I'll add a call in "make check" time for the warnings.
> > > >
> > > > For source code we have warnings for these linter type problems
> > >
> > > Err, these are not just "linter type problems". They are undefined
> > > references, just like undefined C references. So the failure belongs to
> > > compilation time. If somebody modifies a .ui file in a way that gets
> > > such undefined reference, compilation itself should really fail, just
> > > like it does when a variable defintion is missing.
> > >
> > > The "make check" warnings I mentioned above are *other* kinds of issues,
> > > which would indeed only be tested within make check, made warnings by
> > > default, and turned into errors by Jenkins.
> >
> > I don't understand the "turned into errors by Jenkins" part.  How would that
> > be done?
> >
> > (And `make check` should ideally be "silent".  We do not want it to produce
> > any non-fatal warnings.)
>
> Well, perhaps I didn't understand what Miklos meant, then.
>
> Miklos Vajna, on jeu. 22 févr. 2018 09:53:43 +0100, wrote:
> > For source code we have warnings for these linter type problems and then
> > --enable-werror (Jenkins uses it) turns those warnings into errors.
> >
> > Probably the best would be if the .ui checker would follow the same
> > pattern, instead of running it twice during 'make check' (which is a
> > superset of 'make').
>
> What I had understood from that was that at "make" time there are only
> compiler errors, and at "make check" time there are linter warnings, and
> in Jenkins, --enable-werror is used to turn these warnings into errors.
>
> Now that I re-read it with what you said, I understand that
>
> - at "make" time there are some compiler warnings which are turned into
> errors in Jenkins with --enable-werror
> - Miklos suggested that we only call gla11y at "make" time, and not a
> second time at "make check" time.
>
> I'm fine with emitting accessibility warnings at "make" time and let
> Jenkins turning them into errors, I just thought from the discussions at
> FOSDEM that they should be done at "make check" time.

`make` vs. `make check` is completely orthogonal to 
--{en,dis}able-werror:  `make check` just runs more tests than `make`; 
some tests are expensive (and some are a bit brittle), so there are 
reasons why some builds want to be done without running the extra tests 
(though in an ideal world, they would always be included).

--enable-werror causes C/C++ compiler warnings to be turned into build 
failures.  We have a policy of a warning-free git master (and release 
branches), so that developers can --enable-werror and immediately find 
new issues they are introducing for which compilers emit warnings.  (And 
to keep master warning free on all platforms, Gerrit/Jenkins builds are 
also done with --enable-werror.)  Again, in an ideal world, 
--enable-werror would always be enabled for every build, but there are 
practical reasons (like some compiler emitting some hard-to-silence 
warnings only in -O2 production builds) why some builds use 
--disable-werror.

Now, gla11y shall do two things:

1  Find newly introduced bugs in .ui files.  Those should always cause 
the build to fail, regardless of `make` vs. `make check` or 
--enable-werror vs. --disable-werror.  I see no reason to have a build 
mode in which those bugs are either not detected at all, or do not fail 
the build.

2  Find existing bugs in .ui files (and maybe also minor newly 
introduced issues that are not severe enough to be considered fatal bugs 
upfront; I do not know).  Those should initially be suppressed by (1), 
but should eventually all be fixed (IIUC).  So for somebody working on 
fixing those issues, it would be useful to have a way to get a report of 
all those issues.  Maybe a new make target would be useful for that.

Does that make sense?