Re: [Libreoffice] [Libreoffice-qa] 3.5.0 QA ... from BHS 1 to BHS 2

Cor Nouws píše v Út 24. 01. 2012 v 23:10 +0100:
> Hi all,
>
> Pedro wrote (10-01-12 18:57)
> > Michael Meeks-2 wrote
>
> > > > OTH more releases means more features but also more bugs. And because new
> > > > bugs occur, old bugs are left behind.
> > >
> > >    Oh ! so - this is an argument for doing a build every decade ;-)
> >
> > Not really. It's just an argument that if releases are too close, developers
> > will only have time to fix blockers :)
> > So not so critical bugs tend to accumulate. This could mean that it will
> > loose quality as it goes along if it there are no major obstacles ;)
>
> This has been part of our discussion in Paris.
>   http://wiki.documentfoundation.org/QA/Improving_QA-Release-3.5
> more specific
>
> http://wiki.documentfoundation.org/File:Liboconf2011_improving_qa_release_cycle-cornouws_Slides19-28.pdf
> See #4 on slide 23, and slide 27 explaining that point.
>
> That discussion (of course) carried no final conclusion - but the 
> 'agreement' (consent with Petr that it is a good idea) that the 
> developers try to remember/summarise their experience with this, so that 
> that can be part of an evaluation.
>
> The idea/hope is, that faster/smarter fixing of bugs, leads to a shift 
> in the time spending: less weeks on bug fixing and more on features.

Please, look at
http://download.go-oo.org/lo/lo-release-plan-in-colors.png

The picture tries to visualize an expectation where developers spend
time. The strong yellow and red color shows huge activity in bug fixing.
The strong green shows the time where developers prefer work on
features. You see that the strong green is only half size of the strong
yellow and red.

=> developers spend a lot of time on bug fixing


Here are the commit statistics from weeks between 3.4.0-beta1 and
3.5.0-beta1:

week 13, 2011:    19 bug fixes of 339 commits
week 14, 2011:    22 bug fixes of 283 commits
week 15, 2011:    17 bug fixes of 183 commits
week 16, 2011:    16 bug fixes of 182 commits
week 17, 2011:    21 bug fixes of 179 commits
week 18, 2011:    20 bug fixes of 158 commits
week 19, 2011:    19 bug fixes of 194 commits
week 20, 2011:    26 bug fixes of 361 commits
week 21, 2011:    26 bug fixes of 333 commits
week 22, 2011:    8 bug fixes of 301 commits
week 23, 2011:    40 bug fixes of 313 commits
week 24, 2011:    27 bug fixes of 436 commits
week 25, 2011:    21 bug fixes of 223 commits
week 26, 2011:    23 bug fixes of 245 commits
week 27, 2011:    21 bug fixes of 252 commits
week 28, 2011:    11 bug fixes of 352 commits
week 29, 2011:    15 bug fixes of 314 commits
week 30, 2011:    33 bug fixes of 441 commits
week 31, 2011:    15 bug fixes of 226 commits
week 32, 2011:    20 bug fixes of 350 commits
week 33, 2011:    24 bug fixes of 468 commits
week 34, 2011:    16 bug fixes of 375 commits
week 35, 2011:    20 bug fixes of 309 commits
week 36, 2011:    34 bug fixes of 332 commits
week 39, 2011:    5 bug fixes of 357 commits
week 40, 2011:    16 bug fixes of 482 commits
week 41, 2011:    10 bug fixes of 127 commits
week 42, 2011:    21 bug fixes of 268 commits
week 43, 2011:    17 bug fixes of 307 commits
week 44, 2011:    24 bug fixes of 309 commits
week 45, 2011:    20 bug fixes of 333 commits
week 46, 2011:    36 bug fixes of 462 commits
week 47, 2011:    64 bug fixes of 558 commits
week 48, 2011:    59 bug fixes of 569 commits
week 49, 2011:    52 bug fixes of 460 commits

Note that the real numbers are higher because developers sometimes do
not mention the fixed bug number in commit messages. It would be better
to get statistics from bugzilla but it is not that easy for me. Rainer
is the expert here.

Anyway, it shows that developers fix bugs all the time. They do not
spend months working only on features!



> However, there is inevitably a strong tangling with the QA work.
> And - despite my optimistic nature and the many hours spent on both QA 
> and getting that process at a higher level/notice - I have some serious 
> concerns on how secure our over all process is in this regard.
>
> This results in bugs that should have been handled/getting floated much 
> earlier. Fix may be easy / ready in master / needed, but do not find 
> their way to the bug fix releases.
>
> Examples:
> - 45068 Update from 3.4.4 on Win not possible without ...
>      initial bug 2011-04-29 3.4.0 beta3,
>      workaround published 2012-01-22

There was no activity on this bug until 2011-11-24; It was fixed the day
after on 2011-11-25.

=> The problem was that it was hidden in the swamp of other bugs and
newer assigned to a developer who could look at it


> - 41054 Saving problem ods with new sheets
>      initial report 2011-09-20
>      fixed for 3.5 2012-01-10
>      fixed for 3.4 still pending

There was no real activity until 2012-01-11. The bug was fixed the same
day.

=> again the same problem with hidden bug in the swamp


> - 39118 Charts do not update
>      initial report 2011-07-10
>      fixed for master/3.5 2011-12-13

This one was assigned correctly on 2011-07-11. The bug had set medium
priority and normal severity

=> there were more important bugs that needed to be fixed before

Kohei, Markus, and Eike are excellent guys. They really take care about
bug reports and fix one after each other. I sometimes can't belief how
many bugs they fix every day.


>     fixed for 3.4.6 2012-01-16

We are very careful about backporting fixes because we do not want
regressions. It means that it is a bit painful and time consuming. We
need a good balance here.


> Because I know some people are quite sensitive for the impression of 
> being accused personally: this is not the case. I do not accuse someone.
> I just show where our process, our mutual activity, falls short.
> Maybe our process at the moment is even better then 6 months ago (good 
> change). Still, it's not good enough.
>
> This is caused by the amount if bugs filed and the lack of time to 
> handle those properly. So important issues do not always get the 
> attention they deserve. It is not that bugs are not important enough, so 
> that people ignore spending enough time on them.
> Another reason: lack of triage / bundling of issues, which is both 
> important for developers (fixing) ans users (simple how-to's for work 
> around).
> (Will do some kick off on that point later).


I do not say that our processes are perfect. They can be surely
improved. On the other hand, we need to live in the reality. We have
many users with many demands. They need both features and bug fixes. We
have the given number of contributors. 

My feeling is:

+ the time for bug fixing and feature work is relatively well balanced
+ developers take care of bugs => their attitude is great
+ there are many new unit tests and subsequent tests that helps to find
  bugs at built time; they help to avoid regressions
+ the few bug triage people do amazing job; there are still only few
  but we get new ones from time to time
+ we still do not have well organized QA; They are partly lost in the
  new processes (early .0 release; many bugfix releases). The l10n teams
  are not used to working together so closely (many localizations were
  built later after the main release in the past). We try to improve
  this by the bug hunting sessions, improved wiki, improved litmus
  server.
+ people still do not have correct expectations about the .0 release;
  I am afraid that many people still expect that it should be perfect


Note that the release is not faster than it was in the past. We are
going to have one 3.x release every 6 motnhs. There is just the feeling
that it is faster because we ship .0 early with known bugs and we do
more bug fix releases. Though, we are very strict about what fixes go to
the bug fix releases, so they do not need that many testing as if we
would allow any changes there.


IMHO, the challenges are:

+ get more contributors (developers, bug hunters, testers, ...); this is
  newer ending story
+ continue in better organizing QA
+ set correct expectations of the .0 release
+ ???

Otherwise, I hear relatively positive feedback about 3.5.0 betas and
rcs. I feel that we are in much better situation than with 3.4.0.



Best Regards,
Petr