Re: [Libreoffice] Number of days for testing RC

Hi Jean,

I am sorry for the late repply. I have been busy with LO-3.3-rc2.

Jean-Baptiste Faure píše v Čt 16. 12. 2010 v 06:05 +0100:
> Le 13/12/2010 15:04, Petr Mladek a écrit :
> > This is why I proposed 7 days instead 5 days. 7 days have the Weekend
> > included by definition :-)
> Not really if the period start just before the WeekEnd. If a new RC is
> released on Friday for QA-test, most of the testers will not be able to
> start their tests before Monday.

Fair enough. We will try to do not release on Friday. Alternatively,
would it help if we pre-announce the upcoming rc2 few days before it is
available?

It might always be delayed because of technical problems. Though, I
think that we could predict it pretty well two days before it is
published.

> > Hmm, I think that OOo-3.3 release is not a good example:
> Yes, from RC cycle point of view, it is an example that LibO should not
> follow.
> The process is not sufficiently clear:

Yeah, we are still setting the processes up.

>  what QA-team must do when a stopper is found and known?

The current solution is described at
http://wiki.documentfoundation.org/Release_Criteria#Blocker_Bug_Nomination

>  If a new RC is build immediately, a second
> stopper can be fixed only on this new RC so it is not interesting to
> continue to test the old RC. It is more true because the new RC can
> introduce regression to the old one.

You are right that any fix could bring regression. Though, I think that
it always makes sense to continue with testing:

        + there might be more blocker bugs; they are usually
          independent; it is better to find all blockers as soon as
          possible
        + the fix for the stable branches should be tested and reviewed
          before they are committed; it should reduce the number of
          regressions to minimum; the question is if we really need to
          repeat the whole QA for each rc; I think that it should depend
          on the number and complexity of the fixes; this information
          should be mentioned in the announce mail
        + it takes few days to produce new rc (time needed to fix the
          bug, do some testing, build in Windows, Linux, MAC; upload to
          mirrors)


> So I think we need a very clear QA test process: known dates for RC,

I am not sure if we want a regular schedule, e.g. do the rc release
every Thursday and every second Thursday. I think that such approach
would wok well for alpha/beta builds.

In case of release candidates, I think that it is better to do them
according to the current state. I mean to release them when all blockers
are fixed or when there are more that 2 weeks without release.

Note that we plan to upload also daily builds in the future. They might
be used to verify fixes even before the official release.

> when it is useful to search blockers and when we can stop to search
> stoppers until a new RC is released.

I think that it makes sense only when the application does not start at
all, so that the blocker bug blocks the testing completely.

> > There is an idea of more micro bug fix releases. The published minor
> > release should be well usable for 99% of normal users. The micro
> > releases could be done every next month or so. They would include just
> > safe and reviewed bug fixes and should need not full QA. The second or
> > third micro release should be well usable for 99% experienced users.
> Ok for me. The only condition I see to make the end-users happy, is that
> the status of each version released must be very clear for the end-user.
> Individual users or small organizations may want use always the last
> micro-release while bigger organizations like administrations will have
> a longer cycle of update.

Yes, we need to make it clear.

> > > > Also I think that it does not make sense that every national team would
> > > > do the whole testing. IMHO, 95% of the functionality is language
> > > > independent, so most of the testing can be shared and distributed.
> > > Right. For OOo, NL QA-team do only Release Sanity Test. But the current
> > > test period on OOo 3.3.0 shows that the last blockers have been found by
> > > the Community and these blockers could not be found by automatic or TCM
> > > tests; they have been found by users who have done tests on their
> > > documents they use in the real life.
> > > To do that we need time.
> > I see. Well, I think that even the released OOo version included pretty
> > annoying bugs and some of them would be considered as blockers. Such
> > bugs were usually fixed in the micro release x.y.1.
> >
> > We just need to define the right compromise that would be acceptable for
> > all sides developers, testers, and users

sure

> > Just to get better picture. How much time would you need to finish all
> > known manual tests?
> For OOo and Release Sanity Test, time needed is about 2 to 4 hours to
> complete all the 25 tests in the scenario.

Okay, this sounds doable within the week if you can plan it.

> But, as said the last blockers are out of the scope of these tests and
> have been found by test-cases and files from the real life.

I guess that many of these bugs were found by normal users that used OOo
in their daily work. I think that it actually supports the idea for more
frequent releases and tested/reviewed fixes. 

The manual testing makes sense and is really helpful, definitely.
Though, it could not verify the complete functionality within days or
even weeks.


Best Regards,
Petr