Re: [Libreoffice] Number of days for testing RC

Le 13/12/2010 15:04, Petr Mladek a écrit :
> Hi Jean-Baptiste,
>
> Jean-Baptiste Faure píše v Pá 10. 12. 2010 v 20:02 +0100:
> > Hi Petr, all,
> >
> > Le 10/12/2010 15:24, Petr Mladek a écrit :
> > > Hi,
> > >
> > > Jean has modified http://wiki.documentfoundation.org/Release_Criteria
> > > and increased the number of days for reporting blocker bugs from 7 to
> > > 10.
> > My first name is Jean-Baptiste :-)
> Ah, I am sorry, shame on me.
>
> > > He says that they need to do the complete WE inside the period,
> > > see http://wiki.documentfoundation.org/Talk:Release_Criteria
> > >
> > > Heh, what is "WE"? :-)
> > Sorry, it is an acronym for a typical French expression : Week-End ;-)
> This is why I proposed 7 days instead 5 days. 7 days have the Weekend
> included by definition :-)
Not really if the period start just before the WeekEnd. If a new RC is
released on Friday for QA-test, most of the testers will not be able to
start their tests before Monday.
> > I think we need more time to allow volunteers to do manual tests. They
> > do these tests during their spare time so often during the week-end.
> > > OOo has only 5 days, see
> > > http://wiki.services.openoffice.org/wiki/Release_criteria#Stopper_issues
> > > How is it there with building and approving the national builds?
> > Perhaps am I influenced by the big number of RC we have raised for OOo
> > 3.3.0, but I think that if we take our time to deeply test our release
> > candidates, we will have better stable version and enjoyed and not
> > discouraged testers.
> Hmm, I think that OOo-3.3 release is not a good example:
Yes, from RC cycle point of view, it is an example that LibO should not
follow.
The process is not sufficiently clear: what QA-team must do when a
stopper is found and known? If a new RC is build immediately, a second
stopper can be fixed only on this new RC so it is not interesting to
continue to test the old RC. It is more true because the new RC can
introduce regression to the old one.

So I think we need a very clear QA test process: known dates for RC,
when it is useful to search blockers and when we can stop to search
stoppers until a new RC is released.
> OOo-3.3 development started on 2009-09-21 (branch for OOo-3.2)
> OOo-3.3 feature freeze was on 2009-06-24 (branch for OOo-3.3, feature
>                                           freeze)
> OOo-3.3-rc1 released on 2009-10-18
>
> So, development took 9 months. Initial stabilization took 4 months. RC
> phase takes nearly 2 months and is still not finished.
>
> Note that it was not full 9 months of development because many
> developers were busy with stabilizing OOo-3.2 and OOo-3.2.1.
>
> Of course, it was affected by the acquisition of Sun by Oracle but it is
> far from the original plan to release minor release every 6 months.
>
> The long cycle is discouraging for many volunteers. They want to have
> their work living and used by users. Also it is much easier to fix bugs
> shortly after a feature is finished than one year after implementation.
>
>
> There is an idea of more micro bug fix releases. The published minor
> release should be well usable for 99% of normal users. The micro
> releases could be done every next month or so. They would include just
> safe and reviewed bug fixes and should need not full QA. The second or
> third micro release should be well usable for 99% experienced users.
Ok for me. The only condition I see to make the end-users happy, is that
the status of each version released must be very clear for the end-user.
Individual users or small organizations may want use always the last
micro-release while bigger organizations like administrations will have
a longer cycle of update.
> > > Also I think that it does not make sense that every national team would
> > > do the whole testing. IMHO, 95% of the functionality is language
> > > independent, so most of the testing can be shared and distributed.
> > Right. For OOo, NL QA-team do only Release Sanity Test. But the current
> > test period on OOo 3.3.0 shows that the last blockers have been found by
> > the Community and these blockers could not be found by automatic or TCM
> > tests; they have been found by users who have done tests on their
> > documents they use in the real life.
> > To do that we need time.
> I see. Well, I think that even the released OOo version included pretty
> annoying bugs and some of them would be considered as blockers. Such
> bugs were usually fixed in the micro release x.y.1.
>
> We just need to define the right compromise that would be acceptable for
> all sides developers, testers, and users.
>
> Just to get better picture. How much time would you need to finish all
> known manual tests?
For OOo and Release Sanity Test, time needed is about 2 to 4 hours to
complete all the 25 tests in the scenario.
But, as said the last blockers are out of the scope of these tests and
have been found by test-cases and files from the real life.

Best regards
JBF

-- 
Seuls des formats ouverts peuvent assurer la pérennité de vos documents.