Re: [Libreoffice] QA-hints ? -> question for devs !

Hi Jean-Baptiste,

    Thanks for the nice ideas and suggestions! My comments as below.

Hi All,

    This discussion is probably leading the next step we will adjust test case
    management (Litmus) structure of Libreoffice, ideas and experience sharing
    are welcome :) And I will summarize where we will go finally.

On Mon, Jun 13, 2011 at 01:54:46PM +0200, Jean-Baptiste Faure wrote:
> Le 13/06/2011 12:47, Yifan Jiang a écrit :
> Hi Yifan,
>
> I think we should do some simplifications in Litmus.

Yes! we need to adjust it to a more explicit way :)

> In fact I do not understand well how branches are supposed to work in
> Litmus. When you manage testcases and testgroups, you can see that there
> is no one in all branches but only in branch 3.3.1 (in Litmus meaning).

You are right, the branch information is from an relatively old test case set
(3.3.1). What I did for current runs is simply branch from an old test run
without touching test cases. So currently we have a simple set of testcases
reused everywhere. This actually needs to be improved.

> On the other hand when you manage the testruns you can see that each one
> contains the three testgroups defined in branch 3.3.1 (EN, DE and FR
> testgroups).

Ye, a bit confusing :)

> It seems to me that it would be more clear and easy to manage if we
> defined things as follows:
> - branches corresponding to x.y versions of LibreOffice: 3.3, 3.4, 3.5,
> and so on.
> - the branch x.y+1 would be defined by cloning the x.y branch and next
> adding new testcases needed by QA of the new functions implemented in
> the x.y+1 version.

My thinking is to:

    1. have a relatively stable regression test case set which cover most
    important function areas such as installation, launch, file open, save
    etc.

    2. have a changing feature cases (new functions) with the
    evolution of Libreoffice builds. These cases can be various among
    different Libreoffice build. And we can start testing features from beta
    phase.

> - we manage RC and bugfix versions (x.y.1, x.y.2, etc.) only by creating
> dedicated testruns: there is no reason to change the testcases set when
> we pass from version x.y.0 to rc x.y.1 rc1 to bugfix x.y.1 to ... etc.
> So there is no reason to have a new branch for a new rc or bugfix
> - we need to define the buildID for each testrun, which does not seem to
> be the case at this moment.

Great idea! Actually we may not really want to have big change of regression
test cases set even between big versions as x.y and x.y+n.

A problem is if it is worth to repeatedly run regression cases in diffrent rc
builds unders the same big version? My initial thinking about this is to
define a rc test run for each minor version build, QA could run the tests
always on the corresponding latest rc build they have. This is based on the
assumptions:

    1. code change between rc release would be safe

    2. currently we have not enough resource to repeatedly run all regression
    tests on each rc build.

But I guess it is not a big problem since when can always clone between rc
runs, the matter is how to define the run name. How about let us use 'rc' only
at very beginning and see if they are going well. Then we can expand the test
dedicating to specific rc builds if needed.

> According to that, we would have:
> - 2 branches 3.3 and 3.4
> - for the branch 3.3 :
>       + one testrun 3.3.3-rc1
>       + one testrun 3.3.3-rc2
> - for the branch 3.4 :
>       + one testrun 3.4.0
>       + one testrun 3.4.1-rc1
>       + one testrun 3.4.1-rc2
>       + etc.
>       + one testrun 3.4.2-rc1
>       + etc.
> - then a branch 3.5
>       + etc.

Here is my thinking based on your proposal:

Take into consideration libreoffice build 3.3 and 3.4:

Branches:

    - 3.3 Regression test branch (assuming it contains stable regression cases
                                  written from scratch)

        + Test run - 3.3.0-rc regression test 
        + Test run - 3.3.1-rc regression test 
        + Test run - 3.3.2-rc regression test                

        ...
    
    - 3.3 New Feature test branch

        + Test run - 3.3 feature test ( new features in 3.3 build )

    - 3.4 Regression test branch (containing stable regression cases, could be
                                  cloned from 3.3 and did some improvement)
    
        + Test run - 3.4.0-rc regression test 
        + Test run - 3.4.1-rc regression test
        + Test run - 3.4.2-rc regression test

        ...
    
    - 3.4 New Feature test branch

        + Test run - 3.4 feature test ( new features in 3.4 build )

> I think we need to agree on the general organization before
> that the system has become too big and too widely used.
> I can do these modifications if we have a general agreement.
> As that assume to remove several branches and to rebuild testruns, it
> would be prudent to not be several to do changes at the same time.

That'd be great and thanks for your offer of help! Meanwhile we won't want to
lose the existing testing results data :)

> What do think about this proposal? Did I overlook some point which make
> my beautiful organization completely stupid?

They are indeed beautiful, I appreciate your ideas! :) Thank you!

Best wishes,
Yifan

-- 
  Yifan Jiang
  Libreoffice
  Contact: yifan - irc.freenode.net/libreoffice
  =============================================  
  http://www.libreoffice.org/
  http://www.documentfoundation.org/