[Libreoffice] Litmus testcase structure improvement.

Hi all,

Referring to the attached discussion thread initialed by Jean-Baptiste (great
thanks!), we are trying to make testcase management tool Litmus
(https://tcm.documentfoundation.org/run_tests.cgi) better managed and
organized. The following explained all current states. As usual, comments are
welcome! Thanks for all the support!

[For QA people running cases]

    The run created by new method for 3.4.1:

        https://tcm.documentfoundation.org/run_tests.cgi?test_run_id=13

    Basically no practical changes from before. Meanwhile please
    notice:

    a. make sure the build version you are using is suitable for
    the corresponding test run. For example, for the test run

        - LibreOffice 3.4.0 RC Regression test

    libreoffice build 3.4.0.x is needed.

    b. Please make sure the platform, operating system and build
    id information, which should be filled before the testing,
    exactly reflecting the testing environment.

    c. Attaching a bug id to a failed case result would be very
    nice!

[For Litmus admins]

    We are trying to make every thing much more clear and easy to
    update. The current test case organization changes:

    a. Divide test case set to Regression test and New Feature
    test ([1])

    b. Regression Test

        A branch *Master Regression TC* was created
        specifically. So that we can all work on a consistent
        test case base, from which new branches can be always
        *cloned*.

        Keep all new/updated cases in this branch would be
        necessary to make everything inside reusable.
        
        That is to say, we use the Master branch to keep a
        reusable and continuously maintanable test cases storage,
        but NOT for creating test runs.

        When a new build (big version as x.x) will come, we

            1. clone (Recursively) a new regression test branch
               for the build from Master branch, like

               3.4 Regresstion Test (branch)
               3.5 Regresstion Test (branch)

               ...
                
            2. through the branches, create a test run for the corresponding
            builds:

               libreoffice 3.4.0 rc regresstion test (run created from 3.4 branch)
               libreoffice 3.4.1 rc regresstion test (run created from 3.4 branch)
               libreoffice 3.4.2 rc regresstion test (run created from 3.4 branch)

               libreoffice 3.5.0 rc regresstion test (run created from 3.5 branch)
               libreoffice 3.5.1 rc regresstion test (run created from 3.5 branch)
               libreoffice 3.5.2 rc regresstion test (run created from 3.5 branch)

                ...

            *NOTICE*
            
              If some new cases needed, please create them in the
              Master Testcase branch. Thus we can use the new
              regression test case set next time creating
              branches([2]).

              I have noticed a new set of pt-BR test cases
              created in '3.4 Regression test' branch today, I will
              'sync' them to the Master branch later.

    c. New Feature Test

        A branch *Master Feature TC* was created
        specifically. This is more like an outline without cases
        (but with group subgroup info) filled in. So for each new
        big version (x.x),

            1. clone from the Master Feature branch:

                3.4 Feature Test (branch)
                
                3.5 Feature Test (branch)
                
                3.6 Feature Test (branch)

            2. we create a single run for each big version:

                libreoffice 3.4 feature test (run created from 3.4 branch)

                libreoffice 3.5 feature test (run created from 3.5 branch)

                libreoffice 3.6 feature test (run created from 3.6 branch)

                ...

            3. we 'seek' and write new feature cases in the
              corresponding subgroup according to git log,
              mailing list, release notes or so.

            4. we update important feature case to Master
               Regression TC branch

[Footnotes]

    [1] Regression and Feature test are there for quite different
    purpose:

        The Regression test would hold relatively stable set of
        test cases, which should be ran in each of the release
        build.

        The New Feature test would be dynamically changed based
        on a big version update (3.3->3.4). Meanwhile New feature
        test might be run from even early development build
        phase.

    [2] As you may notice, by this process we didn't really
        change the running branch.  However I didn't find an easy way
        to sync test cases between branches :( A workaround method to
        'link' new test cases is:

        1. Add a new test cases in the Master branch, and
        assign the cases to a proper subgroup

        2. Edit current running branch's subgroup to link a new
        case from the Master branch to the current running
        one. (manage subgroups -> select Product as libreoffice
        and Banch as currently running -> edit subgroups ->
        manage testcases for this subgroup)

        Please also let me know if things could be better.

-- 
  Yifan Jiang
  Libreoffice
  Contact: yifan - irc.freenode.net/libreoffice
  =============================================  
  http://www.libreoffice.org/
  http://www.documentfoundation.org/

--- Begin Message ---

• Yifan Jiang <yfjiang -AT- novell.com>
• Tue, 14 Jun 2011 14:58:44 +0800

Hi Jean-Baptiste,

    Thanks for the nice ideas and suggestions! My comments as below.

Hi All,

    This discussion is probably leading the next step we will adjust test case
    management (Litmus) structure of Libreoffice, ideas and experience sharing
    are welcome :) And I will summarize where we will go finally.

On Mon, Jun 13, 2011 at 01:54:46PM +0200, Jean-Baptiste Faure wrote:

Le 13/06/2011 12:47, Yifan Jiang a écrit :
Hi Yifan,

I think we should do some simplifications in Litmus.

Yes! we need to adjust it to a more explicit way :)

In fact I do not understand well how branches are supposed to work in
Litmus. When you manage testcases and testgroups, you can see that there
is no one in all branches but only in branch 3.3.1 (in Litmus meaning).

You are right, the branch information is from an relatively old test case set
(3.3.1). What I did for current runs is simply branch from an old test run
without touching test cases. So currently we have a simple set of testcases
reused everywhere. This actually needs to be improved.

On the other hand when you manage the testruns you can see that each one
contains the three testgroups defined in branch 3.3.1 (EN, DE and FR
testgroups).

Ye, a bit confusing :)

It seems to me that it would be more clear and easy to manage if we
defined things as follows:
- branches corresponding to x.y versions of LibreOffice: 3.3, 3.4, 3.5,
and so on.
- the branch x.y+1 would be defined by cloning the x.y branch and next
adding new testcases needed by QA of the new functions implemented in
the x.y+1 version.

My thinking is to:

    1. have a relatively stable regression test case set which cover most
    important function areas such as installation, launch, file open, save
    etc.

    2. have a changing feature cases (new functions) with the
    evolution of Libreoffice builds. These cases can be various among
    different Libreoffice build. And we can start testing features from beta
    phase.

- we manage RC and bugfix versions (x.y.1, x.y.2, etc.) only by creating
dedicated testruns: there is no reason to change the testcases set when
we pass from version x.y.0 to rc x.y.1 rc1 to bugfix x.y.1 to ... etc.
So there is no reason to have a new branch for a new rc or bugfix
- we need to define the buildID for each testrun, which does not seem to
be the case at this moment.

Great idea! Actually we may not really want to have big change of regression
test cases set even between big versions as x.y and x.y+n.

A problem is if it is worth to repeatedly run regression cases in diffrent rc
builds unders the same big version? My initial thinking about this is to
define a rc test run for each minor version build, QA could run the tests
always on the corresponding latest rc build they have. This is based on the
assumptions:

    1. code change between rc release would be safe

    2. currently we have not enough resource to repeatedly run all regression
    tests on each rc build.

But I guess it is not a big problem since when can always clone between rc
runs, the matter is how to define the run name. How about let us use 'rc' only
at very beginning and see if they are going well. Then we can expand the test
dedicating to specific rc builds if needed.

According to that, we would have:
- 2 branches 3.3 and 3.4
- for the branch 3.3 :
      + one testrun 3.3.3-rc1
      + one testrun 3.3.3-rc2
- for the branch 3.4 :
      + one testrun 3.4.0
      + one testrun 3.4.1-rc1
      + one testrun 3.4.1-rc2
      + etc.
      + one testrun 3.4.2-rc1
      + etc.
- then a branch 3.5
      + etc.

Here is my thinking based on your proposal:

Take into consideration libreoffice build 3.3 and 3.4:

Branches:

    - 3.3 Regression test branch (assuming it contains stable regression cases
                                  written from scratch)

        + Test run - 3.3.0-rc regression test 
        + Test run - 3.3.1-rc regression test 
        + Test run - 3.3.2-rc regression test                

        ...
    
    - 3.3 New Feature test branch

        + Test run - 3.3 feature test ( new features in 3.3 build )

    - 3.4 Regression test branch (containing stable regression cases, could be
                                  cloned from 3.3 and did some improvement)
    
        + Test run - 3.4.0-rc regression test 
        + Test run - 3.4.1-rc regression test
        + Test run - 3.4.2-rc regression test

        ...
    
    - 3.4 New Feature test branch

        + Test run - 3.4 feature test ( new features in 3.4 build )

I think we need to agree on the general organization before
that the system has become too big and too widely used.
I can do these modifications if we have a general agreement.
As that assume to remove several branches and to rebuild testruns, it
would be prudent to not be several to do changes at the same time.

That'd be great and thanks for your offer of help! Meanwhile we won't want to
lose the existing testing results data :)

What do think about this proposal? Did I overlook some point which make
my beautiful organization completely stupid?

They are indeed beautiful, I appreciate your ideas! :) Thank you!

Best wishes,
Yifan

-- 
  Yifan Jiang
  Libreoffice
  Contact: yifan - irc.freenode.net/libreoffice
  =============================================  
  http://www.libreoffice.org/
  http://www.documentfoundation.org/

--- End Message ---