[Libreoffice] Libreoffice QA process outline - a coarse draft.

Hi all,

   The following is some outline/brainstorm level of stuff to genereate a QA
   process documentation. I think it is necessary to have you all to review
   this to avoid overlapped work, misunderstanding and mistakes. And I also
   left a bunch of questions and blanks to be filled inside it :)

   Please help add your comments! Thanks!

-------------------------------------------------------------------------

Hi Sophi,

   When I was generating a draft of the process, there's several things are
   not quite clear when I reviewed our irc session:

   > 13:25 < sophi> yifan: we distinguish between l10n and native language teams,
   > it's not the same members (even if it is the same language)

   Would you help distinguish native language teams and l10n team? Do they
   have overlapped work between each other? Do they have overlapped work
   with pure function test? Thanks!

   > 13:26 < yifan> sophi: Is there somewhere on wiki I can read to know them?

   > 13:26 < sophi> yifan: there will be one QA session dedicated to l10n, with
   > the participation of the localizer, then after that will be regular members
   > of QA in native language project that will do the worek

   Does this mean the native language testing has dependency on l10n testing
   results?

   Do you have any news from German team about the process documentation, and
   from Rimas about the availability of Litmus?:)

   And would you help to outline the localization part below if the structure
   is divided properly :) Thank you!

-------------------------------------------------------------------------

Libreoffice QA process - draft and propose to apply from Libreoffice 3.4

The process document aims to be a practical guildline for Libreoffice QA
activities, naturally the process itself canbe evolves through our QA
practice.

[New Feature QA Process]

   - New feature testing takes place from dev phase to final release

   - New feature testing can be happened with the latest nightly build of
     Libreoffice

   - The new features list of each build canbe usually found at ????

       For example:
           http://wiki.documentfoundation.org/ReleaseNotes/3.4   

[Release QA Process - Localization]

   - I am not sure about the l10n process, but as a tip, information from
     Petr shows that current l10n process might be optimized by the fact all
     LO localizations are built at the same time, so most functionality just
     work the same.

   - Moreover if there is some overlapping between native language testing
     and function testing. We may want to optimize them as well.

   - TBD...
   
[Release QA Process - Function test]

   - The latest Release build (RC) for testing should be available at:

       http://www.libreoffice.org/download/pre-releases/    

   - For each release, a test run will be created in Litmus TCM
     tool(http://tcm.documentfoundation.com). QA would need to run them in at
     most 10 days after the build ready (based on Release Criteria)

     (howto litmus http://wiki.documentfoundation.org/Litmus)

       + Who're going to create test run?

       + When should the test run ready?

       + Who're going to define test cases?

   - QA announces RC build pass/fail based on Release Criteria

     http://wiki.documentfoundation.org/Release_Criteria

       + Where do we announce the results (mailing list, wiki)?

       + Who will be responsible for announcing?

   - Fixed blocker Bug review

[Release QA Process - Automation test]

   - The latest Release build (RC) for testing should be available at:

       http://www.libreoffice.org/download/pre-releases/

   - The testtool is what we used for automation testing:

       http://wiki.documentfoundation.org/QA/Testing/Using_Testtool

   - Upload (TCM?) and analyze log

   - Automation QA announce automation pass/fail based on Release Criteria

       + Where do we announce the results (mailing list, wiki)?

       + Who will be responsible for announcing?    

   - TBD ...

[Generic Libreoffice QA Process]

   QA is being done at any time from develpment phase to final release. This
   section aims to provide a guideline to unify those testing related actions
   to be followed.

   - Blockers Nomination process

       http://wiki.documentfoundation.org/Release_Criteria

   - Bug report/triage process

       http://wiki.documentfoundation.org/BugReport        
       http://wiki.documentfoundation.org/BugTriage
       
       We may want to add sections in Bug report/triage pages:
       
       Regressions

           The bugzilla has a 'regression' keyword in the 'Keywords' field.

           Our attitude for a regression problem? I think for the same problem,
           the regression problem should have a higher priority than a normal
           one.

       Bug Severity

          Bug Reporter's responsibility to mark it.

       Bug Priority

          Bug Triager's job - We may need a priority setting guide similar to release
          criteria. Do we have it already somewhere?


          
-- 
 Yifan Jiang
 Libreoffice
 Contact: yifan - irc.freenode.net/libreoffice
 =============================================  
 http://www.libreoffice.org/
 http://www.documentfoundation.org/