Re: [Libreoffice] Libreoffice QA process outline - a coarse draft.

Hi Yifan,
On 11/04/2011 13:59, Yifan Jiang wrote:
> Hi all,
>
>     The following is some outline/brainstorm level of stuff to genereate a QA
>     process documentation. I think it is necessary to have you all to review
>     this to avoid overlapped work, misunderstanding and mistakes. And I also
>     left a bunch of questions and blanks to be filled inside it :)
>
>     Please help add your comments! Thanks!
>
> -------------------------------------------------------------------------
>
> Hi Sophi,
>
>     When I was generating a draft of the process, there's several things are
>     not quite clear when I reviewed our irc session:
>
>     >  13:25<  sophi>  yifan: we distinguish between l10n and native language teams,
>     >  it's not the same members (even if it is the same language)
>
>     Would you help distinguish native language teams and l10n team? Do they
>     have overlapped work between each other? Do they have overlapped work
>     with pure function test? Thanks!

It depends of the size of the team in fact. For large teams like French 
or German, localizer and QA members are not the same persons. I'm doing 
the localization, where a team of about 10/15 members is doing QA and 
doesn't care about the localization process. If the language team is 
small, the members doing localization, QA, even documentation will be 
the same persons.
>     >  13:26<  yifan>  sophi: Is there somewhere on wiki I can read to know them?
>
>     >  13:26<  sophi>  yifan: there will be one QA session dedicated to l10n, with
>     >  the participation of the localizer, then after that will be regular members
>     >  of QA in native language project that will do the worek
>
>     Does this mean the native language testing has dependency on l10n testing
>     results?

I mean that there will be a session where the QA will focus more on the 
strings of the new functionalities introduced to be able to correct the 
localization in time before the freeze.
>     Do you have any news from German team about the process documentation, and
>     from Rimas about the availability of Litmus?:)

I didn't have time to check for the documentation but will do later 
today. Rimas has work on Litmus to enable links to test documents stored 
inside the tool, it works now :) So we are ready for a more general use 
of it. Rimas should announce the availability of Litmus today I thin, on 
the project list.
>     And would you help to outline the localization part below if the structure
>     is divided properly :) Thank you!
>
> -------------------------------------------------------------------------
>
> Libreoffice QA process - draft and propose to apply from Libreoffice 3.4
>
> The process document aims to be a practical guildline for Libreoffice QA
> activities, naturally the process itself canbe evolves through our QA
> practice.
>
> [New Feature QA Process]
>
>     - New feature testing takes place from dev phase to final release
>
>     - New feature testing can be happened with the latest nightly build of
>       Libreoffice
>
>     - The new features list of each build canbe usually found at ????
>
>         For example:
>             http://wiki.documentfoundation.org/ReleaseNotes/3.4
>
> [Release QA Process - Localization]
>
>     - I am not sure about the l10n process, but as a tip, information from
>       Petr shows that current l10n process might be optimized by the fact all
>       LO localizations are built at the same time, so most functionality just
>       work the same.
>
>     - Moreover if there is some overlapping between native language testing
>       and function testing. We may want to optimize them as well.
>
>     - TBD...

We need a special build called KeyID build for l10n QA. This build 
contains a unique ref for each string in the UI, see
http://wiki.services.openoffice.org/wiki/KeyID_Build
> [Release QA Process - Function test]
>
>     - The latest Release build (RC) for testing should be available at:
>
>         http://www.libreoffice.org/download/pre-releases/
>
>     - For each release, a test run will be created in Litmus TCM
>       tool(http://tcm.documentfoundation.com). QA would need to run them in at
>       most 10 days after the build ready (based on Release Criteria)
>
>       (howto litmus http://wiki.documentfoundation.org/Litmus)
>
>         + Who're going to create test run?
>
>         + When should the test run ready?
>
>         + Who're going to define test cases?
Hum, we should try to set a team dedicated to the test maintenance, 
enhancement and organizing the localization.

>     - QA announces RC build pass/fail based on Release Criteria
>
>       http://wiki.documentfoundation.org/Release_Criteria
>
>         + Where do we announce the results (mailing list, wiki)?
Collect the results on the wiki may be more easier for every body to 
participate.
>         + Who will be responsible for announcing?

Usually each language team announced the pass/fail of the tests.
>     - Fixed blocker Bug review
>
> [Release QA Process - Automation test]
>
>     - The latest Release build (RC) for testing should be available at:
>
>         http://www.libreoffice.org/download/pre-releases/
>
>     - The testtool is what we used for automation testing:
>
>         http://wiki.documentfoundation.org/QA/Testing/Using_Testtool
>
>     - Upload (TCM?) and analyze log
>
>     - Automation QA announce automation pass/fail based on Release Criteria
>
>         + Where do we announce the results (mailing list, wiki)?

Here also, I find it easier to manage this on the wiki
>         + Who will be responsible for announcing?
>
>     - TBD ...
>
> [Generic Libreoffice QA Process]
>
>     QA is being done at any time from develpment phase to final release. This
>     section aims to provide a guideline to unify those testing related actions
>     to be followed.
>
>     - Blockers Nomination process
>
>         http://wiki.documentfoundation.org/Release_Criteria
>
>     - Bug report/triage process
>
>         http://wiki.documentfoundation.org/BugReport
>         http://wiki.documentfoundation.org/BugTriage
>
>         We may want to add sections in Bug report/triage pages:
>
>         Regressions
>
>             The bugzilla has a 'regression' keyword in the 'Keywords' field.
>
>             Our attitude for a regression problem? I think for the same problem,
>             the regression problem should have a higher priority than a normal
>             one.
+1
>         Bug Severity
>
>            Bug Reporter's responsibility to mark it.
>
>         Bug Priority
>
>            Bug Triager's job - We may need a priority setting guide similar to release
>            criteria. Do we have it already somewhere?

I don't think that we have such a guide yet.
Thanks for your work on this Yifan, I'll try to be more available this 
week. I'll be on irc in my early morning tomorrow (from 2:00 AM UTC) and 
will stay quite a long moment if you have any question left.

Kind regards
Sophie
--
Founding member of The Document Foundation