Test planning and documentation

Transcrição

1 TQS - Teste e Qualidade de Software (Software Testing and Quality) Test planning and documentation. Problem reporting and follow-up. Test measurements and test management. jpf@fe.up.pt João Pascoal Faria 1 Test planning and documentation 2

2 The IEEE Standard for Software Test Documentation Specification This was the situation in the 1983 version. In the 1998 version there is a single test plan. (adapted from Ilene Burnstein, Practical Software Testing) 3 The IEEE Standard Document types (1/2) Test planning: Test plan - prescribes the scope, approach, resources, and schedule of the testing activities - identifies the items to be tested, the features to be tested, the testing tasks to be performed, the personnel responsible for each task, and the risks associated with the plan Test specification: Test design specification - refines the test approach and identifies the features to be covered by the design and its associated tests - identifies the test cases and test procedures, if any, required to accomplish the testing and specifies the feature pass/fail criteria. Test case specification - documents the actual values used for input along with the anticipated outputs - also identifies constraints on the test procedures resulting from use of that specific test case (Test cases are separated from test designs to allow for use in more than one design and to allow for reuse in other situations.) 4

3 The IEEE Standard Document types (2/2) Test specification (cont.): Test procedure specification - identifies all steps required to operate the system and exercise the specified test cases in order to implement the associated test design (Test procedures are separated from test design specifications as they are intended to be followed step by step and should not have extraneous detail.) Test reporting: Test item transmittal report - identifies the test items being transmitted for testing in the event that separate development and test groups are involved or in the event that a formal beginning of test execution is desired Test log - used by the test team to record what occurred during test execution Test incident report - describes any event that occurs during the test execution which requires further investigation Test summary report - summarizes the testing activities associated with one or more test design specifications 5 The IEEE Standard Test plan contents 1. Test plan identifier 2. Introduction (why) Overall description of the project, the software system being developed or maintained, and the software items and/or features to be tested Overall description of testing goals (objectives) and the testing approaches to be used References to related or supporting documents 3. Test items (what) List the items to be tested: procedures, classes, modules, libraries, components, subsystems, systems, etc. Include references to documents where these items and their behaviors are described (requirements and design documents, user manuals, etc.) List also items that will not be tested 6

4 The IEEE Standard Test plan contents 4. Features to be tested (what) Features are distinguishing characteristics (functionalities, quality attributes), closely related to the way we describe software in terms of its functional and quality requirements Identify all software features and combinations of software features to be tested. Identify the test design specification associated with each feature and each combination of features. 5. Features not to be tested (what) Identify all features and significant combinations of features that will not be tested and the reasons. 6. Approach (how) Description of test activities, so that major testing tasks and task durations can be identified For each feature or combination of features, the approach that will be taken to ensure that each is adequately tested Tools and techniques Expectations for test completeness (such as degree of code coverage for white box tests) Testing constraints, such as time and budget limitations Stop-test criteria 7 The IEEE Standard Test plan contents 7. Item pass-fail criteria Given a test item and a test case, the tester must have a set of criteria to decide whether the test has been passed or failed upon execution The test plan should provide a general description of these criteria Failures to a certain severity level may be accepted 8. Suspension criteria and resumption requirements Specify the criteria used to suspend all or a portion of the testing activity on the test items associated with this plan Specify the testing activities that must be repeated, when testing is resumed Testing is done in cycles: test fix - (resume) test (suspend) fix -... Tests may be suspended when a certain number of critical defects has been observed 9. Test deliverables Test documents (possibly a subset of the ones described in the IEEE standard) Test harness (drivers, stubs, tools developed especially for this project, etc.) 10. Testing tasks Identify all test-related tasks, inter-task dependencies and special skills required Work Breakdown Structure (WBS) 8

5 The IEEE Standard Test plan contents 11. Environmental needs Software and hardware needs for the testing effort 12. Responsibilities Roles and responsibilities to be fulfilled Actual staff involved (?) 13. Staffing and training needs Description of staff and skills needed to carry out test-related responsibilities 14. Scheduling Task durations and calendar Milestones Schedules for use of staff and other resources (tools, laboratories, etc.) 9 The IEEE Standard Test plan contents 15. Risks and contingencies Risks should be (i) identified, (ii) evaluated in terms of their probability of occurrence, (iii) prioritized, and (iv) contingency plans should be developed that can be activated if the risk occurs Example of a risk: some test items not delivered on time to the testers Example of a contingency plan: flexibility in resource allocation so that testers and equipment can operate beyond normal working hours (to recover from delivery delays) 16. Testing costs (not included in the IEEE standard) Kinds of costs: - costs of planning and designing the tests - costs of acquiring the hardware and software necessary - costs of executing the tests - costs of recording and analyzing test results - tear-down costs to restore the environment Cost estimation may be based on: - Models (such as COCOMO for project costs) and heuristics (such as 50% of project costs) - Test tasks and WBS - Developer/tester ratio (such as 1 tester to 2 developers) - Test impact items (such as number of procedures) and test cost drivers (or factors, such as KLOC) - Expert judgment 17. Approvals Dates and signatures of those that must approve the test plan 10

6 The IEEE Standard Test design specifications One or more documents A test design specification describes how a group of features and/or test items is tested by a set of test cases and test procedures May include a (test case to) features/requirements traceability matrix Contents: Identifier Features to be tested - Test items and features covered by this document Approach refinements - Test techniques Test case identification Feature pass/fail criteria 11 The IEEE Standard Test case specifications Contents: Test case specification identifier Test items - List of items and features to be tested by this test case Input specifications Output specifications Environmental needs Special procedural requirements Intercase dependencies 12

7 The IEEE Standard Test procedure specifications Describe steps required for executing a set of test cases or, more generally, the steps used to analyze a software item in order to evaluate a set of features Contents: Test procedure specification identifier Purpose Specific requirements Procedure steps - Log, set up, proceed, measure, shut down, restart, stop, wrap up, contingencies 13 The IEEE Standard Test item transmittal report Accompanies a set of test items that are delivered for testing Contents Transmittal report identifier Transmitted items - version/revision level - references to the items documentation and the test plan related to the transmitted items - persons responsible for the items Location Status - deviations from documentation, from previous transmissions or from test plan - incident reports that are expected to be resolved - pending modifications to documentation Approvals 14

8 The IEEE Standard Test log Records detailed results of test execution Contents Test log identifier Description - Identify the items being tested including their version/revision levels - Identify the attributes of the environments in which the testing is conducted Activity and event entries - Execution description - Procedure results - Environmental information - Anomalous events - Incident report identifiers The IEEE Standard Test incident report Also called a problem report Contents: Test incident report identifier Summary - Summarize the incident - Identify the test items involved indicating their version/revision level - References to the appropriate test procedure specification, test case specification, and test log Incident description - inputs, expected results, actual results, anomalies, date and time, procedure step, environment, attempts to repeat, testers, observers - any information useful for reproducing and repairing Impact - If known, indicate what impact this incident will have on test plans, test design specifications, test procedure specifications, or test case specifications - severity rating 16

9 The IEEE Standard Test summary report Contents Test summary report identifier Summary - Summarize the evaluation of the test items - Identify the items tested, indicating the environment in which the testing activities took place Variances - of the test items from their original design specifications Comprehensiveness assessment - Evaluate the comprehensiveness of the testing process against the comprehensiveness criteria specified in the test plan if the plan exists - Identify features or feature combinations that were not sufficiently tested and explain the reasons Summary of results - Summarize the results of testing - Identify all resolved incidents and summarize their resolutions - Identify all unresolved incidents. Evaluation - Provide an overall evaluation of each test item including its limitations - This evaluation shall be based upon the test results and the item level pass/fail criteria - An estimate of failure risk may be included Summary of activities - Summarize the major testing activities and events - Summarize resource consumption data, e.g., total staffing level, total machine time, and total elapsed time used for each of the major testing activities Approvals 17 Some lessons by Cem Kaner et al Test documentation Use the IEEE Standard 829 for test documentation Based on the experience of many people Don't use the IEEE Standard 829 Assumes a waterfall-like approach No guidelines for tailoring according to the needs of a project Too many documentation Absence of cost awareness Difficult to conciliate with automated testing Summarize your core documentation requirements in one sentence with no more than three components Example: "The test documentation set will primarily support our efforts to find bugs in this version, to delegate work, and to track status" 18

10 Some lessons by Cem Kaner et al Test planning Three basic questions to ask about test strategy are "why bother?", "who cares?", and "how much?" The real test plan is the set of ideas that guides your test process Design your test plan to fit your context Use the test plan to express choices about strategy, logistics, and work products Test to the maturity of the product Test the gray box Design your test strategy in response to project factors as well as product risks 19 Problem reporting and follow-up 20

11 Automated bug reporting /tracking (Mantis) (source: Ron Paton) 21 Figure 6-1. Lifecycle of a Bugzilla Bug Bugs lifecycle (Bugzilla open source) (source: /docs/2.22/pdf/bugzilla- Guide.pdf ) 22

12 Manual bug reporting (source: Cem Kaner, Testing Computer Software) 23 Tracking test suites (source: Ron Paton) 24

13 Lessons by Cem Kaner et al (adapted) Bug advocacy Your advocacy drives the repair of the bugs you report Report the problem with enough detail so that the programmer can reproduce the bug Make clear the impact/severity of the bug Report the problem clearly, but don't try to solve it Never use the bug-tracking system to monitor programmers' or tester s performance The best approach may be to demonstrate your bugs to the programmers Every bug deserves a report Report perceived quality gaps as bugs Never assume that an obvious bug has already been filed Report design errors Minor bugs are worth reporting and fixing Always report nonreproducible errors; they may be time bombs Make sure the bugs are fixed When the programmer says it's fixed, make sure it isn't still broken Bug reports should be closed by testers Don't insist that every bug be fixed. Pick your battles 25 Defect statistics (for defect tracking) 26

14 Defect statistics (for process improvement) 27 Test measurements and test management We cannot manage what we do not measure 28

15 The Goal/Question/Metric (GQM) method Technique to select the most appropriate metrics to use Select and describe goals quantitatively - Example: Understand product reliability For each goal, ask (a) question(s) to check if the goal was achieved - Example: How is the product behavior? For each question, identify (a) metric(s) to allow answering the question - Example: Mean Time Between Failures (MTBF) 29 Test measurements Measurements for monitoring test costs Number of test cases developed Size of the test harness (in KLOC) Time spent in testing Measurements for monitoring test adequacy Percentage of source code coverage, requirements coverage, etc. Measurements for monitoring product quality Number (or density) of pre-release defects found (by testing) Number (or density) of post-release defects found (by users) Overall number of defects found Measurements for monitoring test effectiveness Number of defects found by testing / Time spent in testing Number of defects found by testing / Number of test cases developed Number of defects found by testing / KLOC (of the software under test) Number of defects found by testing / Number of overall defects found (by testing and users) Evolution of previous measurements along time 30

16 Stop-test criteria (1) The project runs out of time and resources Not a good criterion The detection of a specific number of defects has been accomplished Not a good criterion (why?) All the planned test that were developed have been executed and passed All specified coverage goals have been met Example: 100% branch coverage in unit testing and all the requirements covered by system tests The rates of defect detection for a certain time period have fallen below a specified level Example: we stop testing when we find 5 defects or less, with impact equal to, or below severity level 3, per week 31 Stop-test criteria (2) Fault seeding ratios are favorable Fault seeding: defect injection based on historical defect data Example: at least 90% of the defects injected have been detected by the test suite A specified reliability level has been met Example: the mean time to failure (MTTF) is longer than 24h Estimated with statistical testing based on a usage/operational profile Requires a high-maturity test organization A specified fault-free confidence level has been met The software is fault free, or has less than a certain number of defects per KLOC, with a certain confidence level (example: 95%) Estimated with fault seeding techniques Requires a high-maturity test organization Use a combination of criteria, depending on each project! 32

17 Balance for optimal stop-test decision Stop testing too late: Wasteful of resources Delay time to market Increased costs Delayed schedules Stop testing too early: Defects remain and cause loss or damage to life and property Customer dissatisfaction High costs to repair Costs of hot line calls 33 Release decision Usually, it is not cost-effective to fix all bugs found before releasing the product Some bugs are very difficult to fix Difficult to locate (example: random bugs) Difficult to correct (example: third party component) Reduce overall risk Fix all high severity bugs Fix (the easiest) 95% of the medium severity bugs Fix (the easiest) 70% of low severity bugs 34

18 Release decision Classificação de Anomalias Grau Nível Critério Impacto na Gestão do Projecto 1 Muito Crítico Critical Não funciona; Impede a utilização correcta; Provoca anomalias graves. Impede a entrada do projecto em Produção. (sem autorização de Take-Off) 2 Crítico Severe Não funciona como requerido; Não impede a utilização, mas tem impacto nos processos. Pode ou não impedir a autorização de Take-Off, mas impede que o projecto seja Fechado. 3 Pouco Crítico Annoyance Não afecta a utilização normal, mas a falha tem de ser resolvida. Não impede a autorização de Take-Off. Pode não impedir o Fecho do projecto, se for opção de Manutenção Evolutiva. 4 Melhorias Cosmetic Novos pedidos face ao âmbito do projecto. Melhorias de Interfaces. Opção: Manutenção Evolutiva (fonte:, Universidade do Minho, 7 de Junho de 2003) 35 Curvas S O que são? Variáveis que representam progresso evoluem ao longo do tempo tipicamente segundo uma curva em S Test Cases Test Metrics Graph - Passed Days Exemplo de variáveis para monitorização do progresso dos testes: casos de teste realizados com sucesso ou ao número de defeitos encontrados (testes falhados) Permitem uma gestão objectiva, possibilitando estimativas de tempo e recursos necessários para o cumprimento do plano 36

19 Curvas S Porquê? Alguns pontos residuais em aberto que tornam, novamente o processo lento. TERMINAR??? Test Metrics Graph - Passed 100 Test Cases Days Aparecimento de alguns defeitos graves que impedem o andamento rápido 37 Curvas S Processo 1) Elaborar a Curva Teórica - representa um desempenho uniforme e óptimo (teórico) do progresso dos testes. Exemplo: (Day Number / Total Days in Test Effort) (Day Number / Total Days in Test Effort) + e^(3-8 * Day Number / Total Days) 2) Representar a curva obtida a partir dos resultados em termos de testes resolvidos ou defeitos encontrados 3) Comparar ambas, analisar desvios, identificar causas e propor acções correctivas Test Metrics Graph - Passed Test Metrics Graph - Defects Test Cases Passed Failures Days Num Passed Theoretical Curve Days Total Failures Theoretical Curve 38

20 Curvas S Causas e acções correctivas Test Metrics Graph - TCs Passed TCs Days TCs Passed Theoretical Curve Possíveis Causas: Surgimento de defeitos que impeçam a realização de vários casos de teste Alterações na equipa de testes Acções Correctivas: Reavaliação da prioritização dos casos de teste Utilização de mais recursos ou recursos com mais experiência 39 Curvas S Causas e acções correctivas Test Metrics Graph - Failures Failures Days Total Failures Theoretical Curve Possíveis Causas: Erros de estimativa Disponibilização de bug fixes ainda com problemas por resolver Acções Correctivas: Reavaliar as estimativas tendo em conta o tipo de aplicação ou projecto. Aumentar o esforço em testes unitários e/ou peer-code reviews 40

21 Zero Bug Bounce É uma técnica utilizada no âmbito da gestão e monitorização dos defeitos encontrados durante as actividades de teste Representação do nr. de defeitos em aberto no fim de cada dia de trabalho de teste e correcção de defeitos. Através do processo de monitorização é possível determinar: Quais os defeitos que necessitam de ser corrigidos Quando são corrigidos Se o produto está suficientemente estável para entrar em produção Zero Bug Bounce Open Defects Zero Bug Bounce Basicamente representa o ponto, durante a fase de teste e correcção de defeitos, em que a equipa de desenvolvimento corrigiu todos os defeitos em aberto ficando a par com a taxa de descoberta de defeitos da equipa de testes O Bounce (Salto) ocorre quando a equipa de testes descobre mais defeitos que obrigam novamente a actividades de correcção levadas a cabo pela equipa de desenvolvimento Ripple Effect (Ondulação) Os saltos vão-se tornando cada vez menores em amplitude até a aplicação estar suficientemente estável para entrar em produção. 42

22 Zero Bug Bounce Detecção de Problemas Zero Bug Bounce Open Failures Esta figura revela uma potencial situação de risco, nomeadamente, o atraso com que a equipa de desenvolvimento corrige os defeitos detectados. 43 Zero Bug Bounce - Detecção de Problemas Zero Bug Bounce Open Defects Há uma elevada possibilidade de ainda se encontrarem defeitos no sistema 44

23 Tool support Mercury Test Director Compuware QACenter Test Management Test Case Test Automation verifies verifies detects Configuration Management Version (or patch) deploys fixed in detected in Bug Tracking Bug develop Project Management Task develop execute fix implement affects Requirements Management Requirement (or feature) 45 References and further reading Practical Software Testing, Ilene Burnstein, Springer-Verlag, 2003 Chapter 7 (Test Goals, Policies, Plans, and Documentation) describes the IEEE standard Appendix II (Sample Test Plan) provides an example of a test plan documented according to the standard IEEE Standard For Software Test Documentation - IEEE Std Lessons Learned in Software Testing: a Context-driven Approach, Cem Kaner, James Bach, John Wiley & Sons Inc, 2002 Chapters 4 (Bug Advocacy), 6 (Documenting Testing) and 11 (Planning the Testing Strategy) Software Testing, Ron Patton, SAMS, 2001 Part V (Working with Test Documentation): Chapters 16 (Planning Your Test Effort), 17 (Writing and Tracking Test Cases), 18 (Reporting What You Find) and 19 (Measuring Your Success) Testing Computer Software, 2nd Edition, Cem Kaner, Jack Falk, Hung Nguyen, John Wiley & Sons, 1999 Chapters 5 (Reporting and analyzing bugs) and 12 (Test planning and test documentation) "Operational Excellence through Efficient Software Testing Metrics, Ramesh Pusala, Infosys, August 2006 "S-Curve & the Zero Bug Bounce: Plotting Your Way to More Effective Test Management", Questcon Tecnhologies, 2006 Compuware QADirector - Mercury TestDirector