1、1清 华 大 学 软 件 学 院2004年3月9日2uSoftware Metrics Classification uGoal-Based Framework uMeasurement Validation3nThe primary question in goal-based measurement “What do we want to know or learn?”instead of“What metrics should we use?”Because the answers depend on your goals,no fixed set of metrics is unive
2、rsally appropriate nInstead of attempting to develop general-purpose metrics,one has to describe an adaptable process that users can use to identify and define metrics that provide insights into their own development problem4Determining what to measure Determining how to measure Identifying entities
3、 Classifying entities to be examined Determining relevant goals Inquire about metrics Assign metrics5 Types of Entities Types of Attributes6 A collection of software related activities usually associated with some timescale Different SE processes:development,maintenance,testing,reuse,configuration a
4、nd management process,etc Any artifacts,deliverables or documents that result from a process activity Entities required by a process activity7 Attributes that can be measured entirely in terms of the process,product or resource itself separate from its behaviour Attributes that can be measured with
5、respect to how the process,product or resource relates to its environment through its behaviour8Process Entities AttributesPossible metricsdevelopmentprocesselapsed timeCalendar days,working daysmilestonesCalendar datesdevelopment effortStaff-hours,days,or monthstest processvolumeNumber of tests sch
6、eduledprogressNumber of tests executedNumber of tests passed9Process Entities AttributesPossible metricsdetailed designelapsed timecalendar days,working daysdesign qualityDefect density:number of design defects found in down-stream activities divided by a metric of product size,such as function poin
7、ts or physical source lines of code.maintenancecostdollars per yearstaff-hours per change request10Product entitiesAttributesPossible metricsSystemSizeNumber of modulesNumber of function pointsNumber of physical source line of codedefect densityDefects per KLOCDefects per function pointModuleLengthP
8、hysical source lines of codeLogical source statementsPercent reusedRatio of unchanged physical lines to total physical lines,comments and blanks excluded11Product entities AttributesPossible metricsUnitNumber of linearly independent flowpathsMcCabes complexityDocument LengthNumber of pagesLine of co
9、deStatement typeType namesProgramming languageLanguage name12Resource EntitiesAttributesPossible metricsAssigned staffTeam sizeNumber of people assignedExperienceYears of domain experienceYears of programming experienceCASE toolsTypeName of typeIs used?Yes/no(a binary classification)TimeStart date,d
10、ue dateCalendar datesExecution timeCPU clocks13ProcessesInternal attributesExternal attributesConstruct SpecificationsTime,effort,number of requirements changes,Quality,cost,stability,Detailed DesignsTime,effort,number of specification faults found Cost-effectiveness,cost,TestingTime,effort,number o
11、f bugs found,Cost-effectiveness,stability,cost,14ProductsInternal attributesExternal attributesSpecificationsSize,reuse,modularity,functionality,syntactic correctnessComprehensibility,maintainability,DesignsSize,reuse,modularity,coupling,cohesiveness,functionalityQuality,complexity,maintainability,C
12、odeSize,reuse,modularity,coupling,functionality,algorithm complexity,control-flow structurednessReliability,usability,maintainability,Test DataSize,coverage level,Quality,15ResourcesInternal attributesExternal attributesPersonnelAge,price,Productivity,experience,intelligence,TeamsSize,communication
13、level,structuredness,Productivity,qualitySoftware Price,size,.Usability,reliabilityHardwarePrice,speed,memory size,Reliability,OfficesSize,temperature,lightComfort,quality16 GQM Approach SEI CMM GQM and CMM17Goal-Question-Metric(GQM)approach provides a framework for deriving metrics from organizatio
14、n or business goals References V.R.Basili and D.Weiss(1984),A Methodology for Collecting Valid Software Engineering Data,IEEE Trans.Software Engineering,vol.10,pp.728-738.V.R.Basili and H.D.Rombach(1988),The Tame Project:Towards Improvement-Oriented Software Environments,IEEE Trans.Software Engineer
15、ing,vol.14,pp.758-773.18GQM ProcessG1Meet PlannedProject ScheduleG2Develop ZeroDefect SoftwareG3Provide Data forBetter EstimatesQ1.1Q1.2Q2.1Q3.1Q3.2M1M2M3M4M5M619 List major goals of development or maintenance project.Derive from each goal the questions that must be answered to determine if the goal
16、s are being met.Decide what must be measured in order to be able to answer the questions adequately.20 To(characterize,evaluate,predict,motivate,etc.)the(process,product,model,metric,etc.)in order to(understand,assess,manage,engineer,learn,improve,etc.)it.Example:Evaluating maintenance process in or
17、der to improve it.21 Examine the(cost,effectiveness,correctness,defects,changes,product metrics,etc.)from the viewpoint of the(developer,manager,customer,etc.)Example:Examine the cost from the viewpoint of the manager.22 The environment consists of the following:process factors,people factors,proble
18、m factors,methods,tools,constraints,etc.Example:The maintenance staff are poorly motivated programmers who have limited access to tools.23Evaluate working environment in order to identify opportunities to improve the productivity of our development teamExamine the ratio of work time to break time of
19、 our employees,the accommodations,incentives,extra-curricular activities offered,workspace(room and desk size,ventilation)where our employees work from the point of view of the employees themselvesEnvironment and constraints:Payroll applications programming in C+100 software developers with 5 or mor
20、e years experience in C+Customers are businessesDo not maintain a reusable module databaseExamine new projects completed and sold from 1/1/1998 to 31/12/200224Evaluate the impact of various CASE tools on the productivity of the development teamExamine the effectiveness of using various CASE tools to
21、 help in the development of our product from the point of view of the developers and testersEnvironment and constraints:Payroll applications programming in C+100 software developers with 5 or more years experience in C+Customers are businessesDo not maintain a reusable module databaseExamine new pro
22、jects completed and sold from 1/1/1998 to 31/12/200225We can relate the GQM template to the Entity&Attribute framework.A Goal or question can be associated with at least one pair of Entities and Attributes.Once a measure is obtained,it can be related back to questions and goals.26Goal:Better code in
23、spection Subgoals:Better inspection planning Better monitor and control of the code Improving code inspection27 Q1:How much does the inspection process cost?Q2:How much calendar time does the inspection process take?M1.1:Average effort per KLOC M1.2:Percentage of re-inspection M2.1:Average effort pe
24、r KLOC M2.2:Total KLOC inspected28 Q1:What is the quality of the inspected software?Q2:What degree did the staff conform to procedure?Q3:What is the status of the inspection process?M1.1:Average fault detected per KLOC M1.2:Average inspection rate Average preparation rate M2.1:Average inspection rat
25、e M2.2:Average preparation rate M2.3:Average lines of code inspected M2.4:Percentage of re-inspection M3.1:Total KLOC inspected29 Q1:How effective is the inspection process?Q2:What is the productivity of the inspection process?M1.1:Defect removal efficiency M1.2:Average fault detected per KLOC M1.3:
26、Average inspection rate M1.4:Average preparation rate M1.5:Average lines of code inspected M2.1:Average effort per fault detected M2.2:Average inspection rate M2.3:Average preparation rate M2.4:Average lines of code inspected30Total KLOC inspected11000NiiLOC inspect Average LOC inspected1000Total KL
27、OCinspectedN Average perparation rate11000NiiTotal KLOC inspectedpreparation timeNumber ofi i ns pect orsAverage inspection rate11000NiiTotal KLOC inspectedInspection duration 31Average effort per KLOC1NiiInspection effortTotal KLOC inspected Average effort per fault detected11NiiNiiInspection effor
28、tTotal f aul t f oundAverage faults detected per KLOC1NiiTotalTotal KLOC inspected f aul t s det ect ed Percentage of reinspectionsintNumber of reinspection disposNumber of inspections 100 Defect removal efficiency1100NiiTotalTotal Coding faults dectected faul ts detected 32Three goals A:Maximize cu
29、stomer satisfaction B:Minimize engineering effort and schedule C:Minimize defects33 QA1:What are the attributes of customer satisfaction?MA1:Functionality,usability,reliability,performance,supportability.QA2:What are the key indicators of customer satisfaction?MA2:Survey,quality function deployment(
30、QFD).QA3:What aspects result in customer satisfaction?MA3:Survey,QFD.QA4:How satisfied are customers?MA4:Survey,interview record,number of customers severely affected by defects.QA5:How many customers are affected by a problem?MA5:Number of duplicate defects by severity34 QA6:How many problems are a
31、ffecting the customer?MA6-1:Incoming defect rate MA6-2:Open critical and serious defects MA6-3:Defect report/fix ratio MA6-4:Post-release defect density QA7:How long does it take to fix a problem?MA7-1:Mean time to acknowledge problem MA7-2:Mean time to deliver solution MA7-3:Scheduled vs.actual del
32、ivery MA7-4:Customer expectation of time to fix 35 QA8:How does installing a fix affect the customer?MA8-1:Time customers operation is down MA8-2:Customers effort required during installation QA9:Where are the bottlenecks?MA9:Backlog status,time spent doing different activities 36 QB1:Where are the
33、worst rework loops in the process?MB1:Person-months by product-component-activity.QB2:What are the total life-cycle maintenance and support costs for the product?MB2.1:Person-months by product-component-activity.MB2.2:Person-months by corrective,adaptive,perfective maintenance.QB3:What development m
34、ethods affect maintenance costs?MB3:Pre-release records of methods and post-release costs.37 QB4:How maintainable is the product as changes occur?MB4.1:Incoming problem rate MB4.2:detect density MB4.3:Code stability MB4.4:Complexity MB4.5:Number of modules changed to fix one detect QB5:What will pro
35、cess monitoring cost and where are the costs distributed?MB5:Person-months and costs QB6:What will maintenance requirements be?MB6.1:Code stability,complexity,size MB6.2:Pre-release defect density 38 QB7:How can we predict cycle time,reliability,and effort?MB7.1:Calendar time MB7.2:Person-month MB7.
36、3:Defect density MB7.4:Number of detects to fix MB7.5:Defect report/fix ratio MB7.6:Code stability MB7.7:Complexity MB7.8:Number of lines to change 39 QB8:What practices yield best results?MB8:Correlations between pre-release practices and customer satisfaction data QB9:How much do the maintenance p
37、hase activities cost?MB9:Personal-months and cost QB10:What are major cost components?MB10:Person-months by product-component-activity QB11:How do costs change over time?MB11:Track cost components over entity maintenance life-cycle.40 QC1:What are key Indicators of process health and how are we doin
38、g?MC1:Release schedule met,trends of defect density,serious and critical detects.QC2:What are high-leverage opportunities for preventive maintenance?MC2.1:Defect categorization MC2.2:Code stability QC3:Are fixes effective with less side effects?MC3:Defect report/fix ratio 41 QC4:What is the post-rel
39、ease quality of each module?MC4:Defect density,critical and serious detects.QC5:What are we doing right?MC5.1:Defect removal efficiency MC5.2:Defect report/fix ratio QC6:How do we know when to release?MC6.1:Predicted defect detection and remaining defects.MC6.2:Test coverage.42 QC7:How effective is
40、the development process in preventing defects?MC7:Post-release detect density QC8:What can we predict will happen post-release based on pre-release data?MC8:Corrections between pre-release complexity,defect density,stability,and customer survey data.QC9:What defects are getting through and there cau
41、ses?MC9:Detect categorization 43Meaningful?Operations on metrics?Direct or indirect metrics?Objective and subjective metrics?Software metric classification?Goal-based framework?44Further reading 1.R.E.Park,W.B.Goethert,W.A.Florac.Goal-Driven Software Measurement-A Guidebook.CMU/SEI-96-HB-002,(189 pages)2.J.Barnard,A.Price.Managing Code Inspection Information.IEEE Software,1994.(11 pages)45
