Agile Metrics: Assessing Progress to Plans

Metrics provide perception into the belief of a functioning system, in addition to the efficiency of the processes which are in place. These insights allow oversight on the efficiency of a program because it delivers contracted capabilities. This weblog publish explores the position of metrics in a authorities program’s evaluation of progress in an iterative, incremental supply of a posh cyber-physical system.

The publish is predicated on our experiences on the SEI supporting packages utilizing incremental supply approaches tailor-made to the missions of the sponsoring organizations. Metrics which are key to incremental supply emphasize assessing progress to plan and assessing high quality, each of which embrace flow-based metrics influenced by Lean and Agile considering. Though this publish focuses particularly on assessing progress to plan, metrics that assist assessing high quality additionally present useful perception right into a program’s profitable realization of a system. The technical adequacy of the work is a main concern that spans each progress to plan and high quality.

Correspondence Between Evaluation Scope and Time Horizon for Selections

Assessing progress to plan for iterative, incremental supply usually includes measurements in opposition to near-, mid-, and long-term plans quite than in opposition to a single built-in grasp plan. Iterative-development strategies set up clear standards for the acceptability of labor at completely different ranges of element. These standards function the idea for assessing progress in opposition to plans at acceptable instances.

The hyperlink between the near-term accomplishments and the long-term plan for this system is a vital focus of program administration. This succession of plans from near- to mid- to long-term takes the place of a single built-in grasp plan. Every of those ranges of oversight has an aperture and focus distinctive to its related plan and the time horizon. The measurements that provide perception for every of those plans are described within the sections beneath, ranging from the underside of Determine 1 and transferring upward.

1. Engineering—These metrics are primarily based on assembly technical and engineering targets and efficiency targets. The technical adequacy of the detailed work is measured to make sure that well timed high quality work results in maturation of capabilities. These metrics are focused to an viewers of crew leads, subject-matter specialists, and engineering managers.
2. Program Administration—For packages aspiring to a continuous-delivery strategy, these metrics concentrate on monitoring product components delivered by a pipeline. Measuring the progress of tales, options, and capabilities supplies perception as to how nicely this system is progressing. These metrics are focused to an viewers of center and higher program-management personnel who oversee the supply of completed items over time, as they’re built-in into the system.
3. Acquisition—These metrics present perception in regards to the targets of the acquisition exercise. Such measures are nicely established within the U.S. Division of Protection (DoD) and are straight linked to long-term program estimates and contracts. These metrics are usually focused to an viewers of govt management and higher administration (e.g., materials leaders, senior materials leaders, program govt officers)—with program managers typically reporting the important thing knowledge. In addition they concentrate on time horizons which are longer than the timeline of undertaking administration.

Engineers and subject-matter specialists, program managers, and acquisition professionals should assess work progress from these three complementary views. The engineering perspective is considered one of technical progress. It’s a main focus in packages that advance state-of-the-art applied sciences. This system-management perspective assesses progress in opposition to the deliberate deliverables and timelines established for this system. Lastly, the acquisition perspective comes from the purpose of the enterprise for which this system was established. Totally different metrics and expertise are required to evaluate progress in these three complementary areas.

1. Engineering Progress Metrics

Assessing the maturation of advanced designs demonstrating the achievement of unprecedented technical outcomes requires an engineering perspective. Engineers derive main indicators primarily based on the applied sciences used and the character of the issue. Consequently, generic indicators, akin to productiveness and high quality, decide progress solely partially since they usually don’t account for the precise applied sciences in use. Engineering assessment of artifacts derived from necessities evaluation, design actions, and examples of implementation are vital inputs. Authorities personnel charged with engineering oversight measure engineering progress in opposition to near-term plans. The outcomes of those assessments finally drive program milestones.

Examples of critiques the place engineering progress is usually assessed embrace the next:

• Technical critiques—The main deliverables listed within the roadmap for every functionality will bear technical critiques by way of a programs engineering technical assessment (SETR) course of—a assessment assembly known as by the contractor through which authorities stakeholders consider the adequacy of the deliverables, typically in an iterative trend, to assist fast development of the work in smaller work packages. Authorities oversight personnel contemplate standards for attaining the extent of maturity implied by conventional milestones (e.g., preliminary design assessment [PDR] and vital design assessment [CDR]) because the technical work underway produces artifacts recording the accomplishments. These accomplishments then function enter to the subsequent engineering exercise.
• Demonstration occasions—Demonstrations of the options and capabilities as they mature make seen the technical pulse of functionality maturation. Whether or not demonstrating a small piece of performance in a take a look at harness or working a full simulated mission in a sturdy lab, builders present deeper perception into accomplishments by way of demonstrations than could be obtained from a written report of their success. Acceptable subject-matter specialists consider aims for particular performance or efficiency in demonstrations (e.g., on specified technical efficiency measures). Some organizations gather scores for utility (or worth) of the demonstrated performance.
• Verification occasions—Typically carried out in specialty labs, or with manufacturing {hardware} on a take a look at vary, the information collected at these occasions are foundational. Evaluation in opposition to baseline necessities, assembly documented test-coverage thresholds, and detailed evaluation of take a look at knowledge inform choice makers. Closure of the method is usually contingent on adjudicating the disposition of noticed anomalies and offering certification-related knowledge meant to guarantee the integrity of the work merchandise.

2. Program-Administration Progress Metrics

This system-management perspective, in distinction to the engineering perspective, is knowledgeable by a roadmap of milestones, assessment occasions, and deliverables which are designed to offer a gentle supply of knowledge to verify progress or set off motion. The most typical sources of those progress indicators embrace completion standing for items of labor (e.g., tales, options, use instances, capabilities) and their satisfaction of related completion standards, together with the related schedule and value knowledge.

The definition of accomplished supplies the mechanism to outline all the factors that have to be glad earlier than the merchandise being developed may be thought-about full. In observe, figuring out completion requires a sturdy understanding, agreed to by each authorities and contractor, to make sure that work is, actually, full. It ought to be famous that when a developer is utilizing all of the capabilities of an software lifecycle-management device, the device usually comprises the information wanted to produce most program-management progress metrics. The instruments can typically generate these metrics robotically, although there might be want of extra instruments to offer summary-level knowledge relying on attributes of this system. Measuring progress to program-management plans may embrace the next:

• Completion of options inside an increment—The time horizon many contemplate when planning an engineering or take a look at construct, with the related work merchandise to assist verification, is usually known as an increment. Typically, increments are made up of iterations that final one or two weeks. Many frameworks present a wealthy elaboration of those ideas. Measuring the variety of options that met the definition of accomplished throughout an increment is a quite common strategy. This strategy supplies a tangible indicator of verifiable progress and is usually displayed in a burndown chart. The satisfaction of acceptance standards, in addition to the definition of accomplished for options, usually signifies a supply of performance larger than the “unit stage” (implying a stage of integration). When mixed with different knowledge, function completion counts present insights as follows:
  • Evaluating the variety of options accomplished to the quantity dedicated to throughout that timeframe supplies perception into the estimation course of.
  • For a given functionality, evaluating the variety of accomplished options to the anticipated variety of options supplies a tough estimate of the completion of the potential. Not all options are of equal issue, and never all options required for the potential could also be evident firstly. Oversight personnel can use this similar strategy for different containers of labor, akin to epics and use instances.
• Completion of person tales inside an iteration (dash)—Whereas such granular knowledge is usually of extra use to the groups performing the work, the federal government can achieve perception by wanting on the consistency of story-completion knowledge. If there are wild fluctuations, it’s possible that there’s instability within the improvement course of and that progress shall be impeded or that tales fluctuate broadly in dimension/issue and warning is required to interpret the counts. The importance of integration for evaluating the utility of software program work merchandise typically makes progress to plan on the iteration stage a extra speculative endeavor. Methods for steady integration are supposed to deal with this important position of software program verification by way of improvement approaches and instruments that successfully enhance the technical scope of early testing actions, to speed up the subsequent tier of verification suggestions.
• Traceability matrix report—This report permits this system supervisor to verify for stakeholders that work is being carried out in accordance with a standard understanding of program priorities. Accounting for timelines and energy required for issues akin to upgrades to the event or take a look at environments, upkeep of evolving take a look at belongings, and different enablers of technical progress may be difficult. This report additionally supplies assurance to assist mixture measures for different audiences, in order that oversight personnel could make inferences relating to the completion standing of the contracted capabilities.
• Defects reported/escaped—Defects recognized after completion typically point out both a weak point within the definition of accomplished or failure to stick to that definition of accomplished. A bigger variety of defects which have escaped erodes confidence within the completion knowledge and accompanying metrics and, finally, slows progress when improvement effort have to be expended to right the defects.

3. Acquisition Progress Metrics

Metrics used for perception in regards to the targets of the acquisition plan typically embrace this system managers’ software of earned-value administration to iterative and incremental supply. An earned-value administration system (EVMS) is a program-management device for monitoring prices, schedule, and scope in opposition to an preliminary plan. The construction of the information usually aligns with the work breakdown construction (WBS) and grasp schedule. Traditionally, the earned-value construction was targeted on laptop software program configuration gadgets (CSCIs). This methodology, nonetheless, doesn’t readily assist an Agile-development effort. For Agile developments, utilizing the product backlog to populate the EVMS supplies monitoring with a technical perspective, as beneficial by Agile and Earned Worth Administration: A Program Supervisor’s Desk Information.

To facilitate supply of working software program, sizing constraints for every tier of the necessities breakdown (generally described with the phrases capabilities, options, and person tales) promote a constant framework for understanding progress. After these constraints are assigned, structuring the WBS to disclose progress for contracted capabilities turns into a lot simpler. Sizing constraints are essential for 2 functions: first to normalize the work packages, making certain that they fall inside a recognized dimension vary in order that they’re comparable (and counting them results in helpful summaries); and second so the WBS depicts a constant stage of element throughout the workload with traceability throughout the product components.

One strategy that exhibits promise is to make use of the just-in-time planning of Agile improvement at the side of the EVMS rolling-wave technique. This strategy permits acquisition personnel to include the most-current program efficiency and product information. Options dedicated to throughout increment-planning exercise are up to date within the EVMS to replicate the most recent price and dimension info as it’s baselined. Higher precision within the near-term plans present finer grained evaluation of acquisition progress, whereas the uncertainties of long run plans are the main focus of iterative assessments primarily based on most up-to-date progress. The connection to the product components, quite than CSCI construction of the software program merchandise, promotes concentrate on operationally related descriptions of progress achieved.

A cadence of quarterly increments composed of two-week iterations is widespread. When options are sufficiently small to be accomplished inside an increment, a significant stage of element for monitoring may be achieved—whereas assuring a recognized stage of verification and maturity for delivered merchandise. Progress in opposition to scheduled releases, into take a look at or to manufacturing, may be understood by way of evaluation of the feature-delivery progress, and main indicators for future-delivery efficiency.

Using well-established program oversight mechanisms is a subject deserving extra protection than the dialogue of EVMS discovered right here. Complete price of possession, and the operational efficiency of the system in use typically overshadow the event prices. Forecasting and monitoring in opposition to outcomes, expressed by way of the wants of the system person, present essentially the most significant measure of success. Search for a future weblog publish on this matter.

Creating an Efficient Measurement Program

Assessing progress to plan for iterative/incremental supply usually includes measurements that assess progress in opposition to near-, mid-, and long-term plans. An efficient measurement program identifies the aperture and focus distinctive to every related plan and time horizon. The gathering of those metrics depends on sustaining the arrogance of stakeholders by way of well timed communication of knowledge that demonstrates accomplishment of targets. The stakeholders’ information of program context and a shared understanding of the actions and outcomes driving the metrics allow the perception required to evaluate progress to plans.

In a future weblog publish, we are going to delve additional into metrics for assessing high quality.

Further Sources

Learn the SEI weblog publish, Agile Metrics: A New Strategy to Oversight.

Learn the SEI weblog publish, Do not Incentivize the Fallacious Behaviors in Agile Growth.

Learn the SEI weblog publish, 5 Views on Scaling Agile.

Watch the SEI webinar, Three Secrets and techniques to Profitable Agile Metrics.

Learn the SEI technical notice, Scaling Agile Strategies for Division of Protection Packages.

Take heed to the SEI podcast, Scaling Agile Strategies.

Learn different SEI weblog posts about Agile.

Learn different SEI weblog posts about measurement and evaluation.