Table of Contents
4.4.2 Non-functional Requirements Assessment
Assessing the Alternatives
Non-Functional requirements are by nature, hard to measure and hard to assess for compliance. The complexity of the assessment problem is compounded because products or systems as well as the environment they operate within are not static. Therefore, an assessment that is done today, may no longer be accurate in the future. When assessing Commercial Off-The-Shelf (COTS), Government Off-The-Shelf (GOTS), Modified Off-The-Shelf (MOTS) or NATO Off-The-Shelf (NOTS) or the inclusion of Open Source Software (OSS) the potential for changes (particularly enhancements) needs to be assed also.
One way to accomplish this is to provided an assessment which is weighted for the “ease of implementation” for new features. For example, a system or product may not have done much to support 4.3.4.1 Confidentiality, but the vendor of the product must determine that it is an easy upgrade to add it to the product. On the other hand, the support for confidentiality might be extremely difficult. Sometimes, the feature may be easy to solve but requires time and money to accomplish. As a potential stakeholder, they can direct resources to help overcome this shortfall.
The Vendor's Assessment
The Vendor's assessment team must discuss with each other and use the DIDO-RA workbook to determine the weight of the requirement.
This is done by weighing each property, related to the Requirement, by giving a score between 1-100. 1 representing an easy development process, and 100 being impossible now, merely very difficult in a couple years.
Publish the assessment, and notify the vendor..
The Stakeholder's Focus Group
The Vendor's stakeholder assessment team must create a focus group of stakeholder's. This group is used in conjunction with the stakeholder assessment team to determine the importance of the Requirement and potentially direct more or less resources to this area.
This is done by rating each property related to the requirement with either a '+' option representing a 'more important' status, or a '-' option representing a 'less important' status. On the Coefficient sheet choosing a '+' will cause a '-' to be put in the opposite cell in the table. EX: cell(1,2)='+' means cell(2,1)='-'. Also properties can't be compared to themselves, these cells have lines drawn through them to represent this.
At the end of filling out each table. Each row is counted for '+', the number of '+' in a row is equal to the coefficient for that property.
When completed publish the assessment and notify the stakeholder's.
The Stakeholder's Assessment Team
Once the assessments and importance coefficients are set, the DIDO-RA is passed to the stakeholder's assessments team. Here they must take the factor weighting and the coefficients to create the Figure of Merit (FoM). These figures are then published to assessment reports that are given to the Vendor, and/or given back to the stakeholder's assessment team for further review.
In the DIDO-RA workbook after the Coefficient/Factor Weight sheets are filled out, those values associated with the requirement, are then sent to the corresponding Requirement Sheets in the workbook.
This data is then used to create the FOM(Figures of Merit). The formula to calculate the FOM = (C1)(FW1) + (C2)(FW2)+ (Cn)(FWn) for each property. These Property FOMs are then added into a Requirement FOM.
NOTE: Comparing Requirement FOMs from one vendor to another will cause a skewed comparison. To compare between vendors you must break the Requirement FOMs into its components. The components of Requirement FOM are the Property FOMs.
The Stakeholder's Award Team
The stakeholder's receive the Assessment Reports as well, if they deem the results satisfactory they will choose the product just reviewed.
