- “Towards Software Excellence” (TSE) is a web-based process assessment facility established by the UK's National Computing Centre (NCC). TSE provides software developers with a simple and easy-to-use means of self-assessing the capability of their development processes. They can understand the “good practice” expected by TSE, compare their processes against TSE's process model, identify their "strengths and weaknesses" and then use this knowledge to improve. TSE is especially suited to small and medium sized enterprises (SMEs), also to development teams or groups within larger organisations. It can provide a first step towards more formal CMM or SPICEbased assessments. Since 2003, over 1500 TSE assessments have been performed. The database of TSE assessments has been analysed to identify the overall strengths and weaknesses of its users' processes, as well as which processes and practices are of most interest to them. Good performing processes include Problem Resolution and Project Management, while low performing processes include Measurement and Process Improvement. TSE's users are most interested in assessing the performance of their “customer-facing” development and acquisition practices and least interested in assessing their improvement and legal practices.
Achievement of good practice and effective processes for software development - especially by smaller organizations lies at the heart of the Towards Software Excellence (TSE) initiative. Established in 2001 by the UK’s National Computing Centre (NCC), with support from government and industry bodies, TSE provides an informal but systematic process selfassessment tool, as well as related advice and support for process improvement. Accessible over the internet, it is aimed at helping software developers and IT enterprises to understand and assess the capability (“strengths and weaknesses”) of their processes and practices - and hence encourage them to improve them further.
The process model used by TSE can be traced to the
ISO15504 (SPICE) standard [
Established in the 1960’s by government, the UK’s National Computing Centre (NCC) is an independent not-forprofit organisation dedicated to improving software and IT practice. QAI Europe Ltd is a small specialist provider of expertise in the application of software process capability models and methods, and helped to define the TSE model.
TSE comprises two key components, both web-based: • a route map through best practice, based on international software process standards • a self-assessment tool, by which a project, team or group can evaluate and record the capability of its software/IT practice
By repeated use of TSE, users can subsequently monitor their progress towards improved process capability - and ultimately achieve software excellence. Users can evaluate separate projects, teams or groups to compare performance or highlight differences in effectiveness.
Assessments entered by users are held on a central database, from which the tool can creates benchmarks for assessment reports, and analyse the overall performance of TSE users. Individual assessments are confidential and are not available to other TSE users or third parties.
The intended benefits for companies using TSE include: • better understanding of the capability of processes and practices, especially their “strengths and weaknesses” • comparison of their performance with equivalent developers or sectors, by means of TSE’s good practice definitions and benchmarks created from other TSE assessments • provision of an intermediate step towards adopting more formal process capability assessment, in particular the ISO15504 (SPICE) or CMM approaches • a flexible, low cost method of understanding process capability and supporting process improvement, e.g. in demonstrating the “before and after” levels of practice
TSE is free from the pressures of certification, and should be seen as complementing existing schemes such as ISO9001/TickIT and the Capability Maturity Model (CMM) – it is hoped that TSE will encourage its users to take up such schemes when they feel the time is right and they have the resources available.
The take-up of TSE has been good - over 1500 assessments since its release. This is believed to be due to the recognition by software companies of the need to understand and improve their processes and practices, supported by the ease of use of the TSE web site and the low cost of access. Software engineers, project managers, quality specialists and others can “road test” it informally and without needing to obtain approval for the time and resources normally needed to start along the route to formal process assessment. 2
The TSE best practice information (“Route Map”) is structured according to a process model derived from international standards (especially ISO12207) and other sources of best practice. Users assess their current situation by answering structured sets of questions (Forms) for up to 25 Topics. Topics belong to 6 Categories (process areas), as listed in Table 1 below.
Users can select any number of Topics for an assessment. Using the web-based tool, they answer questions about their detailed practice within these topic areas. The scores achieved against the TSE definitions of Topics and Categories are calculated by the tool and, if required, can be related to benchmarks constructed from analysis of the TSE database. A standard format of Assessment Report is also available.
To help users to improve their processes, a gap analysis report can be generated, indicating topic areas for consideration as well as suggestions as to what improvements are needed to implement fully the best practice defined within TSE.
(The complete set of TSE Categories, Topics and Forms is listed in Appendix A below.) 3
TSE’s Categories (process areas), Topics (processes) and Forms (practices) are accessed via a Route Map that is browsed using a “directory tree”. This tree lists the contents of the Route Map and provides access to the sets of Questions (Forms) for each practice that is going to be assessed.
On the web page illustrated in Figure 1 below, the “directory tree” is visible on the left of the screen, alongside the questions selected by the user - in this case the Form “Approach to Buying” within the Topic (process) “Acquisition” within the Category (process area) “Customer-Supplier”.
The complete set of Questions in this example is listed in Table 2 below. Depending on the question, the user decides whether the stated practice is performed: • always, usually, sometimes, rarely or never (or is not applicable) • yes or no (or is not applicable). Question 1 2 Does your company have and use an 'acquisition strategy'? i.e. Does your company use a defined and repeatable approach to buying on behalf of customers? Does the strategy require you to state the needs of and goals for acquisition? (Why you are buying in the products or services?) Do you follow your company’s acquisition strategy? Does the strategy require that clearly defined criteria should be used in determining whether an acquisition is the right one? Do you review the acquisition strategy, its goals and established criteria in the light of experience? Are details such as the expectations, the responsibilities and the liabilities of you, your company and your supplier written into a contract? Do you monitor the acquisition process to ensure that it sticks to costs estimates, time schedules and quality criteria - both for the service/product and its supplier?
The answers are combined using a weighting scheme which prioritises the questions as appropriate. This weighting takes into account the number of questions on the Form (between 4 and 10, depending on the Topic) and the relative importance of each question. Therefore the blocks of questions within each Topic check how close the user’s declared practice is to TSE’s best practice definition – the weakest areas of practice which score lowest are obvious candidates for improvement.
While the TSE user has to decide now much justification they have for the answer provided, as far as possible TSE’s questions point to a typical source of “objective evidence” which should be used – in the example above, this evidence is a defined acquisition strategy and a written contract.
TSE explains the best practice it expects by means of a series of guidance pages, also accessible from the Route Map. These pages support assessment by helping a user to understand why particular questions are being asked and support process improvement by helping the user to determine what changes in practice are needed if improved capability is to be achieved.
In the example page illustrated below (Fig.2), the guidance summarises the sub-processes within the CustomerSupplier process area, and then describes the recommended set of activities needed (e.g. starting with “Preparing to buy-in products and services - Acquisition Preparation”). It also identifies the expected outputs which will be achieved (e.g. a set of requirements and strategy or approach to be applied).
By answering questions in this structured way, the TSE user can obtain a profiled set of guidance and therefore an overall view of the capability of their software/IT practices. When the questions are answered using the web tool, TSE switches off irrelevant questions, so that if a question is answered negatively, questions on further related details are ‘deactivated’.
As more users add their (anonymous) profiles to the database, TSE is able to generate better benchmark comparisons using profiles from similar projects or companies. Users’ results are saved so that they may revisit the site (e.g. after a few months) and compare the capability of their current activities with before they improved their processes by using advice from the Route Map.
TSE can also be used as a project-oriented facility by a company running different types of project or using different development processes. Thus TSE can be used to compare different development teams or types of project, with each performing their own assessment (which should also help to make TSE attractive to larger organizations).
TSE can provide an overall improvement report, presenting assessment results and profiled guidance obtained from the Route Map. This report includes a summary which can be used by software suppliers as an ‘honesty box’ to share with customers, partners and financial backers to show their current capability as part of their commitment to quality and improvement.
While TSE was set up primarily to help software developers to understand how well their individual practices compare with industry best practice, the assessment results held in the TSE database can be analysed to provide answers to questions such as: • Which process areas and practices are generally the strongest? • Which process areas and practices are generally the weakest? • Which processes and practices are of most interest to
TSE users? • Which processes and practices are of least interest to TSE users?
By taking a general view of the software practice declared by TSE users, we can assemble a profile of typical capability of their software practice and therefore illustrate the kind of information that an individual user can obtain by using TSE.
The topics (processes) which achieved the highest scores from TSE's users are listed below. The number of assessments varies because most users assess the topics which are of most interest to themselves (although a few users have assessed against all the topics). The scope of these topics is as follows:
Within these processes, the best areas of practice identified by TSE assessments are: • making use of their best practices and expertise • having a system for recording customer requests for support • improving the product or service as a result of customer support experience • initiating projects (defining the scope, estimating size and resources, producing plans) These results are consistent with the often perceived strengths of smaller developers (SMEs) - responsiveness and good communication, both externally and internally - it is particularly important for developers to pay attention to supporting their customers effectively if they are to grow and survive.
It is also good to see a high level of practice at initiating and planning projects (although this does not necessarily mean that problems during projects are avoided).
Perhaps of more value for TSE users is knowing for which topics practice is lowest, and therefore where there is the greatest opportunity for improvement. The least performing practices are listed below.
Topic (Process) Process Assessment Supplier Selection Approach to Buying Process Improvement
Measurement
Number of assessments
The weak performance in Approach to Buying and Supplier Selection (both part of the Acquisition process) may be due to TSE good practice expecting a significant level of formality and planning. Formality can be more difficult to achieve in smaller organizations where the need for structured communication is less obvious and acquisitions are performed less often.
The weak performance of Measurement, Process Assessment and Process Improvement may arise because it can be difficult to find the resources to perform measurement and improvement systematically - a situation which is not helped when smaller organizations have fewer projects and contracts to handle and therefore less information to measure. This is a potentially significant barrier to effective process improvement – how can small developers improve systematically when they have limited data, time and resources?
(Measurement and process improvement are key requirements of the ISO9001 standard for quality management, adopted by a proportion of software developers and IT providers, both large and small. Clearly some developers can achieve acceptable practice in this area.)
It is interesting to note that one of the best performing practices is improving products or services as a result of customer requests (customer support process). This suggests that improvement is driven primarily by product and service related issues, not by process related issues.
It has been interesting to see for which practices TSE assessment was used most – and hence which are of most interest to TSE users. Since TSE can be used “when time permits”, the practices chosen for assessment should be a good indicator of its users’ main interests.
Form (Practice) Delivering software to customers Approach to Buying System Requirements Analysis and Design Software Design Collecting and Understanding the Requirements
Topic (Process) Supply Acquisition Development Development Requirements Elicitation
Number of Assessments 148
These results suggest that customer-facing practices have particular interest for TSE users - establishing and understanding the requirements, buying any products or services, then analysis and design of the software and finally ensuring successful delivery.
The other side of the coin is which practices are of least interest to TSE's users. These are in two main topic areas: Improvement and Legal.
It is possible that TSE users realise their practices are poor in these areas and therefore do not attempt to perform an assessment, but the low proportion of assessments for these practices suggests they have little immediate interest – and that few users have specific process improvement practices in place.
Earlier analysis of the TSE database had indicated low interest in verification and validation processes. While these processes remain a “minority interest”, more users have now assessed them, hopefully indicating increasing interest in this important part of the software development process.
(An initial survey during TSE's definition phase showed interest in adding legal requirements to the scope of the TSE process model. It is therefore surprising to note that in the implemented TSE, the 'Legal' category received the least assessments!)
The information from TSE suggests that the typical TSE user (IT/software developer) has the following “strengths and weaknesses” in their processes and practices • good supply, customer support and operations practices, especially problem handling • good practices for evaluating and understanding customer requirements and determining operational requirements and software specifications • effective practices for project management, including planning and estimating • weak management of acquisition, particularly supplier selection and buying (but some good practices for performing acquisition) • good response to customer support problems by improving products and services • little interest or capability for implementing process improvement
It is pleasing that project management and development practices are not a general cause for concern - although some of the individual assessments by TSE users indicate low capability in these practices.
After a pilot project showed its feasibility, the full TSE facility has gained the backing of a considerable number of UK professional and trade associations. The website has been well received by TSE’s users and many have performed multiple assessments. As might be expected with a standard assessment method, some users have criticised the relevance of the questions to their specific IT products or development needs; however, one of the features of TSE is that you can pick out the parts that are meaningful for your activities and still get value from a partial assessment.
While the majority of users have been from the UK, about a third are from Europe or elsewhere, and most types of software developer, industrial and commercial sectors are represented. (There are no barriers to use of the facility by users from outside the UK.) The tool is available for licensing and/or further development if any national organization outside the UK with a similar independent status to NCC wished to establish an equivalent scheme.
QAI Europe has used this tool for two clients in Portugal who are participating in a EC-supported RTD project, to monitor the capability of their development practices and to show the impact of improvements made during the project. To help teams, groups or organizations use TSE, an “assisted assessment” is available, which helps them get quick, maximum benefit from applying TSE, interpret it to their specific practices and ensures they use it in the most effective way for their needs and situation.
The running costs of the TSE facility and website are now supported by user subscription (£35 + VAT for unlimited assessments for one year). If they wish, users from outside the UK can compare their practice against benchmarks derived from assessments performed by mostly UK software companies. As a significant contributor to international standards such as ISO9001 (quality management), ISO15504 (software capability assessment), ISO17799 (IT security) and ITIL (IT service delivery and management), the UK software industry is among the world leaders in best practice.
The TSE facility can be accessed at www.ncc.co.uk/tse or www.software-excellence.org.uk
This Appendix lists the complete set of Categories, Topics and Forms used by TSE. On the website, each Form is a page of questions which are used to assess a practice.
Category Customer - Supplier
Topics Acquisition Manage
ment Organisation Legal Ensuring a Consistent Future Getting it Right Conforming to Specification Fit for Purpose Checking it Together Independent Review Sorting out Problems Achieving Business Goals Within Time and Cost The Best Possible Result Minimising the Threats Organising to meet the needs The Right Knowledge and Skills The Right Tools Dials as Well as Levers! Maximising the Value Contractual Relationships Legislation Employers
Website
The authors wish to thank A, B, C. This work was supported in part by a grant from XYZ.
Engineer
ing The Route to Improvement Process Establishment Process Assessment Process Improvement Writing it Down