10.1 Evaluating at different stages of the project
A project is often shaped through discussion among those developing the vision and direction of the project. They may agree in general terms about what is to be achieved, but have to make a number of choices before deciding how to proceed. It may be important to allow time for different views to be heard and considered, and for attitudes to change and – hopefully – converge.
Many stakeholders may become involved in evaluation in the early stages of a project, in imagining what it might mean for them and how it might present advantages or disadvantages. The anticipated impact of the project can be usefully evaluated in the early stages, to ensure that the investment of energy and resources can be expected to achieve the intended results.
10.1.1 Evaluation during the planning stage
Evaluation at this stage is usually concerned with whether plans represent good value for money.
It may be appropriate to evaluate inputs to the project, to ensure that their quality and quantities are sufficient to achieve the objectives. In large building projects, many specialist tasks are subcontracted. Specifications are developed, and potential contractors are invited to tender for work. The element of competition can lead to problems if some tenderers are over-anxious to win contracts. They may be tempted to offer very low prices or attractively fast times, making no allowance for setbacks or delays outside their control. If they fail to meet their deadlines there may be a knock-on effect, when other tasks cannot start till they finish.
Those evaluating tenders need to be able to estimate the budget and timing required for a particular piece of work: the cheapest is not necessarily the best, nor the one that promises the fastest completion.
Even when a tendering process is carried out with care, to ensure that the contract results in a partnership that will be successful for everyone, it is impossible to predict all the potential risks. Some contingency sum might be agreed, but many contracts also incorporate a process for negotiating liability for additional costs, when these arise unexpectedly.
Risk is the chance of something occurring that has an adverse effect on the project. Many risks can be foreseen and identified. For example, if the project involves development of computer-based systems, time needs to be allowed for ‘de-bugging’ once the systems are installed.
The main categories of risk can be summarised as follows.
Main categories of risk:
- physical: loss of or damage to information, equipment or buildings as a result of an accident, fire or natural disasters,
- technical: systems that do not work or do not work well enough to deliver the anticipated benefits,
- labour: key people unable to contribute to the project because of, for example, illness, career change or industrial action,
- political/social: for example withdrawal of support for the project as a result of change of government, a policy change by senior management, or protests from the community, the media, patients, service users or staff,
- liability legal action or the threat of it because some aspect of the project is considered to be illegal or because there may be compensation claims if something goes wrong.
10.1.2 Evaluation during implementation of a project
At this stage the project activities are monitored to determine how their timing, quality and cost match the plan. The results of this monitoring are reviewed to see whether the plan needs to be modified.
New environmental conditions may indicate the need to change the organisation's strategic direction. It might be necessary in that case to re-align the project, so that the outcomes relate to the new direction. In some cases it may be necessary to abort the project, if it is no longer appropriate or likely to make a useful contribution.
Incorporating this kind of evaluation as part of the project plan (formative evaluation) can considerably enhance the likelihood of achieving useful outcomes. If formative evaluation is to take place, it should be integral to the project's design. It can facilitate a more organic change process, with testing and refining built in as the project progresses.
However formative evaluation can increase the complexity of a project, because of the need to timetable an extra set of deadlines. It will also add new items to the risk log, particularly the risk of delays. If formative evaluation results in decisions to make significant changes to the project, it may lengthen the time-scale, increase the budget or necessitate additional quality measures.
10.1.3 Evaluation at the end of a project
Different types of evaluation may take place at the end of a project. A common one is determining the extent to which the project outcomes have been achieved.
This is often done in a meeting of the sponsor, key stakeholders and project team leaders, and sometimes informed by reports from key perspectives. An evaluation of this nature may be the final stage of the project, and the main purpose might be to ensure that the project has met all of the contracted expectations and can be ‘signed off’ as complete.
A different type of evaluation may be a review of the process, with the purpose of learning from experience. This is often done by comparing the project plan with what actually happened, to identify all the variations that occurred, in both processes and outcomes. The aim is to draw out how to avoid such variations in future projects or how to plan more effectively for contingencies.
Although monitoring takes place throughout a project, evaluation based on the information thus gained is likely to happen at the end of the project, in a final summative evaluation which identifies:
- what the project has achieved;
- the aspects of the project that went well;
- the aspects that went less well;
- things that you would do differently next time.
10.1.4 Designing a formal evaluation
Reviews and informal evaluations will often be sufficient, but sometimes a formal evaluation will be needed.
A formal evaluation can be both time-consuming and expensive, and so must be carefully planned. There are many different ways of carrying out such an evaluation, and the process chosen will influence the attitudes of those involved and whether their response to the project is positive or negative.
There are a number of decisions that have to be made in designing an evaluation.
- Decisions about goals: what is the evaluation for? who wants the evaluation?
- Decisions about focus: what is to be evaluated?
- Decision about methods: how and from what sources will the information be gathered?
- Decisions about evaluation criteria: how will criteria be set, and by whom?
- Decisions about process: who will do the evaluation, and who will manage the process?
- Decisions about application: what use will be made of the findings?
All of these decisions relate to the overall purpose of the evaluation; and, if each is considered as part of the design process, the answers will enable the process to be planned.
10.1.5 Collecting and interpreting data
In many projects it can be difficult to make comparisons with other seemingly similar projects. However, there may be quality standards that can be used for one of more of the outcomes, perhaps alongside different targets for time-scales and resource use. Benchmarks are another possible source of comparative data; they have been established for many processes, and data are available from industry, sector and professional support bodies.
A number of potential methods could be used to collect and analyse data:
- records kept for monitoring purposes to make comparisons between activities;
- records of meetings and other formal events can provide useful data on the sequencing of decisions and discussion of issues;
- interviews, questionnaires or focus groups;
- observation or role-play might provide useful insights into how activities are carried out.
The balance between qualitative and quantitative data is important, because each can supplement the other and it is difficult to achieve an overall picture if only one type of data is used. The methods you choose to collect information will be influenced by the availability of resources, taking into account:
- the cost of obtaining the information, in relation to its contribution to the evaluation;
- the number of sources from which information should be obtained if sufficient viewpoints are to be represented to ensure that the results are credible;
- the time it will take to obtain and analyse the information;
- the reliability of the information obtained;
- the political aspects of the process – for example, some ways of gathering information may help build up support for the evaluation.
Direct contact with those involved in the project might be the only way in which sufficient information can be obtained to make the evaluation worthwhile.
10.1.6 Analysing and reporting the results
When planning what data to use in the evaluation it is helpful to consider how the data will be analysed.
Usually, there are a lot of data, perhaps in several different forms. If you have set clear objectives, it should be possible to identify the data that are relevant to each issue. It is usual to follow the steps below:
- consider numbers, for example how much has been achieved at what cost;
- consider quality, whether appropriate and not too high or low;
- seek out both positive and negative evidence;
- make comparisons;
- look for patterns in the evidence.
It can be very time consuming to analyse data from interviews and observations but, if the purpose of the analysis is clear, then it is possible to focus only on the relevant material.
It may be that several different evaluation reports must be prepared on completion of a project:
- the client/sponsor report;
- what has been learnt from the project;
- different types of evaluation report for different stakeholders.
For example, some funding bodies require to be told how their funding contributed to the success of a project, and so need a report relating only to one aspect. It is usually for the manager of a project to identify the number and types of report that are required, and to ensure that they are prepared and presented appropriately.
You need to consider the audience and use language that they will understand, avoiding unfamiliar jargon. The report is likely to include:
- an executive summary;
- an explanation of the background to the project;
- an explanation of how the evaluation was planned;
- the methods that were used to collect and analyse data;
- a presentation of the evidence, and how it has been interpreted;
- a conclusion, and recommendations for future practice.
If some aspects of the work encountered problems, be careful about identifying causes if there is an implication of blame. Sometimes it is better to discuss problems that have implications for contractual relationships in confidential reports or face-to-face meetings. Consider how to present the report in a businesslike and attractive format appropriate for its audience.
10.1.7 Following up the report
The evaluation report will often contain recommendations for further actions and these may lead to new project ideas. Recommendations may relate to processes and procedures within the organisation. Project evaluation and debriefing can be a learning experience for the organisation as a whole, as well as for individuals.