Evaluation criteria for an online procurement for an IT project manager who will be conducting a new procurement

Recently, I have been working on several online procurement projects, in the course of which I have helped the client to prepare the procurement documents of either a website or a mobile application, including the technical terms of reference.

As I have worked for the lion’s share of my working life in web agencies and written dozens and dozens of public procurement and private sector tenders, over time I have developed a pretty good picture of the typical mistakes that both private and public sector clients often stumble upon.

On the other hand, having carried out various website audits at the request of the customers, I can see how the root cause of the problems is unfortunately already written into the procurement and has a negative impact on the progress of the entire project.

Of course, the field of the topic is enormously wide, but I’ll start over. Let’s start with public procurements. The activities of the contracting entity are regulated by the Public Procurement Act, but the freedom to prepare a high-quality online procurement is fully present and feasible.

I am sure that almost every project manager of a contracting authority whose task is to carry out a new procurement wants to be a smarter IT client and see many strong bidders in their procurement.
So I’m writing this story to you, the client’s web/IT project manager, who is currently preparing a new online procurement.

The good intentions of the contracting authority are often torpedoed by a number of obstacles, such as in-house disagreements or ignorance of the procurement conditions and criteria, fixing the classic trio – time, budget, scope – inadequacy of technical specifications, ignorance of the trends of modern web technologies, etc. Most mistakes arise either from ignorance, lack of experience, lack of time or pity, but sometimes also from ignorance (“I know what kind of procurement I’m writing, the agency will make up for it afterwards”).

Type error No. 1 – incorrect selection of the conditions of conformity and evaluation criteria of the procurement

First, let’s talk about the concepts.
The contracting authority can use the

• qualification criteria – minimum requirements for a tenderer to qualify
• conditions of compliance – minimum requirements for the submission of a tender
• evaluation criteria – criteria that are important to the contracting authority and it wishes to select the strongest tenderer through healthy competition

The first thing the contracting authority must decide is whether the public contract will be awarded by applying the criterion of economic advantage or the lowest price .
It is important to know that the Public Procurement Act prefers economic advantage, which tries to find the best price-quality ratio.

A contracting authority that chooses the lowest price criterion in an IT procurement can be sure that it will most likely get the cheapest and poorest quality developer on the market as a partner.

Unfortunately, this is done over and over again, and this is also the case in IT procurements. Why?

I suggest that the reasons may be laziness (it takes time to come up, weigh and describe the evaluation criteria), pressure from lawyers or other influential persons (it’s easier this way, let’s not take risks, we have budget limits, we do it cheaply) and ignorance (we have compliance conditions in place and that is enough).

Whatever your reasons, please do not make a low-cost procurement in an online procurement, because no decent bidder wants to participate in it.
Yes, sometimes they rarely participate if it is a well-known client in Estonia that they really want in their portfolio, but this is the exception rather than the rule.

So when would it be reasonable to use the lowest price in the first place? Only if the contracting authority describes the purpose of the procurement contract exhaustively and all requirements unambiguously. Therefore, the lowest price is suitable as an evaluation criterion primarily in the case of simple procurements, such as the purchase of a specific item. However, online procurements are more complex in nature and the lowest price is not suitable here.

Qualification and compliance conditions

The qualification and compliance criteria must be balanced so that a sufficiently large number of agencies pass through this bar to be able to participate in the procurement.
These terms and conditions set out the formal requirements (joint tender, indication of business secrets, Estonian language requirement, etc.) and substantive requirements. There may also be no substantive requirements at all, in which case the winner’s choice is based only on the evaluation criteria.

I recommend using the following options for substantive requirements in the conditions of compliance:

• Team composition and requirements for team competence

If necessary, also:

• number of similar references, each with the required turnover

Note: according to the updated Public Procurement Act, the team’s work experience may only be requested in the compliance conditions (and not in the qualification conditions).

Evaluation criteria

A sufficient number of companies that have been active in the respective field, including low-cost bidders, can (and should) pass the qualification and compliance criteria. Your task now is to determine the evaluation criteria based on which you can choose the partner with the best price-quality ratio.
The procurement specialist or lawyer of the procurement department may have a different understanding of this topic than you, but discuss the reasons and do not forget that you are ultimately responsible for the high-quality completion of the application, not the procurement specialist.

As an important argument, you can note that the European Court of Justice is of the opinion (C-234/03) that contracting authorities are free to choose both the award criteria and decide on their weighting to the extent that allows for an overall assessment of the criteria used in order to determine the most economically advantageous tender.
It is also important to know that the criteria used to determine the most economically advantageous tender do not necessarily have to be quantitative, they can also be the appropriateness of the strategy, methods, tools, high-quality environment, etc., if these requirements concern the proper provision of services. In web development, for example, development methodology is a very important component.

The range of evaluation criteria is wide, such as:

• total cost of the tender
• Price per hour of development and maintenance work
• Test task (e.g. homepage design or technical task)
• Number of similar references
• Team structure and competence
• a description of the organisation and methodology of the development process, including a risk plan
• Detail and realism of the project plan
• the structure of the tender, its clarity and completeness
• jm

For each criterion, the contracting authority must add a weighting and a detailed description of the evaluation methodology.

As you can see, weighing and choosing the evaluation criteria is a much more time-consuming way than simply choosing the lowest price (in addition, there is a human fear of making mistakes), but proper procurement preparation is the basis for the success of your project. Find that time and you will save ten times the time and money later on at the expense of sorting out problems.

I recommend choosing 3-4 evaluation criteria that are the most important for a specific procurement. The customer has complete freedom to choose them based on their needs, as well as to determine the proportions as they see fit. I will bring out a few thoughts about each of them and a reasonable range of proportions.

Total cost of the tender

The first and most common criterion. The share should be in the range of 30..50%, depending on the choice of other criteria and the size of the budget.

Price per hour of development and maintenance work

If the procurement also includes years of development and maintenance services, add this criterion to avoid surprises later on. The share could be between 10..25%

All in all, the total share of price criteria should be between 40…60%.

Test task (e.g. homepage design or technical task)

A relatively common criterion, especially when evaluating design.
If the scope of the procurement includes a visual design and a prototype of the website, then the requirement for a design sample task is justified, but keep in mind that it will still reduce the number of participating bidders. High-quality professionals have their hands full and it is difficult and also expensive for them to let a designer create a creative solution from scratch for an offer. The same applies to developers who are supposed to solve a technical task. The tenderer’s decision also depends a lot on the planned cost of the procurement – if the planned budget is large enough, the test task is not an obstacle.
If you are already asking for a test task, you probably want a higher proportion for it, e.g. between 25..40%

Number of similar references

“Similar” can be either financially or substantively. Similar in content generally means either projects with the same technical platform (WordPress, Drupal, etc.) or similar in functionality (e-shop, self-service, intranet, public web, etc.). Financially similar means projects of the same size as your planned budget.
Usually, a time limit is also set – e.g. completed in the last 3..5 years and also the minimum number of required projects – e.g. 3….8. The specific number depends, among other things, on the size of your planned budget, if the budget is larger (>100k), then it is reasonable to put a smaller number.
NB! One important nuance: always ask for the contact person of the project client in the tender, this gives you the opportunity to check the actual satisfaction of the customer after the completion of the project and thus get more background information about the tenderers. I have seen enough cases where the list of the agency’s projects on the web is impressive, but the clients are usually extremely annoyed and do not remember the person doing it well. Try to avoid this trap. (Another option is to ask for background information about the agencies from an independent expert who knows the market.)
The share could be 15..20%

Team structure and competence

In general, I recommend putting this criterion under the conditions of compliance, because as we all know well, the best performers of the agency are always included in the tender. In reality, however, they start talking about changing the team right at the beginning of the project. At such a moment, the contracting authority can refuse and demand the same team as promised in the procurement, but in reality, few contracting authorities do so. Since they want to start new cooperation in a positive way, they usually settle for a new (and unfortunately less experienced) team.
If you do decide to include it in the criteria, the percentage should be small, e.g. 5…10%.

Description of the organisation and methodology of the development process, including the risk plan

This is a criterion that generates controversial opinions. Some say that this is just writing a beauty story and there is nothing easier than writing a voluminous beauty story from somewhere (even if it is an offer from a tougher competitor if you happen to have one). Others say that based on this story, very important conclusions can be drawn about the quality of the agency’s work organisation. If I used to agree more with the former, then today I have changed my opinion radically. Why? When conducting software audits, I have always compared the description of the development process promised in the tender and the overlap of real life, and I have seen that agencies actually use the perfect description of their own work process, i.e. there is no great copying habit.
If we can assume that the tenderer more or less describes its own actual organisation of work (or at least the dream of it), then this criterion acquires very important importance. To be honest, it’s my favorite.
The description of the development process and risk plan reveals all the weaknesses of the provider’s work process to the experienced eye. You can believe me that based on the “beauty story” already in the offer, you can point out the possible project risks with this particular agency. And if the ideal version is already skewed, one can only imagine what real project life will be like.
An important focus is also on the risk plan. It is immediately apparent whether the tenderer does not understand its substantive significance at all and simply makes a copy-paste from its previous tender or whether it has thought through the risks for this particular procurement and highlighted them accordingly.
If you have the IT expertise necessary for the assessment, I recommend choosing this set of criteria. The share could be 20..40%

Detail and realism of the project plan

Assuming that you do not set too short a deadline for completion in the procurement (more on that later), this is a good opportunity to assess the tenderer’s experience in carrying out similar projects. In the course of preparing the tender, a proper agency will make a rough project and resource plan anyway to estimate the volume of the procurement, and it will not be difficult for them to highlight it in the tender. Actually, I am not in favor of a project plan that is too detailed here, it does not give a comprehensive overview. It is enough to outline the general stages of the project, the stages of functional modules, the stages of final testing, etc. Again, you need technical expertise or your own experience to assess realism. The share could be 10..20%

Structure of the tender, its clarity and completeness

This criterion really shows something, but I’m not sure if it really shows the level of quality of the provider. This shows more tender writing skills and does not say much about the tenderer’s experience. Use with caution. The proportion should be small, e.g. 5..10%

Example of evaluation criteria

I have now described a number of possible criteria and I will also give an example.
Let’s say you’re planning to get a public web implementation. You already have the visual design, UX/UI analysis and prototype ordered in the previous procurement. The business analysis has been completed, the goals are clear, and now you need technical development. In addition, you want 2 years of development and maintenance after the launch of the website.
Your planned budget is about 100k, half of which you have calculated for the creation of the website and half for further development and maintenance.

One possible set of assessment criteria is as follows:

Name

of the tender

risk plan

supplier

Such a choice should give you the necessary freedom of assessment to find a cooperation partner with the best price-quality ratio.

If you think I missed a good criterion, let me know!

This story goes on. Next, I will deal with the typical mistakes “Incorrect selection of procurement conditions” and “Incomplete preparation of technical specifications”.

PS I also recommend a cool read in the “World in the office” blog: “How to update a website”