Performance evaluation is carried out to find the “best” manager to meet a specific portfolio need, or to check if an existing manager still meets the client’s original portfolio need.

Performance evaluation is the process of:

• Understanding how a manager’s performance was achieved
• Testing if their data aligns with their investment philosophy and process
• Determining if their investment process can achieve reasonably consistent and satisfactory returns

The process entails a qualitative and quantitative analysis using several sources of information. A combination of completed questionnaires provided by managers and desktop research forms an understanding of the manager. This is followed by a due diligence to confirm the manager’s claims. The due diligence is used to clarify and investigate any inconsistencies.

Six Ps are used as a broad guideline in the evaluation process: philosophy, process, people, price, performance and the physical environment.

What are managers saying?

A manager provides a set of principles that guides their investment decision making, which is referred to as a manager’s philosophy, or alpha thesis. This philosophy helps you understand how and why a manager generates alpha. It is the foundation of an investment process. Having a well-defined investment philosophy and process enables effective testing of these factors.

Managers make assumptions about what guides markets which impacts their investment philosophy. These assumptions may depict the style of a manager (risk factor exposures including value, growth, momentum, quality), especially in the case of an equity manager. In active and passive investment management, a manager makes assumptions about market efficiency.

How a manager executes to deliver on their philosophy is known as the investment process. Portfolio construction forms a large part of this process. Portfolio construction should provide detail on how the manager allocates capital to assets, the interplay of decision making between the team members and the resulting allocation decisions, liquidity management, and risk measures used in the investment process.

Testing what managers say

An essential step in performance evaluation is testing whether the manager does what they state in their philosophy and process, and if this creates a sustainable competitive advantage.

A change in a manager’s philosophy over time causes uncertainty, but you need to investigate further to determine if this is a cause for concern. If fundamental long-term shifts in a market have led to the manager’s philosophy evolving, this could be interpreted as good because the manager is able to adapt to the market. However, if the philosophy changed on the whim of short-term volatile factors such as performance and flows, this creates a question about the repeatability of the investment process.

Why is repeatability important? If a manager has successfully delivered alpha in the past, a repeatable process carries some weight in having the potential to deliver positive alpha in the future.

Performance can be evaluated from qualitative and quantitative perspectives.

Quantitative analysis

The first point of reference is a manager’s returns over a specified period. Even though net returns are appropriate when considering investment options, gross returns should be utilised when evaluating skill, as fees have no bearing on skill. Additionally, you may be able to negotiate fees.

Quantitative analysis does not only include return analysis. Looking at holdings and transactions are equally important, and in many cases, much more important.

Alpha is another relevant measure. Alpha is defined as the portfolio return in excess of the relevant benchmark return. Methods used to identify sources of portfolio alpha are referred to as return attribution techniques, which attempt to understand the consequences of active investment decisions and provide evidence towards assessing claimed competencies.

In an attempt to deliver returns, a manager takes on risk, which can be defined as “exposure to uncertainty”¹. As with return attribution, risk attribution examines the risk implications of investment decisions and can be used to analyse consistency with a manager’s investment philosophy.

Related to return attribution is investment performance appraisal. Return attribution complements performance appraisal, which is an overall skill assessment. Performance appraisal measures can differ by the measure of risk used. This is important to ensure that risk-taking is not confused with skill. The table below provides brief descriptions of a few risk adjusted return metrics that could be used in performance appraisal.

You need to keep in mind that past performance does not necessarily indicate skill or guarantee future performance because there will be times when skilled managers underperform, and managers with no skill outperform.  However, one of the objectives of performance evaluation is to ascertain whether a manager’s process can be expected to deliver alpha going forward.

At the same time, taking cues from underperformance may not be a correct or useful manner of assessing performance. Even though this may be a delayed signal of lack of skill, or lack of adherence to investment process, or an operational problem, it could just simply be “noise” or a random outcome.

Limitations of quantitative analysis

In the case of developing economies, financial markets are typically illiquid, transaction volume is low, and transaction costs high. These characteristics arise as a result of many factors associated with developing economies, such as low levels of regulation, poor accounting standards, weak investor protection, political risk, and inflation risk. This leads to less informational efficiency and high asset price volatility.

Under these circumstances any conclusion reached through quantitative assessments may be questionable because data may not be up to date or fully representative of the market. This conundrum is very relevant for many African financial markets in an infancy stage. Any quantitative assessment needs an understanding of these limitations.

Qualitative analysis

Some of the critical yet softer aspects of performance evaluation fall under people and the physical environment which includes organisational culture.

When a manager’s investment process is highly dependent on the skills or decision making of an individual or a few key individuals, this would indicate “key-man risk”. The repeatability of a process is at risk should the key individual leave the firm.

Another people consideration is having investment people with the right credentials and experience, vital to creating and executing an investment process. Without this, even a good, consistent and repeatable process could fail to deliver alpha. Why? An individual who does not have the right credentials or has not experienced a market crash, or liquidity crunch, or other somewhat “rare” market phenomena, may not be able to appreciate the impact of these events on the investments they manage.

Remuneration is important to attract and retain staff. Remuneration related to product performance leads to aligned interest between the firm and individual. On the other hand, remuneration is not as vital as flexibility in working hours and funding further studies for some employees. Each individual has different motivational factors.

Why is it important to retain staff? If a firm is unable to retain staff and there is high turnover, institutional knowledge and experience may be lost, which could impact the manager’s performance.

Related to the above is organisational culture. Investment individuals are motivated and have a greater chance of being retained when an organisation’s culture encourages positive traits. At the same time, if there is instability due to staff movements, this does not necessarily imply concern. Some individuals get the opportunity to progress in their career and staff turnover can be a sign that the current firm is a good grooming ground for future leaders. Thus, staff mobility is something that needs to be investigated before reaching any conclusions.

Behind the figures

When quantitatively assessing a manager you must take Global Investment Performance Standards (GIPS), benchmarks and fees into account.

GIPS are a voluntary set of standards that provide an ethical guideline for calculating and presenting investment performance history for investment management firms across the world.

GIPS gives recommendations regarding input data, calculation methodology and composite construction with regards to performance figures. Composite refers to the combination of portfolios (one or more), which have a similar strategy, objective or mandate. In representing historical performance, there are multiple factors that GIPS recommends for consideration to ensure there are no potential misrepresentations.

When carrying out any benchmark relevant assessment, you need to ensure an appropriate benchmark is specified. Benchmark specification errors result in incorrect performance measurement, which negatively impacts the attribution and appraisal analyses. Changing a benchmark to mask underperformance is inappropriate and unethical.

The impact of fees (price in the 6 Ps) needs to be evaluated because the net return is the relevant return for the investor. Investors may not want to invest with a skilful manager if the manager keeps most of the excess returns in fees, merely using clients’ money to generate fees. Management fees are not important in evaluating skill, but they are critical in deciding whether or not to invest in a skilful manager.

Conclusion

Performance evaluation from a manager research perspective comes down to assessing and understanding a manager’s philosophy and process while determining the impact of their process on their ability to deliver alpha over time. It involves a detailed qualitative and quantitative assessment of the manager across many dimensions.

Looking only at past returns ignores a plethora of important aspects of the manager’s investment philosophy and process. Not only should quantitative analysis be as broad as possible along many other dimensions (such as risk, holdings and transactions, mandates across the manager), but qualitative analysis should supplement the evaluation and consider the future to determine whether the proposition is likely to produce the required results.

¹ CIPM Principles Reading, CFA Institute (2017)