A four-step process
Northfield’s methodology is based on the assumption that skill –once identified—will persist.
If a manager’s performance is due to skill, that skill – or lack thereof – will continue. If a
manager’s performance is due to luck, however, the best guess for future performance is the average of an
appropriately constructed peer group. In other words, if a manager’s outperformance is due to luck, it
will eventually revert to the mean.
According to di Bartolomeo, the academic literature has found that performance is persistent over a relatively
short time horizon, “one to three years, depending on who you believe.” Northfield tested its
results over a one-year time horizon.
Each fund is analyzed using a four-step process. Northfield first determines the appropriate peer group for each
fund. An iterative methodology with returns-based analysis is used, a tool first developed by William Sharpe. Di
Bartolomeo described this as a “very numerically intensive” processes, which uses a large group of
funds to find ones that act similarly. For every fund, Northfield determines a distinct and custom peer
group.
“Unless you correctly classify funds, there is no persistence in fund performance,” di Bartolomeo
said. “If you don’t, you might as well be throwing darts.”
The second step is to identify how much history should be used in that fund’s analysis. Northfield does
this with a tool known as CUSUM. Developed in the 1950s, CUSUM is a sequential probability test that was first
used to measure quality control on assembly lines. It looks for trends in the number of rejects. Bad performance
for a mutual fund is like a reject on an assembly line.
The CUSUM analysis answers a very precise question: “At what time in the past was it least likely that the
subsequent performance would have occurred, given the precedent performance?” Stated less
precisely, “How much of the fund’s history is relevant?” Everything before that point in time,
when a “regime change” essentially occurred in the fund’s management, is ignored by Northfield’s
analysis.
Sometimes, di Bartolomeo said, nothing changes in a fund’s performance. Other times, however, it is
incredibly relevant. For example, when Peter Lynch left the Magellan fund, its performance declined rapidly.
The third step is relatively straightforward. Northfield determines the risk-adjusted performance of the fund
relative to a market index and relative to its peer group.
Northfield next asks whether the performance over that time period was due to skill or luck. To do so, it uses
four pieces of information: the risk-adjusted performance of the fund; the volatility of that performance; the
average return of the fund’s peer group; and the dispersion of returns among funds in the peer group.
The dispersion is critical to distinguishing skill from luck. If all the returns for a peer group are clustered
around a mean, then it is statistically more likely that an outlier is due to luck, rather than skill.
Alternatively, if returns are more dispersed, then outliers are most likely to represent skillful
management.
The statistical method by which those four pieces of information are analyzed is known as Bayes Theorem. Bayes
Theorem enjoyed recent popularity when Nate Silver of FiveThirtyEight.com used it to correctly predict the winner in 49
states in the November 2008 race. Di Bartolomeo said this theorem was first applied in mutual fund performance
analysis in 2004.
The end result of the four-step process is what di Bartolomeo calls the PWER score, Northfield’s best
estimate of fund performance going forward. It is a mathematical compromise between both the skill and luck
assumptions and is a result of applying Bayes Theorem.
