Separating signal from noise is at the heart of what we do at AD&Co. One of the key tools we utilize for that purpose is a sophisticated set of model performance trigger reports. These monthly reports not only alert us to model drift but also point to possible causes for the drift.

In the mortgage market, changes in behavior on the part of both borrowers and lenders may create gaps between model projections and actual performance. Early analysis of these gaps provides rich information to our modelers based on which they can either identify temporary drifts in model performance or, more importantly, highlight fundamental changes in behavior that need to be captured in our models.

Every product type is monitored along the following four dimensions, with residuals defined as the monthly difference between projected and actual results:

Moving average: Are residuals deteriorating over time?

Trend: Are residuals consistently becoming more positive or negative?

Bias: Is the model persistently overpredicting or underpredicting actuals?

Magnitude: Are model misses large enough to matter?

Multiple types of triggers are useful for determining the seriousness of a breach as well as the type of adjustments that might be warranted in response. For example, a moving average warning might indicate a developing change in borrower behavior that may require either additional explanatory variables or more complex functions of existing variables to tighten the residuals. Alternatively, updated data may provide key information for refitting the model. A small but persistent bias, on the other hand, may be easily corrected with a simple tuning parameter adjustment.

These triggers also help us monitor performance at the factor level. Rather than simply observing, for example, how GNMA 15-year MBS are performing, we have triggers at different levels of credit score, LTV, note rate, and many other factors. This deep analysis, presented as an easy-to-use dashboard, quickly identifies possible causes of model drift.

In January, we will present a detailed review of our trigger methodology, and show how we use the reports to help set our modeling priorities in order to explain performance and, when necessary, modify our models to accommodate new behavioral patterns. For now, we will leave you with the following example of a segment of a trigger report for FHLMC MBS using LDM 2.2 with COVID tunings applied.

Happy Holidays!

Tom Parrent, Model Risk Management

tparrent@ad-co.com

October 2020 FHLMC Trigger Report