What are the key differences between model vs tool validation?

All models are tools but not all tools are models. Within the MRM context, models typically refer to quantitative solutions while tools denote to qualitative solutions. Both consist of three components: inputs; a process to transform inputs, and outcomes.

Although variances exist across banks, most financial institutions differentiate models from tools by the underlying methodologies. Specifically, tools are featured with simple arithmetic calculations including descriptive statistics and defined financial formulas, tend to follow rules-based logics or stepwise procedure for operations. Assumptions applied to tools are business driven which could be heavily judgmental based. On the contrary, quantitative models generally fall into two categories: conventional regressions and machine learning algorithms, and have a strong underpinning on statistical, econometric, and mathematical theories. Aside from business assumption, models also depend on statistical properties related to the chosen methodologies.

As a result, tool validation focuses more on process transparency and repeatability and control adequacy. Model validation hinges on quantitative evaluation of the modelling technical. Ongoing performance monitoring is another aspect that is more relevant to model validation than tool validation. Depending upon the complexity and risk ranking of the model, other aspects of validation such as inputs, business use cases, implementation could be more comprehensive and involve more analytics as well.

How has the cybersecurity validation requirements impacted institutions?

While model validation has been widely applied to models developed to support many banking operations, its application on cybersecurity models presents unique challenges.  The first challenge is most validators lack cybersecurity domain knowledge to effectively challenge the development and use of cybersecurity solutions. The second challenge is some elements of the prevailing supervisory model validation framework are inapplicable to cybersecurity models. Understanding those challenges will help financial institutions decide what level of oversight they need to apply to cybersecurity models and tools.

How can institutions manage across qualitative and quantitative model risks?

A qualitative model, in general, is considered a visual representation of an event or a series of events, and how they combine to produce an outcome. A quantitative model, in comparison, forms some mathematical mechanism between the inputs and the output. With probability estimates included, a quantitative model can provide an estimate of the relative risks. Qualitative and quantitative risk exist for both qualitative and quantitative models.

To manage across qualitative and quantitative model risk, an institution must have deep business knowledge and expertise, as well as strong technical skills. It takes the combination of both abilities to properly identify and quantify risk. Quantitative model risk is assessed during the evaluation of data, model estimation, and performance testing. Quantifying qualitative model risk is, however, less straightforward. In practice, a comprehensive evaluation of non-technical model assumptions and limitations around data, methodology, and estimates, should provide useful insights on the magnitude of the qualitive risk.

Why should institutions look to include non-financial/ operational risk models?

Non-financial models including operational risk models have been included in most large financial institutions’ model inventories and subject to regular MRM review at present. Non-financial risk arises from banks’ risk-taking activity and are not reflected in the financial positions on banks’ balance sheets. Models developed to quantify or predict such risk, if not understood and managed properly, would lead to huge operational losses and increase in reputation, legal and compliance risk.