Shawn Tumanov, Director, Data & Analytics Governance, BMO Financial Group

Below is an insight provided by Shawn ahead of his 2023 session at Advanced Model Risk USA.

The views and opinions expressed in this article are those of the thought leader as an individual, and are not attributed to CeFPro or any particular organization.

Can you provide an overview of the role of the lines of defence in model risk oversight and validation?

The first line of defence is accountable for 1). Development of Models 2). Implementation 3). Performance monitoring 4). Training on the usage 5).  Owns Model Risk

The second line of defence is accountable for 1). Risk Management activities 2). Model Validation 3). Effective Challenge 4). Enterprise Reporting

The Third Line of defence is corporate audit. Independent and objective assurance on the end-to-end framework.

Why is it important to develop transparency across the organization?

Transparency across the organization is vital to creating effective and efficient processes.  Top reasons for developing transparency:

• When everyone has the necessary information, making informed decisions that benefit the organization is easier and more efficient.

• When everyone has the necessary information, making informed decisions that benefit the organization is easier and more efficient.
• Individuals can collaborate and have open communications. By having open communication, individuals develop trust with their peers, allowing people to work closely together to solve organizational issues.

How can organizations ensure they align their LoD with data, audit and compliance?

It is essential to develop a robust governance framework which includes the following components:

• Strategy – alignment of LoD with Data, audit and compliance is a culture shift for the organization. This should be a strategy that outlines how data should be collected, processed, stored and analyzed. This strategy may be a multi-year strategy to encourage partnerships across the LoDs.
• Roles and Responsibilities – Within each LOD there needs to be documented and socialized roles and responsibilities. By ensuring everyone understands their role in the organization there will be alignment between the different LoD.
• Corporate Policies should be set at the top, which different groups adopt or follow. By having one set of Corporate Policies, there will be alignment on what, when, how and why a framework is executed.
• Audit and Compliance needs to be an enabler/partner of the 1^st line of defence while maintaining their independence.

What are the benefits of developing a control framework for advancing technology?

New Technology may create new risks for the organization. These risks may not be well understood, mitigated, or controlled. A well-established framework will advance new Technology in the following ways:

• Develop Guardrails – Proactive Risk management to monitor and control technology risks. When set up accurately, these guardrails allow for responsible technology development and assist with mitigating incremental risks. One such example would be the Software Development Lifecycle controls (SDLC) which would mandate specific steps/actions to advance Technology.
• Enablement – a developed control framework will enable the advancement of Technology through the established roles and responsibilities. Technology teams will understand the process from ideation to development to implementation, allowing for efficient processes and reducing redundant or inefficient processes.
• Legal/Regulatory Compliance- a developed control framework will consider regulatory and legal requirements. By identifying the relevant regulatory/legal requirements and aligning these to control expectations, the organization will reduce risks of non-compliance with regulatory/legal requirements.

How can institutions ensure Integration across teams?

Two items are critical to ensure integration across teams are 1). Collaboration and 2). Transparency.

Collaboration between the three lines of defence is critical for a successful Model Risk Management Framework. Specific to AI/ML- All lines of defence must collaborate to develop efficient and effective process to ensure the processes work for all groups and do not introduce unnecessary or redundant processes. Key groups requiring collaboration- Model Developers, Model Validation, Legal/Regulatory, Privacy, Data Governance, and Ethics.  These teams need to work together to establish the following:

• Clear Roles and Responsibilities
• Expected Processes steps into the process
• Process flows and decision rights
• Policy, Guidelines, standards, and templates
• SLAs and timeframes

Transparency is key in ensuring integration across teams. Transparency is critical in AI/ML Governance for the following reasons:

• Transparency allows stakeholders to understand each team’s roles and responsibilities and insight into the status and the process.
• Specific to AI/ML development- transparency promotes trust in the AI/ML systems because the users can understand and explain AI/ML outputs and decisions.
• Ensuring transparency of data used to develop and train the model supports the mitigation of biases in AI/ML systems. It is easier to identify bias in data and to take appropriate actions.
• Transparency allows for AI/ML to be trustworthy and fair