Most of our work falls into five packages. Each has a defined scope and ends with a written report. How deep the work goes follows the risk rating your own framework gives the model.

E-23 and AMF readiness assessment

For institutions whose model risk framework is not yet complete. We compare your current practices with OSFI Guideline E-23 and the AMF guideline on the use of artificial intelligence:

  • model identification and inventory;
  • model risk rating method;
  • requirements at each lifecycle stage: design, review, approval, deployment, monitoring, decommissioning;
  • roles, responsibilities and independence of the review function;
  • documentation and reporting to senior management.

You receive a prioritized list of gaps and a remediation plan sized to your institution and its risk profile.

Full validation of a high-risk model

For models your framework rates high-risk, such as those used in credit, fraud or pricing decisions. The review is done by someone who had no part in building the model. It covers:

  • Conceptual soundness: whether the method suits the problem, the data and the intended use, and whether assumptions and limitations are documented.
  • Data: quality, representativeness, lineage, handling of missing values and outliers, leakage between training and evaluation.
  • Performance: replication of the developer’s results, out-of-sample and out-of-time testing, comparison with a simple benchmark model.
  • Stability: sensitivity to changes in data, hyperparameters and population.
  • Fairness: outcome differences between groups, proxy variables, consistency with fair-treatment expectations.
  • Explainability: the level of explanation the use case requires, and whether the model can provide it.
  • Implementation and monitoring: whether the production model matches the validated one, and whether the monitoring metrics and thresholds are adequate.

You receive a validation report with findings rated by severity and an overall opinion on whether the model is fit for its intended use, technical appendices, a findings log, and a closure memo once remediation is verified.

Targeted review of a lower-risk model

For models your framework rates low or medium risk, where a full validation would be out of proportion. The review covers conceptual soundness, data and performance. It relies more on the developer’s own tests than a full validation does, and the report says so where those tests are missing or unconvincing.

You receive a short report with findings rated by severity and an opinion limited to the areas reviewed. If the review gives reason to question the model’s risk rating, the report flags it.

Vendor pre-procurement assessment

For institutions about to buy a model or an AI system. OSFI Guideline E-23 expects institutions to bring the models they buy under model risk management along with the ones they build. Documentation and contract terms are easier to get from a vendor before signing than after.

  • Documentation: what the vendor can provide on methodology, training data, known limitations, its own validation, and how it communicates model changes.
  • Testing on your data: where a trial or test environment is available, performance on your own population and by segment, compared with a simple benchmark or the model being replaced.
  • Contract: the terms a later validation will depend on, such as access to documentation, notice of model changes and performance information. OSFI Guideline B-10 sets out the expectations for third-party arrangements.

You receive an assessment report, a list of documentation gaps to raise with the vendor, and contract points to take to your legal and procurement teams.

Generative AI pre-launch review

For conversational assistants, document search and document processing built on large language models, before they go live.

  • Accuracy and hallucination: evaluation sets built from your own use cases, with reference answers.
  • Bias: differences in treatment by language, profile or phrasing of the request.
  • Robustness and security: prompt injection, data leakage, instruction bypass, behaviour outside the intended scope.
  • Data provenance: which sources the system draws on, access controls, retention.
  • Human oversight: control points, access to a person, disclosure to clients.

You receive the evaluation results, findings rated by severity, and the conditions we recommend before launch.

Models we review

Fraud detection, credit risk rating, forecasting, recommendation and personalization, and generative AI systems. The About page lists the kinds of models we have worked on. If yours is a different kind, ask, and we will tell you whether we can review it properly.

Other services

Revalidation

Models change after they are validated: new data, retraining, performance drift, new uses. We carry out periodic reviews at the frequency your framework sets, and targeted reviews after a material change.

Documentation dossier for model vendors

For vendors that sell models to financial institutions and get documentation requests from each client. We work with the vendor to produce structured model documentation and an independent review it can give to its clients. Each institution remains responsible for its own assessment and can use the dossier as its starting point.

What we don’t do

We don’t build the models we validate, and we don’t certify regulatory compliance. Our report is an independent opinion. The decision to approve a model, and responsibility for compliance, stay with the institution.

How an engagement runs · Contact us