Solutions AMM Inc. provides independent validation of AI and machine learning models for financial institutions. The practice is led by Aaron Mascaro.

Background

Aaron Mascaro holds a PhD in experimental condensed matter physics from McGill University (2019). His research was in scanning probe microscopy: calibrating instruments, removing sources of noise from measurements, and measuring ionic transport in battery materials. Much of that work came down to checking how far a measurement could be trusted, and validation asks the same thing of a model.

He has spent nearly eight years building and deploying models in industry. At a proprietary trading hedge fund, he implemented pricing algorithms for the volatility desk. At one of the largest websites in the world, he worked on search algorithms that served hundreds of millions of results a day. At Shopify, he worked on the credit risk rating models used for capital loans to merchants. He is a named inventor on four patent applications related to AI and machine learning.

Kinds of models

Each kind of model fails in its own ways. The same failures turn up in financial institutions’ models, and a validator needs to know where to look for them.

Fraud detection
Rare events, adversaries who adapt to the model, thresholds that trade losses against customer friction, labels that arrive late.
Credit risk rating
Outcomes known only months after the decision, repayment observed only for the loans that were approved, populations and economic conditions that shift between training and use.
Pricing and trading algorithms
Calibration to sparse or stale market quotes, non-stationary data, backtest overfitting, look-ahead bias.
Search and recommender systems
Feedback loops where the model shapes the data it will be retrained on, position bias in click data, offline metrics that poorly predict real-world results.
Large-scale distributed computer vision
Gaps between training and production data, latency and volume constraints, label quality control.
Forecasting systems
Out-of-time evaluation, regime changes, quantifying uncertainty.
A/B test analysis and data science
Causal inference, statistical power, multiple comparisons, effects that are statistically significant but too small to matter.

Why independent validation

A machine learning model can post excellent results and still fail for reasons its own team is badly placed to see: leakage between training and test data, a population that shifts, a feature that stands in for a protected characteristic. The people who built the model made choices that seemed reasonable to them. Validation tests those choices with the same technical depth.

OSFI Guideline E-23 expects model review to be independent from model development and allows institutions to use objective third parties for it. For an institution without a dedicated validation team, or whose team lacks time or machine learning experience, an independent third party is often the most direct way to get there.

Montréal, in French and English

The practice is based in Montréal and offers its services in French and English, in Quebec and across Canada. Reports are written in the language you choose. Engagements are mainly remote; on-site meetings are available on request.

Publications