PART A · QUESTION 1
A developer hands you a loss model, and says it fitted past data with very little error. You are asked to validate it.
// EXAM HALL22 QUESTIONS · 30 MINUTES
Model Validator Practice Test
Testing someone else's model: order and level, one back-test and scorecard, a validation file, and the calls when the answer is unwelcome.
BY MICHAEL SHANG · SIXTEEN YEARS IN COMMERCIAL BANKING · UPDATED
// THE SHORT ANSWER
A model validator practice test asks whether you can test someone else's model, write down what you find, and keep to it under pressure. This paper has four parts: discrimination, calibration and tiering; one back-test and one scorecard from a risk pack; a validation file with an overlay and a data exclusion, set on an exhibit; and calls on independence, vendor models and findings. There is no coding in it, so it does not replace your own technical round. Every question is original, and none comes from any bank's test. Sit it against a thirty-minute clock, or take it as practice.
BASIS Practitioner judgment from sixteen years in commercial banking, set out in How Bankers Think (Highbank Press, 2026). Every borrower on this site is a composite constructed for teaching, not a client. The market is mid-market commercial lending as practised in New Zealand and Australia; where a convention differs elsewhere, the page says so.
// HOW TO SIT IT
Answer below as practice, and each read opens as you go. Or sit it as a test: 30 minutes, with every read held back until you finish.
PART A
The measures
What a model can be right and wrong about, and which model to look at first.
PART A · QUESTION 2
A model ranks borrowers very well. It also says 2% will default, and 4% do.
What does that tell you?
PART A · QUESTION 3
You are given a model and its documentation to validate. The developers offer to run the tests and send you the results, which would save weeks.
What do you do?
PART A · QUESTION 4
A provisioning model and a marketing response model are both up for review, and you have time to do only one properly this quarter.
Which comes first?
PART A · QUESTION 5
1% of transactions are fraud. A model catches 90% of fraud, and wrongly flags 5% of genuine transactions. Take 10,000 transactions.
What share of flagged transactions are really fraud?
PART A · QUESTION 6
Your early-warning model for business loans scores remarkably well on test data. One strong feature turns out to be 'number of calls from the collections team'.
What do you conclude?
PART B
The risk pack
One back-test and one scorecard, from one quarter's pack.
EXHIBIT H · THE RISK PACK
One quarter's risk report for a mid-sized bank, three tables from the pack. Figures in US$m. The Lost and Grew columns cover the last 12 months. PD is the chance of default in a year; LGD is the share lost if it happens.
| US$m | PD | LGD | Lost | Grew | |
|---|---|---|---|---|---|
| Home loans | 6,000 | 0.4% | 15% | 2 | 4% |
| SME loans | 2,000 | 2.5% | 45% | 24 | 3% |
| Commercial property | 1,500 | 2.0% | 35% | 3 | 28% |
| Credit cards | 500 | 4.0% | 80% | 19 | 5% |
| Book | 10,000 | 48 |
| Details | Figure |
|---|---|
| Commercial property, against a limit of 15% of the book | 15.0% |
| Trading: one-day 99% value at risk, against a limit of 3.0 | 2.4 |
| Back-test: days in the last 250 with a loss above value at risk | 9 |
| Liquid assets 1,800; stressed 30-day outflow 1,500 | 120% |
| Share of deposits held by the ten largest depositors | 22% |
| Application scorecard, Gini: all customers | 0.71 |
| Application scorecard, Gini: new-to-bank, 18% of new loans | 0.38 |
| Item | What the log says |
|---|---|
| 1 | A payment batch of US$1.2m was sent twice. All recovered. The second such event in six months. |
| 2 | A leaver's system access stayed active for three months. No loss recorded. Access logs not yet checked. |
| 3 | The arrears report the board reads is built by hand, in a spreadsheet, by one person. |
| 4 | 400 letters went out with the wrong date. Corrected in a day. |
PART B · QUESTION 7
Exhibit H's back-test shows nine days in the last 250 with losses above one-day 99% value at risk. The desk points out that it stayed inside its limit on every one.
Signal or noise?
PART B · QUESTION 8
Exhibit H: the application scorecard scores 0.71 overall, but 0.38 on new-to-bank customers, who are 18% of new loans. The owner says: 'Overall it is excellent. Approve it.'
As the validator, what do you do?
PART C
The validation file
One file, seven questions. Read what was left out as closely as what was put in.
EXHIBIT Y · THE VALIDATION FILE
A consumer loan default model, up for its first independent validation. Built on 2015–2023 data.
| Grade | Accounts | Predicted | Actual |
|---|---|---|---|
| 1 | 5,000 | 0.2% | 0.3% |
| 2 | 3,000 | 1.0% | 1.1% |
| 3 | 1,500 | 3.0% | 5.0% |
| 4 | 500 | 10.0% | 9.0% |
| Item | What the file says |
|---|---|
| Data | 2020 left out as 'unusual'. |
| Stability | Population stability index on income 0.31; the team's standard investigates above 0.25. |
| Overlay | 0.5 points added by hand to grade 3, included in the 3.0% above. No document explains it, and whoever added it has left. |
| Reporting | The validation team reports to the head of the team that built the model. |
PART C · QUESTION 9
From Exhibit Y: multiply each grade's accounts by its predicted rate, and by its actual rate.
How many defaults did the model expect, and how many came?
PART C · QUESTION 10
In Exhibit Y, grade 3 predicted 3.0% and actually ran at 5.0%, on 1,500 accounts. Grades 1, 2 and 4 are each within five defaults of prediction.
What do you conclude?
PART C · QUESTION 11
Exhibit Y's developers left 2020 out of the data as 'unusual'.
What do you ask?
PART C · QUESTION 12
Exhibit Y shows a population stability index of 0.31 on income, against 0.25 where the team's standard calls for investigation. The ranking score has barely moved.
What do you do?
PART C · QUESTION 13
Exhibit Y lists a manual overlay: 0.5 points added to grade 3. No document explains it, and the person who added it has left.
What is your finding?
PART C · QUESTION 14
Exhibit Y notes that the validation team reports to the head of the team that built the model. Its head says: 'We are all one analytics family here.'
What does that do to the validation?
PART C · QUESTION 15
Having read Exhibit Y, the model owner asks for a simple pass or fail by Friday.
What do you give them?
PART D
The calls
When a model's owner, a vendor or a senior manager would like a different answer.
PART D · QUESTION 16
A new provisioning model must go into the half-year numbers next week. Validation needs another month. The CFO's office says: 'Sign it off, and finish your work afterwards.'
What do you do?
PART D · QUESTION 17
Two models predict small-business default. The complex one scores slightly better in testing, and nobody outside the team can explain a single decision. The simple one can be explained to a banker.
Which do you lean to?
PART D · QUESTION 18
A vendor sells the bank a fraud model, and says its method is proprietary and cannot be disclosed.
What do you ask for?
PART D · QUESTION 19
Your validation report rates a finding as high. A senior manager asks you to call it medium, 'so that it does not go to the board this quarter'.
What do you do?
PART D · QUESTION 20
A team uses a machine learning model built for one purpose, approving personal loans, to set collections strategy too. 'It is already validated.'
What do you say?
PART D · QUESTION 21
A credit scoring model does not use ethnicity. An analysis finds people from one community are declined far more often at the same repayment record. Postcode is a strong feature.
What do you do?
PART D · QUESTION 22
A fraud model went live a year ago and has not been looked at since. Its alert volumes look normal. A colleague says: 'It is not broken, so leave it alone.'
What do you say?
0 OF 22 ANSWERED · 0 WITH THE SENIOR READ · 0 STANDS UP · 0 OFF THE LINE
// THE REST OF THE ROUND
Every question here is original; none is any bank's real test. Written for: Model validator.
- Model validator: the interview questions, against a two-minute clock.
- Situational judgement, for any seat.
- The assessment centre: a whole case in forty-five minutes.
// SUBSCRIBE
The Lab.
One case file and one question, in your inbox every Tuesday morning, New Zealand time. The senior banker's read opens only after you answer.
FREE WITH IT · 10-PAGE PDF
Case 01 of the interview case pack — Northmark Engineering, in full.
No spam. Unsubscribe anytime. Educational content only.