Since the process of credit decision-making encompasses careful evaluation of risk assessment, credit scoring, and scrutiny of financial statements, you might wonder how to filter out the best-performing strategies through comparison of multiple decision-making algorithms in real-time.
Enter Canary Testing and Champion/Challenger models – a dynamic duo in Business Rules Engine (BRE). Let’s dive into how they redefine credit decision-making and how they pave the way for more informed and adaptive credit assessments.
Let us take a deep dive into the concepts of Canary Testing and Champion/Challenger and their use cases.
In digital lending, Canary Testing refers to the method of testing new strategies under-real world conditions to determine how they will perform in a live environment. By following this method, risk teams can assess new possibilities, ensuring minimal to no impact on users.
For example, if you’re a lender partnering with a digital lending platform, you might encounter a relatively new user base. You may face several challenges such as:
Canary testing helps solve these challenges by allowing you to run a variation of an existing or new workflow in the live production environment without disrupting your daily operations.
Though canary testing has its benefits in helping refine credit decision-making, it also brings in some challenges that lenders have to deal with for successful policy testing. Some of them include:
The Challenge | The Solution |
Lenders must ensure that sensitive customer data used in canary testing complies with data privacy regulations such as the GDPR and CCPA. Digital lending guidelines issued by the Reserve Bank of India (RBI) are also mandatory to be followed by lenders, which include policies such as: 1] Collecting borrower data on a need-basis with their approval and an audit record. 2] Refraining from accessing additional data stored in their devices such as call logs, contact information, etc. 3] Ensuring biometrics of the borrowers are not stored in lending apps. | Implementing robust data encryption and anonymization practices and conducting regular audits will help ensure maximum compliance. |
The Challenge | The Solution |
Integrating new models and algorithms into existing systems can make setting up and managing canary tests more complex. | In this case, automation tools can help streamline the setup and execution of canary tests. |
The Challenge | The Solution |
Performing a successful canary test requires additional resources such as technology infrastructure with data management capabilities and specialized personnel. | Leveraging cloud-based solutions can be beneficial in scaling canary tests and reducing infrastructure costs. |
The Challenge | The Solution |
Scaling canary testing to meet large volumes of transactions and diverse customer segments can be difficult. | This problem can be addressed by designing modular canary tests and utilizing scalable cloud services to process large datasets and transactions. |
In the realm of digital lending, the Champion typically refers to the tried-and-tested lending policy and the Challenger refers to testing multiple new strategies in a controlled environment to determine if you can enhance the accuracy or approval rates of credit assessments. This approach can be best-described as a litmus test for financial strategies, also popularly known as A/B Testing and can reveal how alterations in credit evaluation tactics impact overall decisioning outcomes.
Despite being a powerful tool for optimizing credit decision-making, even champion/challenger features come with their own challenges. Some of them include:
The Challenge | The Solution |
Credit decision-making models often incorporate multiple variables and data points, making it more complex. Adding to that, comparing challenger models and champion models can be difficult when the models are sophisticated and include machine learning algorithms. | This problem can be solved by simplifying models and using robust platforms that can facilitate seamless model comparison and highlight key performance metrics effectively. |
The Challenge | The Solution |
Measuring the performances of the champion and challenger models and determining their statistical significance can be a challenge, especially with limited sample sizes or inaccurate data. | Appropriate statistical tools and techniques that can conduct hypothesis testing can be used in this situation. Additionally, the sample sizes can also be enhanced to improve the scope of the tests. |
The Challenge | The Solution |
Garnering the support of stakeholders will be difficult, especially from the ones who’ve been accustomed to existing policies and workflows. They may be resistant to change and reject new models, unless it can help bring in profits. | To better convince stakeholders, pilot results and case studies can be demonstrated to the stakeholders to help them understand the benefits and rationale behind the testing. |
The Challenge | The Solution |
Champion/Challenger testing is a resource-intensive process that requires a significant amount of data storage, computational power, and specialized personnel for setup, monitoring, and analysis. | To solve this issue, cloud-based solutions and automation tools can be leveraged to streamline the testing process and scale technology infrastructure. |
Leveraging the power of canary testing, you can build your own Champion/Challenger policy accordingly and introduce it to a subset of your borrowers. The insights derived from the process can be used to enhance your policy and progressively expand its implementation to a larger borrower base. The combined power of canary testing and challenger features will offer rich analytics that can help you mitigate the risks that can have a drastic impact on business performance and customer experience.
Let us look at a few advantages of how canary testing and Champion/Challenger models in business rules engine benefit your credit decision-making processes:
Back in FY23, the number of complaints against digital lending apps in India rose to 1,062 on account of fraudulent apps despite the crackdown by the Government and the Reserve Bank of India (RBI). However, even digital lenders also are victims of online fraud, which can leave them facing severe financial implications. To tackle frauds, lenders perform tough screening processes, which leads to rejection of good customers as well.
This is where BRE features such as canary testing and champion/challenger features come to the rescue to help strike a balance between fraud prevention and borrower satisfaction. You can use canary testing to evaluate the impact of test parameters, measure the relevance of specific data sources, or assess scope of related drivers.
Once you understand what could work, you can randomly test one Champion and five Challenger versions of the workflow in the live environment to determine which version will help you identify the impact of fraud rates and borrower satisfaction.
You can also quantify the trade-offs between customer experience and fraud losses. For instance, if you’re looking to increase loan application completion rates without the chances of frauds impacting borrowers, you can try out different identity verification workflows. This will help gauge the impact on fraud rates and drop offs by analyzing the genuinity of an individual through selfies, biometrics, and government-issued documents.
In the end, having a BRE with canary testing and champion/challenger features is like having a lending autopilot system. You can set monitors and get real-time analytics whenever required, thus giving you a peace of mind when dealing with critical data.
Underwriting
One of the significant challenges present in underwriting is that credit risk strategies are unlikely to interest senior management unless it involves profit maximization with limited risks. Even after the shift in focus from risk to profit is made, projected incremental gains can be overshadowed by big ideas and the challenge mainly lies in showing how small improvements can cumulatively lead to big changes.
Thus, canary testing and champion/challenger features become critical as they can accurately assess the future impact of a strategy change. Credit risk teams will be able to gather real-time marginal performance data in a series of controlled experiments that can deliver results on the impact of the strategy when rolled out on scale.
For example, if a bank’s credit risk team is looking to change their credit card approval strategy to increase acceptance rates, they can implement canary testing and champion/challenger models over a small, controlled group. By gathering real-time data on metrics such as default rates and customer retention from both testing modules, risk teams can use the results to analyze if the new strategy will work and how it will impact the larger population.
Both canary testing and champion/challenger features provide a robust framework for minimizing the risks faced by borrowers in credit decision-making. Testing new credit policies and strategies in live environments facilitate data-driven decision-making that offer more accurate assessments of a borrowers’ risk. Lenders can also identify flaws in the policies that put their entire borrower base to risk.
The combined powers of canary testing and champion/challenger features also promote continuous testing and refinement of new models, allowing lenders to adapt to market changes and ensuring their risk management strategies remain effective and unaffected by market changes. Through real-time data from BRE features, lenders can also conduct performance evaluations, leading to timely adjustments in credit policies.
By reaping the benefits of canary testing and champion/challenger models, you can revolutionize credit decision-making through real-time insights and live testing of new credit policies and strategies. Ultimately, the goal is to minimize risks for both borrowers and yourself as a lender. These methodological approaches to testing and validation will help boost innovation and continuous improvement in credit risk management.
You can also leverage CloudBankin’s Loan Origination System (LOS) as a testing ground for analyzing the creditworthiness of your borrower base and help set up proper workflows to process loans. Using the LOS’ automation and data analytics capabilities, you can run different canary tests and champion/challenger models to obtain relevant performance metrics such as approval rates, default rates, and processing times. To know more about our Loan Origination System’s capabilities, book a demo today.
Introduction In a world where access to credit can make
Many are faced with unexpected financial difficulties when a day
A robust Loan Origination System (LOS) that can establish strong
© 2024 LightFi India Private Limited. All rights reserved.
(Formerly known as Habile Technologies)
After smartphone penetration, people are not watching their SMS at all. They use SMS only for OTP related transactions. That’s it.
But What can a Lender see in your SMS after you consent to them?
Lender can see income, expenses, and any other Fixed Obligation like (EMIs/Credit Card).
1) Income – Parameters like Average Salary Credited, Stable Monthly inflows like Rent
2) Expenses – Average monthly debit card transactions, UPI Transactions, Monthly ATM Withdrawal Amount etc
3) Fixed Obligations – Loan payments have been made for the past few months, Credit card transactions.
It also tells the Lender the adverse incidents like
1) Missed Loan payments
2) Cheque bounces
3) Missed Bill Payments like EB, LPG gas bills.
4) POS transaction declines due to insufficient funds.
A massive chunk of data is available in our SMS (more than 700 data points), which helps Lender to make a credit decision.
An interesting insight on vehicle loans for lenders.
A trend we are seeing today – the first-hand vehicle ownership is decreasing with time. Why? People are upgrading their vehicles in every few years because of technological advances. And, this can be seen more with the millennial generation.
So, what should a lender do in terms of financing?
– Estimating the residual value of the vehicle at the start of the financing period.
– Charging a borrower only for the residual value (which is the difference between the value after a few years and the current value)
Example: A bike currently is INR 1 lakh. You want to buy the vehicle for 2 years. A lender will estimate the residual value of that bike today and what it would be after 2 years. If the estimated residual value = INR 45,000, the lender will charge you only that (say, INR 55,000 with interest for this instance) during your tenure.
At the end of 2-year period, you have 3 choices:
1. Return the bike and upgrade to a new one without going through the struggle of selling it.
2. Pay the lump sum remaining amount to own the vehicle outright.
3. Extend the financing and own it by keep paying the EMIs for the remaining amount of the vehicle for the next 12 or 18 months.
Benefits for the borrowers?
– Flexibility to use a vehicle and upgrade to a new one.
– Affordability to not pay for the complete value of the vehicle with the intention to use for a lesser amount of time.
– Convenience in owning the vehicle.
Say goodbye to the old lending option and embrace the new way of financing for vehicle by lenders!
How many of us know this?
1) Tiktok does Lending ( is it an entertainment company or social media company or a fintech company?
2) Youtube China does Lending
3) Top 100 internet companies in China(no matter what business they are in) do Lending
The team which was heading Lending in Tiktok was the Advertisement team. If we do Ads, we do X no of revenue. But if we do lending, we’ll get X+30% more revenue. This is on the same Ad spot.
Ad team has transformed into a lending team, and in today’s world, it’s possible because the subject matter expertise can be put in as an API and given to you.
Embedded Lending as a service is becoming popular in India too, and I am happy to be part of this ecosystem.
The answer is No. Only the top 10 crore people have access to many credit products in India. Almost all Banks focus on this market.
Once you go beyond that, the credit access rate has dropped significantly due to multiple factors.
1) Customers who are having low income(30-40K per month)
2) Not earning from an employer who belongs to Category A or B
3) Not from Tier 1 or 2 cities
NBFCs and Fintechs focus on the above segment, pushing another 10 crores of people.
But in India, 70 crores more people are formally or informally employed, which still needs to be tapped.