As global markets began reacting to the uncertainty of the COVID-19 pandemic, organizations involved in supporting trading, settlement, valuation, and other operating activities faced unprecedented volume and volatility while also transitioning their working models to incorporate social distancing practices.
One impacted process was the calculation and certification of an accurate net asset value (NAV) for investment funds. This critical function immediately became more difficult as fluctuations in security prices caused high volumes of price tolerance exceptions even with increasingly sophisticated validation models. Such models are designed to identify securities that are incorrectly valued and could, in theory, cause a NAV error. Examples of this include scenarios where a corporate action was not processed correctly (e.g. stock split), or a security was incorrectly priced.
The sophistication of these validation models has grown exponentially in recent years thanks to advances in machine learning and AI technology. BBH has recently gone through a similar transformation. Knowing that our models were producing thousands of exceptions for our analysts to review each night, with less than 1% representing a true issue, we turned to AI to help alleviate the strain of manual review and to enhance efficiency and control. In 2018, we introduced ANTS, which uses predictive analysis and machine learning to eliminate the noise of false positives and highlights exceptions that might not have been previously detected. With ANTS, analysts spend less time sorting through recurring exceptions and more time on resolving true NAV issues.
In recent weeks, however, we saw a significant increase in pricing exceptions, which makes sense considering the immense volume of trading activity since the crisis began. Realizing we could be facing a prolonged period of volatility, and knowing the answer was not to simply assign more people to reviewing exceptions, we once again turned to the model. The goal was to review adjustments to the algorithm that would further reduce exceptions in a safe and sustainable way. While we are always making incremental adjustments to the model, it was clear that addressing the impact of this crisis required rapid mobilization across multiple functions — from our data scientists to our fund accounting team and everyone in between. In mobilizing these experts, we could further sharpen the processes we instilled at the inception of our AI program. This is at the core of what we call pragmatic transformation — incremental adjustments that yield meaningful results.
In many ways, the events this year have validated our key principles in planning ahead for adoption. Understanding that this era of “new normal” will likely throw some additional curveballs, we remain committed to an iterative approach to refining our technology. Here are the key takeaways we keep front of mind as we look to the future.
1. Recalibrate and adjust to improve over time
No prediction model lasts forever — they will all degrade over time due to seasonality, changes in production data, new events, and patterns which lead to continual adjustments and fine tuning. Just consider the events of the last couple of months: The average daily market change of security prices in March 2020 was over 5%. The next highest month on record was October 2008 at 3.8%. Our previous model’s data set had simply not seen this type of volatility, so adjusting the model quickly was essential. In fact, when back-testing ANTS against our legacy model, we saw a reduction of ~77% in pricing exceptions that needed to be reviewed. Considering we value hundreds of thousands of unique positions, these percentage impacts reflect a significant reduction in work effort. But it was also important to understand bigger patterns that were emerging. This is where ANTS proved its mettle: instead of our analyst and data systems teams spending their time sorting through the exceptions (a task assigned to ANTS), the model identified larger emerging themes. Rather than constructing a new decision-making logic from scratch, we were able to build off a previous iteration and home in on those patterns to make small adjustments. We continue to make these adjustments and are optimistic that a new version of the model can reduce those exceptions by another 5-6%.
2. Data purity and accessibility is a pre-requisite for agile
Data, it’s often said, is the fuel that ignites the AI engine. That’s why real-time views into volumes and trends is paramount in adjusting to a dynamic environment. Data is useless unless it can be easily accessed, normalized, and understood by business users and IT alike. For us, we knew we couldn’t implement AI without a streamlined data inventory. So at the launch of ANTS, we standardized metrics to monitor AI accuracy, application utilization, and efficiency. Doing so at the beginning is paying dividends today because we are able to surface, in real-time, the most pressing pain points. These metrics are critical for the long-term success of AI adoption, giving us insight as to when and how often to retrain a model.
3. Precision beats speed to market any day
Like many in our industry, we use the agile methodology for our technology development. But agile development requires integrated business and technology teams operating in a single framework that incorporates risk and control protocols that facilitate analysis, development, testing, and deployment – no easy feat. Coordinating among these disparate functions cannot be rushed. Nor should precision and accuracy take a back seat to speed to market. Given the business-critical function of NAV calculations, oversight and auditability is central to the sustainability of an AI program. A possible framework for consideration:
- Designate responsibilities between model research and proposal versus model testing and acceptance
- Determine KPIs to objectively compare and select an optimal solution
- Ensure models are archived and documented for proper version control oversight and lineage to audit the AI decisions in a production environment
Some of the early lessons from the impacts of COVID-19 were really a validation of our initial AI and machine learning transformation journey within our fund accounting product. However, it has further influenced our thinking as we continue to refine our approach. In the above example, the process to analyze and react required data extraction, review with data scientists, and some degree of translation into business terms for our subject matter experts to make a decision on the best course of action. In the future, we see benefits in allowing the business user to test and toggle between multiple AI models within the application in real-time.
The extraordinary volumes have also proved the accuracy of our models on a larger scale. This has deepened the confidence our practitioners have in the tools and willingness to increase their utility. These events continue to emphasize the importance of new skillsets required of our workforce. Analysts no longer need to know the steps to carry out a task, but how to analyze the performance of the system to ensure it is accurate and learning correctly.
As firms consider how to respond to operational impacts of these extraordinary market events, it is evident we must continue focus on reskilling and broadening understanding of AI and machine learning in addition to developing subject matter expertise. As we look for ways to scale and improve capabilities in these technologies, this is a necessary balance in skillset and mindset. While the impacts of COVID-19 will certainly alter future operating models across the industry, we have leveraged the opportunity to refine and advance our efforts in line with our pragmatic transformation.