银行如何自训基础模型识别欺诈：Wells Fargo 的 Freddy Lecue 分享实践

Brief Description

Freddy Lecue explains how Wells Fargo built a transaction foundation model from scratch using customer transaction data. The talk covers tokenization, transformer architecture, training and fraud detection, and discusses using a shared backbone for additional banking tasks.

Table of Contents

• Why a foundation model for transactions
• Why not just XGBoost?
• Billions of transactions
• Language models vs. transaction models
• Pre-training from scratch
• Who, what, how, when and where
• Turning tabular data into tokens
• Architecture and scaling
• Training setup
• Fraud detection results
• Recap

Why a foundation model for transactions

And I'm going to show you the work we've been doing lastly on building a our first foundation model Wells Fargo foundation model. So I explain why we did that, how we did it and what we accomplished in doing so. So first foundation model it doesn't necessarily to be language based.

I mean we all playing with Claude and chat GPT and others they all language based foundation models. In financial services we also have language of course from a structure and structure representation data but we have also have a lot of structure representation. A lot of this data is tabular data.

We have a lot of information from our customers. We have information in investment banking, consumer banking and all of these data is data that we have in Wells Fargo and all fun big financial organizations have to and we want to take benefit of all of these huge amount of data that we have and it's not like you just take it and put it in a in a language model and get the best of it.

Of course you can get a lot of benefit out of it but it's not necessarily the best way to do it. So what we decided to do is to look at the data we have all the transactions all the customer transactions that they're doing with their debit card, credit card, digital check all of those coming from many different channels and see if we can build a foundation model that we can use and fine tune for downstream task like credit scoring or that you can use it for fraud detection, anti-money laundering, do marketing. So how we can build one backbone that you can use and streamline across downstream task.

So now you're moving from many models very solid across different organizations in a really one backbone and every organization can just reuse because you learn the representations of customers transactions at scale.

So, that's the intent. And just for that, you may say, "Hey, that's tabular data.

Why not just XGBoost?

Why do you just don't use XGBoost and the usual suspect?" We did that. This is the model we had in production. And but the model I'm going to show you is something that get that beats XGBoost even from a latency perspective, but also from a from a fraud detections, we get much better false reduction of false positive and better detection rate.

So, it was a really good lesson learned in term of having better model using foundation model. And to foundation model, I've been transformer blocks.

Billions of transactions

And that's that's what we built. So, so, we have as I mentioned billion of transactions on a monthly basis our customers interacting with their debit, credit card, and they do check, and they do different sort of transactions. And the question is really how we can use these informations at scale and not for one and every single downstream task.

So, I'm going to cover a bit of the data, the representations, the architecture that we use, how that what the result we get, and the lesson we learned in building this.

Language models vs. transaction models

All right. So, I mean, left-hand side, language foundation model, they're processing language tokens, and then the goal is to learn a representation that's going to predict the next token. And then you have you have supervised fine-tuning, and then you do our LCHEF, LRVR, and all of the great post-training process to get better at the task that you want to, right?

And that's for languages. That's great. but what if your data is very much more specialized in term of representation. So, that's where we build this transaction foundation model.

So, instead of having languages as the base, the base is now transactions. Transactions from our customers. And you still try to predict the next thing and here the next thing is the next transaction or it's the next token in the transaction using all of the channel that you have.

But that's not the end goal. I mean, the similarly to language model where you try to predict the next token, that's not the end of the story, right? You learn a representation and then you use that representation and then you fine tune it and you post train it for any other task, right?

So, it could be classification task for languages, but in our context for transaction foundation model, it could be used for credit line increase, could be used for doing fraud detections and flagging bad actors from an anti-money laundering perspective.

Pre-training from scratch

And the goal is to learn this relationship between transactions over time and transactions across customers. That's really the intent. So, the intent number one is you do pre-train from scratch.

I'll say it again because people sometimes confuse. We pre-train transformer blocks at scale from scratch using our transaction data. So, the weights is zero weights that we use from open weights.

We just use the architecture and then we pre-train it from scratch using our GPU cluster. And then we train it for couple of days and weeks. And then you do the downstream task of post training.

But the goal here and you pre-train it for predicting the next token or the next transactions. And you try to do this. Well, you try to predict the next the next pattern of the customer.

But that's not the intent.

The intent is to do the downstream task later on. And in term of scale, I mean, we operate at a large scale. So, we have billions of transactions, a million of accounts, hundreds of transactions per clients on a monthly basis interacting with their debit card, credit card, or doing check.

Who, what, how, when and where

And the five main dimensions we look at is really who is doing I mean, who is processing and who is asking on who is providing a service. So, really the customer or the merchant. the what, so I mean, what do we what do we buy? What do we transact?

And how, so which channel? Is it credit? Is it debit?

Is it digital? When is it happening? And where is it happening?

And all of those, they basically breaking down in Android of features. And if you look at the existing model we have in production, there's a lot of I mean, and people that been build those model in similar organizations, my understand that we build a lot of very hard crafted uncrafted engineering feature engineering effort on that. And that's required a lot and a lot of time.

So, we step back and look at really the raw features we need, not the one that been really pre-built and post-built for doing the downstream task, but try to understand which raw features.

And we ended up with basically 100 features. Even with at the end of the day, we had thousands of them, but we tried to find which ones are really the raw information unit. And then we let the architecture, the transformer blocks to figure out the relationship between those core features.

And as you can see, we do have a yeah, a bunch of features on those five dimensions. And the vocabulary is very unique to the financial services setup. So, and that's the reason why I mean, we didn't really use existing pre-trained model because those the vocabulary that we use is very, very be to the work that you want to do and even if you merge well all digital and checking and saving and all of those sort of transactions, even if you merge all of those, you still end up with a very limited numbers of features and characteristics.

Turning tabular data into tokens

So at that scale then the question is well how you work with that? So we started everything is tabular data. I mean there's no secret it's tabular data and then the goal is to understand how you going to capture the vocabulary because you do have numerical values, you have categorical values, you have cat I mean you have information like zip code.

Do you encode each and every zip code in a token? Does that make sense? So there's a lot of effort need to push in understanding the vocabulary that you need and vocabulary that you want also to scale because at the training if you have a too huge vocabulary, it would be kind of messy to train such a model with a vocabulary which is far too big.

So there's a lot of effort in understanding this value to token. So which information do you encode in a token?

And then you have the token 3D, the tokenizations and then the embedding lookup. So then when we have a transaction coming up, you do have the vocabulary. So it's a basically it's a lookup and aligning a transactions and all the characteristics of that transactions to a token and an ID and then an embedding.

So it just to give you a number well some numbers. So we have quantitative, qualitative, categorical of different kind, right? So categories with only eight types, some with much bigger types.

You have transaction amount which is a big deal. You have the delta time between transaction. So you have anything related to IP address, postal code to just to give you example of variety and heterogeneity of each and every characteristic could be encoded in a binary token or could be coded in a much larger number of tokens.

Architecture and scaling

And even if you just look at the bit that the zip code, I mean, you have a much larger number of zip code in the in the US than those 2800, but you have fancier, well, you have much better way to structure how you're going to regroup zip code based on distributions, for instance. So, architecture So, so here just what we So, we use the Chinchilla well, mathematics behind the to define how we build it, right? I mean, initially, cuz we had no clue how it's going to work in a tabular space, even though there are existing well, well, open weights and open solutions for doing tabular data foundation models. so we started from what they learned and what I did to what we needed to do.

And I'll give you an example of innovating dimensions, number of decoder layers we used, and all of the recipe from a from an architecture perspective.

so we have And as you can see, we're not trying to build a trillion parameters model. We don't try to compete with any of the leaders in the language models, because we're working really and focusing on transactions, right? Transactions foundation models at billion level.

And one 1 billion is because that's the model that give us that was the well, kind of the smallest and give us the highest boost. And now we're working on a on a next version that's 11 billion, but that's good enough to give a lot of ROI from a false positive reductions for downstream tasks, like fraud detections and detection unit detection rate. So, and the and the type of architecture So, the decoder-only architecture, that's basically for predicting the next the next token or the next transactions.

And that's really the pre-training. So here we initially used 20 times the number of parameters to define the number of tokens we might need. We ended up actually 10 times factor than that.

So by doing a few experimentations, we needed much more much more tokens per parameters. I mean this is this kind of the initial experimentation we did. So I'll pass on this because this is the usual suspects from objective functions causal language modeling and mass language modeling type of optimization work.

Training setup

And yeah, from an architecture perspective, nothing really I mean fancy here. The again, the most complex part was really the tokenization. And then and then adapting and doing the pre-training.

And then how you do the fine-tuning. So if I would say the recipe or the secret sauce is really how you do the tokenization and then how you going to do the post the post training. But anything from a training perspective, it's it the question is just how many how many transaction you're going to put in your in your context or in your historic for doing the training.

And evaluation perplexity base accuracy dashboard on downstream tasks, that's that's the usual suspect. And so Yeah, so use mixed precision distributed data parallel maximize micro batch size get faster convergence yeah, using one rope usual and memory efficiency flash attention activation checkpointing.

I mean here there's nothing really new. It just we used the advices from the from the best and from the one that built even much bigger models. So I'll pass on this one.

Time is short, but I wanted to spend some time yeah on the results. So Yeah, so the when you build this foundation, that foundation is really is of is really is on your backbone. It's something you can use for downstream task.

Fraud detection results

And the downstream task was to do fraud classification. So you have a transactions and then you try to predict if that transaction is going to be fraudulent. So in the model we have in production, it's an XGBoost model and it's trained on large amount of transactions which is fraudulent or non-fraudulent.

And the missing piece is that was the relationship between whatever transaction you try to predict try to classify and the relationship with this transaction with all of the past transactions. And that information is kind of gone when you use an XGBoost model unless you encode it in a very hard way which is which is not so easy. But if you use transformer architectures, you do have this nice attentions from a from transactions to past transactions.

And that's basically what helped and this is what we think this is what really helped pushing the fraud detection rate much higher.

But at the end of the day the downstream task was to say here's the sequence of transactions. I have someone just using the debit card. Is that transaction fraudulent with respect of all his historical information?

And that's that's exactly what you put in at inference time when you try to do the prediction. So that's the yeah architecture. So nothing really fancy here.

You just need to have the good engineering in place to do less than 250 milliseconds for doing the processes because the processing because the decline need to happen within that time frame. But it's it's it's we were well 10 times slower than that from a latency perspective, so it was all good. And just in terms of final result we get.

So, the main metrics and those metrics relate right away to dollar to cost well to savings for the firm.

So, it's detection rate and false positive rate. Of course, you can detect much higher, but in many cases sometimes we get much more false positive. And in this case, we managed to get more weak flag more fraudulent activities and we even reduced the false positive.

So, that was a big win. That was a big win because at the end of the day, if you decline a transaction which is not fraudulent, then that person is going to call. There's often operation cost associated to it.

You lose the transaction related to that non-fraudulent action. So, it's a it's a very it's a very well, something that you don't really want to happen. So, so again, that's this that's compared to the state of the art of XGBoost model using fraud detection.

So, it's it's basically huge it's millions of dollar savings.

So, it's it's a very big boost. So, just to finish up on that slide. So, So, that foundation model has been built on billions of transactions from Wells Fargo customers.

Recap

That's number one. It's it's pre-trained from scratch. So, we don't pre-train from any open weight existing model.

We use existing architecture, but the way they zero rate, we pre-training on couple of days. I mean, we did it in a couple of weeks initially, but then that's pre-trained. And then you and then when it's pre-trained, it's pretty good in predicting the next transaction.

And when you done with that, then you can use it for many downstream task. We did it for fraud detection. Now we investigating for credit line increase, investigating for anti-money laundering, investigating for marketing.

So, now the same foundation is used across different organizations even Wells Fargo. And yeah. Thank you for having me and thank you.