为什么没有算力市场，AI 创业公司可能走向消失：Evan Conrad 谈信用与 GPU 供给

Brief Description

Evan Conrad explains why compute markets require both cluster operations and a liquid capacity market, and how credit risk, deposits, lead times and provisioning constraints threaten new AI companies.

Table of Contents

• Intro
• Physically settled compute
• Supercomputing as a service
• How a struggling AI lab became a GPU broker
• Two miracle problems
• Sold out, deposits and delays
• It's all one risk problem
• Credit, not power, is the bottleneck
• What a compute market has to solve
• Why you have to operate the cluster
• Why nobody does physical settlement
• From $10M to $1B deposits
• Why startups need this

Intro

So, this is a talk about compute markets. but it's kind of more of a talk about why to make a compute market. if you don't know me, I'm Evan. we run a company called San Francisco Compute. we make a physically settled order, physically settled forwards market. so if you were here for the last talk, last talk was basically why all the operators of GPU clusters don't really work very well. when we say physically settled, we mean we're the operator.

Physically settled compute

We're both the operator of the hardware in the same way that a cloud is the operator of the hardware and then we're also the market at the same time. It's kind of like a grid for compute. So the general gist of this is that in order to run and operate a supercomputer we think problems start right at the design of the supercomputer right in the design of the bill of materials it's control of the backplane fabric as well it's also control of bare metal orchestration it's also services on top VMs and so on there's a lot of different parts of the stack that you have to do effectively otherwise the folks who have a data center and would like to make more money on that data center by becoming the GPU cloud end up failing for the customer. So that's what we do.

We offer what's called supercomputing as a service.

Supercomputing as a service

We go to folks who want to become a cloud. That's like folks who want to become what Nvidia would call an NCP or venture firms that want to have clusters for their portfolio and want to be able to reallocate that capacity when a company like Instinct is taking off. or AI companies who would like the residual value on the compute because they've realized that's really important to their financials but don't want to operate a supercomputer. So that's physical settlement and then on top of that we run a live order book for capacity that you can buy reserved either right now or in the future.

How a struggling AI lab became a GPU broker

this together is not at all what we thought we were going to be doing. Originally, we were this company called Junlark that was trying to make kind of like Sunno or UDIO, but prior to Sunno or UDIO. at the time, what you needed to do to realistically start a lab was you needed to raise a big deposit, a huge one, which was $10 million at the time. we thought that was a crazy amount of money. and so we thought, no, that's stupid. We're going to raise 500k and obviously what we're going to do is we're going to rent from somebody else. obviously today it's not $10 million, it's a billion dollars. And nonetheless, we had raised our 500k.

We buy a big cluster and then we're underwater on our cluster every month. and if we didn't sublease our cluster, we would have gone bankrupt. so if you've heard of GPU brokers, we were the first GPU broker because we were forced into being a GPU broker. So the first year of the company was us just trying to sell our cluster, otherwise we would go bankrupt. so I would wander around with these like actual pieces of paper at Nurups which both say H100 and A100 at the same time because that's how I made I made the poster so quickly. and so this talk is not really about necessarily like how a compute market works. It's how a compute market gets to exist.

Two miracle problems

we like to think about it like there's this pamphlet and on the pamphlet on the front side it's like man starting a Neolab sucks. There's all these different like financial and economic problems and on the back of the p back of the pamphlet it says compute markets. It's not super clear how one gets to the other until you open the pamphlet and then like oh my holy there's like so much stuff in this pamphlet and it keeps going and there's so many folds. It's just a very difficult and challenging problem because you need to solve two like miracle problems to get working.

You have to both make a market which is a uniquely difficult problem because it has a cold start problem. You also have to build a cloud and then you have to not become a cloud. which is a whole separate challenge. But these are the problems that the labs are facing and it's about to be the problems that everyone is about to facing. So if you work at a startup these are your problems now. right now everybody is sold out of compute. this is probably not going to change for a while.

Sold out, deposits and delays

so that means that you're going to have to buy a new cluster, a net new one, and there's a lead time. You have no way to pay for that cluster to come online now. You're going to pay for it to come online later. There's nothing in the market that can realistically get that cluster online for you now. meanwhile, you're going to pay your entire bank balance as a deposit. the reason you're paying your entire bank balance as a deposit is because the lender has no idea how to render underwrite AGI. how in the world are you supposed to do that?

You've got two person startups in San Francisco who or maybe New York who say I'm going to build super intelligence. No lender knows how to underwrite super intelligence. so they have no idea what the asset is worth. meanwhile your cluster is very likely to be delayed. That's because the lender needed to underwrite against the credit risk of the customer and so they needed to know who is the customer first before you deploy the cluster. So right when the customer signs is the riskiest point in the entire operation. you wanted it to be the case that you built the cluster first and then you sell it.

This also helps reduce all sorts of different types of problems where clusters break. Meanwhile, most of the people who are setting up clusters are people who have the actual asset that makes the thing cheap. Like you've got the data center space already. What that doesn't get is supercomputing expertise. as we saw in the last talk if you've watched that one. meanwhile, you're going to run into the exact same problem that Junark did, which is you're going to either way overby or you're going to weigh underby. that's because AI doesn't have margins that look like SAS companies.

And so you don't have very much margin for error. And so the chances that you either underby and somebody else wins because they did it correctly or you over buy and then you just become stability AI or you blow up. you end up like character AI are really high. So these are all the same problem. They're all just different types of risk.

They're technical risk, their financial risk, their credit risk. they're all one big bundled risk problem.

It's all one risk problem

which is the problem that we end up solving and it's also the problem that we think compute markets broadly the entire industry solves. so we'll walk through each one of these and walk how that gets to a compute market because if you don't understand these, it's kind of like unclear why you would want a compute market. and it almost seem like this problem that somebody thinks, "Oh, wow. Compute markets are cool, so I'm going to go start one." We think they should be derived from the actual problems that people face, which is first, everyone is sold out. Why are they sold out? Because the lender couldn't finance the cluster unless you had a long-term offtake agreement.

In CPUs, this wasn't the case because CPU buyers are SAS companies. SAS companies have high margins. And because they have high margins, you can build a product that rewards you for having a great that rewards you with high margins for your your CPUs. think about it like this way. If you're a SAS company and your underlying cloud doubles their prices, you don't go bankrupt.

You hate your cloud, you're mad, but you don't go bankrupt. If you're an AI company and your underlying cloud doubles their prices, you go bankrupt unless you go to some other underlying cloud. And what that means is that you're actually price sensitive. And because you're actually price sensitive, and there's nothing keeping you at any particular cloud.

So you can't sell on demand because your customers will immediately leave to whoever is cheaper. And so that means that in order to finance GPU clusters, you have to have long-term take-pay contracts, hopefully with someone who has investment grade credit. However, if you don't have liquidity on those, if you're selling reserved and your customer can't get out, then that also means that some other customer who wants the capacity right now because they're like instinct and their product is taking off can't pay for anything. They have to finance a new cluster which has a lead time because the physical world just exists and it's going to take a while.

Sometimes you even have to build the damn data center.

Credit, not power, is the bottleneck

and then because you're financing a new cluster, you're going to pay a deposit because almost nobody in AI has any reasonable credit. And because this is a brand new technology, all these companies are new. Meanwhile, even as you start to scale up that ladder, you're still going to run into credit issues all the way through. It's the number one bottleneck of AI at the moment.

It's not power. You hit credit way before you hit AI or way before you hit power. So, what that means is every inference company, every Neol, they're bottlenecked by how much cash do they have in the bank account. And so they go back to the venture capitalist who says, "Oh, I'm going to fund at the model layer, the application layer, the inference layer because I don't want to fund all of that capex stuff." and so the venture capitalist then funds all the capex stuff by giving the the new neolab or new inference company the check that they then send to the cloud which is the deposit for the cluster and that's how you underwrite the customer.

So the clusters then are delayed because the c the cloud couldn't actually deploy the cluster until they had the customer in hand. And so that was the riskiest point. So therefore what should have happened was the cloud had bought the cluster ahead of time and then waited until it was landed in the data center and then sold it. And that means you have way less likelihood of delays.

Meanwhile, the cluster is very likely to break when you get the cluster as the customer because you don't have liquidity. And so, if the failure rate is higher than what you expect and you don't have the ability to transition a node in, then you don't have anywhere you can pull nodes from, which means that if you need new nodes to make up for the failed ones, you have to RMA them, which means you have to get a new node. And then that gets us back to several steps before where you have the lead time problem again. And so, everything breaks for everybody all the time.

Once you have gotten through all of that as a new AI lab, your problems are you know you nobody was sold like everyone was sold out. you then had to wait a lead time, you paid this big deposit.

What a compute market has to solve

the thing was delayed and then it broke and finally you got through all of that. and the likelihood that you predict your demand correctly is very low. And so you're either going to be way overprovisioned or way underprovisioned and you have no way out. there's no way to sell your capacity to somebody else and there's no way to buy more capacity if you need it. So, computer markets to solve these problems, you need to solve a bunch of different problems all at once. you need to underwrite the value of the GPU cluster rather than underwriting the value or underwriting the credit risk of the customer. this is how it could work. it's not guaranteed to work like this. In other industries, what happened is everything just consolidated.

You got one big creditworthy customer that does the whole damn industry. That's what the path of AI is currently on track to do. It's instead of going to be a liquid market where anybody can participate, it's going to look like TSMC. You're going to get one buyer or a few buyers that are the clouds.

You're going to get a few model labs that are the AI labs. And then you won't get other startups. this is what we think is going to happen. And this is why 3 years ago when we were doing our AI lab, we had a chance. We could have said, "No, we're not gonna do our new we're not gonna like, you know, stick it out for several years just trying to keep our company alive. we're just going to go do a new AI lab." But we saw this problem and we said, "Oh if we don't do anything about this, then we will just hit this 3 years from now."

Why you have to operate the cluster

And here we are 3 years from now hitting this problem and where everyone's going to hit this problem. So as if you watched the last talk it's very hard to do solve this problem unless you are operating both the market and the clusters at the same time. so this is Dylan Page. Dylan is a co-chair of the CNCF infrastructure group. That's the organization that makes Kubernetes formerly worked at Lambda doing infrastructure there. and the reason and he joined San Francisco compute because GPU compute isn't sufficiently standardized reliable for the market operator to be divorced from the physical infrastructure.

It doesn't work if you aren't the one who's buying the thing. Sorry, it doesn't work if the person you're buying from isn't operating it. To give an example of why this doesn't work is I'm sure some of you are flying in from overseas or sorry out of town and you have a hotel room. wouldn't it be great if you could, you know, sell that hotel room to somebody else. I go to you and I want to buy that hotel room. and I say, "Can I get room service?

Can I get my room cleaned? I've got a problem with my room. who do I call? if you don't operate the hotel, you can't fix any of those problems. That's why brokering clusters doesn't work. You have to operate the cluster like the hotel.

You have to run the whole damn thing. Otherwise, you can't actually make a market. It doesn't work. You can't do physical settlement. so why does nobody do physical settlement? the problem is there's a very strong incentive to never do physical settlement.

Why nobody does physical settlement

that's because if you build out physical settlement for compute, what that literally means is you're building out the entire operating capacity of a GPU cloud and if you are building out the entire operating capacity of a GPU cloud, you should just be a GPU cloud because it looks to be the case that you will make an ungodly amount of money. you will sign billion dollar deal after billion dollar deal. And so, it. Let's not make a GPU or let's not make a compute market. let's instead just become a GPU cloud, which is what basically everybody does because the numbers are astronomical and it's a very reasonable thing to do. the reason why we ended up building physical settlement was because we knew that if we didn't start this, we would then not be able to start an AI lab, which was the real damn thing that we wanted to do in the first place. We wanted to make a lab.

We didn't get to make a lab unless this exists. So, when we started, it was 10 million as the deposit that you had to pay to start a new lab.

From $10M to $1B deposits

This is like how you get over the credit risk problem. then the cluster sizes got bigger. The sort of minimum ante into the space got bigger. Then it was 100 million. Now it's a billion.

We don't think it can continue. We don't think there is a world in which you can get a $10 billion round right out the bat before you raise before you have revenue or traction or your product or anything. and what that means is that if you don't solve this problem, we are currently on track to not have any startups in AI. now one of the things that you could think could happen is maybe there can be new inference companies and new model labs. And the way that they solve the credit problem is that instead they just ship the risk to the application layer. They say look I will I need to get over my credit problem.

So what I'm going to do is I'm going to only sign customers who are investment grade credit, you know, Fortune 500. I'm going to only sign customers on five-year contracts for tokens. I'm going to only sign customers for tokens that require a big deposit upfront. that is the end of everything. if you have inference layer getting bottlenecked by credit, what happens is you can't get tokens unless you have credit or you have a big deposit. And the problem of that is that tokens are used by the whole damn economy.

There is no rule that says that startups have to exist.

Why startups need this

if every startup is suddenly bottlenecked by credit in order to access intelligence, what happens is existing firms just ship everything. They themselves get to make the product because they've got credit and then they make the revenue from that product and then it makes their credit better and that's a flywheel that makes consolidation happen throughout the entire economy. If you don't fix this bottleneck, there aren't any startups. This is why we started doing this and because at one point or what we think is going to happen is you're going to end up in like 3 to 5 years you're going to look out and say I want to start a new startup and to get there you need to buy so many tokens in order to compete with the existing partner or like existing players.

So you need to buy a hundred million dollars of tokens. we think this is what you are actually going to have to do in order to start a new company because other people are spending hundreds of millions of dollars of tokens. If you just draw the line out, this is what happens. So, you need a big giant deposit because you're two people who have no idea what they're doing and so nobody trusts you. So, you're going to go and raise $100 million from your venture capitalist. meanwhile, nobody wants to work with you because you're a two-person startup.

Then, you're going to be asked to pay for that hundred or you're going to be asked to pay for a 5-year contract on that 100 million if you're not paying it all as a deposit. And then you just don't get new model labs. You don't get new startups. you get a consolidation in the industry. So we at one point were this little lab and what we did is we lived in San Francisco and we knew all the other people who were also just two little folks who had started a startup and wanted to be able to participate in the most important technology of our time. so I want these people to exist.

I want this person to exist and I want this person to exist. I want this startup to exist and this startup to exist. This one and this one. If you get these companies going, maybe they'll become big folks that build your cancer cures or they build something that nobody thought of or something that wasn't just sort of AI slopped out.

This is kind of the problem that compute markets solve. And if you don't solve it, we don't think there's going to be any startups.