Interesting that the hardware is NVidia Blackwell, not Google TPUs. That means Google will likely have an energy efficiency and cost advantage, and keep their proprietary hardware out of other people's reach.
Getting a whole business set up to build TPU hardware for third parties (design, build, sell, support, etc.) is probably not worth it when there is overflowing demand for TPUs in their cloud already.
Businesses running their own hardware probably prefer CUDA as well for being more generally useful.
Part of the reason for this is likely due to customers preference to have CUDA available which TPUs do not support. TPU is superior for many use cases but customers like the portability of targeting CUDA
My limited understanding is that CUDA wins on smaller batches and jobs but TPU wins on larger jobs. It is just easier to use and better at typical small workloads. At some point for bigger ML loads and inference TPU starts making sense.
Not really. Reverse engineering a modern chip is no small feat. Any company capable of it is also capable of designing their own from scratch. However getting something taped out (and debugged) on a modern process is massively expensive.
I did my undergrad internship on federated learning. I was tasked with implementing in a simulator different federated algorithms, so to have a way to compare them in a meaningful way. The last that had to be implemented was FedMA. We didn't manage to do it. That algorithm is absolutely devilish. Every issue that I solved made other two issue arise, and neither my supervisors could help. The sheer idea of matching neurons in different networks might (and does) make sense, but the way the approximate costs are calculated require other 2/3 math papers that I could follow for only the first lines of the abstract. I'm happy for the time I spent in my internship there. I'm also happy it's over
The general understanding of how it works is surprisingly easy though, you can find the paper here https://arxiv.org/abs/2002.06440
That's the point of the privacy scheme. It would only be able to learn things common to multiple clients. Private data wouldn't make it through the noise.
A bit thin on detail, but will this require confidential VMs with encrypted GPUs? (And I wonder how long before someone cracks SEV-SNP and TDX and pirate copies escape into the wild.)
The Common Crawl is going to become increasingly contaminated with LLM output and training data that is more likely to have less LLM output will become more valuable.
I see this misconception all the time. Filtering out LLM slop is not much different than filtering out human slop. If anything, LLM generated output is of higher quality that a lot of human written text you'd randomly find on the internet. It's no coincidence that state-of-art LLMs increasingly use more and more synthetic data generated by LLMs themselves. So, no, just because training data was produced by a human doesn't make it inherently more valuable; the only thing that matters is the quality of the data, and the Internet is full of garbage which you need to filter out one way or another.
But the signals used to filter out human garbage are not the same the signals that would be needed to filter LLM garbage. LLMs generate texts that look high-quality at a glance, but might be factually inaccurate. For example, an LLM can generate a codebase that is well-formatted, contains docstrings, comments, maybe even tests; but it will use a non-existent library or be logically incorrect.
The number of folks that have the hardware at home to run it is going to be very low and the risk of companies for leaking it is gonna make it unlikely IMHO.
I don't know – if there's still dumb money being thrown towards AI in non-tech and non-privacy-heavy industries, especially ones traditionally targeted by ransomware, there'll always be a chance of datasets getting leaked. I'm thinking retail and consumer product-oriented companies. (There's always non-Western governments without strong security orgs, too.)
Financial firms with significant on-prem datacenter use will love this as well. My company still stays away from the cloud -- we have 6 DCs in the building, and run everything else out of colocated racks.
I don’t think so. To my knowledge GCP has no approval for classified networks, which is by far the hardest part. Contrast with Azure OpenAI has been approved to run on government networks for over a year now.
This feels like a play for companies in highly regulated industries, GCP has a notable list of biopharma customers.
>Today at Google Cloud Next, we're thrilled to announce another significant milestone for Google Public Sector: the authorization of Google Distributed Cloud Hosted (GDC Hosted) to host Top Secret and Secret missions for the U.S. Intelligence Community, and Top Secret missions for the Department of Defense (DoD).
> Our GDC air-gapped product, which is now authorized for US Government Secret and Top Secret missions, and on which Gemini is available, provides the highest levels of security and compliance.
Banking as well, this is the kind of offering they've been looking for a while. Google just saw the demand decided to jump in while OpenAI and Anthropic probably calculated they don't have the manpower to deal with the support for this.
With a few exceptions for companies with highly secretive data, you do have to be a government agency or working in a highly regulated government-adjacent area for secured private clouds to be a requirement carved in stone and therefore worth investing a ton of extra money into though.
Neither approaches the secrecy needed by government installations. Health care and banking leak PII regularly and never really suffer any consequences.
i'll add that on-prem is getting 10-100x easier than it was 10-20 years ago (still very hard), and "i want to run this in my own datacenter" is becoming accessible to much smaller companies than just F500 enterprises
Curious if this was forced on Google Cloud by Sundar, or was it something that Google Cloud as an org wanted to do?
At first glance, it seems Google Cloud might lose some revenue from customers who can now deploy Gemini in-house. On the other hand, it's not a complete loss, since presumably Google Cloud is still involved in providing some underlying tech? Not to mention, some customers would never consider using off-premises setup anyway.
Absolutely many would, especially those with deep pockets. The biggest concern I'm hearing from companies adopting AI, for basically any use case, is data leaving their network. Especially (but not only) in the EU.
I don't understand how Google is willing to do this but won't sell TPUs to other days centers. It should be obvious from Nvidia's market cap that they're missing a huge opportunity.
The only reasons I can think of is they see them as their secret sauce, they don't want to support them for customers long-term, or they don't have the foundry capacity.
It's definitely #3. The GPUs have to first satisfy Google's own computing needs, and only then can they start selling them to others. Given how much training and inference the company is doing and how much demand there is internally it's very unlikely they are able to manufacture loads of extras, especially not profitably.
Would Google seriously have trouble raising the funds to build a chip fab? This seems like something they could do if they actually want to but I’d guess that would take actual leadership when they appear to have none.
Especially in today’s political climate, building this in a purple state would ensure longevity too. The Trump admin would probably let them break ground immediately if they had the plans and I doubt democratic leadership would disagree either.
If I was an investor and Google said they are going to now compete with Nvidia and TSMC I would take that as a sign they the leadership has completely lost the ability to see what their core competency is. Investing 100-200+ billion into fabs just to be on an equal playing field, is not it.
Would be a poor allocation of capital. Especially since, as they build up capacity for their own jobs, they get to see the excess to customers.
The raw computation is just a bunch of matrix multiplications in a row, most of the algorithmic complexity/secret stuff would be around scaling & efficiency.
For training the model the HW is much more important as you need to scale up to as many chips as possible without being bottlenecked by the network.
This would just be inference, and it doesn't need to be very efficient as its for on prem usage not selling API access. So you could strip out any efficiency secrets, and it would probably look like a bigger Gemma (their open source model).
I wonder if they would/could try and strip out stuff like whatever tricks they use for long context + video support (both of which they are a bit ahead of everyone else on).
The model itself is likely built upon their own open source system JAX so they should be usable in Nvidia. Of course cost efficiency is going to be a different story.
That’s the first thing I thought of as well. I had to integrate one into our custom CMS early in my career. I vaguely remember explaining to management that I was not responsible for the order or quality of search results and tweaking queries (now prompts?) with hints to restrict searches to certain paths. It was such an opaque device, but provided better results than MySQL did at the time.
Seems pretty high, this is an air gapped product so at some point the employees of whatever government they are giving it to would need to SSH into the VM's to load new weights etc. Lots of ways to make it tricky/watermark the weights though.
The google search appliance might have been one of the worst products I've ever used in my career. If they're going to make a box, I hope they put some effort into it.
Probably the most praise I’ve ever seen about Elastic.
I do respect the amount of power and utility, and it’s definitely a workhorse, but it’s like a horse with one human leg, a bad eye, extra bones but also not enough bones, and a French accent but only knows Korean. Once you get used to the fact that you can’t do what you intend to, but you can do what elastic wants, it becomes a lot more manageable.
I don't think GP is talking about the "search with google" box on third party sites. They're talking about a physical on-prem search server (box) that google used to distribute.
I wonder if the improvements in semantic search have changed that at all. For a big company though, you might need a pretty beefy setup to perform the initial indexing.
The Netflix appliance is pretty good in my experience. No reason Google couldn’t pull something similar off themselves, unless they’re being very Google about it.
Given the myraid of issues they seem to have, I am not sure I would classify Pixels as having polish. But yes, they definitely have the talent to make some good hardware. It's just a matter of whether their priorities match those of their users.
I can't think of a major phone brand that hasn't had some kind of major issue over the years. The batteries of the Note 7, iPhone "antenna gate" (and the more recent lack of advertised AI debacle), etc.
I think Pixels are pretty polished, at least compared to all the cheapo off-brand Android options out there. Some people like Samsung better but I can't stand their UI. Apple would be fine if I could sideload...
They don't sell them. But, if the developer / hotelier had a sufficiently large network, think providing service equivalent to the number of rooms at a US state university system network (multiple universities), then they might qualify: https://openconnect.netflix.com/en/
There are plenty of hotel groups big enough for that, but their properties are geographically distributed and I can't imagine they'd benefit from running fibre for their own multi-site network. Better to just connect each property to a local ISP like everyone else.
Maybe there are some exceptions. Disney World? MGM Resorts in Las Vegas?
FTA:
As part of the announcement, Google said Nvidia
will bring Gemini models to the company’s Blackwell graphics processing units, or GPUs. Companies can buy the chips through Google or other channels.
Richard has a student with an idea involving AI and joins his company as an advisor but can't keep his opinions to himself. Ends up ruining the company because everything he touches turns to shit.
On paper, Stephen Tobolowsky seems like he shouldn’t be successful enough of an actor to warrant an autobiography. But man do I love Ned “The Head” Ryerson in all his incarnations. What a strange, tall, little man.
That the world does not have a Stanley Tucci, Stephen Tobolowsky buddy comedy trilogy has made it all the poorer. But it’s been a while since someone tried to remake The Odd Couple…
I don't think you're wanting to converse in good faith, but on the off chance this is a question - yes, GCP was revenue losing for a number of years, but since Q1 2023 they've been profitable. It takes money to bootstrap anything - obviously - this is the case for the vast majority of companies and their offerings, especially so for one which requires vast amounts of compute resources, SREs, legal, etc.
Advertisers paid money for Google for totally unrelated services. Google invested that money in a number of ways. One of them was to build this very profitable non-advertising business. The advertisers didn't fund that business any more than the advertisers funded US treasuries, or the dozens of startups that Google has invested in as a VC.
This is a thread about using your money for better things than paying an ad company. The comment that started this argument you want to have pointed out that it’s self sustaining. But I pointed out that wasn’t always true. Tfsh backed my claim.
So today maybe there isn’t a problem to which your money isn’t being spent with the ad org but it was that way for a very long time to which we can grant the OP some grace as it’s a rather recent change.
There is even still an argument to be made that while you may not be giving money to the ad org you are still giving money to Google thereby helping them deflect the damage they cause the world in their other orgs.
No, even if you were Google Cloud paying customer #1, your money was going to Cloud. It wasn't supporting anything to do with ads.
The ads were providing income to Google which allowed Google to bootstrap Cloud until it was profitable on its own, not vice-versa.
When you buy (or bought) Cloud services, that doesn't affect Google's ad revenue or advertising behavior at all, not for the better and not for the worse. They're basically unrelated orgs within the corporation. Using Cloud isn't promoting ads or whatever you seem to think, not now and not previously.
But it’s not about killing Google’s ad revenue, it’s about hurting Google as a whole. It’s a complete monster, regardless how many heads the hydra has.
You could have saved us all a lot of time by simply stating upfront that you hate Google as a whole, rather than discussing the technicalities of which parts have to do with advertising or not.
Would you prefer VCs to have fronted the money to bootstrap it? How is it relevant today if ads are no longer enabling their financial viability? Ads largely finance Google's consumer offerings, not their enterprise offerings. Most enterprise Google customers understand the difference.
I believe these are pure word tricks to suggest privacy without actually delivering it.
As context, you need to remember that Google deleted their "Don't Be Evil" motto and became a defense contractor. The customer will most likely receive a black box owned and set up by Google. That means they have no way of knowing if the system inside is phoning home or being remote controlled by an US government agency, or not. You can then say that the model is hosted in your own data center, which might make some people feel good, but using it with personal information is still a violation of the GDPR.
If Google, however, would make these boxes fully offline capable and I was also allowed to wipe all hard disks myself before returning it, that would convince me of their good intentions.
Why is don't be evil relevant here? If Google never had that motto would you care less? It's not even factual that they dropped it from the code of conduct. It was just moved to the end rather than at the beginning. Moving it wasn't some magical event that signaled a change in Google's ethical values. Do the right thing was just seen as less ambiguous and placed more prominently.
As others have stated, being able to see that the appliance is phoning home or not is trivial. No one who is in the market for this won't ensure it meets some rigurous bar.
You’re talking about Fortune 50 companies here. I don't think Google is going to be messing around spying on them in direct violation of the no-doubt sophisticated contract that will be signed between them.
That was not with Google's consent and it was quickly shut down by enabling encryption between nodes in Google's internal networks. Your average company is far more likely to be susceptible to state actors than Google is.
According to the documents leaked by Edward Snowden, that espionage was sniffed in-transit in plaintext across the Internet's trunk and filtered against XKEYSCORE queries for eventual collection. Google's surprise came from the expectation that cross-datacenter traffic was sent over direct circuits and not susceptible to interception.
It was totally unrelated to PRISM, which was more like a voluntary law enforcement access portal that autoapproved every request. The participating companies since made public statements saying they no longer operate the portal, thereby forcing intelligence agencies to use National Security Letters instead. That's certainly closer to the intent of the laws passed by Congress.
> FISA orders and authorizations can be used to compel electronic surveillance and the disclosure of stored data, including content from services like Gmail, Drive, and Photos.
That's very different from prism. It's also why Google has spent a lot of energy trying to make it impossible for them to see the contents of your data. The government cannot conpel information Google doesn't have access to. I'm also not sure it's relevant for the topic of this post.
You’re making a lot of assumptions there. It’s trivial to monitor traffic patterns from modern appliances, even if it’s encrypted.
Also, companies have been sharing data with cloud security organisations for years now. There a robust means of assessing the risk. License agreements are a very real thing.
I don't fully disagree, but the only reason why this product is noteworthy is precisely because companies don't trust cloud providers with their data anymore. And while you might be able to prevent data exfiltration by monitoring the traffic patterns, you probably can't prevent sabotage that way.
Are you implying that Google will sell a product that is designed to ‘sabotage’ their own customer’s business? The legal and reputational damage far outweigh the value of stolen information.
Or do you mean that it could be a vector of attack? That can happen with literally any piece of software, hardware, or appliance you install in or out of your datacentre.
> Are you implying that Google will sell a product that is designed to ‘sabotage’ their own customer’s business?
The US government is constantly telling us that the likes of Huawei and Hikvision are doing precisely that, despite being subject to the same risks of reputational damage.
Of course, the same could be said of everything else in the data centre. It's not like Google are somehow more vulnerable than Juniper or Cisco or Unifi or Dell or Intel or whoever.
It's the same folks it always has been. Google is just trying to win those customer's business that would never have otherwise chosen Google. I'm sure these on prem solutions are not nearly as cost efficient as running the same workloads in Google data centers. Most companies would not pay that difference unless forced to via regulatory requirements.
That and there are various regulatory, political etc. reasons. Also I'm not sure about the "anymore" IMHO a lot more companies trust cloud providers with their data than they did 10-20 years ago .
Well, TFA appears to be thin on the details, but who says whatever they deploy is phoning home? If you run their model on prem, it wouldn't be a difficult feat to monitor its network traffic. Not to mention limiting it. It would be tricky if it phoned home by design, but if this is all abstracted through tool use or something, it can certainly be audited. And the kind of company that wants this usually doesn't just run random software without understanding and inspecting closely what it does.
This is being sold as an air gapped product, it has to work offline by definition.
Sure you could hide some way of phoning home and deploy it into the SCIF, but would you really want to risk a firing squad to improve some advertising metrics?
https://cloud.google.com/blog/products/ai-machine-learning/r...
Businesses running their own hardware probably prefer CUDA as well for being more generally useful.
They're in limited supply. Even Google doesn't have enough for their own use.
https://federated.withgoogle.com/
The general understanding of how it works is surprisingly easy though, you can find the paper here https://arxiv.org/abs/2002.06440
https://www.nature.com/articles/s41586-024-07566-y
Those enterprises won’t take the risk of being sued for using a model without proper permission.
https://bughunters.google.com/blog/5424842357473280/zen-and-...
That means they have direct fiber connections to the Tier 1/Tier 2 guys. The big ones have direct fiber connections to the NYSE.
If they're not rolling their own connection, they're using BT Radianz, IPC Systems, Colt Technology Services, etc.
This feels like a play for companies in highly regulated industries, GCP has a notable list of biopharma customers.
https://cloud.google.com/blog/topics/public-sector/google-pu...
> Our GDC air-gapped product, which is now authorized for US Government Secret and Top Secret missions, and on which Gemini is available, provides the highest levels of security and compliance.
You don't have to be a government agency to not want your company's data all over the place.
At first glance, it seems Google Cloud might lose some revenue from customers who can now deploy Gemini in-house. On the other hand, it's not a complete loss, since presumably Google Cloud is still involved in providing some underlying tech? Not to mention, some customers would never consider using off-premises setup anyway.
Would Google seriously have trouble raising the funds to build a chip fab? This seems like something they could do if they actually want to but I’d guess that would take actual leadership when they appear to have none.
Especially in today’s political climate, building this in a purple state would ensure longevity too. The Trump admin would probably let them break ground immediately if they had the plans and I doubt democratic leadership would disagree either.
So what gives?
If I was an investor and Google said they are going to now compete with Nvidia and TSMC I would take that as a sign they the leadership has completely lost the ability to see what their core competency is. Investing 100-200+ billion into fabs just to be on an equal playing field, is not it.
Would be a poor allocation of capital. Especially since, as they build up capacity for their own jobs, they get to see the excess to customers.
Performance may differ but Google (and Nvidia) are very interested in having good performance on both platforms.
For training the model the HW is much more important as you need to scale up to as many chips as possible without being bottlenecked by the network.
This would just be inference, and it doesn't need to be very efficient as its for on prem usage not selling API access. So you could strip out any efficiency secrets, and it would probably look like a bigger Gemma (their open source model).
I wonder if they would/could try and strip out stuff like whatever tricks they use for long context + video support (both of which they are a bit ahead of everyone else on).
Probably the most praise I’ve ever seen about Elastic.
I do respect the amount of power and utility, and it’s definitely a workhorse, but it’s like a horse with one human leg, a bad eye, extra bones but also not enough bones, and a French accent but only knows Korean. Once you get used to the fact that you can’t do what you intend to, but you can do what elastic wants, it becomes a lot more manageable.
What was so bad about the search appliance though? Physical? Software?
How would you say it compares to those?
Maybe there are some exceptions. Disney World? MGM Resorts in Las Vegas?
What made it one of the worst products you’ve ever used?
https://en.m.wikipedia.org/wiki/Google_Search_Appliance
https://homestarfanstuff.fandom.com/wiki/The_Cheat
I think by the end I was far more invested in the characters rather than the plot though.
It was the very start of the AI hype cycle, and in fact they built the app: https://news.ycombinator.com/item?id=14636228
That the world does not have a Stanley Tucci, Stephen Tobolowsky buddy comedy trilogy has made it all the poorer. But it’s been a while since someone tried to remake The Odd Couple…
I mean, would you buy cookies from a brand that is known for producing rodenticides?
I mean, the company that makes Raid also makes Saran Wrap and Ziploc bags. Corporate conglomerates can do lots of things.
The entire Google Cloud org is funded by regular customers paying money, not advertising.
That can’t be true, how did they bootstrap it? How do they pay for R & D for their half baked offerings?
Advertisers paid money for Google for totally unrelated services. Google invested that money in a number of ways. One of them was to build this very profitable non-advertising business. The advertisers didn't fund that business any more than the advertisers funded US treasuries, or the dozens of startups that Google has invested in as a VC.
This is a thread about using your money for better things than paying an ad company. The comment that started this argument you want to have pointed out that it’s self sustaining. But I pointed out that wasn’t always true. Tfsh backed my claim.
So today maybe there isn’t a problem to which your money isn’t being spent with the ad org but it was that way for a very long time to which we can grant the OP some grace as it’s a rather recent change.
There is even still an argument to be made that while you may not be giving money to the ad org you are still giving money to Google thereby helping them deflect the damage they cause the world in their other orgs.
The ads were providing income to Google which allowed Google to bootstrap Cloud until it was profitable on its own, not vice-versa.
When you buy (or bought) Cloud services, that doesn't affect Google's ad revenue or advertising behavior at all, not for the better and not for the worse. They're basically unrelated orgs within the corporation. Using Cloud isn't promoting ads or whatever you seem to think, not now and not previously.
You could have saved us all a lot of time by simply stating upfront that you hate Google as a whole, rather than discussing the technicalities of which parts have to do with advertising or not.
In a sense, yes, it was bootstrapped by ads and now pays for itself.
As context, you need to remember that Google deleted their "Don't Be Evil" motto and became a defense contractor. The customer will most likely receive a black box owned and set up by Google. That means they have no way of knowing if the system inside is phoning home or being remote controlled by an US government agency, or not. You can then say that the model is hosted in your own data center, which might make some people feel good, but using it with personal information is still a violation of the GDPR.
If Google, however, would make these boxes fully offline capable and I was also allowed to wipe all hard disks myself before returning it, that would convince me of their good intentions.
As others have stated, being able to see that the appliance is phoning home or not is trivial. No one who is in the market for this won't ensure it meets some rigurous bar.
It was totally unrelated to PRISM, which was more like a voluntary law enforcement access portal that autoapproved every request. The participating companies since made public statements saying they no longer operate the portal, thereby forcing intelligence agencies to use National Security Letters instead. That's certainly closer to the intent of the laws passed by Congress.
The NSA does not need consent from Google. Google is simply ordered to comply. See https://policies.google.com/terms/information-requests?hl=en...
> FISA orders and authorizations can be used to compel electronic surveillance and the disclosure of stored data, including content from services like Gmail, Drive, and Photos.
If you look at the content requests under FISA, you can see that there were over 118000 requests for user data between July 2023 and December 2023. https://transparencyreport.google.com/user-data/us-national-...
Also, companies have been sharing data with cloud security organisations for years now. There a robust means of assessing the risk. License agreements are a very real thing.
Are you implying that Google will sell a product that is designed to ‘sabotage’ their own customer’s business? The legal and reputational damage far outweigh the value of stolen information.
Or do you mean that it could be a vector of attack? That can happen with literally any piece of software, hardware, or appliance you install in or out of your datacentre.
The US government is constantly telling us that the likes of Huawei and Hikvision are doing precisely that, despite being subject to the same risks of reputational damage.
Of course, the same could be said of everything else in the data centre. It's not like Google are somehow more vulnerable than Juniper or Cisco or Unifi or Dell or Intel or whoever.
That and there are various regulatory, political etc. reasons. Also I'm not sure about the "anymore" IMHO a lot more companies trust cloud providers with their data than they did 10-20 years ago .
Sure you could hide some way of phoning home and deploy it into the SCIF, but would you really want to risk a firing squad to improve some advertising metrics?