♪♪
This is the Newcomer Podcast.
We've got a special episode this week.
I'm Eric Newcomer.
♪♪
On this episode, we've got the second set of interviews
from this Reebro Valley AI Summit
that I co-hosted with my friends at Volley.
First up, we've got Replicao, Amjad Masod,
and Hugging Face CO, Clemente Long in conversation with me.
It's really a story of the war
with Microsoft and OpenAI,
the sort of open source alliance against OpenAI,
which despite the name is not open source,
so it can be a little confusing there.
But Replica is sort of the home for open source coding.
Hugging Face is a repository for open source AI projects,
and together both companies are trying to compete
against Big Tech and big supporters of open source.
So that's the first half of this podcast episode.
And the second is a conversation with adept CO David Luan
and Greylock partner Sam Motometi.
It was a fun discussion of both adept,
which is a fascinating company.
It's got a different spin on the foundation model
and some of Sam's perspectives,
both on his investment and adept,
but also the broader AI market.
We also dug pretty deep into OpenAI and Google
where David worked.
I asked him whether he trusts OpenAI
with the future of artificial intelligence,
and I'll let you assess his answer for yourself.
Give it a listen.
Thanks to Samsung Next
for being the presenting sponsor
to the Cerebral Valley AI Summit.
Samsung Next invests in the boldest
and most ambitious founders.
Tell us about your company.
We'd love to meet.
Reach out at SamsungNext.com.
I'm John Massad at Replet,
Clem DeLong at Hugging Face.
Thank you so much for coming on stage.
We promised a spicy panel for the last one, right?
You guys?
I feel like OpenAI has loom so large.
And in particular,
you're competing pretty directly.
Replet helping coders,
competing directly with GitHub co-pilot
and Microsoft,
I don't know, powerhouse with OpenAI.
How do you grapple with their sort of dominance
in this space as a company?
Yeah, I mean, initially we started building on GPG3,
like, and like, we started building on GPG2 actually in 2019.
And in GPG3, like early 2020,
and like we were ready to go.
But they weren't letting people like release applications.
I don't know if you remember,
but you had to go through this ethics board
before you launch an OpenAI application at the other days.
And so we're sort of like held back
by some capabilities, problems,
by some of the economics,
by some of the processes.
At the same time, Microsoft was building co-pilot.
And so, you know, we were building,
we had the ideas, we had the platform,
we had everything.
And it just became exceedingly clear to us
for us to actually compete,
we can't build on OpenAI.
So we started training our own models.
We're very small companies, especially at the time,
but we decided to kind of want to do that.
You know, fork GPT2, try to fine tune that.
Eventually, like, things kind of happen,
we're Salesforce to release this like co-generation model.
Why did Salesforce, you know,
train a co-generation model?
Who knows?
But turns out, it's actually like pretty good.
And so we did extra work on it.
It was really slow.
We improved the speed, like two or three orders of magnitude.
We hosted it, and then we built the front end
around it and everything.
And we started fine tuning it based on our data,
and we put it out to market, and we got a product out.
But had we known that, you know,
it was gonna be hard to build an OpenAI
because of their relationship with Microsoft,
we would have done that earlier.
Like, you feel like they unfairly blocked you?
No, I don't feel like we unfurried to block me.
I feel like their deal with Microsoft
just gives Microsoft an advantage.
And if you want to compete with Microsoft,
you can't use OpenAI.
And we'll come back to Google soon.
But what is your view on the state of like Open Source?
I mean, we had, you know,
a mod its stability on earlier.
How strong as like sort of a system of options
is Open Source today, do you think we will see
an Open Source version of Chat GPT for soon,
or GPT for soon?
Yeah, I mean, to your first question,
I think the nomination of OpenAI is really overblown,
especially here in San Francisco, in Silicon Valley.
I was looking at it yesterday
since the release of Chat GPT, right?
Which is supposed to be this one model to do everything.
We've seen on Hockingface the release
of over 100,000 models, right?
So companies are not training models just for fun, right?
Like they're training models
because actually it improves their performance
as Chat describes, even if there's one model
from OpenAI right now that is like making the rounds
and that everyone is talking about.
Ultimately, what we're seeing is that
for most companies, most use cases,
when you want something that is fast, cheap,
and works for your use case,
actually, companies build their own models.
So small models can be fast, cheap,
but can a small model be better than GPT for in a use case?
In most use cases, yes.
Not so much today on very general use cases,
but in my opinion, that's fine
because for example, if you wanna do a customer support,
but you don't really care about it
telling you about the meaning of life
for the weather in San Francisco, right?
You want it to be really good
at your specialized customized use case.
So there's all this hype around
kind of like big, generalist models,
which is interesting, it's good, it's powerful,
it makes sense for use case like Bing, for example,
because they wanna answer all the questions in the world.
But the truth of it is that for most companies,
when they have a specialized use case,
it makes more sense for actually,
for them to build their own models,
train their own models.
So in my opinion, in a few years,
we'll just have a world where every single company
is going to have their own GPT for, or their own chat GPT.
And that's what's gonna allow them to build differentiation,
to customize, to align these models
with their own company values, right?
And not rely on the open AI values.
In my opinion, that's a better world too.
Okay, so you partnered with Google.
Ben Thompson wrote in Stretecory
that Google should have acquired you,
and that in some ways, not acquiring you,
was a strategic error.
Did Google try to buy Repplet?
I can't comment on that, but we've been saying no.
We've been saying no to acquisitions for a long time.
I mean, we famously said no to a billion dollar acquisition,
like when we were like six people.
Yeah, so is there a bigger number of that?
I don't know, I mean, do you have an offer?
LAUGHTER
We'll take a look.
But seriously, I think the potential for Repplet is really huge.
And we're just getting started.
We're just warming up, and it doesn't feel like it's the right time to do that.
I mean, the partnership with Google will give us huge acceleration.
It's like a very, very much win-win partnership.
And so that's exciting, and we'll see what that brings us.
I mean, both of you are companies
where I think people are really cheering for you,
excited about the companies.
And then they say to me, but ask them about the business model a little bit.
Like, can you talk about the hugging face business model?
And like, how are you guys going to make a lot of money?
That's still an open question.
But we're starting to show that we can do it.
And the way we do it is very typical, I would say, of platforms
with the high level of freemium model, right?
We have 15,000 companies using us today.
And the percentage of them are paying us.
So we have 3,000 customers today.
And they're paying us for premium enterprise features, right?
So complex user management, for example.
Or they're paying us for compute.
They want to run hugging face on faster GPUs, faster hardware.
Like, you strike a deal with someone who provides the compute
and then your server reseller or what's your relationship there?
Yes, we provide compute associated with the usage of the platform.
So the overall model is, and I imagine,
is going to be a bit similar for Ripple.
It's very similar to whether GitHub has just crossed a billion
dollar in revenue, right?
So they're proving that this model works.
And I bet most of it is compute.
Most of it is actions.
Yes, yes.
That's a bet.
Where is Ripple in the monetization?
Yeah, we just released our pro plan, which is like $20 a month,
which is exploding now.
There's like a lot of latent demand,
kind of switching from user growth and adoption
to monetization.
Turns out, wow, it's like a lot easier than we thought.
And it's cool that this climate allows that, where it was like,
grow, grow, grow, be everywhere.
So like, oh, maybe you should figure out
the unit economics in the business model.
We actually like that now that we're figuring that part out.
And we're making a ton of progress on that.
And so right now, it's more prosumer, individual developers,
and startups.
A lot of YC startups are launching on a raplet right now.
Just yesterday, there was a company called.
That means like they do all the coding in that.
Yeah, they do all the coding in a raplet.
And a lot of them do like go to market on a raplet.
So VOCode, for example, yesterday is this, you know,
conversational programming framework.
They did their main launch on raplet.
Another company called Leopii did their main launch on raplet
as well.
And so the cool thing is that you can find your initial
customer side.
In the same way that Hacker News used to be that,
I think raplet is increasingly taking that place.
And I think there's like hugging face.
There's tremendous opportunities for monetization.
At the limit, we're probably a cloud company.
Like I think selling compute and selling cloud services
is very attractive.
And it also scales very well.
Like consumption-based pricing is really attractive.
What are the coolest projects on hugging face right now?
Or are you just here?
You're very in touch with like the cutting edge of AI.
Like what should people be watching there that you think
really is sort of like the next thing?
And what are you watching that's hosted on hugging face
that you think's worth paying attention to?
Everything but text, right?
Like any scan flag with open AI focus, like looking at text.
But actually, it's a very small subset of AI in general.
And looking at what's everything that is not text
gives kind of like a clearer picture of what is AI today.
So for example, I'm really excited this week about text
to video.
If you remember for text to image,
there was the first viral model called Dali Mini.
We had the Dali Mini time for text to video
where you're starting to be able to generate video.
That kind of look bad.
All right, look.
It looks amazing.
I love the Will Smith eating spaghetti video.
Did you see that?
No.
If you haven't seen that, look at us.
You're going to stare at it for an hour and laugh.
Yes.
It's scary.
It's disturbing.
But somebody just typed in text and this video was created.
Exactly.
It creates conflict.
And this is open source or something?
Open source on hugging face, luckily.
And the thing is what's going to be interesting
is that these first models are kind of like outputting
very weird, very uncanny videos.
But it's going to attract a lot of attention.
And really, really fast, it's going to get better.
And we're going to get in a few months to the same level
of quality that we're getting right now on text.
In a few months.
Exciting.
Yes.
Yes.
In a few months.
Anything that excites you in particular?
I bet like, just like the development of AI
or like projects that people are working on.
Oh.
Well, I think Nat Freeman put it as capability overhang.
I think that capabilities of the models on the platform
right now are actually exceeded the product deployment.
So I think there's a lot more product
to build until we catch up to the capabilities.
So on the code generation side or coding side.
We think that as much as Copilot is amazing,
we think it's fairly primitive concerning what we can build.
The way Copilot works is that you add it as an extension
to your editor.
And it's sort of sitting there trying
to predict your keystrokes, right?
It's basically a glorified sort of typing aid.
Whereas what we're trying to do with Ghost Rider
Replitz coding assistant is like an actual agent that's
sitting on your computer and reading your files,
trying to make suggestions to your code,
trying to learn from how you program in order
to automate things for you before you even say them.
How far away is Replitz being a coder itself, right?
I mean, there's always this fear of like the platform
competing with the people who use the platform.
But it feels like, OK, if I can be a great assistant
to coders, someday your company will be a great coder.
Or how do you think about just, yeah,
sort of AI coding on its own, not as an assistant
as an actual coder?
Yeah, so I think that will happen.
But that question is really a question
about the future work generally.
And I think what happens is a lot of the repeated stuff
that we know how to do, we just train people how to do it.
And people generally do it without applying
a lot of creativity.
That stuff gets automated, right?
Because it's easy to automate.
We have a lot of data.
We have a lot of knowledge on how to do it.
You don't have to write the 300 trillionth left pad function,
right?
Everyone writes left pad functions.
And so this stuff will get automated.
But what that means is that humans
can go do more creative, more advanced things.
And I think that's a good thing about technology
is that we automate the things that are automatable
so that we can go on to creative things.
I don't think we're at a point, and by the way,
all the hype and all the excitement about GPT4 and all that,
and we're really excited about it,
we're big beneficiaries of this whole thing.
And people go generate code GPT4, put in Replen and run it.
The reality is it's still not very good
at completely novel code.
Like if you're writing essentially super novel functionality,
it's not as good at it as much as things
that it's seen in its training sets.
So I think we're very far from the point
where you have a completely autonomous programmer.
I think it's probably coming in the next couple of years
where it will feel like you're coding with an assistant.
But it's a far cry from automating all programming jobs.
Is hugging vase available in China?
Or what sort of the breadth of open source
and how much do you think these open source AI projects
are going to continue to be available all over the world?
So personally, I think the main risk for AI today
is concentration of power.
These technologies are powerful.
And what we need for them to be sustainably deployed
in our society is for more people
to understand how they work, understand
what they've been trained on, and understand
how to limit and mitigate them.
So it's really been our mission to bring more transparency
to the AI world.
Because otherwise, you end up with a world
where these technologies are built behind closed doors.
And it creates these scafflike narratives
completely disconnected to the reality.
Like what Amjad was describing is
that they're kind of glorified or to complete.
And at the same time in the public sphere,
you see letters with people describing it
as a robo-cup of these things,
a conflict that is going to take over and destroy the world.
And in my opinion, it's created partly
because there's a lack of transparency and education
about how these technologies are built and how they work.
By the way, a lot of the safety concerns come from hype.
And a lot of the big companies are beneficiaries of hype.
So they are, in a sense, beneficiaries of the anxiety.
So for example, Microsoft Research
got an early version of GPT-4 naturally.
And they wrote a paper calling it,
first contact with artificial general intelligence.
And then they commented that out.
They left it in the latex.
And then they changed its sparks of AGI.
So they're using science and archive and research.
And they put it out a pre-print as a marketing opportunity.
They're marketing this system as artificial general
intelligence, and then they're feeding all the fears
of the people that have been talking about the problems of AGI.
And then it's creating a very toxic environment,
where for them it's marketing, but for a lot of people,
it's life or death.
Right.
Yeah.
And I want to pull on that thread.
But sorry, I just, but China.
Is it available in China?
Yes.
It's available in China.
And you're saying it should be, was really your answer.
Because you want people to understand it.
Yes.
OK.
Thank you.
And then in terms of this problem of, like, yeah,
people are hyping AGI as possible.
But then at the same time, you were sort of saying,
we haven't even fully seized all the benefits
of the technology we have.
So it feels like it does sort of swing back and forth
from the same people where it can feel like, oh my god,
like this thing, there's power.
We haven't even gotten out of it yet that we don't understand.
But then the same time, you don't want to over-hype it.
It's just hard for people to totally understand,
yeah, what's next?
Or sort of how much more powerful it can get?
Yeah, but I think sort of like the company's calling it AGI
is misguided.
And like there's a definition of AGI, and perhaps dishonest.
And people have been writing about AGI for 70 years now,
right?
And so to call 54 AGI is a huge stretch.
I don't think even OpenAI wouldn't call it that.
But Microsoft did, right?
And then you have people who've been worrying about AGI
like Nick Bostrom, he has your Tkowski
and the Max Psych Mark that wrote the letter.
And basically you're telling them everything
that you're worried about we just built, right?
And then of course they're going to freak out.
Right.
Yes, I think the main challenge is that this future forward
looking futuristic sci-fi driven narratives,
they blind us from the priorities which
are mitigating the challenges of today.
So for example, at Hregkinface, something
that we work really heavily on is
to mitigate some of the biases in these models, right?
If you give to stable diffusion prompt
to create an image for a CEO, for example,
it's going to create a man's face, right?
And not really give you any woman faces.
And by picturing this Robocop thing,
we focus the public narrative on something
that hasn't happened, that might never happen.
And we don't focus on the real challenges of today,
like biases, like misinformation,
like lack of transparency and control of power.
So that's, in my opinion, why it's dangerous.
What do you do about that?
I mean, that's a great point.
You pull down something that doesn't account for bias,
so you want to give feedback to the creators
or what's the actual protocol for getting people
to consider the bias.
There are a bunch of different measures.
We actually released three weeks ago
bias detector for image generation models
by a team member called Sasha in the Hregkinface team,
which allows you to try to detect the biases of these models.
And it's really useful for people who are underrepresented,
for them to realize what kind of biases are there.
And then we advocate for publicity of that
from the model builders, so that the company that are going
to integrate these systems into their products
can take that into account.
So for example, if you take a resume classification model,
to classify if a resume is good or not,
or candidate is good or not,
if you see that it's biased in terms of gender
distribution, then you know that you're
going to not put that as the main filter
and that you'll need human in the loop
to make sure that it's not biased against women.
So this is called model cards and data set cards.
We have over 100,000 of them on the Hregkinface hub.
It's pioneered by someone called Dr. Margaret Mitchell,
who is the co-founder and colleague of the AI-E60
at Google before, who is now at Hregkinface.
So that the builders actually take that into account
when they build their products with this technology.
I mean, you both have expressed concern about the centralization
of power within AI.
Is the answer only everybody else
needs to get really good at it?
Or is there anything to be done about, yeah,
I mean, are you advocates of like legislation?
I don't know, are you starting to talk to members of Congress
or it's just everybody needs to out-compete them?
I actually worry about that.
So I worry about regulatory capture.
You see a lot of the big players are already going around.
Please regulate us in a way that might help them.
In the history of regulatory capture,
it shows that that's what you do.
That's the rational thing to do.
When you get big is to pull out the ladder, right?
So I worry about that more than anything else.
And I think the, again, this co-dependence,
this hype slash fear is playing into the hands
of those who want to create regulatory capture, right?
So you create more anxiety and therefore people ask
for regulation and you're like the hero, oh yeah, let me,
here's, we already wrote the regulation,
you know, which basically says no other startup could compete.
But I think open source is like a great way of doing that.
And I think like, you know, to clumps point,
open source is actually like a great way
to also stress test these systems
and also create systems that protect us
from that negative harm.
So like, there's a lot of systems right now built
around stable diffusion to detect stable diffusion images
to kind of do reverse stable diffusion,
show what the sources are and credit the artists.
There's tremendous amount of innovation and open source
around safety and security and all that.
Do you agree with that view that regulation
would basically help Microsoft and the incumbents
and therefore against it or do you have a point of view there?
It's a complicated question.
I think we're in a fast evolving field
that needs some regulation.
And I think for me, the most important thing for good regulation
is that the regulators can understand the systems
which sounds challenging today because of the lack of-
Like they barely understand Facebook,
the idea that they're gonna understand, yeah,
a tragic behavior-
Yeah, it's different between conflict,
the previous generation of tech
and the new generation of AI, in my opinion,
because in traditional software, because it's rule-based,
you can kind of like test the product
and understand how it works for AI systems.
The creators don't even know.
You can test it for a day, for a week, for a month.
It doesn't lead you to really understand it
if you don't have access to the data set,
if you don't see what it's been trained on,
how it's been trained on.
So in my opinion, for the AI era,
we really need to advocate and push for more transparency
because that's how we're gonna-
Into what people are training on?
Yes.
Okay, I'm gonna open up to a question.
What's your partnership with Amazon?
Or how serious is your relationship with Amazon?
Pretty serious.
We've been working with them for three years.
We are platforms, so we're also working with all the cloud providers.
But we've had a very fruitful collaboration
in conflict making it easier for companies to use open source
plus AWS to build AI systems internally.
Questions?
Questions?
Great. Yeah.
Hello, I feel like there's this tension between,
you know, we want the open AI and the big companies
to be more transparent, but then we also are concerned
about proliferation and bad actors being great.
Let me take that model and go do something evil with it.
And, you know, the good guys are like,
we need more transparency, but we need more responsibility.
And I just haven't really seen how you reconcile,
like just open everything up, but also, you know,
gatekeep access so that bad actors don't run away
with these things.
Yeah, there's a very good paper from someone called Irene Soliman,
who was actually at open AI before,
and who's at taking face right now,
on the different risks and challenges
of different release strategies.
It poses different challenges, right?
When you keep it close source, behind closed door,
you create concentration of power,
you make it more difficult for civil society
underrepresented populations to participate
for regulators to actually regulate.
When you make it more open and more transparent,
you are more inclusive.
You, in some way, reduce the risk
because you create power and counterpowers at the same time.
So even in an actor, which is by definition, much smaller
than the actors who control it right now, right?
Because right now, it's the biggest players, right,
that are controlling the biggest companies,
the biggest governments and countries.
So actually, the risk of having smaller players misuse it,
is counterbalanced by the fact that you can create
the counter power to actually mitigate these risks.
I know it's not always straightforward,
but if you look at the long-term safety
of the development of technology,
more openness and more transparency
is actually much more sustainable in the long run
and creating much less risks than keeping this power
in the hands of the very few number of big companies.
Great.
Hi there.
I just have a question for Amjad, actually.
Replet has emerged as this huge AI power player.
So kudos to you, building your own models, Ghostwriter.
It seems quite different from the initial focus of Replet
as really an educational tool and specifically a tool
for people learning how to code.
It seems like the initial audience was beginner developers
and now it's actually very advanced.
It's to get developers who want to pull in the latest AI.
So I'm just curious how you feel about that
and what is Replet?
Is it an edtech company, an AI company?
What was the plan and what happened there?
Yeah, our plan from the start is to build an end-to-end platform
for software development.
Our set of North Star since day zero was idea to product.
So how do you have an idea in your head and how do you put out a product out
in the shortest amount of time possible?
So initially, what's the first hurdle to that?
It turns out the first hurdle is setting up the top environment.
So we solved that and put that in the cloud.
And then what's the second hurdle of that?
Well, the second hurdle is hosting the code somewhere
or running the code.
So we solved that.
By the way, along the way, it turns out these are enormously valuable
for education.
Education was somewhat of a focus, but it was sort of an accidental kind
of product market fit because we just wanted to solve a problem for developers.
And then along the way, it turns out that, like, AI, actually someone tweeted out
our pitch stack from 2016 and had, like, a master plan.
And the second plan is, like, once we have a lot of users,
we're going to train all these models and help people code.
And so when we started looking at, especially DPT-2,
it was pretty obvious for us that we need to invest in those.
Because, again, anything that shortens the distance between an idea
and a product is something that we're going to invest in.
So the long-term plan of Replit is, like, you have an idea
and you talk to your phone and you create an app
and, like, you know, an hour later, you have your first paid customer.
That's really the North Star.
That was the plan on all of us.
Yes. You can see my writing go back to 2012.
The last question for the conference,
urge the audience, like, what would you have them work on?
Or, like, your time's limited. Like, there's so much excitement now.
If you could say, like, go work on this small, large,
like, what would you charge people to chase after?
For me, I would urge everyone to start working on biology.
Chemist 3 with AI.
I think it's an underfunded area that could be really beneficial
for the world in the next few years.
I think building, like, the original kind of vision of Siri
and Alexa and these things, now it's possible.
And especially with the open source models that are small,
you know, whisper and llama and alpaca in these things.
And also, like, building educational programs.
So just, like, putting these things on a smartphone.
So try to figure out how to put a large language models on a smartphone.
There's a lot of interesting work in open source,
where we're just talking about a guy in Bulgaria who, like, not part of the, like, Silicon Valley lead,
you know, found llama on the internet and did this process called quantization.
And now llama runs on a, like, a 7 billion parameter model,
runs on a pixel 6 phone.
And so imagine, like, now, like, sort of a kid in Africa being able to learn English,
like, talking to llama on their phone.
I mean, that's pretty freaking amazing, you know, and I would love to see, not just open source,
but also startup companies, so I'm building that.
I just want to say thank you to everybody.
You know?
Alright, that was my interview with Replitzio, Amjad Masad,
and Hugging Face CEO, Clemente D'Long.
Now we're going to go on to my conversation with Adepseo David Luan
and Greylock partner, Sam Motometti.
David Luan, founder of Adepth, former OpenAI, former Google,
Sam Motometti, Greylock investor, I'm board member of Adepth, I believe,
snorkel, cresta, Greylock's also an investor, and Tom, super excited.
Let's start with Adepth, right?
Can you just explain the product to people, right?
I mean, this is not open for public release.
In terms of my own thinking, I feel like we have this list of, like, foundation models
Do you put yourself in the foundation model category because you have a little bit
of a different focus?
If you could just walk people through that, I think that would be super helpful.
Yeah, totally.
So for Adepth, we're training our own foundation model, but we're not duking it out
with the, like, open AI and anthropics of the world that are sort of training LOMs
or increasingly now these, like, multimodal-based models that are sort of, like,
general web stuff.
I think that's going to be a really interesting space because people are going to, like,
increasingly fight to have, like, more and more fungible models that are going to be
doing similar capabilities in that area.
What we're doing at Adepth is we're actually working on training a model that can do
anything a human can do on a computer.
So the goal is, like, teach this one neural network, like, how to use all software,
how to use all different tools that, like, someone on the machine would use every day,
and ours is, like, specifically focused on making a really good for knowledge workers.
And so, I mean, what was appealing to you, like, how do you assess these companies?
I mean, it feels like there's so many hot AI startups right now, like, what's the method here?
Or, like, what did you see in Adepth?
I'll start with Adepth and then maybe talk more generally.
So, you know, I think Adepth for us was a very, very easy decision.
Like, if you take a step back, we believe one of the largest opportunities in software
is to build usable general intelligence, and the way that expresses itself as a product
is a co-pilot for every knowledge worker.
That makes us 10 times more efficient and 10 times more effective.
And we think to do that, you have to build a vertically integrated company.
You have to build a multimodal model that understands how to learn from humans
and then can operate across all software tools.
And then you have to build all of the scaffolding that's required
to actually operate that in real-world enterprise environments.
And that's exactly what Adepth is doing.
And David and his team are one of a very, very small set of people
who actually have the operational understanding of how to scale these models from scratch.
And combined with that, they're extremely commercial and customer-oriented.
So, Adepth was a very easy decision for us and we're fortunate that David selected us to be his partner.
More generally, Eric, I think the answer varies quite a bit based on the layer of the stack
that you're operating at.
So, like, we broadly think of it as applications and infrastructure.
I talked about Adepth as one example.
But in general, when we look at the application layer, it's like the basics of software investing.
You know, people talk about these AI companies, like they're these magical creatures
where different rules apply.
And it's just not the case, right?
Like, we look for companies that are focused on very specific customer problems,
have a point of view on how they're going to own a valuable end-user workflow,
and where there's some gravity to the data that they're training on or operating on.
And when we find those things, we get really excited to invest.
But if you're just building a thin layer and you don't have any of those things,
like, it's not something that's going to be a fit for us.
And so, what's this data play with Adepth today?
Like, in terms of customers, like, it's in, what do you call them, beta?
And, like, what can people do?
And, like, what are, obviously, part of the dream is that the use cases are sort of unlimited
in terms of, you know, anything a human can interact with a computer with graphically.
But what do you see people sort of putting into use for?
So, right now, we're at a point where we've got a bunch of really cool enterprise partners
that are really excited about Adepth and working together on that.
So, we're working together with places like Workday and Atlassian,
and one more that we're going to sign and announce soon and Microsoft and some others.
And so, like, basically, what works really well with Adepth right now is, like, you know,
if a human can do the thing on their computer and, like, a human can show the Adepth model
how a particular thing is done, we can just feed that into training,
and then now Adepth will be able to do that again on-
Well, like, an example that was given to me, I think this reel is, like, Redfin?
Yeah, that was kind of our hello world thing.
I think it's kind of funny.
People are adopting Redfin as, like, the hello world browser control thing
that they want to demonstrate, but, like, I think a really simple example is, like,
if you go find a bunch of LinkedIn's for someone that you want to hire for, like, a job posting,
you can say, like, okay, now move all these people over to Greenhouse,
or, like, add them all to Salesforce as a new lead and, like, start a campaign.
And you can put these down in text, and because a model has learned,
it's seen a lot of interactions with Salesforce,
it's seen a lot of interactions with, like, general things on the web,
and it has its context to be a little shuffle information around.
It can do all sorts of stuff like that.
But I think this is kind of the, like, early table-stake stuff.
It's going to be really helpful to knowledge workers who spend a lot of time,
like, doing tedious stuff on their computer all day,
to, like, already have this level of capability.
But I think where things get really exciting is when you, like,
level up the, like, abstraction that you could do with models like this, right?
Like, maybe instead of just, like, doing that level of thing,
maybe it's, like, just building a whole financial model,
or, like, create a part for me in CAD and, like, optimize a hell out of it with a simulator.
Like, these are all things you should be able to do as the model sees more and more interactions
between humans and computers.
You know, OpenAI came out with plugins,
and my understanding is there's a sense that, like,
couldn't a company just, like, build a plugin instead of actually going through
this, like, graphical sort of thought process?
Just, like, help us understand why you think your approach is superior to, like,
language-based plus plugin.
I think one giant part is just looking at the success of things, like,
Siri and, like, Google Home and stuff like that.
Those are very, like, language-in-language-out interfaces.
I think, like, there's a reason why we only trust them with a very limited set of things.
It's because, for the most part, these giant models are not yet reliable, right?
Like, for the end user, like, you want to be able to, like, one, there's, like,
a tremendous amount of richness in existing interfaces that have already been created, right?
Like, it's the best way to consume content.
It's the best way to see a lot of, like, visual stuff, right?
And, similarly, like, because a depth does the thing for you on your computer,
you have a high amount of trust that it's actually, like, instead of it just spitting out,
oh, the answer's 42, right?
You can actually see everything.
You can, like, play it back, you can see, like, the moves that it made.
Exactly, yeah.
So, I wish, like, we could just, like, show some people right now,
but, like, what it does is it, like, just actuates your actual machine to go get that thing.
And do I, like, just...
Does it watch me do the activity and then it starts building from there or...?
You can do that.
So, we're working on a thing where you just hit the record button and you show a depth of workflow,
and then when you hit Stop Record, you're, like, now go do this for at least, like,
next 16 things that I want to go do the same thing on.
And so, could it do, like, a spreadsheet-level task like that?
Works on spreadsheets and, like, other key knowledge worker tools.
What's the graylock relationship with OpenAI right now?
Or, like, I mean, obviously, Reed, you know, was so early.
He was on the board.
He just stepped off the board.
I interviewed him on my podcast.
I mean, you all are investing super aggressively in these companies.
I mean, this clearly competes directly with OpenAI.
Like, what's the relationship and, like, what have you all learned from that, like, working with them?
Yeah.
So, to point out, my partner, Reed Hoffman, was an early investor in OpenAI
in the nonprofit foundation, was on the board until recently.
At graylock, we've been investing in AI for over a decade now, both at the application layer
and the infrastructure layer.
We're close to the OpenAI folks.
We have a ton of respect for what they've done.
A lot of our companies are partners with them in different ways, and at the same time,
we believe AI is going to transform everything around how we work and live,
and we're investing, you know, with that thesis in mind.
You know, I feel like, with a depth, I mean, people read the headlines,
they're here to sort of hear the headlines.
Like, you know, the information had the story about some of your co-founders leaving.
I think they've raised money from Thrive.
Like, what's the backstory there?
What's that mean for a depth?
So, I think this era of AI is, like, the period where I'm most excited about,
because I feel like, you know, for a long time, we were just working on foundations, right?
And then in 2012, happened, we got Alex Net, and then 2017, we got Transformers,
but, like, it was this era from, like, nothing worked at all to this era where, like,
it felt very much artisan.
It was, like, you and your three best friends could write a research paper
that changes the world.
And, like, that kind of became the dominant mode of AI progress.
And I think once Transformers came out, we kind of had a lot of the core building blocks, right?
Like, the transistor has been invented.
Like, it's now really time to go build everything on top.
And...
I like that line.
Yeah.
Like, your belief is, like, the core technology has been figured out.
There's still a tremendous amount of research that needs to be done,
but, like, the philosophy behind the depth, I think, is different from, like,
what my old teammates were doing and also what AI labs that are doing research are doing.
It's like, everything we do is ultimately in service of building this thing
that everybody will use every day at work.
And so, there's plenty of problems, like, a lot of offline reinforcement learning problems
that we're working on solving.
And, like, we do tons of research, but it always flows back from that level.
And the subject here is that the people will laugh for more research focused.
And you see...
Much more bottom-up.
Yeah.
Bottom-up research, like, very classic Google Brain-type thing.
And I think tremendous respect for, like, being able to do that really well.
But if you go think about, like, we've entered the industrialization age of AI,
like, it's now time to build factories, like, that's the attitude.
I'm curious across your other companies.
Like, I framed up this tension from the beginning.
Like, foundation models versus research companies versus, like, product companies.
Do you have pure product companies?
I mean, obviously, David has so much experience himself at OpenAI and Google
and is very technical, but I'm curious, like, would Greylock invest in a pure product company?
And how do you think of those types of companies?
Yeah, absolutely.
I assume what you mean by pure product as well.
I know, I know.
It's like a very...
You know, you know, I mean, this sort of wrapper.
Like, could you see a wrapper business being a successful startup investment?
Yeah, I think it's very use-case and vertical-specific.
I'll give you a general framing for how I think about it, which is...
One of the questions we debate internally is, where does AI turbocharge existing
companies and existing workflows, and where is it a window for new companies to get built?
So, the lens I use is, if you're going after a workflow that's already owned by somebody
and the way that AI incorporates into that workflow doesn't lead to a profound shift in the workflow,
it fits more in the turbocharge model.
So, writing is a great example, right?
Like, it turns out writing assistants is a great use-case for AI.
Notion, Google Docs, and the rest have amazing editor experiences, and they can integrate AI in
and offer those superpowers to their users without needing a completely dramatically different workflow.
I'll contrast that with, like, I saw Chris from Runway here earlier today,
and, you know, what Chris is doing around video and content creation,
sure, Adobe can, like, staple on text-to-ass a creation onto their products,
but he's reimagining that workflow from scratch for an AI-first world
centered around a natural language interface, and the product and experience feels completely different.
Or, like, sometimes when people think of a depth, they think of, okay, RPA vendors,
like UiPath and Audubon, and the rest.
Right, yeah, I initially thought that...
Explain that.
Like, I think people would like that disintegration.
Yeah, I mean, if you think about kind of the product mandate of, we're going to go automate a bunch of work
and make people more efficient, and you ask who else has a similar product mandate,
the RPA robotic process automation category has such a mandate.
But if you go look at UiPath and the entire approach of deploying bots on these
deterministic processes that are extremely brittle and not end user-facing,
and you compare that to the ergonomics and robustness of something like a depth
that's powered by a natural language interface and can operate across different modalities
with the end user in the loop, it's, like, a dramatically different workflow,
and so it's very obvious to us why that's a new company versus an existing company.
So we think about that lens when we evaluate application layer opportunities,
we have invested in tons of application layer companies.
You mentioned Tom and Kresta as other examples, but both of those companies kind of fit into that framework.
Great.
I'm curious about OpenAI, and, like, the evolution we've seen with the company,
I mean, in particular, like, the shift from nonprofit to, like, for-profit,
I'm curious, like, had that transition already happened when you were at OpenAI,
or what cultural evolution, like, have you watched, did you watch while you were there?
I mean, the company changed dramatically, so I joined in 2017 to run engineering
when there was only 35 people.
Wow.
And it was tiny, very, very early days, like, teams hadn't really yet been, like,
clearly pulled out of the ether.
And I think, like, honestly, during that time, like, it was not really clear what our unique spot in the world
was going to be.
You know, like, we had some great researchers, we were kind of doing some, like,
Google Brain-E style work, but it was like, okay, is this just Google Brain SF?
Like, it really was never meant to be that, right?
And so I think what we did a really good job of was we, I think, realized,
one of the first places to realize that AI was changing, like, leaving this, like,
you and your three best friends write a paper world, and really entering the age of, like,
when most ideas just start default-working to, like, the 60 to 70% level,
what you really want to do is you want to go build a full Research Plus engineering team
to swing for the fences on the, like, technical outcome, not the scientific novelty.
And so I think, like, that was sort of like, phase one.
I think that's what led to, like, a lot of early breakthroughs on the opening.
I said, including the robot handwork and some of the early GPT work as well.
But I think from there on out, once the industrialization thing started staring us in the face more and more,
like, resourcing was going to be really, really important.
And I think that, like, that was behind a lot of that.
Resourcing, aka, getting the money to fund, like, compute-
Sending checks to Jensen.
Yeah, really.
So, like, that's the thing that you really, really had to do.
But then we got to this new era where I think that people, I'm sure everybody in the audience is fully realizing now,
but, like, it's my strong belief that, like, if you want to build AGI,
regardless of how you define it, you need users.
And you need humans in the loop.
And I think that now, actually, building a product company is the critical path to getting a general intelligence.
Do you trust OpenAI still?
Or do you have, like, faith in them as sort of the carrier for this, like, huge cultural moment?
Or, I mean, I don't want to talk about the letter too much, but there's clearly anxiety with this pause idea.
Like, do you think OpenAI, they're good stewards of this future?
Well, I think what's really interesting is that, so just due to how neural net scaling laws work, right,
like, everybody's seeing the stuff, like, our rule of thumb that we've learned from training large models of OpenAI
and also at Google is you kind of need to be, like, two-doubling's ahead.
If you hold everything else constantly, like, dataset, architecture, all that good stuff,
literally all your fighting on a scale, you kind of have to be two-doubling's ahead to be, like, materially more,
for the model to feel materially more brilliant.
And so I think this is both a blessing and a curse.
And, like, when I was running the Google large models effort, I literally had a presentation where I was like,
this is how we're going to win, this is also how we're going to get screwed.
Because you're saying other people can catch up, and they're not as far ahead as we believe
or what you don't need.
Because you don't need orders of magnitude more resources, right?
Because if you're within a 4X factor, your models seem, if holding all else similar,
like, fairly similar in terms of how smart they are.
And so I think what we're, I think what this year is really going to show us is just proliferation.
Like, I think proliferation is the name of the game.
And when proliferation is the name of the game, then, like, exactly what the front runner does,
doesn't necessarily steer the direction of the field as much as we think of it.
Like, this is a very competitive situation, lots of people can catch up.
That's fundamentally what you're saying.
Also, just this amazing energy that we're seeing from the open source community.
Like, good stuff is happening.
Like, we're like, it's also, we're starting to see this thing where, like, of course,
the thing we care about most, and I'm just talking about LMs, not so much about adapt,
but like for LMs, right?
Like, you, the tasks people are excited about are, like, higher and higher level ones, right?
But, like, a lot of these, like, lower level tasks on language, we're starting to see some saturation on the models.
And I think that process is only going to increase.
So, there are just be more things people can reach for.
And I think the fact that it's now in the hands of the community means that the pace of progress
is just going to keep on speeding up.
Are you seeing the rest of your portfolio, like, non-gender-in-of-AI companies pivot in?
And, like, what advice are you giving people in terms of just, like, embracing this moment?
And, I don't know, yeah, are you seeing people where it's a mistake to go in because, yeah,
it's not their specialty?
Yeah, I'd say most of our portfolio, every single board meeting, there's a conversation around
should we be doing something around generative AI?
Now, newcomers having that kind of...
Yeah, exactly.
The lens of that conversation is, how does it help us deepen the customer value we're driving, right?
And so, I'll give you an example.
I'm on the board of a company called Appiro.
It's an application security company.
And so, one of the things they do is they identify and prioritize vulnerabilities in code.
Six months ago, we realized that we could leverage a bunch of the advancements in things like codecs
to go beyond just vulnerability identification, to automatically generating code to solve
and remediate vulnerabilities.
So, that's an example of a company that didn't start as a generative AI company,
owns a really valuable end-user workflow, and then finds a way to turbocharge their customer
value, leveraging these models.
Most of our enterprise-oriented companies have some story like that, but at the same time,
like, you know, I think it's important not to try to force it onto your product if it doesn't
naturally help you further your customer value mission.
I haven't talked a lot about, like, jobs today, or job replacement through AI,
but that's clearly on a lot of people's minds.
And I feel like even though I don't know that a depth would be unique in sort of, you know,
the threat to some people's jobs, I think the model where it's like, we can watch the worker,
you do this thing, we figure out how to do it, we can do it.
Like, it feels like a little scarier to me.
What do you say to people who worry about the job replacement?
I'm sure you've thought about this at every company you've been at,
but what's your thinking on the state of AI replacing people's quality jobs?
Honestly, I think it's something that I think about a lot.
I think it's definitely something that I am concerned about.
I think that's actually a big part of, like, look, like the adept mission is not actually,
and I think it's going to be really difficult to have all these models literally replace jobs.
I think what's more likely is I think it's just going to give all knowledge workers a higher-level interface
so that instead of spending, like, for a salesperson, right, instead of spending, like,
30% of your week, like, dealing with updating Salesforce, you spend 1% of your week asking
a debt to just do that for you, and you spend 99% of the time talking to customers,
which is really why you're doing this job to begin with.
Like, I think that's going to be the story that's going to play out.
It's going to be really much more about the degree to which the Industrial Revolution took away,
like, the need to go do a lot of pure hands-on work that could be sped up.
Like, I think it's just going to take some of the, like, most tedious parts of everyone's job,
the bottom part out.
Do you think it's mostly disruptive to white collar work at this point, or, like,
are there, like, factory tasks that you're able to do?
For the most part, what we're doing, because it's, like, really leveraged for knowledge workers,
we do the low-level stuff at the high-level stuff is really all handled by, like,
like, what should we do and why, and how do I work with humans part, is like,
that's going to be fundamentally human for a really, really long time.
The part that we're doing is, like, handling, like, clicks and presses and types and formfills
and, like, all that tedious crap that, like, I don't think anybody can sign up for it.
So if I wanted to get Taylor Swift tickets, I could have used this.
It sounds like, yeah, ticket scalpers, I love this product.
I mean, the positioning of, like, the big tech companies, or, like,
I mean, any views on, like, I mean, it seems like Microsoft is so strong here,
but, like, any views on where Apple or Facebook are,
and also just, like, related to that, like, how you're advising your start-ups to strike deals
for, like, long-term, like, compute sort of contracts.
Yeah, it's a good question.
So I think your observation is correct, where if you look at the playing field today,
you'd give Microsoft a ton of credit, Google's number two.
There are others as well, right, like Oracle's playing an important role as well,
and then it's actually surprising to us, some of the names you mentioned,
are kind of nowhere to be found.
And so I would expect we see them realize that they're behind and make moves to catch up,
but to date, those three companies have been the best to partner with,
and we have different companies in our portfolio.
Microsoft, Google, Oracle.
Yeah, that's right.
On the cloud side.
I mean, you have a network, like, Workspace is a user of your product,
or, like, what's your sort of partnership strategy?
I think right now, a lot of companies that make software tools sort of are in a really
interesting moment where I think there's the ability for them to offer something to end users
that's, like, a way easier way to go use the same thing,
while still letting experts, like, be fully functional in their software, like,
who, like, no one loves, I don't know, just choose an example, right, something like Workday,
like, they know how to go get everything done, right?
But I think at the same time, people who may not know how to use the software as well,
I think having something like a depth that makes it possible for them to just put in language
where they need to get done in the tool gives a lot of leverage to the end user
and cuts down the need for the L&D budget, right?
And so I think that's made it natural for us to go work a lot of companies that way.
I'm going to open this up in a second.
I just want to go back to an earlier question in that I feel like I asked you whether you trust OpenAI,
and the answer was somewhat everybody else will be able to catch up.
And, yeah, is that the right inference that in some ways your answer was not yes?
So I think on that in particular, I think OpenAI, like, I spent a ton of time at OpenAI
helping build the safety team and the policy team, which ended up rolling up to me,
so really, really enjoy a lot of the folks who are steering that ship over there.
I think at the end of the day, like, though, I still believe, like, most importantly that, like,
this is still really early, really enough in the story of AI,
and I think that there's a lot that's going to happen when the rubber hits a road on the stuff,
and I actually think that there's going to be a disproportionate amount of influence
that the community is going to have in terms of what happens.
Any questions? Great. Yep.
An earlier panelist talked about helpful, honest, and harmless being kind of the three pillars of...
And the topic.
Yeah, of constitutional AI.
If you had to rate each of those where we are today on a scale of one to ten, where would you put it?
I think that helpful is an intent, and I think all the models intend to be helpful.
I think the degree to which they're actually helpful. Still very early.
I mean, it's just like, the action space for AI systems should be such that they can help you with a lot more stuff than writing text.
And I think we're now seeing new signs of people calling APIs, I think that's a really great direction.
I think that as a whole, for the other two parts, they really come from learning from human feedback.
And I think that this is still relatively under-tapped, right?
The only way we use to train models is we, like, hit scale until something broke, and then we put it out there.
But I think that being able to learn from users to follow human preferences is going to close a lot of the gap on the latter two.
You spent like a year at Google, right? Or after OpenAI?
There's definitely a meme online that is like OpenAI there to like 3am, grind it out, and Google.
It's like the show Silicon Valley, like on the roof. I don't know. How true is that?
So at OpenAI, we definitely had our 9th to 5s.
We also had our people who just really pushed hard.
I would say at Google there was also some of the latter, but definitely the ratio is slightly different.
I was there during peak COVID though, so it was hard to tell over.
And any takeaways on sort of the, you know, like, seems like Google Brain, there's sort of a merger within the organizations within Google.
Do you think that's significant?
So for a long time, I always said that that would be the sign that Google is really mobilizing.
They're awake.
And it's happened.
As far as, I mean, I don't have any inside information on that.
But from what I can tell, I think that was a really big.
I think that's just like, that's where the stuff needs to go.
Any other questions? Great.
Yeah.
I'm a founder. It seems kind of blindingly obvious that we're moving into a new paradigm where like the cloud native companies are now going to be competing with these kind of new AI native companies.
And can you kind of unpack what you, to spoke about with Workday, where you're offering them kind of an escape path to not be displaced by an AI native version?
Like, to what degree is that kind of the core?
Would you kind of frame that as the core opportunity that you're working on and can you talk more specifically about like what it's going to take for these companies that have built, been built natively into the cloud to kind of compete in this new paradigm?
I think Psalms are much better a person to comment on.
And I'll say one quick thing, which is I think that a lot of the opportunities right now come from which things should be unbundled because of AI, which they should be bundled because of AI.
And I think that working with Workday, for example, is a perfect example of where the product shape, I think, is actually very good.
And so as a result, working together makes a lot of sense.
But I think in other areas, I think vertical SAS, that's AI first, due to bundler or unbundler, could do really well.
Yeah, and the only thing I'd add to that is the lens I use is where is there real gravity?
So if you take something like Workday, there's immense gravity in the data and the fact that there are a system of record for your employees.
And then the opportunity for AI is to reimagine the workflows and interfaces on top of that gravity.
And like David was alluding to this, but I think as a more general point, most people think of AI, they think, oh, I'm going to make people who know how to do something more efficient.
And that's definitely a big part of it.
It also is a much more elegant and accessible interface for a lot of workflows.
And so what actually ends up happening is there's a lot of latent demand for people who want to be able to operate on these systems,
but who can't because they don't understand or don't have the sophistication or training to use them in their current form.
But they know how to talk to a natural language interface oriented assistant who can help them use Workday or Sheets or whatever.
And so I think it ends up also expanding the TAM of people who can interact with these systems.
Do you think that companies that build their cost structures natively in an AI world are just going to blow away?
Do you think companies that have built their cost structures in a cloud native world will be able to compete with companies that have built their cost structures, AI first?
I don't fully know what you mean by cost structure, but I would say there's going to be a lot of variance around whether or not companies are able,
like cloud native companies will be able to compete in an AI native world versus not.
It's very company specific.
Great.
We've had our transistor moment already in AI, which is pretty crazy to think about.
All right.
Thanks very much.
Yep.
Good bye.
All right.
That's our episode.
Thanks so much for listening.
I'm Eric Newcomer.
These talks were from the Cerebral Valley AI Summit that I hosted with Volley, a voice AI game company.
I want to shout out Samsung Next, Oracle and Nvidia.
Our theme music is from Young Chomsky.
Thanks so much to Tommy Herron for editing all these episodes and our YouTube videos.
And finally, shout out to Riley Kinsell, my chief of staff for all his help and support on this.
Thanks for listening.
We'll see you next week on the Newcomer podcast.
You can like, comment, subscribe on YouTube where we're posting a bunch more interviews from the Cerebral Valley AI Summit
and follow us on our newsletter at newcomer.co.
Thanks a bunch.
Goodbye.
Goodbye.
Goodbye.
Goodbye.
Goodbye.
Thank you.
.