![]() ![]() I intend to take nothing away from their genius, but it is built on others' work: the vaunted Transformer paper wasn't "here is attention" it was called " Attention is all you need.". ![]() They made tremendous advances, and we owe them a ton-no question about it, but claiming it could have (or should have) been done alone seems off to me. The narrative I hear now from Googlers that Google developed AI is also a bit strange to me. Now, we see the power of a larger community almost every week (and today, shameless plug, with Tri Dao in Flash Attention used everywhere).Ĭontributions to AI's Core. Being the tip of the spear is expensive and hard in one place, let alone every place! The other side was a range of folks from NVidia to Meta to Microsoft, etc. It was an all Google stack, they had to be great everywhere and all the time. To me, the reason was kind of clear: they were using TensorFlow on TPUs with models they designed. When we put out DAWNBench (which became MLPerf), my understanding is that Google was shocked it didn't solidly win the benchmark. But also, there was something to learn about collaboration with the community. First, I think the openness of PyTorch won the day and its focus on making researchers productive and listening to the community (Soumith and team are amazing!). What happened? Well, two things, I think. Fast forward, and it's not as used as PyTorch. It wasn't the first: there were packages like Torch, Theano, Caffe, and others. When it first came out it was huge! Wow! I rewrote so much of our code in Tensorflow when it came out. My view of Google's Tensorflow experience was the following. They have built world-class products that we use most of our day. Again, these are brilliant people to whom we owe a ton of gratitude. Let me give a (perhaps unfair) take on some of Google's recent work in AI on infrastructure, AI's core algorithms, and the open-source path they have followed in other examples. They hold a special place in our department. I have valued my interactions with their researchers, executives, and founders immensely through the years. We're moving as Alex Ratner says from GPT-X to GPT-You. The power is not with who has the model, but who can deliver the value people want-and that hopefully means more AI-infused services in the world that are cheaper, safer, and more useful. So the moats are shifting and a bit uncertain, and this is what is so exciting about open source. I'm thrilled that we have open copilots, and I hope they take off, but VScode is amazing. I use ChatGPT regularly, but I still don't know many folks who use Bing for search. However, the moat of search is a real one. Putting these two points together, I wanted to frame some thoughts on the "moat." It's true that the technical moat for AI models is rapidly commoditizing. This community is vibrant and exciting! Let's live up to the best of what we can do. Also there were immensely valuable discussions and contributions from folks we as a community owe a ton to like Stella and the EleutherAI folks to further refine data mixtures.Last year at this time, I was constantly engaged in hand-wringing about whether Academia and Open Source would contribute to what happens next, and yes-we very much have-and I believe will continue to do. Personally, I was thrilled to see our little bricks of contributions like RedPajama, FlashAttention and longer sequence models featuring prominently in reproductions from Together, MosaicML, and Berkeley and in the BigCode models. Also, congratulations to our friends at Berkeley with Vicuna and Stanford for Alpaca getting such a wonderful shout out! Awesome to see!.I'm excited that large companies are waking up to the Linux moment in AI. The business and resources moat aren't going anywhere, but I'll talk about this viewpoint a bit more later. To me, the claimed technical moats of big tech are eroding (and maybe overstated). The "no moats" draft was released/leaked, and AI internet went crazy. These last few weeks have been a whirlwind! Even this week, a few things happened that were personally exciting to me. ![]() A Llama wearing red pajamas wades through a moat. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |