Add upcoming GPT-3 model

[stefan-it]

🌟 New model addition

Model description

The GPT-3 paper just landed on ArXiv: https://arxiv.org/abs/2005.14165.

Would be great to integrate it into Transformers, whenever models are available.

Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Open source status

• GitHub repository is available: here
• the model implementation is available: (give details)
• the model weights are available: (give details)
• who are the authors: (mention them, if possible by @gh-username)

[LouisCastricato]

My god, the paper hasn't even been up for a day...

Said being, +1

[moinnadeem]

So who can run 175B parameters and what do I have to do for a favor?

[AdamDanielKing]

The full model will be at least 350 GB (16-bit parameters). You'd need to partition it across more than (350 GB) / (16 GB) ~ 22 GPUs just to run it! Not to mention the egress costs of making a model that size available.

Of course, the paper shows 8 different-sized models, 4 of which are smaller than GPT-2, so some of those could be practical. 🙂

[GraphGrailAi]

Is there any Colab to test at least GPT-3 XL ?

[AdamDanielKing]

Is there any Colab to test at least GPT-3 XL ?

They haven't released any code or pretrained models yet. See the issue on the official repo: openai/gpt-3#1

[minimaxir]

Note that the released models may be FP16, which may require forcing FP16 for use/finetuning (and therefore hardware-limited), or casting up to FP32.

[andifunke]

Of course, the paper shows 8 different-sized models, 4 of which are smaller than GPT-2, so some of those could be practical. slightly_smiling_face

One of the main benefits of the smaller gpt-3 models compared to their gpt-2 counterparts could be the increased context length of 2048 tokens.

[enzoampil]

Yeah, personally, I wouldn't be able to use the models that won't fit in a Tesla P100

[AdamDanielKing]

The GPT-3 repo is now archived (read-only) so perhaps OpenAI isn't planning on releasing anything this time around.

[ljlueloff]

The GPT-3 repo is now archived (read-only) so perhaps OpenAI isn't planning on releasing anything this time around.

That is a crying shame, because my system could do-er... :(

[flarn2006]

Hopefully they have a better excuse than last time.

[ljlueloff]

Hopefully they have a better excuse than last time.

@flarn2006 You mean the....ooohhhh we created something scary and have soggy diapers excuse with GPT-3?

[ljlueloff]

@flarn2006 If they don't make excuses or drag their feet, and I finish my system build in a relatively congruent time frame, hopefully I can help...

[fen0s]

A little update: OpenAI's now running their own API with GPT-3 on it. https://beta.openai.com
You can apply for access, but seems like they're aiming mostly for big companies, not researchers. Sad, way too sad.

[stefan-it]

But who put the "Open" in OpenAI then 🤔

[yassineAlouini]

I guess we will need to "fundraise" enough GPU-compute to run the GPT3 model. 😄

[fen0s]

It should be possible to run lower-models on regular GPUs, like 1b model. But we don't have the model itself, and seems that OpenAI is against releasing it and would rather commercialize it :(

[sagarreddypatil]

I wonder if you could hardcode the 175B model into an electronic chip(like an ASIC but more specific)

[yassineAlouini]

I wonder if you could hardcode the 175B model into an electronic chip(like an ASIC but more specific)

Very interesting as an idea. @StealthySemicolon do you have reference to other similar work done in the past?

[sagarreddypatil]

I wonder if you could hardcode the 175B model into an electronic chip(like an ASIC but more specific)

Very interesting as an idea. @StealthySemicolon do you have reference to other similar work done in the past?

No, just a hunch. Even if I did know how to do this, it's not like OpenAI would publicly release the model weights...

[shashankMadan-designEsthetics]

Guys when is this gonna be integrated!?

[fen0s]

When OpenAI decides to release GPT-3 open-sourcely, but this won't happen it seems, they just want to sell access to big corporations.

[shashankMadan-designEsthetics]

https://bdtechtalks.com/2020/08/17/openai-gpt-3-commercial-ai/amp/

Here it goes...

[bhack]

https://arxiv.org/abs/2009.07118
https://github.com/timoschick/pet

[OverlordQ]

Hopefully they have a better excuse than last time.

Because Microsoft gave us money.

[Clickative]

GPT-3 is not coming out anytime soon :(

[shashankMadan-designEsthetics]

this thread signifies capitalism's pros and cons at the same time...😅

[albusdemens]

The full model will be at least 350 GB (16-bit parameters). You'd need to partition it across more than (350 GB) / (16 GB) ~ 22 GPUs just to run it! Not to mention the egress costs of making a model that size available.

Of course, the paper shows 8 different-sized models, 4 of which are smaller than GPT-2, so some of those could be practical. 🙂

@AdamDanielKing is there a way to estimate the size of the GPT-3 XL model?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ItsIgnacioPortal]

we're still waiting.. :(

[srulikbd]

it seems that a replication of GPT3 might be open source soon!! :
https://www.eleuther.ai/
https://github.com/EleutherAI

[flarn2006]

Nice! Hope that works out!

…

On Mon, Jan 4, 2021, 5:00 PM srulikbd ***@***.***> wrote: it seems that a replication of GPT3 might be open source soon!! : https://www.eleuther.ai/ https://github.com/EleutherAI — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4658 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFHZUO5O7O247EHFDWH7Q3SYI27BANCNFSM4NNTN5GQ> .

[NielsRogge]

Closing this as GPT-3 won't be open-sourced unfortunately.

Have a look at an open-source effort (a 176-billion parameter multilingual language model called BLOOM) to replicate it here:

• blog post: https://bigscience.huggingface.co/blog/bloom
• model: https://huggingface.co/bigscience/bloom.

Besides that, EleutherAI and other groups (such as this one) have been working on several open-source variants of GPT-3.

[Yusuf-YENICERI]

Don't worry, if they made it, some other people going to make it, inshaAllah.

There are already replications, so wait for that.

[sarahwang93]

anyone told me is there GPT-3 available? the official one

[Yusuf-YENICERI]

@sarahwang93 No. It's not open sourced and they won't probably. Because they are able to make money using that.

Replying to myself: Yes you are right. Other people made millions of it, Alhamdulillah.

[sarahwang93]

@Yusuf-YENICERI Hope that they could opensource after they made enough money, my phd dissertation is waiting for it.

[Yusuf-YENICERI]

@sarahwang93 Why do you need it? You can't run it simply. It's a real huge model. Maybe 700GB VRAM required to run it. If you want to know about how its made you can check the paper of it.

There are other open source models. You may want to check them.

[NielsRogge]

There's also the Open LLM leaderboard which benchmarks all openly available LLMs on 4 benchmarks: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.

Of course this is not perfect as it only includes 4 benchmarks, but is still gives a nice overview of the best open-source LLMs out there.

[Yusuf-YENICERI]

@NielsRogge
https://chat.lmsys.org/?arena
This is better for simplicity, InshaAllah.