GPT-3: The API That Learned to Write

There is a model that is learning to write this week, and the model is GPT-3: the language system from OpenAI, the successor to GPT-2, the neural network with 175 billion parameters that writes the paragraphs and the code and the answers, the API that opened to a private beta on June 11, 2020. The announcement came as a blog post and a waitlist: the developers who signed up, the researchers who studied the samples, the industry that took notice. GPT-3 is the June 2020 story, and the story is the lesson: the machines that write like us, and the businesses that will build on them.

GPT-3 is the subject of this article: how it was built, why it writes, and what the API means for the companies that will build on it.

1. The Announcement

The announcement is the event, and the event is the beta: OpenAI that opened the API on June 11, 2020, the developers who joined the waitlist, the access that was private, the applications that were invited, the requests that were reviewed. The announcement is the shape of the release: no open weights like GPT-2, no download, no local server, the model that lives in the cloud and answers through the interface, the access that is controlled by the company. The announcement is the signal: the lab that moved from the paper to the product, the research that became a service, the AI that is sold by the call.

The announcement is also the bet: the API that is the business model, the usage that will be priced, the customers that will pay, the platform that will be built around the access, the revenue that will follow. The announcement is the June 2020 lesson: the models that are too big to share, the access that is the gate, the waitlist that is the marketing, the exclusivity that creates the demand. The announcement that opened the beta was the beginning, and the beginning was the change.

2. The Scale

The scale is the number, and the number is the billion: the 175 billion parameters in GPT-3, the weights that are tuned, the layers that are stacked, the compute that trained it, the cost that only the giants can pay. The scale is the leap: GPT-2 with its 1.5 billion parameters, GPT-3 with more than a hundred times that, the model that grew and the abilities that grew with it, the pattern that held from the small models to the large. The scale is the evidence: the more data and the more compute, the better the text, the law that the researchers call scaling, the curve that keeps going up.

The scale is also the barrier: the training run that costs the millions, the machines that Microsoft provides, the Azure cluster, the partnership that funds the work, the laboratory that spends what the startups cannot. The scale is the June 2020 lesson: the models that concentrate in the few hands, the capabilities that follow the capital, the industry that will rent what it cannot build. The scale that produced the writing was the moat, and the moat was the model.

3. The Few-Shot Learning

The learning is the method, and the method is the prompt: the examples that are placed in the input, the model that reads them, the task that is understood without the fine-tuning, the training that is not needed. The few-shot learning is the change: the old way that retrained the model for every task, the new way that writes the instruction, the few examples that are enough, the adaptation that happens at the time of the call. The few-shot learning is the surprise: the model that generalizes, that follows the pattern, that imitates the style, that answers without the bespoke tuning.

The few-shot learning is also the product: the developer who describes the task, the API that performs it, the customization that costs nothing extra, the barrier that falls for the small teams. The few-shot learning is the June 2020 lesson: the shift from the training to the prompting, the skill that moves from the data scientist to the writer, the access that is democratized, the playing field that is leveled for the small teams. The few-shot learning that skipped the fine-tuning was the unlock, and the unlock was the product.

4. The Capabilities

The capabilities are the proof, and the proof is the demo: the model that writes the article, that translates the sentence, that answers the question, that writes the code, that completes the pattern, that does what the prompt asks. The capabilities are the range: the prose and the poetry, the email and the essay, the JavaScript and the Python, the summaries and the dialogues, the tasks that were separate tools and are now one model. The capabilities are the quality: the text that reads like a human wrote it, the code that compiles, the answers that hold together, the output that is hard to distinguish from the human.

The capabilities are also the question: the mistakes that are confident, the facts that are invented, the bias that is inherited, the model that is fluent and not always right. The capabilities are the June 2020 lesson: the power that is real and the limits that are real, the supervision that is needed, the human in the loop, the review that is mandatory. The capabilities that amazed the beta were the promise, and the promise was the caveat.

5. The Road From GPT-2

The road is the history, and the history is the caution: GPT-2 that was announced in February 2019, the model that was withheld, the misuse that was feared, the fake news that it could write, the staged release that followed, the months of the debate. The road is the change: the lab that held back the model, the criticism that followed, the release that came anyway, the harm that did not materialize, the lesson that was learned. The road is the memory: the earlier fear, the later calm, the balance that the company now tries to strike.

The road is also the pivot: from the withheld weights to the controlled API, from the download to the waitlist, from the paper to the product, the safety that is built into the access. The road is the June 2020 lesson: the release that is a process, the safety that is a policy, the trust that is managed, the responsibility that the leader carries, the example that the industry follows. The road from GPT-2 was the education, and the education was the design.

6. The API Business

The business is the model, and the model is the meter: the API that charges by the use, the tokens that will be counted, the billing that will follow the calls, the revenue that scales with the usage, the service that is paid for like the electricity. The business is the platform: the developers who build on the API, the applications that embed the writing, the startups that will be founded, the ecosystem that grows around the access, the Microsoft cloud underneath, the Azure that is the engine room. The business is the strategy: the AI that is not a product but an input, the capability that is rented, the intelligence that becomes a utility.

The business is also the tension: the waitlist that limits, the access that is rationed, the customers that wait, the competitors that will copy, the prices that will fall. The business is the June 2020 lesson: the AI as a service, the models as the infrastructure, the companies that will be built on the rented intelligence. The API business was the beginning, and the beginning was the market.

7. The Business Implications

The implications are the customers, and the customers are the companies: the marketers who need the copy, the support teams who need the answers, the developers who need the code, the publishers who need the drafts, the tasks that will be automated, the time that will be freed. The implications are the workflows: the first draft that the machine writes, the human who edits, the response that the bot drafts, the agent who approves, the productivity that multiplies. The implications are the change: the content that is generated, the customer service that is augmented, the coding that is assisted, the roles that shift from writing to reviewing.

The implications are also the caution: the output that must be checked, the brand that is at stake, the regulation that may come, the jobs that will change, the responsibility that stays with the company. The implications are the June 2020 lesson: the automation that starts with the draft, the human that remains the editor, the value that is in the judgment. The implications for the businesses were the point, and the point was the opportunity.

8. The Lesson

The lesson is the law, and the law is the scale: the models that improve with the size, the compute that compounds, the capabilities that emerge, the curve that rewards the biggest bets, the few who can afford the race, the many who will rent the results. The lesson is the June 2020 meaning: the AI that is a service, the access that is the gate, the safety that is the policy, the responsibility that is the price of the power. The lesson is the practice: the prompts that are the new skill, the evaluation that is constant, the human oversight that never leaves.

The lesson is also the perspective: the technology that writes, the businesses that will adapt, the change that is gradual and then sudden, the future that is being built in the beta. GPT-3 is the June 2020 story, and the story is the lesson: the model that learned to write, the API that made it available, the year that opened the door, the industry that will not look back. The waitlist will shorten, and the writing will change.

Tags

#ai #engineering