What Is an LLM? Large Language Models Explained

What Is a Large Language Model?

A large language model is a type of deep learning model that takes text as input and generates text as output. It does this by learning statistical patterns across massive amounts of text during training, then using those patterns to predict the most probable continuation of any given input.

The "large" in the name refers to parameter count. Parameters are the numerical weights inside the network that determine how it responds to any input. GPT-3 had 175 billion of them. GPT-4 is estimated at roughly 1 trillion parameters, though OpenAI has not confirmed this publicly. More parameters means more capacity to capture nuance, handle longer context, and produce more accurate predictions, though beyond a certain scale the relationship becomes less predictable.

All modern LLMs use transformer architecture, first described in the 2017 paper "Attention Is All You Need" by Vaswani and colleagues at Google Brain. The key innovation was the self-attention mechanism. Rather than processing tokens sequentially like earlier recurrent networks, transformers let every token in a sequence attend to every other token simultaneously. This makes training parallelizable across thousands of GPUs and dramatically improves performance on long texts.

The practical result: a model that can answer questions, write code, summarize documents, translate languages, and conduct multi-turn conversations using the same underlying mechanism. Given a sequence of tokens, predict what comes next.

Parameters, weights, and what the model actually learns

A parameter is a number. An LLM with 175 billion parameters is a function with 175 billion adjustable numbers. During training, those numbers get tuned through billions of prediction examples until the model's outputs closely match the actual next tokens in the training data.

What the model learns is not facts in any retrievable sense. It learns distributions. Given this sequence of words, these next words are more probable. The model has no index of facts, no database it queries. It has statistical patterns compressed into billions of numbers. This matters enormously for understanding what LLMs can and cannot do reliably.

How LLMs Are Trained: Pre-Training, Fine-Tuning, and RLHF

Training an LLM happens in three stages. The first is the most expensive by a large margin.

Pre-training is the compute-intensive phase where the model processes hundreds of billions of tokens of text: web pages, books, academic papers, code repositories, and other sources. For each token, the model predicts the next one. The prediction error is calculated, and all parameters are adjusted slightly to reduce that error. This process runs across thousands of GPUs in parallel for weeks or months.

GPT-3 was pre-trained on approximately 300 billion tokens, including content from Common Crawl (a web crawl covering over 50 billion web pages), Wikipedia, and digitized books. Training took roughly 355 GPU-years of compute on NVIDIA V100 GPUs, according to Lambda Labs' 2020 analysis. In practice, thousands of GPUs ran in parallel to complete the job in a matter of weeks.

Fine-tuning follows pre-training. The pre-trained model is adapted on a smaller, curated dataset for specific behaviors or tasks. Fine-tuning costs are a fraction of pre-training.

RLHF (Reinforcement Learning from Human Feedback) is the step that turns a text-completion engine into a useful assistant. Human raters score the model's outputs for helpfulness, accuracy, and safety. A reward model is trained on those ratings. The LLM is then trained further to produce outputs that score highly against the reward model. This is what makes ChatGPT respond helpfully rather than simply continuing the statistical pattern of whatever you typed.

Tokenization: how text becomes numbers

Before any of this works, text must be converted to numbers. Tokenization splits text into subword units. Common words become single tokens; rare or complex words are split into two or three. In GPT-style tokenizers, "unbelievable" might become three tokens: "un", "believ", "able". The model never processes letters or words directly. It processes token IDs from a vocabulary of 50,000 to 100,000 entries.

Context window is the maximum number of tokens a model can process in one pass. Early LLMs had context windows of 2,000 to 4,000 tokens. GPT-4 supports 128,000 tokens. Gemini 1.5 Pro supports 1 million tokens (Google, February 2024), allowing it to process an entire book or large codebase in a single call.

The Major LLMs: GPT-5.4, Claude, Gemini, and Llama Compared

Seven models define the current LLM landscape across open-source and closed-source categories.

The shift from 4,096 to 1,000,000 context tokens between 2020 and 2024 is not a marginal improvement. A model with a 1-million-token context can read and reason over an entire book in a single pass. Gemini 1.5 Pro achieved this in February 2024 (Google). Early LLMs lost coherence and produced hallucinations when context exceeded a few thousand tokens; long-context improvements came from both architectural changes and better training data curation.

All closed-source models (GPT-5.4, Claude, Gemini) do not disclose parameter counts publicly. GPT-4 was the previous OpenAI flagship, estimated at 1 trillion parameters before being discontinued. Meta's Llama family is fully open-source and auditable, making Llama 3.1 405B the largest publicly verifiable model as of mid-2024.

"The models that will define the next few years are already being trained right now." (Dario Amodei, Anthropic CEO, August 2023)

The open-source track has narrowed the gap with closed-source frontier models significantly. Llama 3.1 405B outperforms GPT-4 on several benchmarks and can be run on private infrastructure, which is why it has become the default choice for enterprises that cannot send data to external APIs.

LLM Training Costs: The Real Numbers for 2026

Training costs are the largest single expense in bringing a frontier LLM to market. The numbers have changed substantially from 2020 to 2026.

"It's more than a hundred million dollars, and I think it's probably more than most people realize." (Sam Altman, OpenAI CEO, on GPT-4 training cost, cited in Fortune, April 2024)

Fine-tuning a pre-trained open-source model costs between $500 and $5,000 depending on the model size and task, according to the same analysis. This is why most enterprise deployments start with a fine-tuned Llama or Mistral model rather than training from scratch.

The number most guides don't show

Training efficiency has improved faster than raw model size has grown. GPT-3 cost approximately $4.6 million per training run in 2020. A model achieving equivalent benchmark scores today costs under $500,000 to train, according to localaimaster.com's 2025 cost analysis, which found roughly 40% year-over-year efficiency gains driven by hardware improvements and better data curation. At that pace, training costs halve roughly every two years.

This does not mean frontier AI is getting cheaper. It means the frontier keeps moving. OpenAI, Anthropic, and Google respond to each efficiency improvement by training larger models, so total training spend keeps climbing even as cost-per-unit-of-capability falls. Dario Amodei's 2023 forecast of a $10 billion training run by 2025 reflects this pattern.

For a full comparison of training costs versus the ongoing inference costs once a model is deployed, the AI training vs. inference guide covers both phases in detail.

The hardware running these training jobs is covered in the AI accelerator guide, including H100 and B200 GPU specifications and rental pricing.

How LLMs Power the AI Products You Use

Every major AI product in widespread use today has an LLM as its core component. ChatGPT, Claude, Gemini, Microsoft Copilot, and Perplexity are interfaces built around language models. The infrastructure running these products at consumer scale is one of the most expensive computing operations ever built.

When you type a query into ChatGPT, your text is tokenized and sent to a data center running thousands of NVIDIA H100 or H200 GPUs. The model processes the tokens through its transformer layers, generates a probability distribution over the next token, samples from that distribution, appends the token to the sequence, and repeats until it produces a complete response. On a single H100 GPU, a frontier LLM the size of GPT-5.4 runs inference at roughly 20 to 30 tokens per second. A 500-word response requires approximately 650 tokens, which takes 20 to 30 seconds on one GPU.

ChatGPT processes hundreds of millions of queries per day. At that volume, Microsoft (which runs ChatGPT on Azure) deploys tens of thousands of H100 GPUs purely for inference. OpenAI's monthly inference cost is estimated in the hundreds of millions of dollars. According to Goldman Sachs Research, AI inference workloads will account for an increasing share of global data center power consumption through 2030.

This is why the hyperscale infrastructure behind these products costs tens of billions of dollars to build and operate. AWS, Azure, Google, and Meta have collectively committed $290 billion in data center investment through 2027, driven almost entirely by LLM demand.

The LLM inference cost per query

Inference at consumer scale has its own economics. GPT-4 API pricing (the previous generation) ran at $0.03 per 1,000 input tokens and $0.06 per 1,000 output tokens as of 2024. An average ChatGPT query processes approximately 500 input tokens and generates 500 output tokens, costing roughly $0.045 in API-equivalent compute. At 500 million queries per day (estimated from reported user counts), the daily inference cost across ChatGPT is approximately $22.5 million, or around $8 billion per year. This is a rough estimate, since OpenAI uses proprietary infrastructure rather than the public API, but it illustrates the scale of the compute bill behind a widely used LLM product.

What LLMs Cannot Do: The Four Biggest Misconceptions

The most common misconceptions about LLMs create real problems when businesses deploy them.

LLMs do not know things

They have no memory between sessions by default, no access to live information, and no internal database of facts. They generate plausible-sounding text based on statistical patterns in training data. An LLM that states a specific fact with high confidence may be completely wrong. This is called hallucination: the model produces fluent, confident text that is factually incorrect. It happens because the model is optimized for next-token probability, not factual accuracy.

LLMs cannot reason reliably on novel problems

They appear to reason, but this is a trained pattern, not a mechanism. If you present a problem type the model has seen thousands of times in training, it responds as if it understood. On genuinely novel problems requiring step-by-step logical deduction, current LLMs fail in ways a logical reasoning system would not. Chain-of-thought prompting improves this by forcing the model to generate intermediate steps before a final answer, but the improvement is partial.

LLMs do not think between tokens

There is no deliberation process. The model runs a forward pass through its transformer layers and samples a token. It does not have a working memory or an internal deliberation state beyond what is in the context window. Responses that appear thoughtful are the product of training, not of any reasoning that happens at inference time.

Bigger is not always better

Scaling parameter counts has produced real capability improvements, but diminishing returns have appeared at the frontier. The Chinchilla paper from DeepMind (2022) showed that for a given compute budget, the optimal strategy is to train a smaller model on more data rather than a larger model on less data. Many of the largest models released between 2020 and 2022 were undertrained relative to their parameter counts. This insight drove the efficiency improvements in Llama 2 and 3, which outperform much larger models on compute-matched comparisons.

Retrieval-Augmented Generation (RAG) addresses the knowledge limitation directly by retrieving relevant documents at inference time and including them in the context window. This reduces factual errors on domain-specific queries without requiring a new training run.

The LLM Timeline: Key Milestones from 2017 to 2026

The modern era of large language models spans less than a decade. The pace of change has been rapid enough that models released in 2022 are already considered obsolete for most purposes.

The four years between ChatGPT's launch in November 2022 and early 2026 compressed more capability improvement than the previous five years of LLM research combined. A model available today through an API for a few cents per query would have required millions of dollars of dedicated infrastructure to approximate in 2020.

The cost and capability trajectory suggests that the models available in 2028 will be to GPT-5.4 roughly what GPT-5.4 is to GPT-2. The infrastructure required to train them will cost more, but accessing them through an API will cost less.

Frequently Asked Questions

Related Articles