Claude 3.5 Sonnet: Benchmarks, Pricing and Artifacts

A mid-tier model priced under the flagship it beat

Anthropic released Claude 3.5 Sonnet on June 20, 2024, the first model in what it called the Claude 3.5 family. Sonnet was positioned as the mid-sized tier, sitting below the still-unreleased Opus rank in Anthropic's naming scheme. [1]

Anthropic priced it at $3 per million input tokens and $15 per million output tokens, matching Claude 3 Sonnet's price and roughly a fifth of Claude 3 Opus's $15 and $75 rates. The company also said it ran at twice Opus's speed. [1]

Atlas interpretation: On Anthropic's reported tests, the lower-priced Sonnet outscored its previous flagship, Claude 3 Opus. That narrowed the usual choice between the two models' price and reported benchmark performance at launch; the announcement alone does not establish how they compared on every real-world task. [1]

What the benchmark claims actually cover

Anthropic reported Claude 3.5 Sonnet ahead of Claude 3 Opus on graduate-level reasoning (GPQA), undergraduate knowledge (MMLU), and coding (HumanEval), and ahead on standard vision benchmarks too. On an internal agentic coding evaluation, it solved 64% of problems against Opus's 38%. [1]

Atlas interpretation: Every one of those figures is Anthropic grading its own model against its own older model on tests it chose. That does not make the comparison false, but it is not the same claim as independent, reproducible benchmarking, and the announcement did not publish a head-to-head score against competing vendors' models. [1]

Artifacts moved output out of the chat log

Alongside the model, Anthropic shipped Artifacts in Claude.ai: a side panel that opens next to the conversation and holds a piece of generated content, such as a code file, a document, or a diagram, separately from the back-and-forth chat. A user can keep working in the panel while the conversation continues beside it. [1]

Atlas interpretation: Chat interfaces up to that point treated every output as another message in a scrolling log, which made anything long or iterative, a spreadsheet formula, a multi-file script, awkward to revise. Giving the output its own persistent, editable surface was an interface change more than a model one, and it previewed the direction Anthropic's coding tools later took. [1]

Independent accounts agree on the shape, not the specifics

Secondary coverage of the release corroborates the June 20, 2024 date and the headline framing: a mid-priced Anthropic model reported to outperform the company's own prior flagship, Claude 3 Opus, on standard benchmarks. [2]