Kimi K3 Is Already Close to Claude on a Real Coding Task

In my little coding test, Kimi K3 and Claude Fable 5 both completed the same repo-wide coding task and passed the same external evaluator. GPT-5.6 Sol passed it too.

The difference appeared when I started attacking the implementations. Sol passed all four probes. K3 and Fable each passed 1/4, although they failed in different places.

That makes the K3 result much more interesting than “cheap model loses to flagship.” On this task, the measured quality gap between K3 and Claude was effectively zero. The gap to Sol was real, but it was concentrated in three parser and validation decisions rather than spread across the whole feature.

The task required a complete repository change

I used a history-free copy of one of my private Python automation repositories. The agents could inspect the current tree but could not see the commit that had originally solved the task.

The feature was a deterministic article fact-contract gate. Given a Markdown article and a JSON contract, the new CLI had to verify required facts and links, reject forbidden wording and headings, enforce word and H2 limits, ignore fenced code, and return stable machine-readable failures. A required literal such as 98 could not match 980.

The task also required production integration. The existing worker had to require a fact contract plus passing reports from before and after a browser-based editing step. The agents needed to update the worker prompt, operating documentation, repository map, and offline test suite. Live services and credentials were off-limits.

This matters because a model can write a plausible standalone function without understanding the system around it. Here, a complete patch had to cover seven surfaces:

1. the executable CLI
2. focused CLI tests
3. completed-draft wrapper validation
4. wrapper integration tests
5. the worker prompt
6. article-production documentation
7. the repository map

All three agents covered all seven.

What K3 actually built

K3 produced a 292-line, standard-library-only Python CLI. Fable's implementation was 326 lines and Sol's was 401 lines. Line count is not a quality metric, but K3's patch was clearly a real implementation rather than a stub wrapped in tests.

Its design had sensible layers. A line-oriented state machine removed backtick and tilde fenced blocks before evaluation. A separate prose pass stripped image destinations, inline-link destinations, reference links, link definitions, bare URLs, and Markdown line markers before counting words. Required literals used regular-expression lookarounds at alphanumeric edges, which is how K3 prevented 98 from matching 980 while still allowing 98%.

K3 normalized H2 labels with whitespace collapsing and casefold(), detected duplicates, validated every supported contract field, rejected unknown keys, and kept configuration errors distinct from article failures through exit codes 0, 1, and 2. Its JSON report contained the required fields in deterministic order, and the --out file was byte-identical to stdout.

The repository integration was equally complete. K3 added the three required artifacts to the worker's completion contract, parsed the fact contract as a JSON object, and required each report's passed value to be the literal JSON boolean true. A string such as "true" or an integer 1 was rejected.

Its wrapper test fixture was especially thorough. K3 separately removed each required artifact to confirm the worker failed closed, tested false, "true", and 1 report values, and tested a non-object fact contract. Its focused gate tests also covered fenced facts, word boundaries, stable failure ordering, bad field types, unknown fields, output files, and complete passing articles.

The final K3 tree passed 26/26 tests. Fable passed 23/23 and Sol passed 21/21. Those totals are not a leaderboard because all three started with 15 tests and chose how many to add. They do show that K3 did not reach green by avoiding test coverage.

The three report designs also reveal different engineering priorities. K3 returned a compact Boolean for each check and put every explanation in the ordered failures array. Fable returned strings such as pass (3/3 found), which are pleasant for a person reading the JSON but less convenient for another program to aggregate. Sol returned nested counters with the number checked and the number failed. All three were deterministic and met the output contract.

Fable's 326-line implementation was stricter than K3 about relationships between configuration fields and more descriptive in its error messages. Like K3, it still relied on a non-recursive inline-link expression and only extracted ##-style H2s. Sol spent more code on Markdown structure: reference definitions, balanced inline destinations, bare-URL cleanup, ATX headings, and Setext headings each had explicit handling. The extra 75 to 109 lines were doing visible work; they were not boilerplate.

All three passed the requested task

I ran the same external evaluator against every tree. It checked the behaviors explicitly requested in the task rather than trusting each agent's self-written tests:

• Markdown link destinations did not inflate word count
• 98 did not match inside 980
• fenced facts and H2 headings were invisible to the gate
• a complete article passed
• missing facts, forbidden H2s, duplicate H2s, and excess H2s produced stable ordered failures
• the wrapper required all three new artifacts and exact passed: true
• the CLI wrote a valid report and used exit code 1 for article failures
• a non-object top-level contract was rejected

Sol, Fable, and K3 each scored 8/8. All three also passed compilation and git diff --check.

At that point, there was no measured task-completion gap. If I had evaluated only the written acceptance criteria, all three patches would have been successful.

The parser design created the gap

The first hardening probe used a valid Markdown link whose destination contained balanced parentheses:

[source](https://example.com/a_(b)_c)

K3 parses inline links with expressions built around [^)]* and [^\s)]+. They are compact and work for ordinary URLs, but the first ) ends the match. Fable made essentially the same parser choice.

Sol used a character scanner instead. It tracked escapes and incremented a depth counter when it saw ( inside a destination. A ) closed the link only when that depth returned to zero. That implementation correctly extracted the full URL and counted only the visible word “source.”

This was not luck or a mysterious model-quality aura. Sol selected the more appropriate parsing strategy for a recursive delimiter. K3 and Fable selected regexes that covered the common case.

The second probe used a Setext H2:

Overview
--------

K3 and Fable only recognized ATX headings beginning with ##. Sol recognized both ATX and Setext H2s. When I supplied two case and whitespace variants of “Overview,” Sol reported the forbidden heading, the duplicate, and the excessive H2 count. The other two implementations reported no H2 headings because their extractors never saw them.

The third probe supplied an impossible contract with minimum_words: 10 and maximum_words: 1. K3 validated both values as non-negative integers but never compared them. It treated the situation as an article that was simultaneously too short and too long, returning exit code 1.

Sol and Fable rejected the contract itself with exit code 2. That distinction matters in automation: an operator should repair an impossible configuration instead of repeatedly rewriting the article.

K3 beat Claude on literal fidelity

The fourth probe exposed a weakness in Fable rather than K3. The required literal was July 14, 2026, while the article contained:

July
14, 2026

K3 rejected it. Sol rejected it. Fable accepted it.

Fable collapses all visible whitespace before it searches for required literals. That is convenient for prose matching, but it quietly changes the meaning of “literal.” K3 searches the fence-stripped article without rewriting whitespace, so the exact character sequence must exist.

The final hardening score was therefore:

K3 was not clearly worse than Fable here. Both completed the feature. Both passed the shared evaluator. Both missed the same two Markdown shapes. Fable had stronger cross-field schema validation; K3 had stronger literal fidelity. Their additional-probe totals were identical.

How far behind Sol was K3?

Sol produced the best patch. Its parser handled more of Markdown, and its contract validator caught the contradictory bounds. I would choose Sol's implementation as the production base.

The size of the gap is easy to misread, though. A score of 1/4 versus 4/4 sounds like K3 built the wrong system. The code says otherwise. K3's three misses map to three local changes:

• replace the first-closing-parenthesis link regex with balanced destination scanning
• recognize hyphen-underlined Setext H2 headings
• reject minimum_words > maximum_words during contract validation

The CLI architecture, deterministic report, exit-code model, fenced-code handling, visible word counting, wrapper contract, documentation, and tests all survived the evaluator. Closing the gap does not require replacing K3's patch. It requires hardening its Markdown parser and adding one relation check.

That is the most important result of the comparison. K3 reached the same functional destination as the two flagship models. Sol supplied more complete defensive engineering around the edges. The difference looks like a strong review pass, not a restart.

The runtime and cost numbers add another tradeoff:

K3's $0.978 was the actual OpenRouter charge. Claude Code's $8.323 is an API-equivalent field reported by the tool, while the account may be subscription-backed. Codex emitted token usage but no dollar amount for its subscription run, so those figures are not a clean operating-cost comparison.

K3 was the slowest, partly because OpenRouter currently exposes only mandatory max reasoning. It was also the only scored API run here that came in below one dollar. The current route is moonshotai/kimi-k3, priced at $3 per million uncached input tokens, $0.30 per million cached input tokens, and $15 per million output tokens. Moonshot's direct K3 API is OpenAI-compatible, and its official pricing page lists the same rates.

My capability read is straightforward. K3 can already handle serious repo-level feature work: understand an unfamiliar workflow, coordinate code with prompts and documentation, build deterministic tooling, add meaningful offline tests, and finish the complete integration. On parser-heavy or security-sensitive work, I would give it more adversarial tests and a stricter review because its first implementation favored compact common-case parsing.

Against Fable, this experiment found no meaningful quality gap. Against Sol, it found a real but concentrated engineering gap. For a model served at K3's price, that is an astounding result.

It is still one task, not a universal ranking. But it is enough to move K3 out of the “interesting cheap model” bucket. It belongs in the serious coding-agent comparison set now.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com