Lately, most of my projects have been built by agents - hermes-mesh, Homelander and a couple more. Somewhere along the way, I started to suspect that typed languages make agents smarter: you just type everything, and suddenly the agent «gets» your code. I wasn’t sure, and for that I needed numbers. That is why I went for them in this blog post.

I made a toy Python codebase and took one real project from open source, then let Claude Code and Codex work on them. The idea was to see if types do help agents go faster, use fewer tokens, or just be less wrong. The results surprised me personally, but before that, take your guess:

Billing

Our patient is billing - a subscription billing engine, about 3,000 lines of customers, plans, invoices, coupons, tax, dunning, refunds, webhooks. I built it ad hoc, so no model could have seen this exact code before in its training. To test the hypothesis we’re gonna need these 6 versions of it:

1. untyped - every annotation is stripped.
2. partial - signatures only, each annotation erased to a built-in type (CustomerId becomes str, PaymentMethod | None becomes dict | None).
3. well-named - untyped, but rewritten for descriptive names and docstrings.
4. typed - the strict stuff: NewType IDs and units like InvoiceId, Cents, EpochMillis, Literal statuses, TypedDict records.
5. typed + mypy - the same, plus mypy installed with one line about it in the agent’s instruction.
6. typed + ruff - the same, but the advertised tool is ruff, a simple, quick linter that knows nothing about types.

The untyped and partial versions are derived mechanically from the typed one with LibCST. A separate AST comparison is used to prove that they are the same program, just with less information.

def default_payment_method(ctx, customer_id):
    customer = ctx.store.require_customer(customer_id)
    method_id = customer["default_payment_method"]
    if method_id is None:
        return None
    return ctx.store.payment_methods.get(method_id)

def default_payment_method(ctx, customer_id):
    """The customer's default payment method record, or None if there is none."""
    customer = ctx.store.require_customer(customer_id)
    method_id = customer["default_payment_method"]
    if method_id is None:
        return None
    return ctx.store.payment_methods.get(method_id)

def default_payment_method(ctx: Context, customer_id: CustomerId) -> PaymentMethod | None:
    customer = ctx.store.require_customer(customer_id)
    method_id = customer["default_payment_method"]
    if method_id is None:
        return None
    return ctx.store.payment_methods.get(method_id)

def report_usage(ctx, subscription_id, price_id, quantity, timestamp=None, idempotency_key=None):
    ...
    record = {
        "id": ctx.ids.usage_record(),
        "quantity": quantity,
        "timestamp": timestamp if timestamp is not None else to_millis(ctx.clock.now()),
    }

def report_usage(ctx, subscription_id, price_id, quantity, timestamp=None, idempotency_key=None):
    """Record `quantity` units of metered usage and return the usage record.

    `timestamp` is epoch milliseconds and defaults to now.
    """
    ...

def report_usage(
    ctx: Context,
    subscription_id: SubscriptionId,
    price_id: PriceId,
    quantity: int,
    timestamp: EpochMillis | None = None,
    idempotency_key: str | None = None,
) -> UsageRecord:
    ...

def refund_invoice(ctx, invoice_id, reason="requested_by_customer"):
    """Refund everything that was paid on an invoice back to the original payment method."""
    invoice = ctx.store.require_invoice(invoice_id)
    ...
    refunded = 0
    for payment in payments:
        remaining = payment["amount"] - payment["refunded_amount"]

def refund_invoice(ctx, invoice_id, reason="requested_by_customer"):
    """Refund everything that was paid on an invoice back to the original payment method.

    Account credit that was applied to the invoice is returned to the ledger rather than
    the card. Returns the credit note covering both.
    """
    ...
    refunded_total = 0
    for payment in payments:
        refundable_amount = payment["amount"] - payment["refunded_amount"]

def refund_invoice(ctx: Context, invoice_id: InvoiceId, reason: str = "requested_by_customer") -> CreditNote:
    """Refund everything that was paid on an invoice back to the original payment method."""
    invoice = ctx.store.require_invoice(invoice_id)
    ...
    refunded = Cents(0)
    for payment in payments:
        remaining = Cents(payment["amount"] - payment["refunded_amount"])

The agents are Claude Sonnet and Claude Haiku through Claude Code 2.1.306 and Codex CLI 0.162.1 with gpt-6-sol - all on default effort. The queries (we call them tasks) have 3 types:

1. 16 tasks that look like tickets - questions, lists, bug fixes and changes.
2. 20 random bugs, planted automatically by mutating the code in random places. Each comes with a test that fails because of that mutation.
3. 5 changes from the history of vunnel, a real open-source vulnerability data tool. It was chosen according to 2 criteria:
   • It should be big enough to not fit in one pass.
   • It should be small enough that the models weren’t pretrained on it.

Each task was run 2 or 3 times per codebase, and I averaged the results for each metric. I mean, I agree 16 metrics may have been overkill, but more info is better.

Nobody gets it wrong

Turns out, current agents can solve almost everything - typed or not.

Check out that table: almost every cell that actually ran is 100%. The «not run» ones are intentional - each experiment only tested the versions it needed (tokens aren’t free), so hover over a cell to see the excuse. Occasionally an agent flubbed a task, but it happened in both typed and untyped runs, so it barely counts.

Then at least it’s cheaper?

That was my next hope. If types can’t buy correctness, maybe they buy speed. Surely an agent that sees EpochMillis in a signature won’t have to go hunting for what unit a timestamp is. Right?

Nope. With types, the agents spent about the same - same tokens, same time, maybe a few percent off. And when it did change, it sometimes even got worse.

What if you just use good names instead of types? It is basically the same story. Partial types make no difference either.

The checker

Only one thing made a difference: tell the agent mypy is around, and it’ll run it on basically every change. Every. Single. Time.

But maybe agents just run whatever tool you mention? That’s why I tried the typed + ruff version. And yes: Sonnet ran ruff just as eagerly as mypy. So it’s not about types at all. Agents are a bit paranoid and simply love to check things.

But only the type checker actually kept the code clean. Without it, about one in ten Sonnet changes and one in twenty Codex changes slipped a type error into the typed code. Sonnet made the same mistake every time: passing something that could be None into a function that doesn’t want None. The tests and ruff missed it completely, but when we enabled mypy, not a single one got through.

What the agent actually reads

The logs we got record every step, so we can peek at what the agents actually do with all those types. Frankly, in most of the runs they did nothing with them. Sonnet almost never opened the file where the types live; it only read the annotation next to the function.

Maybe it’s just too small?

Fair question - I wondered the same. billing is small enough to read in one gulp. vunnel is another, much bigger case; it costs the agents three or four times as much. Still, it was largely the same story: types didn’t make anything faster, cheaper, or more correct.

Every run

If you don’t trust my summaries (you shouldn’t) - here’s the whole thing.

Limitations

You’ve probably noticed this is a case study, not a benchmark like SWE-bench. So here’s what it can’t say:

1. billing and the tickets were written for this experiment with Claude, and Claude is one of the agents tested. Code in its own style may be unusually easy for it to read. The random bugs, vunnel and Codex are there to dilute that.
2. I had only 5 to 20 tasks per comparison, so only big effects could show up.

So, a seatbelt

Three agents, two projects and 766 runs later, I can say that types didn’t make agents faster, cheaper or right more often. What does help an agent is a checker. Provide one if you have one, and the feedback loop will be closed much better.

And that’s not new for people either. Back in 2017, Gao, Bird and Barr found that about 15% of public JavaScript bugs would have been caught by TypeScript or Flow.

Which is not a lot, but enough to care: a seatbelt doesn’t make you drive faster - it’s there for one particular kind of situation.

One thing that’s bothered me a lot recently is hearing people say Rust is the best language for AI to use. As we saw today, it will 100% use more tokens and take more time just to process all that verbosity. And such a strict type system absolutely must (p. 21) make AI trip up a lot too, not to mention humans (say hi to your mom, borrow checker).

Perhaps that would be a nice topic for next time.