In May 2026, HackerRank ran Orchestrate, a 24-hour hackathon in which participants built an AI support triage agent from scratch. The agent had to classify tickets, assess urgency, and decide whether to reply or escalate, using a corpus of 774 documents as its only knowledge base.

12,885 people registered from 48 countries. Every submission was evaluated on four independent signals: code quality, output accuracy, AI chat transcript, and a 30-minute AI judge interview.

The data produced some findings we expected. And some we didn’t.

What We Expected: Different Tools, Different Results

We asked participants to submit their AI chat transcripts, a log of how they worked with their AI coding tool throughout the 24 hours.

Across 658 valid transcripts, 13 distinct tools appeared. The most popular overall was Antigravity at 33.7% of categorized transcripts, followed by Codex at 19.2%, Claude Code at 14.4%, and GitHub Copilot at 11.1%.

Then we looked at the top 50 submissions. The distribution looked completely different.

Claude Code was used by 44% of top-50 participants, more than three times its overall share. Antigravity, the most popular tool in the full cohort, appeared in only 12% of top-50 submissions. Seven of the ten highest-ranked submissions used Claude Code. Not one top-10 submission came from an Antigravity user.

What We Didn’t Expect: The Tool Wasn’t the Driver

The obvious interpretation, that Claude Code is a better tool, is too simple.

When we looked more closely at how the top participants worked, a behavioral pattern emerged that was distinct from lower-ranked participants regardless of which tool they used. Top participants:

• Planned before writing a single line of code
• Set explicit constraints and architectural decisions upfront
• Iterated against sample data and inspected outputs as they went
• Treated the AI as an engineering partner that needed direction, not a code generator that needed a prompt

The behavior and the tool were correlated. But the behavior was the actual driver. The leaderboard separated participants by how they worked with their tools, not which tool they picked.

What the Four Signals Revealed

Across 1,349 participants with all four required submissions, no pair of evaluation metrics had a Spearman rank correlation above 0.45. That matters: if all four signals moved together, the evaluation would look diverse but be measuring the same thing four times. Instead, each signal captured a different dimension of what it means to build with AI.

The most striking correlation gap: Interview vs. Code ZIP scored only 0.283. How well someone built the code was a weak predictor of how well they could defend it. The candidates who produced strong code and the candidates who could explain their architecture under pressure were substantially different populations.

Every Top 10 candidate was in the top quartile across all four signals. Among the Top 50, 37 were top-quartile on all four, and 11 more on three. The best participants weren’t great prompt writers, or great coders, or great interviewees. They were balanced builders, strong across every dimension.

What the Interview Revealed

The 30-minute AI judge interview produced some of the clearest signal of the entire evaluation.

Top-ranked candidates sounded like technical reviewers. They didn’t just say “we used RAG.” They explained what kind of retrieval they used, why they chose it, where it fit in the pipeline, and what its failure modes were. A high-signal answer: “We used BM25 because the corpus was small and keyword-heavy, then added a reranker only after seeing retrieval misses on specific sample rows.” A low-signal answer: “The agent searches the data and gives the best answer.” Both described retrieval. Only the first demonstrated engineering judgment.

Top candidates also iterated with evidence. They referred back to specific regressions: “We tried increasing top_k, but it caused regressions on the Visa fraud rows, so we reverted it and kept the deterministic escalation gate.” They sounded like people who had debugged the system, not prompted an AI to build it.

Weaker candidates narrated what their AI tool did: “Claude created the pipeline.” “The agent decided the answer.” Top candidates spoke in terms of their own decisions: “I chose this architecture because…” “I rejected that approach because…” “I made the LLM output schema-enforced JSON because…”

The ownership language was consistent across the top of the leaderboard and almost entirely absent at the bottom.

What This Means for Hiring

Orchestrate wasn’t designed as a hiring study. But it produced one of the clearest datasets we’ve seen on what AI fluency actually looks like in practice, and how to evaluate it.

The findings translate directly to technical hiring:

• Evaluation built around a single signal (code output, test results, algorithm performance) captures one dimension of developer capability. High performers are strong across multiple dimensions.
• The ability to build something and the ability to explain it are different skills, with weak correlation. Hiring processes that don’t test both are getting incomplete signal.
• Behavioral patterns during the build process, planning before coding, iterating with evidence, directing the AI with specificity, are measurable and predictive of output quality.
• Ownership language is a strong signal. “I chose this because” versus “Claude built it” is not a stylistic difference. It reflects whether the candidate understands the system they shipped.

The developers worth hiring in the agentic era aren’t the ones with the best tools. They’re the ones who know how to use them.

See how HackerRank’s agentic interview surfaces these signals. Book a demo