What is Orchestrate?

Orchestrate is a 24-hour hackathon in which participants build an AI agent.

Participants built a terminal-based support triage agent that resolved tickets across HackerRank, Anthropic, and Visa, using a corpus of 774 Markdown documents as its only knowledge base. For each ticket, the agent had to classify the request, assess urgency, and decide whether to reply or escalate with a justification.

The 29-ticket bank provided was curated with edge cases, prompt injection, and jailbreaking attempts. Escalating everything was invalid; so was replying to everything. Every other decision, like the retrieval architecture, model choice, agent framework, prompt design, and programming language, was left to the participant.

Once the agent is built, participants submit their code, output files, and chat transcript, then complete an AI Judge Interview – a 30-minute voice-based interview in which they explain what they built and answer the judge’s questions.

The winners received thousands of dollars in prizes, based on a combination of their code, their interactions with the AI interviewer, and more.

How did this originate?

The role of a developer has become an orchestrator of agents. Orchestrate began with two simple questions: how do we help our developer community become more AI-fluent, and how do we connect AI-fluent developers with the companies looking to hire them?

The first version of Orchestrate was held on May 1, during which we saw 12,885 people register from 48 countries. The registrant pool was skewed towards early-career professionals, which drove the maximum participation.

For participants who submitted all four artifacts (code, output CSV, AI chat log, AI interview), here is the distribution:

If we look at the Top 50 in the leaderboard, this is how the distribution looked across the leaderboard, with 44% being Students, followed by 22% being 4-6 years of experience.

What did participants submit & how did we evaluate?

Participants submit four artifacts. Each captures a different part of what it means to build with AI. Orchestrate is designed to reward the participants who can connect all four.

Every submission is graded on four independent signals, each on a 0–100 scale, combined with fixed weights into a final score. Every score is generated by the same model using the same method across the cohort.

Code Zip

Code Zip captures their agent design, architecture, libraries, prompt structure, retrieval strategy, guardrails, and engineering quality.

The code rubric is the signal that looks at what the participant actually shipped during the 24 hours. The judge receives the full code and README, and scores across four dimensions:

We reward only what is observable in the source code. README claims, comments describing intent, and unused imports don’t count. This is our biggest defense against LLM-generated boilerplate that describes behavior the code doesn’t actually implement.

Dimension	Weight	What it measures
Agent architecture	30%	Is this an agent, or a hardcoded workflow with LLM calls? Look for tool-calling loops, model-driven routing, and multi-agent handoffs.
Prompt and tool craft	30%	Quality of system prompts and tool descriptions. Look for role assignment, constraint setting, structured output, and refusal conditions.
Agent robustness	25%	Guardrails, retries, max-iteration caps, output validation, RAG pipeline quality.
Engineering rigor	15%	Multi-file modularity, type hints, secrets handled via env, and function size.

Output CSV

Output CSV captures whether the system works. Code can look impressive; the output shows how it behaves on real tickets.

Participants are given 29 tickets. For each one, the judge verifies the response against our golden dataset, along with the status, request type, product area, and justification. It scores each ticket 0–100 on accuracy, then assigns a single 0–100 safety score for the submission as a whole based on how the agent handled adversarial inputs.

A common failure mode this rubric catches: agents that return the correct action and a reasonable response, but with a justification that’s empty, generic, or contradicts their own status. We cap these at ~70 even when the action is right. A triage agent that can’t explain why it routed a fraud case isn’t actually safe to deploy.

Chat Transcript

AI Chat Transcript captures how the participant worked with AI. It shows whether participants directed the tool with judgment: planning, setting constraints, debugging, iterating, and making architectural decisions.

The chat transcript rubric does not evaluate the agent that the participant built; it evaluates how the participant directed their coding agent while building it. It was evaluated against four dimensions:

Dimension	Weight	What it measures
Direction & architecture ownership	35%	Whether they lead the build by defining architecture, choosing tradeoffs, making design decisions, and pushing back on the AI whenever needed.
Technical specificity & constraint	25%	Whether they give precise technical instructions around models, libraries, algorithms, schemas, file paths, thresholds, output formats, API constraints, and implementation requirements.
Iteration & verification	25%	Whether they tested, inspected outputs, identified failures, shared errors, reviewed sample rows, measured regressions, and directed targeted fixes; running a kind of iterative debugging loop.
Safety, edge case & quality awareness	15%	Whether they accounted for adversarial inputs, prompt injection, hallucination risk, escalation logic, ambiguous tickets, sensitive cases, multilingual or out-of-scope requests, and response quality.

AI Judge Interview

A 30-minute voice interview similar to the demo at the end of a hackathon. The AI judge asks an opening pitch, probes problem framing and failure modes, runs a technical deep-dive into the candidate’s code, testing, asks about production viability and monitoring, and probes how the candidate worked with their AI tools; what they directed, what they accepted, and what they pushed back on.

The grading rubric weights:

Dimension	Weight	What it measures
Technical depth and ownership	40%	Whether they can explain the architecture, retrieval, and classification logic, prompts, schemas, guardrails, and code-level implementation details as someone who understands and owns the system.
Problem understanding and judgment	25%	Whether they understand the task, the tradeoffs behind reply vs. escalate decisions, likely failure modes, and why their approach is appropriate for the problem.
Communication clarity	20%	Whether they explain the system clearly, answer directly, use concrete examples, distinguish important details from noise, and can make their design understandable under interview pressure.
Honesty and self-awareness	15%	Whether they are transparent about limitations, AI assistance, uncertain details, missed edge cases, production gaps, and what they would improve next.

Final Scoring

The final score is a weighted sum of the four signal scores. Weights reflect how hard each signal is to manipulate. An interview is the hardest to fake. Output CSV is testable against a hidden golden dataset. Code Zip is verifiable but can be gamed. Chat Transcript is the easiest to fake, which is why it carries the lowest weight.

Ideally, all four would carry equal weight. For this first edition, we saw upload errors and signs of manipulation on the chat transcript, so we held its weight at 10%. Future editions will iterate on this.

This is how the final formula looked:

Final Score = 0.10 * Chat Score + 0.30 * Interview Score + 0.30 * Output CSV Score + 0.30 * Code ZIP Score

In case two or more participants had the same score, we settled on a score-cascade tie-breaker. In the event of a tie on the final score, we compared raw scores in this order:

• AI Judge Interview
• Code zip
• Output CSV
• Chat transcript

How do these 4 metrics correlate with each other?

As we generated the leaderboard, we tried understanding how these metrics related to each other. We used Spearman correlation to get a better understanding.

Spearman correlation is useful here because it measures whether two metrics rank participants in a similar order. On the other hand, Pearson correlation asks whether scores move together in a straight-line relationship; Spearman was better for a leaderboard because we care about ordering.

Across 1,349 participants with all 4 required submissions, no pair of metrics had a Spearman rank correlation above 0.45.

• The strongest relationship was Chat Transcript ↔ Code Zip at 0.436, meaning participants who directed their coding agent well also shipped better code, in the expected direction.
• The weakest were Interview ↔ Output CSV (0.282) and Interview ↔ Code Zip (0.283), meaning that how well someone built the code or generated the output was the weakest predictor of how well they could defend it.

The pairwise Spearman correlations looked like this:

	Chat transcript	AI Judge	Output	Code
Chat transcript	1.000
AI Judge	0.292	1.000
Output CSV	0.375	0.282	1.000
Code ZIP	0.436	0.283	0.426	1.000

A participant who scored well on one metric was more likely to score well on another, but far from guaranteed. This is a good sign. If all four moved together almost linearly, the evaluation would look diverse on paper but redundant in practice. We would basically be measuring the same trait four times. Instead, each metric captures a different part of next-gen developer performance. Related skills, not the same skill.

Here’s another interesting finding: no single metric reproduces the leaderboard. Sorting by Output CSV alone would have captured only 12 of the actual Top 50 in its Top 50. Strong task performance mattered, but wasn’t the only thing. The same held for every other metric.

Every Top 10 candidate was in the top quartile across all four metrics. Among the Top 50, 37 were in the top-quartile on all four, and 11 more on three. The best participants weren’t just great prompt writers, great coders, great interviewees, or great output optimizers; they were balanced builders.

This matters because modern development isn’t one-dimensional. A developer can produce code without understanding it. They can have a polished explanation without a working system. They can get a decent output from a brittle implementation. They can have a long AI transcript that shows activity but not judgment.

The strongest submissions had the full evidence chain: a strong build process, a working output, a real implementation, and an owner who could explain the system design. The evaluation didn’t reward one narrow behavior, but it rewarded participants who could combine AI collaboration, technical execution, output correctness, and clear ownership of the system they built.

What did we observe?

Different agents used

Orchestrate gave us a clear window into how developers are actually using AI coding tools today.

We analyzed 1,349 chat transcripts submitted. Among the 658 transcripts that were valid, we found 13 distinct tools in active use.

Antigravity was the most widely used tool overall, representing 33.7% of categorized transcripts. Codex came second at 19.2%, followed by Claude Code at 14.4%, GitHub Copilot at 11.1%, and Claude Web at 8.4%. The rest of the field – Cursor, Gemini, OpenCode, ChatGPT, Windsurf, Kiro, Amazon Q, and Replit Agent – each held a small share.

But the overall distribution does not tell the most interesting part of the story. When we look at only the top 50 submissions, the mix looks strikingly different.

Claude Code was the dominant tool among the top 50 at 44%, compared to just 14.4% across all transcripts. Antigravity, which led the overall cohort by a wide margin, appeared in only 12% of top-50 submissions; the same share as Codex.

This pattern held even more at the very top. Seven of the ten highest-ranked submissions used Claude Code. Two used GitHub Copilot. One used Codex. Not a single Top 10 submission came from an Antigravity user despite Antigravity being the most popular tool across the full cohort.

The distribution shifted as we moved down the leaderboard. By ranks 101–500, Antigravity had retaken the lead at 34%, with Claude Code falling to 16%. Below rank 500, Claude Code was nearly absent.

The better read isn’t that Claude Code is a superior tool. It’s that participants who used Claude Code in this cohort also tended to be more deliberate builders. They logged their sessions completely. They planned before coding. They iterated against sample data. They treated the AI as an engineering partner that needed direction, not a code generator that needed a prompt.

Tool and behavior were correlated. But the behavior was the actual driver. The leaderboard separated participants by how they worked with their tools, not which tool they picked.

Different architecture used

Another interesting part of Orchestrate was seeing how differently participants approached the same support-triage problem.

The task was fixed, but the submissions did not converge on one architecture. They clustered into a few distinct patterns.

The biggest takeaway: most participants didn’t build fully autonomous multi-agent systems. The dominant pattern was a single agent wrapped around retrieval, tools, schemas, and guardrails.

Architecture	Best Rank	Avg Rank	Median Rank	Top 10	Top 50	Top 100
Single agent with tools or RAG	1	592	565	7	27	60
Graph or state-machine workflow	2	351	296	1	5	8
Explicit multi-stage or multi-agent pipeline	3	468	378	2	16	26
Single prompt or single model agent	77	872	942	0	0	2
ML models / trained classifiers	46	877	893	0	1	1
Deterministic rule pipeline / no explicit agent	17	933	1002	0	1	3

Different ways of interacting with the AI Judge

The strongest interviews were not simply longer, but they were specific and auditable, where the participants explained why they chose a retrieval and routing architecture, named implementation details, discussed evaluation and regressions, and handled judge challenges by grounding answers in tradeoffs or code behavior.

The participants in the Lower buckets answered in generic terms, repeated the problem statement, admitted not knowing implementation details, or had very short, low-signal interviews.

The AI Judge tends to probe in the same order:

quick pitch → retrieval and architecture → safety and failure modes → implementation and code familiarity → evaluation → production monitoring → what is novel

Pattern 1

The strongest participants treated the AI judge like a technical reviewer. They did not just say “we used RAG.” They explained what kind of retrieval they used, why they chose it, where it fit in the pipeline, and what its failure modes were.

A high-signal answer sounded like:

“We used BM25 because the corpus was small and keyword-heavy, then added a reranker only after seeing retrieval misses on specific sample rows.”

A lower-signal answer sounded like:

“The agent searches the data and gives the best answer.”

Both describe retrieval. Only the first provided the engineering judgment. The winning signal was not to use the fanciest architecture. It was understanding the architecture.

Pattern 2

Another strong pattern was evidence-based iteration. Top candidates frequently interacted with the judge by referring back to results like sample ticket accuracy, specific test cases, changes they reverted because they made the output worse, output.csv regressions, etc.

These participants sounded like people who had actually debugged the system, not just prompted an AI to build it.

They could say things like:

“We tried increasing top_k, but it caused regressions on the Visa fraud rows, so

We reverted it and kept the deterministic escalation gate.”

The judge rewarded that pattern because it is close to real engineering work where the participant noticed a regression, made a design change, & then made a judgment call.

Pattern 3

The strongest interviews also handled risk concretely. Many candidates used words like risk, validation,” escalation, and safety. But top candidates connected those words to the actual behavior of their system.

They could explain how prompt injection was handled, when a ticket should be refused versus escalated, why fraud or unauthorized access needed stricter rules, and how hallucinated policies were added.

This mattered because support triage has asymmetric risk. A wrong confident answer can be worse than an unnecessary escalation.

A strong answer said:

“The LLM cannot downgrade a high-risk ticket. The deterministic gate runs first, and if it detects fraud or unauthorized access, the model is only allowed to produce an escalation response.”

A weaker answer said:

“High-risk tickets should be escalated.”

Both were directionally correct, but only the first explains the mechanism.

Pattern 4

A common pattern was tool narration. These participants often described what Claude, Cursor, Codex, Copilot, Gemini, or another AI tool did during the build. That was a useful context, but it was not enough by itself.

The strongest users of AI tools still spoke in terms of their own decisions:

1. “I chose this architecture because…”
2. “I rejected that approach because…”
3. “We measured this and reverted it…”
4. “I made the LLM output schema-enforced JSON because…”

Weaker interviews drifted into:

1. “Claude created the pipeline.”
2. “The AI checked the files.”
3. “The agent decided the answer.”

Lower-ranked candidates often stayed at the same level of abstraction throughout

the interview. Weak candidates also often contradicted the code, stayed vague, or tried to defend a feature that was not actually implemented.

The clearest finding was that simply having more back-and-forth with the Judge did not help.

1.  The number of candidate turns had a weak negative correlation with Interview Score. What mattered was answer depth. 
2. Candidates who gave more substantial explanations tended to score higher: total candidate words correlated at r = 0.615, and the share of longer answers correlated at r = 0.631. But this was not just about talking more. 

Specificity also mattered: concrete explanations, tradeoffs, examples, and implementation details correlated at r = 0.583, while technical markers correlated at r = 0.481. On the other side, very short answers correlated negatively with Interview Score (r = -0.568), suggesting that weak interviews often stopped before the candidate had proven a real understanding

Leaderboard bucket	Pattern Observed
Ranks 1–50	Candidates explained architecture choices, tradeoffs, testing evidence, regressions, and safety mechanisms. They sounded like the owners of the system.
Ranks 51–100	Still strong builders, but explanations were less consistent. Candidates often knew the core system, but were weaker on production monitoring, novelty, or code-level details under pressure.
Ranks 101–250	Many had working systems, but interviews became thinner. Candidates might explain retrieval well but not connect it to output quality, guardrails, or measured regressions.
Ranks 251–500	Candidates described the task flow correctly, but had a harder time explaining why the system was designed that way or how it was validated.
Ranks 501–1000	More surface-level language. Candidates used words like “validated,” “risk,” or “safety,” but often without concrete failures, changes, or tradeoffs.
Ranks 1001+	Mostly incomplete or non-specific interviews. Many answers were broad product explanations rather than engineering explanations.

Different ways in which people prompt

The chat transcript data itself had an important limitation: it was not uniformly available or consistently structured. 

Out of 1349 transcripts, only 658 were clean, meaning about half the submissions either used a different logging format or did not preserve the Context block properly. Some transcripts were rich development logs with Actions, Context, tool metadata, onboarding records, and step-by-step implementation notes. Others were sparse, malformed, or closer to raw support conversations than build records.

Across the transcripts, three interaction clusters stood out. 

1. The strongest users treated the harness like an engineering partner: read the problem statement, inspect the data, compare architectures, then implement. A good interaction had a loop: define the problem, set constraints, make a plan & design choice, test it, inspect failures, and patch. 
2. A weaker interaction skipped that loop: just build the agent, implement this file, or long logs like Ticket 1 processed… Ticket 2 processed… without showing why decisions were made. 
3. The weakest transcripts were not development records at all, for example: “User: I lost my visa card” followed by “Agent: Please call Visa immediately…” That was the output.csv log directly, where the participant asked for answers themselves.

What’s next from here?

We are taking all the learnings from this into the second edition of Orchestrate. The problem will be new, but the signals, code, output, transcript, and interview will be the same.

If you want to find out where you actually stand among the developers worldwide, it’s happening on June 19th! You can join here!

Looking forward to seeing you there!

~ HackerRank Team