The state of frontier models across the SDLC

Prefer to listen?

Large Language Models (LLMs) are increasingly being integrated into software development workflows, promising to transform how code is written, tested, and maintained. This research analyzes the current capabilities of LLMs in software engineering through the lens of leading benchmarks, assessing their strengths, limitations, and potential across different phases of the software development lifecycle.

Methodology

While existing benchmarks provide insight into LLM strengths and weaknesses, they tend to offer only a fragmented perspective. To develop a more comprehensive understanding, we systematically mapped LLM capabilities to each phase of the software development lifecycle (SDLC)—from planning and design to development, testing, deployment, and ongoing maintenance. This approach allowed us to contextualize model performance within real-world development workflows and identify where LLMs are most and least effective.

The following benchmarks were analyzed to evaluate LLM coding capabilities: SWE-Bench-verified, SWE-Lancer, LiveCodeBench, and HackerRank-ASTRA.

LLM performance across the SDLC 

Overview by SDLC phase:

Phase-by-phase analysis:

Benchmark analysis: evaluating LLM coding capabilities

SWE-bench, SWE-bench+ & SWE-bench-Verified

SWE-bench was created to systematically evaluate LLMs’ capabilities in resolving software issues, comprising 2,294 real-world GitHub issues and their corresponding pull requests from 12 widely used Python repositories. However, a systematic evaluation revealed significant quality issues with the dataset:

• 32.67% of successful patches involved “solution leakage,” where solutions were directly provided in issue reports or comments
• 31.08% of passed patches were deemed suspicious due to inadequate test cases
• Over 94% of issues were created before LLMs’ knowledge cutoff dates, raising potential data contamination concerns.

When these problematic issues were filtered out, the resolution rate of SWE-Agent+GPT-4 dropped dramatically, from 12.47% to just 3.97%. The steep decline in performance underscores the need for rigorous benchmarking methodologies that control for data leakage and test adequacy—otherwise, we risk painting a misleading picture of LLM capabilities.

Subsequently, SWE-bench+ was developed to address these limitations. It collects GitHub issues created after LLMs’ training cutoff dates to prevent data leakage.

SWE-bench-Verified is another offshoot of SWE-bench built by OpenAI consisting of 500 human-validated samples from real GitHub issues. Unlike its predecessor, this benchmark addresses critical flaws in the original SWE-bench dataset:

1. Solution Leakage Mitigation: Eliminates 32.67% of cases where solutions appeared verbatim in issue discussions
2. Test Adequacy Verification: Removes 31.08% of previously accepted solutions that passed due to insufficient test coverage
3. Temporal Contamination Prevention: Only includes issues created after LLM training cutoff dates (post-2023)

SWE-Lancer

OpenAI’s SWE-Lancer benchmark represents a significant advancement in evaluating LLMs’ capabilities on practical, real-world coding tasks. This benchmark includes over 1,400 tasks sourced from Upwork with a combined value of $1 million, spanning independent coding activities and managerial decision-making tasks.

SWE-Lancer emphasizes economic value assessment, tying model performance directly to monetary worth in the freelance software engineering ecosystem. The benchmark employs end-to-end testing verified by professional engineers to ensure practical validity.

Despite recent advancements in AI language models, initial findings revealed significant limitations:

• The best-performing model, Claude 3.5 Sonnet, achieved only 26.2% success on independent coding tasks
• Most models struggle with tasks requiring deep contextual understanding or the ability to evaluate multiple proposals

The benchmark includes diverse tasks such as application logic development, UI/UX design, and server-side logic implementations, ensuring comprehensive capability assessment.

LiveCodeBench

LiveCodeBench offers a holistic, contamination-free evaluation approach that continuously collects new problems. Currently hosting over 300 high-quality coding problems published between May 2023 and February 2024, this benchmark focuses on broader code-related capabilities beyond mere generation.

Key features of LiveCodeBench include:

• Annotation of problems with release dates, allowing evaluation on specific periods to measure generalization on unseen problems
• Assessment of various code-related scenarios: code generation, self-repair, test output prediction, and code execution
• Comparative analysis of model performance across different scenarios

The benchmark has revealed interesting performance patterns:

• DeepSeek models showed significant performance drops on LeetCode problems released after September 2023 (its release date), indicating potential contamination in earlier problems
• GPT models maintained relatively stable performance across different periods
• Model performance correlations across different scenarios varied significantly – Claude-3-Opus outperformed GPT-4-turbo in test output prediction but not code generation.
• Mistral-Large demonstrated superior performance on natural language reasoning tasks like test output prediction and code execution.

Overall, closed API-access models consistently outperformed open models on LiveCodeBench evaluations.

LiveCodeBench leaderboard (as of April 2025)

HackerRank-ASTRA

The HackerRank-ASTRA benchmark fills a critical gap in LLM evaluation. Unlike many that focus on standalone coding problems, it tests models on multi-file, cross-domain projects that reflect real-world development workflows.

A distinguishing feature of this benchmark is its rigorous assessment of model consistency through 32 runs (k=32) and median standard deviation analysis. Initial evaluations on 65 problems yielded notable findings:

• GPT-4.1, DeepSeek-R1, and o3-mini currently top the leaderboard, each achieving over 77% average scores, with GPT-4.1 showing the highest average score (81.96%) and pass@1 rate (71.72%).
• DeepSeek-V3, Claude 3.7 Sonnet ,and GPT-4.5-preview remain strong performers, each near 77.5%, with Claude 3.7 showing remarkable consistency (0.10).
• GPT-4o ranks mid-pack with an average score of 69.52%, showing lower pass@1 (50.91%) and higher variability (consistency = 0.20) compared to top performers.
• LLaMA-3-70B, the only open-weight model on the board, trails with a 61.65% score and 46.54% pass@1, highlighting the performance gap that remains between open and closed models.

This benchmark’s focus on consistency highlights an essential dimension of model evaluation beyond raw performance, emphasizing reliability for real-world software development tasks. 

Conclusion

The current generation of frontier LLMs demonstrates promising capabilities in specific software development tasks, particularly in code generation for well-defined problems and particular aspects of test prediction. However, significant limitations remain in handling complex, multi-file projects and tasks requiring deep contextual understanding or architectural decision-making.

While LLMs can currently augment developer productivity in specific contexts, particularly for code generation, snippet completion, and documentation tasks, they are not yet capable of replacing human developers across the full spectrum of software development activities. Their most effective application remains as collaborative tools that enhance human capabilities rather than autonomous systems that drive the development process.

As these technologies continue to evolve and benchmarks become increasingly rigorous, we can expect steady improvements in LLM capabilities across the software development lifecycle. This will transform how software is built, tested, and maintained in the coming years.