Large Language Models in Software Engineering: Productivity, Cognitive Offloading, and Governance

Central argument

Large language models can accelerate code production, but code production is not the same as software delivery.

Their value depends on whether the generated work remains understandable, independently verifiable, maintainable, and proportionate to the consequences of failure.

---

Four Theses

This article advances four theses.

1. LLMs do not remove the historical bottleneck of software engineering. Reading, understanding, validating, and maintaining code remain more consequential than producing it.
2. LLMs introduce cognitive offloading at an unprecedented scale. Used without deliberate safeguards, they can weaken technical autonomy, ownership, and the capacity to judge their own output.
3. LLM adoption is a governance decision, not merely a tooling decision. Organizations too often act on vendor claims, fear of missing out, and individual perceptions of speed rather than measured delivery outcomes.
4. A presumed productivity gain must not eclipse other obligations. Confidentiality, security, intellectual property, compliance, sovereignty, and exposure to foreign law remain business constraints even when a tool is convenient.

These theses do not imply that LLMs are useless or that every use is irresponsible. They imply that the burden of proof should increase with the authority granted to the model and with the cost of an undetected error.

The Bottleneck Is Not Typing Code

Programming produces source code. Software engineering sustains a system through changing requirements, teams, dependencies, threats, and operating conditions. The distinction matters because an assistant that makes code cheaper to produce does not automatically make a software system cheaper to understand or change.

Developers spend substantial effort constructing mental models: what a component does, why it was designed that way, which assumptions it encodes, and what else may break when it changes. In an industrial study at Microsoft, understanding the rationale behind code was the most frequently reported serious problem; 66% of respondents identified it as such, while 82% said that understanding why code was implemented in a particular way required significant effort.¹ A study at Google similarly found that developers searched code frequently—on average, five search sessions and twelve queries per workday—to answer questions such as where behavior lives, how an API is used, and why a failure occurs.²

Generating an implementation faster does not eliminate these questions. It may defer them. If a developer accepts a plausible patch without acquiring the corresponding mental model, the immediate task can appear cheaper while the next review, incident, or modification becomes more expensive. The cost has moved from production to comprehension.

This is particularly visible in generated boilerplate. LLMs are useful when a framework demands repetitive declarations, mappings, or adapters. The local gain can be real. But boilerplate is also an architectural signal. Before automating it indefinitely, a team should ask whether a library, generator, shared component, language feature, or different boundary could remove the repetition at its source. Generating the same category of glue in every repository may conceal a missing abstraction rather than resolve it.

Tests do not remove the need for understanding either. A generated test can confirm that generated code behaves consistently with the model’s interpretation of a requirement, yet both artifacts may share the same misunderstanding. Automated tests are evidence against defined failure modes; they are not an independent oracle for business intent. If nobody reads the implementation or validates the tests against the requirement, accountability has not been automated—it has been left unassigned.

The scarce resource in software engineering is therefore not keystrokes. It is informed attention: the capacity to understand a system well enough to make and defend a change.

Productivity Evidence Is Context-Dependent

Conflicting Measures, Conflicting Results

Claims about productivity are often presented as if they measured a single property. They do not. Time to complete an isolated task, number of pull requests, accepted suggestions, lead time, review burden, incident rate, and long-term maintainability describe different outcomes. Optimizing one can degrade another.

A controlled experiment conducted by researchers associated with GitHub, Microsoft, and academic institutions asked participants to implement a JavaScript HTTP server. Developers with GitHub Copilot completed the task 55.8% faster than the control group.³ The result demonstrates that an AI assistant can accelerate a bounded implementation task. It does not demonstrate an equivalent improvement in a mature codebase, across a team, or over the lifetime of the code. The study is also vendor-associated and should be read with that context, without dismissing its experimental result.

The opposite result has appeared in a substantially different setting. METR ran a randomized controlled trial involving 16 experienced open-source developers completing 246 real tasks in repositories they had worked in for years. With early-2025 tools, allowing AI increased completion time by 19%, even though participants believed that AI had made them faster. METR explicitly cautioned against generalizing the finding to all developers or all work. A 2026 follow-up also suggested that newer tools may provide greater benefits, but selection effects and unreliable time measurements prevented a confident estimate.⁴ The important lesson is not that AI always slows experts. It is that perceived speed, benchmark performance, and observed performance can diverge.

Maintenance complicates the picture further. A 2025 preprint analyzing open-source activity reported increased output from less-experienced contributors after Copilot adoption, alongside more rework: core developers reviewed 6.5% more code and showed a 19% reduction in their own original-code productivity.⁵ Because this is a preprint and an observational study, its causal claims require further scrutiny. Nevertheless, it identifies a credible mechanism: local acceleration can transfer verification and rework to the people with the most system knowledge.

Google’s 2025 DORA research, based on survey responses from nearly 5,000 technology professionals and more than 100 hours of qualitative data, describes AI as an amplifier of the organization in which it operates.⁶ Strong internal platforms, fast feedback, clear product direction, and healthy engineering practices can convert faster generation into value. Weak controls and feature-factory incentives can convert it into more unstable or low-value output. DORA is industry research presented by Google Cloud, so its affiliations and survey-based method should remain visible when interpreting it.

Measure Outcomes, Not Activity

An organization cannot settle these contradictions by selecting the study that supports a predetermined policy. It must define what productivity means in its own system and measure it at the appropriate level. At minimum, an adoption experiment should observe:

• lead time from accepted requirement to production;
• review time and review queue length;
• rework, rollback, and change-failure rates;
• defects and security findings introduced after the change;
• operational load and incident recovery time;
• maintainers’ ability to explain and modify the resulting system;
• total tool, inference, integration, and governance cost.

Lines of code, tokens consumed, prompts submitted, and suggestion acceptance are activity metrics. They are not business outcomes. Mandating tool usage and simultaneously raising delivery targets assumes the conclusion that the experiment is supposed to test.

Cognitive Offloading and the Ownership of Knowledge

All tools offload cognition. Documentation externalizes memory, compilers automate translation, static analyzers detect classes of errors, and calculators remove repetitive arithmetic. The relevant question is not whether offloading occurs, but which capability is being offloaded and whether the human retains enough understanding to evaluate the result.

LLMs differ from conventional deterministic tools because they can produce an entire persuasive artifact—an explanation, design, implementation, and test suite—through the same interface. Fluency compresses the visible distance between a request and a finished-looking answer. That convenience can encourage a progressive change in behavior: after many acceptable outputs, a developer reads less carefully because the next output is expected to be acceptable too. The errors that remain are then precisely those that evade superficial review: misplaced business rules, unnecessary coupling, insecure defaults, or a design that makes the next change disproportionately expensive.

Evidence of this effect is emerging, but it should not be overstated. A peer-reviewed CHI 2025 study surveyed 319 knowledge workers about 936 real examples of generative-AI use. Higher confidence in the AI was associated with less reported critical thinking, while higher confidence in one’s own ability was associated with more; the work also found that critical thinking shifted toward verification, integration, and task stewardship.⁷ This is self-reported, cross-sectional evidence across knowledge work, not a longitudinal measurement of developer skill loss.

Adjacent evidence from education illustrates a possible mechanism. In a randomized field experiment with nearly one thousand secondary-school students, unrestricted access to a GPT-4-based assistant improved performance while the tool was available but produced worse unaided performance than the control condition. A tutor with guardrails largely mitigated that effect.⁸ Mathematics students are not professional engineers, and the result cannot be transferred directly. It does show that immediate assisted performance and retained independent capability can move in different directions.

The organizational risk is cumulative. If code, tests, technical choices, and trade-off analysis are all delegated, who owns the system’s knowledge? A repository can contain functioning code while the team loses the ability to explain why it is correct. Documentation generated by the same process does not fully solve this problem because it may repeat the same mistaken model of the system.

Useful safeguards are behavioral rather than ceremonial. Reviewers should be able to explain the changed behavior without relying on the generating conversation. Tests should be checked against requirements and failure modes, not merely executed. Teams should preserve opportunities for unaided debugging, design, and implementation, especially for less-experienced engineers. Tool usage should not become a performance target. The objective is not to preserve manual labor for its own sake; it is to preserve the judgment needed when automation fails.

Understanding Is the Wrong Operational Guarantee

It is tempting to settle the argument by declaring that LLMs either understand or do not understand code. That claim is stronger than the evidence permits and less useful than it appears.

Bender and Koller argued that a system trained only on linguistic form cannot, from form alone, learn meaning in the human sense.⁹ More recent work presents several competing positions and argues that language models can possess inferential and referential semantics under broader accounts of grounding.¹⁰ This is an active scientific and philosophical debate, not an established premise on which an engineering control should depend.

The operational facts are sufficient. An LLM does not provide a correctness proof, a stable architectural model, or an accountable decision-maker. It produces outputs conditioned on training, context, instructions, sampling, and surrounding tooling. Peer-reviewed research continues to treat confident but unsubstantiated generation as a reliability problem requiring external detection.¹¹ In code analysis, experimental work has found stronger performance on syntax than on dynamic semantic structures, alongside fabricated program facts, although this particular evidence remains a preprint and covers older model generations.¹²

None of this means that a model cannot generate correct or insightful code. It means fluency is not a guarantee. The user remains responsible for grounding an answer in the actual repository, requirements, runtime behavior, and operational constraints.

The distinction becomes more important with agents. A conversational answer can be rejected. An agent may read untrusted content, modify files, run commands, retrieve secrets, or call external services. The probability of any single incorrect step is only part of the risk; authority and blast radius determine the consequence. OWASP’s guidance on prompt injection therefore recommends least privilege, separation of untrusted content, adversarial testing, and human approval for high-risk actions.¹³ Natural-language instructions should not be treated as a security boundary.

Architecture remains a human responsibility for the same reason. A model can propose a local pattern without seeing equivalent code in another program, the political reason a boundary exists, or the future migration already agreed by another team. Larger context windows and repository tools improve access to evidence, but they do not create organizational accountability. Cross-project standardization still requires people who can identify duplication, negotiate ownership, and establish a maintained shared abstraction.

Security Costs Scale with Generated Surface Area

Security research does not support treating generated code as safe by default. An early study of GitHub Copilot generated 1,689 programs across security-relevant scenarios and found that approximately 40% contained vulnerabilities.¹⁴ This experiment tested an older model in controlled scenarios, so its percentage is not a universal rate for current tools. Later empirical work examining 733 snippets attributed to AI coding assistants in public GitHub projects reported security weaknesses in 29.5% of Python snippets and 24.2% of JavaScript snippets, while also showing that an assistant could repair some findings when given static-analysis warnings.¹⁵ That study is also a preprint and relies on attribution and analysis methods with limitations.

The defensible conclusion is narrower: generated code can be functional and insecure, and model improvements do not remove the need for established software-security controls. Threat modeling, dependency review, static analysis, dynamic testing, fuzzing, secret scanning, peer review, and production observability remain necessary.

Faster production can increase risk even if the average defect rate improves. If the volume of code entering review grows faster than review capacity, the absolute number of missed defects can rise. The same applies to tests: a large generated test suite can create an appearance of assurance while omitting the failure mode that the implementation also overlooked. Independent validation matters more than the volume of generated evidence.

Agents add a second security surface beyond the code they write. They process potentially hostile repository content and operate with credentials, network access, and tool permissions. Compromise can therefore involve indirect prompt injection, data exfiltration, unauthorized commands, or changes that appear legitimate in a normal development workflow. The agent should be treated as an untrusted automation principal: isolated, observable, minimally privileged, and unable to approve its own high-impact actions.

Governance, Confidentiality, and Sovereignty

The market for generative AI contains obvious commercial incentives. Model providers, cloud platforms, and hardware vendors benefit when organizations increase adoption and inference consumption. This does not invalidate their products or research, but it makes independent measurement and procurement discipline essential. Fear of missing a market or career opportunity is not a risk assessment.

Confidentiality and Deployment Boundaries

Confidentiality starts with the service boundary. The French data-protection authority, CNIL, advises users not to submit confidential personal, organizational, or legally protected information to public generative-AI services and states that the user organization generally remains legally responsible for misuse by its personnel.¹⁶ Source code can contain credentials, vulnerabilities, customer identifiers, unreleased product logic, contractual information, and trade secrets. A policy that approves a chat interface without defining what data may cross that interface is incomplete.

ANSSI’s security guidance for generative-AI systems similarly identifies passwords, API tokens, personal data, legal and financial information, and protected research among sensitive inputs requiring control.¹⁷ Deployment model matters: consumer SaaS, enterprise SaaS, a dedicated hosted model, and a self-hosted model create different exposure, logging, administration, and update paths. None is automatically safe.

Sovereignty and Intellectual Property

Sovereignty is also more precise than the physical location of a data center. ANSSI’s SecNumCloud framework explicitly considers exposure to non-EU extraterritorial law, corporate control, operational autonomy, administration, and the location of processing and technical data.¹⁸ A foreign hyperscaler may offer strong technical security while remaining unsuitable for a workload whose threat model includes foreign legal compulsion. Conversely, an open-weight model does not by itself guarantee sovereignty: the organization must still control infrastructure, dependencies, updates, expertise, and operations.

Intellectual-property questions extend in both directions. Organizations need contractual clarity about whether prompts and outputs are retained or used for training, what indemnities exist, and how generated code is screened for licensing or provenance concerns. The EU AI Act imposes transparency and copyright-related obligations on providers of general-purpose AI models, including a policy for compliance with Union copyright law and a public summary of training content.¹⁹ Those provider obligations do not determine whether a particular generated component is safe or lawful for an organization’s use.

Governance Before Scale

Governance must therefore precede scale. NIST’s Generative AI Profile frames risk management around governing, mapping, measuring, and managing risks across the system lifecycle rather than relying on a one-time tool approval.²⁰ For software development, that means documenting at least:

• approved tools, deployment modes, and data classifications;
• retention, training, subprocessors, administration, and legal jurisdiction;
• permitted tasks and prohibited repositories or components;
• identity, least privilege, audit logs, and human approval points for agents;
• independent quality and security gates;
• metrics, review dates, incident handling, and an exit strategy;
• responsibility for every generated change that reaches production.

Top-down adoption mandates invert this process. They assume value before measuring it, shift verification costs to teams, and encourage people to use the tool even where the risk is unjustified. Leadership should define desired outcomes and constraints, then allow evidence to determine where assistance is beneficial.

An Adoption Model Based on Criticality

A single organization-wide rule is unlikely to be defensible. The correct posture depends on reversibility, independent verifiability, data sensitivity, and the cost of silent failure.

Criticality	Examples	Default posture	Minimum conditions
Low	Disposable prototypes, routine documentation, isolated CRUD scaffolding, reversible boilerplate	Controlled use is reasonable	No sensitive inputs; generated artifacts remain reviewable; normal testing and dependency controls apply
Moderate	Business applications, internal platforms, data pipelines, shared libraries	Assistance, not autonomous acceptance	Independent human review; requirement-based tests; security scanning; observable team-level outcomes; accountable owner
High	Kernels and operating systems, cryptography, identity and authorization, safety-critical control, highly regulated or classified systems	Prohibit generative and agentic code changes by default	Any exception requires a documented threat model, isolated tooling, evidence of independent verification, explicit risk acceptance, and regulatory approval where applicable

This model is deliberately asymmetric. A CRUD endpoint that can be regenerated, reviewed, and rolled back is not equivalent to a scheduler, cryptographic primitive, authorization engine, or flight-control component. The question is not whether the model is impressive. It is whether the organization can detect and contain its failure before harm occurs.

Even in low-criticality work, adoption should attack the root cause of repetition where possible. Even in high-criticality environments, this position does not prohibit all research into LLMs or every indirect, non-code use. It prohibits treating convenience as sufficient evidence for inserting probabilistic generation into a critical engineering path.

Conclusion

LLMs are powerful software tools. They can reduce the activation energy of unfamiliar tasks, draft explanations, generate boilerplate, and accelerate some implementations. Rejecting those capabilities would be as unhelpful as treating them as proof of universal productivity.

The critical mistake is to confuse cheaper code production with cheaper software engineering. Every generated change still has to be understood, validated, secured, operated, and maintained. If generation accelerates while those capacities remain fixed—or deteriorate through cognitive disengagement—the organization has not removed the bottleneck. It has increased pressure on it.

Adoption should therefore be earned by evidence at the level where value and risk actually appear: the team and the production system. It should preserve independent technical judgment, protect sensitive knowledge, and become more restrictive as failures become harder to detect or reverse. For low-risk, verifiable work, LLM assistance can be a pragmatic advantage. For critical components whose correctness and sovereignty must be defensible, abstention is not technological conservatism. It is engineering discipline.

The engineers who remain free are not those who refuse every tool. They are those who can still reason, design, debug, and decide without surrendering responsibility to one.

Footnotes

1. Gina Venolia, Robert DeLine, and Thomas LaToza, Microsoft Research, “Software Development at Microsoft Observed”, 2005. ↩

2. Caitlin Sadowski, Kathryn T. Stolee, and Sebastian Elbaum, “How Developers Search for Code: A Case Study”, ESEC/FSE 2015. ↩

3. Sida Peng et al., “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot”, 2023. The controlled study was conducted by researchers affiliated with Microsoft, GitHub, and MIT. ↩

4. Joel Becker et al., “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, 2025. The study covers 16 developers, 246 tasks, and tools available from February to June 2025; its authors explicitly warn against broader generalization. See also METR, “We Are Changing Our Developer Productivity Experiment Design”, 2026, which reports indications of improvement with newer tools but considers its estimates unreliable because of participant-selection and measurement problems. ↩

5. Feiyang Xu et al., “AI-assisted Programming May Decrease the Productivity of Experienced Developers by Increasing Maintenance Burden”, preprint, 2025. ↩

6. DORA and Google Cloud, “State of AI-assisted Software Development 2025”, 2025. The report is based on survey and qualitative research and is presented by Google Cloud with research partners. ↩

7. Hao-Ping Lee et al., “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers”, CHI 2025. ↩

8. Hamsa Bastani et al., “Generative AI without Guardrails Can Harm Learning: Evidence from High School Mathematics”, Proceedings of the National Academy of Sciences, 2025. This is adjacent educational evidence, not a study of professional software engineers. ↩

9. Emily M. Bender and Alexander Koller, “Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data”, ACL 2020. ↩

10. Anders Søgaard, “Do Language Models Have Semantics? On the Five Standard Positions”, ACL 2025. The paper argues for a position that attributes inferential and referential semantics to language models, illustrating that the question remains contested. ↩

11. Sebastian Farquhar et al., “Detecting Hallucinations in Large Language Models Using Semantic Entropy”, Nature 630, 2024. ↩

12. Wei Ma et al., “Large Language Models: Understanding Code Syntax and Semantics for Code Analysis”, preprint, 2023. ↩

13. OWASP GenAI Security Project, “LLM01:2025 Prompt Injection”, 2025. ↩

14. Hammond Pearce et al., “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions”, 2021. ↩

15. Yujia Fu et al., “Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study”, preprint, 2023. ↩

16. CNIL, “Questions and Answers on the Use of Generative AI Systems”, 2024. ↩

17. ANSSI, “Security Recommendations for a Generative AI System”, 2024. ↩

18. ANSSI, “FAQ Before Starting SecNumCloud Qualification”, accessed 2026. ↩

19. European Union, Regulation (EU) 2024/1689—the Artificial Intelligence Act, 2024, particularly Articles 53 and 55. ↩

20. NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile”, NIST AI 600-1, 2024. ↩