Machines That Discover
The scholarly record on automating science — can discovery be mechanized, what has been claimed, and what actually holds up.
Canonical brief, v3. Upgraded from v2 (library listing cards) with rigorous close readings of full-text sources. Compiled via SBCC Luria Library databases through EZproxy with live Comet cookies, using skills/sbcc-library/. Pass 1: 57 raw captures (/tmp/opencode/research/). Pass 2: 41 raw captures (/tmp/opencode/research2/).
Thesis: The literature has treated discovery as at least partly algorithmic since Simon (1973), and the mechanization debate is empirically alive — but it keeps rediscovering the same wall: machines are good at searching a well-specified space, and the hard, unsolved part is defining the objective, encoding novelty, and retaining the conditions for serendipity. Close reading sharpens this: the novelty/validation oracle is now a live engineering subfield, and the binding constraint is not model capability but incentives.
Database provenance is tagged inline: (JSTOR), (MUSE), (GALE), (EBSCO).
What close reading changed
Close reading of 58 sources confirmed the brief’s core thesis but sharpened it considerably, turning abstract philosophical problems into concrete engineering challenges with an active methods literature. The v2 brief, built from library cards, was directionally correct but lacked the specific, passage-backed evidence that now grounds the synthesis.
Verdict distribution: adds-nuance: 29, misrepresented-in-brief: 1, retrieval-failure: 3, supports-brief: 25.
The most significant changes are:
- The thesis holds, but is now an engineering checklist. The v2 thesis stands, but its key concepts—the “oracle,” “novelty,” and “serendipity”—are no longer just philosophical desiderata. Close reading revealed a vibrant, recent literature that treats them as engineering problems. Works like Zhang (2026), Ahmadpour (2026), Tyagin & Safro (2024), and Jing (2025) provide concrete, domain-specific frameworks for building validation oracles and measuring novelty, turning the brief’s conclusion from a high-level insight into a practical roadmap.
- The “AI co-scientist” architecture is old. The programmatic claims in the 2026 Daedalus issue are not new. Sparkes et al. (2010) described and implemented the full
generate → design → run → testclosed loop in their “Robot Scientist” a decade and a half earlier. The current wave’s contribution is scale and model capability, not a fundamentally new architecture, which provides a crucial pre-LLM baseline for evaluation. - The historical precedent is stronger than stated. The skepticism warranted by the current hype cycle has a direct historical parallel. Brannigan’s (1989) critique, “AI: Promises versus Accomplishments,” reads as a direct response to the 2026 Daedalus issue, showing the field has cycled through these claims before.
- One claim was based on a secondary source. The brief’s entry for Merton & Barber (2005) cited the book but was based on a fetched text that was only a review of the book. This misattribution has been corrected.
- Three sources were based on incorrect texts. The entries for Anthony (2021), George (2025), and Hoffman (2017) were based on entirely wrong documents due to retrieval failures. The claims have been removed and the failures are documented below.
1. Foundations — does discovery have a “logic”?
The classic debate turns on whether there is a logic of discovery (a method that generates hypotheses) or only a logic of justification (a method that tests them).
- Carmichael, “The Logic of Discovery,” The Monist 32(4), 1922 — earliest framing of discovery as a comprehensible method. (JSTOR)
- Close read: Carmichael distinguishes the logic of discovery (“inference from the known to the unknown”) from the logic of demonstration. Crucially, he argues against a single universal method, proposing instead “not so much a logic of discovery as logics of discovery each relative to some field or subject matter of investigation” (p. 576). This adds nuance to the brief’s “a comprehensible method,” suggesting domain-specificity is a foundational concept.
- Hanson, “The Logic of Discovery,” J. Philosophy 55(25), 1958 — draws the modern proof/discovery line; followed by the Schon (1959) / Hanson (1960) exchange that muddied it. (JSTOR)
- Close read: Hanson argues for a rational “logic of discovery” concerning the reasons for proposing a hypothesis, distinct from the reasons for accepting it. He states, “If the establishment of H through its predictions has a logic, so has the argument which leads to H’s proposal initially” (p. 1083). This supports the brief’s framing but emphasizes the rationality of the initial proposal, not its mechanization.
- Simon, “Does Scientific Discovery Have a Logic?”, Philosophy of Science 40(4), 1973 — the pivotal pro-mechanization move: treats a logic of discovery as a normative theory of discovery processes. (JSTOR)
- Close read: Simon provides the direct philosophical foundation for mechanization by defining a logic of discovery as “a set of normative standards for judging the processes used to discover or test scientific theories” (p. 473). By framing discovery as a search for parsimonious recodings of data, he argues it can be evaluated for efficiency, making it a branch of computational complexity theory. This strongly supports the brief’s characterization.
- Kelly, “The Logic of Discovery,” Philosophy of Science 54(3), 1987 — argues the traditional arguments against a logic of discovery are “bankrupt.” (JSTOR)
- Close read: Kelly uses recursion theory to argue that philosophical objections to a logic of discovery are “bankrupt” (p. 435) because they ignore computational considerations. He demonstrates the feasibility of “strongly adequate hypothesis generation machines” for various classes of problems, directly supporting the brief’s claim by providing a formal, computational defense of the mechanization premise.
- McLaughlin, Philosophy of Science 49(2), 1982 — frames it as Laudan vs. Simon, both sharing a discovery/justification divorce. (JSTOR)
- Zahar, “Logic of Discovery or Psychology of Invention?”, BJPS 34(3), 1983 and Shah, J. General Philosophy of Science 39(2), 2008 — Popper, often cast as rejecting a logic of discovery, is argued to reject only one kind of it. (JSTOR)
- Close read (Zahar): Zahar argues that discovery is a “largely deductive” (p. 249) process guided by rational heuristics derived from metaphysical principles. He aims to show that discovery is not inductive or purely psychological, but operates “purely deductively (at the meta-level)” (p. 245), providing a strong rationalist, if not computational, basis for a logic of discovery.
- Close read (Shah): Shah argues that Popper only rejects an “epistemic logic of discovery” that guarantees truth before testing. Popper’s own evolutionary epistemology, Shah contends, implicitly supports an “error-eliminative logic of discovery” and an “error-corrective logic of discovery,” which are normative constraints on hypothesis generation. This refines the brief’s claim by specifying which logics of discovery even a critic like Popper can accommodate.
- Newell–Simon program: Simon, Langley & Bradshaw, Synthese 47(1), 1981 (“discovery as problem solving”); Bradshaw, Langley & Simon, Science 222(4627), 1983 (BACON); Zytkow & Simon, Synthese 74(1), 1988; canonical volume Scientific Discovery: Computational Explorations of the Creative Process (MIT Press, 1987). (JSTOR)
- Close read (Simon et al. 1981): This paper explicitly frames its central question as “whether we need a special theory to explain the mechanisms of scientific discovery, or whether those mechanisms can be subsumed as special cases of the general mechanisms of human problem solving” (p. 2). It uses the BACON and AM programs as evidence that discovery is a form of heuristic search, strongly supporting the brief’s “discovery as problem solving” tag.
- Close read (Bradshaw et al. 1983): This paper demonstrates BACON simulating the discovery of Black’s law of temperature equilibrium and inventing the concept of specific heat. It concludes that “The processes that BACON uses are much the same as those that have emerged from other research on human problem-solving… described as processes for selective heuristic search” (p. 975), providing a concrete example of the Newell-Simon program’s output.
- Close read (Zytkow & Simon 1988): This paper argues that computer discovery systems like BACON and GLAUBER create a new, formal research program for the philosophy of science. It draws a direct “analogy between formal logic and heuristic search” (p. 71), arguing that discovery and justification are intertwined in these systems, directly supporting the brief’s claim of a pro-mechanization move.
- The dispute it caused: Slezak, Social Studies of Science 19(4), 1989 claims computer discovery empirically refutes the Strong Programme; Gorman (SSS 1989) and Simon (SSS 1991) continue it. (JSTOR)
- Close read (Slezak): Slezak defends his claim that AI programs like BACON challenge the Strong Programme’s assertion that scientific content is “caused by certain features of the social milieu” (p. 673). He argues that these programs demonstrate that discovery can be explained by cognitive, rule-governed psychological factors, supporting the brief’s summary of the dispute.
- Expert systems & limits: Waltz & Buchanan, “Automating Science,” Science 324(5923), 2009; Forsythe, SSS 23(3), 1993 documents “knowledge acquisition” as a stubborn social/interpretive bottleneck. (JSTOR)
- Close read (Waltz & Buchanan): This perspective piece traces the history of automating science from early AI to modern “closed-loop” systems. It argues that “Human-machine partnering systems that match the tasks to what each partner does best can potentially increase the rate of scientific progress dramatically” (p. 44), framing automation as a continuous, respectable lineage.
- Close read (Forsythe): This ethnographic study of expert systems labs argues the “knowledge acquisition problem” is a fundamental “cultural and epistemological clash” (p. 445). Knowledge engineers see knowledge as explicit, transferable rules, while in practice it is socially constituted. This leads them to propose “a technical solution for problems that in my view are largely non-technical” (p. 467), a direct precedent for today’s limits.
- Recent: Clark & Khosrowi, Synthese 200(6), 2022 (“decentring the discoverer”); Gomes, Daedalus 155, 2026 (knowledge-centric AI). (JSTOR)
- Close read (Clark & Khosrowi): This paper argues against traditional “agent-centred” views of discovery, proposing a “collective-centred view” where the “discovering entity is a collective in the first instance” (p. 11), which can include AI systems as genuine contributors. This supports the brief’s “decentring” tag and adds a framework for credit allocation in human-AI teams.
- Close read (Gomes): Gomes argues for “knowledge-centric AI,” which integrates “reasoning grounded in scientific principles, domain structure, and constraints while learning from data” (p. 2). This is proposed as a necessary evolution beyond purely data-driven methods, which have “intrinsic limitations for scientific discovery” (p. 2), strongly supporting the brief’s tag.
Added in Pass 2: - Chirimuuta, “The Reflex Machine and the Cybernetic Brain: The Critique of Abstraction and its Application to Computationalism,” Perspectives on Science 28(3), 2020 — philosophical critique of computationalism via the reflex-machine/cybernetics tradition: the abstraction move at the heart of “discovery as computation.” (MUSE) - Borg, “Discovery and Instrumentation: How Surplus Knowledge Contributes to Progress in Science,” Perspectives on Science 27(6), 2019 — instruments produce “surplus knowledge” beyond intended use; a mechanism for how genuinely new observations arise rather than being searched for. Relevant to the novelty oracle. (MUSE) - Close read: Borg argues that progress is driven by a “dialectic of discovery and embodiment,” a feedback loop where new knowledge is embodied in instruments, which in turn enable new discoveries. This process “allows science to transcend the limitations of native human epistemic abilities” (p. 861), providing a strong historical and philosophical grounding for the mechanization premise. - Ferraz-Caetano, “The AI Explanatory Trade-Off on the Logic of Discovery in Chemistry,” Philosophies 8(2), 2023 — AI in chemistry forces a trade-off among description, classification, prediction and explanation. Extends the Simon line into computational chemistry. (GALE) - Alvarado, “AI as an Epistemic Technology,” Science & Engineering Ethics 29(5), 2023 — AI/ML/LLMs are “first and foremost epistemic technologies,” distinguished by what they are designed for; a normative frame for discovery tools. (EBSCO) - Abdou, “A Long History: From Universal Language to Artificial Intelligence,” Technology and Culture 67(2), 2026 — mechanized-reasoning projects from universal-language schemes to modern AI. (MUSE) - Heyck, “Intelligence, Natural and Artificial: Thinking with Machines in History,” History of Social Science 1(2), 2025. (MUSE)
Read: discovery was operationalized as heuristic search and demonstrated in running programs — but largely on historical cases, exactly the objection Gorman raised and which recurs today.
2. Serendipity — the part that resists systematization
- Pearce, “Chance and the Prepared Mind,” Science 35(912), 1912 — the canonical Pasteur (1854) statement. (JSTOR)
- Close read: This address on medical education uses Pasteur’s dictum—“In the fields of observation, chance favors only the mind which is prepared” (p. 943)—as its central theme. It argues that scientific imagination and trained observation are what allow a researcher to grasp the significance of an unexpected event, strongly supporting the brief’s relational read of serendipity.
- Cannon, “The Role of Chance in Discovery,” Scientific Monthly 50(3), 1940 — traces Walpole’s coining of “serendipity.” (JSTOR)
- Close read: Cannon explicitly traces the word “serendipity” to Horace Walpole’s 1754 letter about “The Three Princes of Serendip,” who “were always making discoveries, by accident or sagacity, of things they were not in quest of” (p. 204). This directly supports the brief’s claim and provides the canonical origin story.
- Merton & Barber’s Travels and Adventures of Serendipity, reviewed by Epstein, Contemporary Sociology 34(5), 2005 — discovery “by chance or sagacity… not sought for”; “microenvironments of discovery.” (JSTOR)
- Copeland, “On serendipity in science,” Synthese 196(6), 2019 — serendipity as an emergent property at the intersection of chance and wisdom, not luck alone. (JSTOR)
- Close read: Copeland argues that serendipity is an “emergent property of scientific discoveries, describing an oblique relationship between the outcome of a discovery process and the intentions that drove it forward” (p. 3). It is a retrospective category that disrupts epistemic expectations, adding nuance to the brief’s summary by emphasizing its community-level and retrospective nature.
- Barber & Fox, “The Case of the Floppy-Eared Rabbits,” AJS 64(2), 1958 — serendipity “gained and lost”; interpretation, not method, decides. (JSTOR)
- Close read: This comparative case study of two scientists observing the same phenomenon shows how preconceptions and research interests determine the outcome. One scientist’s “pre-established special research interests” led him away from the discovery, while the other’s “continuing concern” led to success (p. 135-136). This strongly supports the brief’s read that interpretation, not method, is decisive.
- Design-for-serendipity cluster: Liestman, RQ 31(4), 1992; Austin, Devin & Sullivan, Organization Science 23(5), 2012; Yi, Jiang & Benbasat, ISR 28(2), 2017; Duff & Johnson, Library Quarterly 72(4), 2002. (JSTOR)
- Close read (Liestman): Liestman proposes six approaches to understanding serendipity in library research, arguing that it can be “better understood and experienced more frequently” (p. 524). The “sagacity” approach, combining perseverance with knowledge of information organization, is presented as the most pragmatic, supporting the idea that serendipity can be cultivated.
- Close read (Austin et al.): This study finds that innovators in conducive conditions “actually seek out accidents intentionally and rely on them in their work” (p. 1510). It proposes design principles for digital systems that “Support, Not Reduce, Some Outcome Variation” and “Offer the Option of Induced Variation (Intentional Accident)” (p. 1518), directly supporting the brief’s “design-for-serendipity” framing.
- Close read (Yi et al.): This experimental study finds that the combination of product tags and “socially endorsed people” enables users to “discover more unexpected interesting alternatives” (p. 425). It provides empirical evidence that specific design features can be integrated to create a serendipitous search experience.
- Close read (Duff & Johnson): This study of historians argues that what appears as serendipity is often “accidentally found on purpose” (p. 495) through the deliberate, iterative strategy of building contextual knowledge. This adds nuance by reframing serendipity as an outcome of expert, systematic (though nonlinear) research.
- Forward-looking framing: Krenn & Champion, “Philosophy of Autonomous Science: Ten Questions for the Coming Age of Artificial Scientists,” Daedalus 155, 2026 — asks how to make novelty computable for artificial scientists. (JSTOR)
- Close read: This programmatic essay argues for a new field to translate epistemic aims like “understanding, curiosity, surprise, interest, creativity, and novelty” into computable objectives (p. 2). It directly asks, “can general notions of scientific novelty be computable?” (p. 9), supporting the brief’s summary and framing novelty as a core philosophical and computational challenge for autonomous science.
Added in Pass 2: - Gest, “Serendipity in Scientific Discovery: A Closer Look,” Perspectives in Biology and Medicine 41(1), 1997 — fills the gap between Cannon (1940) and Copeland (2019). (MUSE) - Close read: Gest argues that discussions of serendipity “tend to overemphasize ‘accident’ as the prime factor” (p. 21), while the crucial element is “sagacity,” which Walpole included in his original definition. Using historical cases, he shows that “sagacity, curiosity, and perseverance usually prove to be the decisive forces” (p. 8), adding important nuance about the active role of the prepared mind. - Fyfe, “Technologies of Serendipity,” Victorian Periodicals Review 48(2), 2015 — serendipity is historically bound up with technologies of reading/search: “chance” is mediated by the tools in use. (MUSE) - Close read: Fyfe argues that serendipity has been “operationalized,’ or built into, the research platforms which reflexively shape our methods” (p. 261). Citing tools like the “Mechanical Curator” and “Serendip-o-matic,” the paper shows that “chance discovery is not a bug; it is a feature” (p. 261) of modern digital scholarship, strongly supporting the brief’s design-for-serendipity theme. - Greenhalgh, “Science and Serendipity: Finding Coca-Cola in China,” Perspectives in Biology and Medicine 62(1), 2019 — case study of unexpected findings; serendipity as relational and interpretive. (MUSE) - Close read: This first-person account of anthropological research highlights “methodological openness, serendipity, narrative sense-making, and personalism” (p. 131) as key to discovery. The author notes, “The process was also full of serendipity, chance moments that occasionally led to fresh discoveries or shifts in direction” (p. 137), providing a rich case study of serendipity in practice. - George, “Serendipity How? Data Insights During the Age of Artificial Intelligence,” PTJ: Physical Therapy & Rehabilitation Journal 105(11), 2025 — defines serendipity as “looking for one thing and then finding another,” and asks how it fares under data-driven/AI workflows. (GALE) - Corrales-Hernández et al., “Development of Antiepileptic Drugs throughout History: From Serendipity to Artificial Intelligence,” Biomedicines 11(6), 2023 — narrative history; serendipity dominated early AED discovery, AI now entering. (GALE) - Close read: This review acknowledges the historical importance of serendipity in discovering drugs like phenobarbital and valproic acid. However, it argues for a “shift from serendipitous discoveries to data-driven research” (Section 6), framing serendipity as a past, methodologically limited approach. This provides an important counterpoint to the design-for-serendipity cluster.
Read: serendipity is relational — an unexpected encounter made meaningful by a prepared mind. Over-targeted search starves the accidental adjacency it needs. The design literature prescribes cultivating collisions (diversity, exploration), not tightening precision.
3. AI for science today — claims vs. documented limits
The current-wave manifesto is the Daedalus 155(1/2) special issue, “AI & Science: What Is the Future of Discovery?” (2026, open access on JSTOR): - Manyika, “Introductory Notes” — AlphaFold showed AI can learn biology’s “grammar”; but academic/national labs are “ill-suited” to compute-intensive AI and leak talent to industry. (JSTOR) - Close read: Manyika’s introduction frames the issue, noting AlphaFold “demonstrated that AI could learn the ‘grammar’ of biology from data” (p. 2) but also highlights challenges, including the “brain drain to the private sector” (p. 3). It also provides historical context, citing the 1988 Daedalus issue on AI and its skeptical tone as a precedent for the current hype cycle. - Aspuru-Guzik, “From Alchemy to AIchemy” — the self-driving laboratory claim. Objectives are human-defined. (JSTOR) - Close read: Aspuru-Guzik traces the history of information processing to the modern “self-driving laboratory.” He frames current systems as augmenting humans into “cyborgs” (p. 10) but speculates about future “android” scientists. This supports the brief’s read that objectives are currently human-defined, while presenting a forward-looking autonomy claim. - Kohli, “Unlocking Scientific Intuition…” — Google’s AI co-scientist (Gemini 2.0, multi-agent, 2025, “guided by human-defined goals”). (JSTOR) - Close read: This essay from a Google DeepMind VP describes AI as a “revolutionary lens for deciphering the nature of reality” (p. 78). It explicitly introduces “Google’s AI co-scientist, built on Gemini 2.0 and launched in 2025… a multi-agent system, guided by human-defined goals” (p. 88), directly confirming the brief’s summary of this programmatic claim. - Ho, “Building an AI Polymath” — AI’s scientific impact “is fragmented… the scientific enterprise remains siloed.” (JSTOR) - Close read: Ho advocates for “polymathic foundation models” to overcome the current state where AI’s impact “on the natural sciences is fragmented” (p. 202). The essay is a programmatic call to build the necessary data infrastructure and models, conceding the very limits the brief highlights. - Gomes, “Knowledge-Centric AI” — “purely data-driven methods [have] intrinsic limitations for scientific discovery.” (JSTOR) - Close read: Gomes argues that “relying solely on purely data-driven methods has intrinsic limitations for scientific discovery” (p. 2). She proposes “knowledge-centric AI” that integrates scientific principles and constraints with deep learning, using her lab’s Deep Reasoning Networks (DRNets) as a concrete example. This strongly supports the brief’s claim.
Empirical/robotic validation: Points et al., PNAS 115(5), 2018 (AI exploration of protocells finds novel collective behavior); Mehr, Caramelli & Cronin, PNAS 120(17), 2023 (Bayesian explorer for reactivity discovery). Real, peer-reviewed — but narrow and domain-specific. (JSTOR)
Critiques / STS / hype precedent: - Brannigan, “AI and the Attributional Model of Scientific Discovery,” SSS 19(4), 1989 — section title literally “AI: Promises versus Accomplishments”; a direct historical rhyme with 2026. (JSTOR) - Close read: This 1989 critique of AI’s “extravagance of Slezak’s claims” (p. 601) about autonomous discovery directly parallels modern debates. Brannigan quotes a contemporary critic of BACON: “You give BACON pairs of error free numbers and it comes up with a mathematical expression… You can call that discovering laws of physics if you want to, but you really have to squint hard to call it that” (p. 609). This provides a powerful historical precedent for skepticism. - Desai et al., “The epistemological foundations of data science,” Synthese 200(6), 2022 — “black box” epistemology. (JSTOR) - Strasser & Edwards, Osiris 32, 2017; Elliott et al., BioScience 66(10), 2016; Hoffman, STHV 42(4), 2017; Anthony, ASQ 66(4), 2021 (black-boxing in knowledge work). (JSTOR) - Close read (Strasser & Edwards): This essay historicizes “Big Data,” arguing that “concerns regarding the collection, storage, and uses of vast amounts of data are not new” (p. 345). It questions the novelty of the Big Data era and emphasizes that what counts as data is a social and historical construct, providing critical context for data-driven science claims. - Live replication dispute: Wu, Yang & Uzzi, PNAS 120(33), 2023. (JSTOR) - Yoon et al., J. Medical Ethics 48(9), 2022 — interpretability “should have primacy” in high-stakes ML. (JSTOR)
Added in Pass 2 — the 2025–26 methods layer: - Sparkes, Aubrey, Byrne et al., “Towards Robot Scientists for autonomous scientific discovery,” Automated Experimentation 2(1), 2010 — reviews the full autonomous loop (AI generates hypotheses from a domain model, designs experiments, runs them robotically) 15 years before the “AI co-scientist” wave. The current contribution is scale/orchestration, not new architecture. (EBSCO) - Close read: This 2010 paper defines a “Robot Scientist” as a system that automates the full scientific cycle: “it generates hypotheses… designs experiments… runs the physical experiments… analyses and interprets the resulting data, and repeats the cycle” (Abstract). It documents the success of the robot Adam in identifying gene functions in yeast, providing a crucial pre-LLM baseline for the “AI co-scientist” concept. - Tyagin & Safro, “Dyport: dynamic importance-based biomedical hypothesis generation benchmarking technique,” BMC Bioinformatics 25(1), 2024 — states “the automated evaluation of HG systems is still an open problem, especially on a larger scale,” and proposes a benchmarking technique. The best single citation for the novelty/validation-oracle gap as an engineering problem. (EBSCO) - Close read: This paper introduces Dyport, a framework to benchmark hypothesis generation systems by quantifying the “discovery importance” of new findings based on their impact on future research. It directly addresses the problem that “automated evaluation of HG systems is still an open problem” (Abstract), providing a concrete engineering solution for the novelty/validation oracle. - Zhang, “A chemically-aware validation framework for benchmarking large language models in materials synthesis planning,” Journal of Cheminformatics 18(1), 2026 — explicitly “moving beyond generic NLP benchmarks that fail to capture chemistry-specific” validity. Domain-specific oracle construction. (GALE) - Close read: This paper presents a “domain-tailored verification framework for evaluating the scientific quality of AI-generated synthesis protocols, moving beyond generic NLP benchmarks” (Abstract). By decoupling conceptual plausibility from parametric accuracy, it provides a concrete, engineerable “proto-oracle” for materials science, supporting the brief’s thesis. - Zhu & Weinan, “From AI for science to autonomous chemical innovation: closing the loop in energy and chemical engineering,” Clean Energy 10(4), 2026 — the rate-limiting step is closing the experimental loop, not prediction/generation. (GALE) - Close read: This perspective piece argues that “the rate-limiting step in energy and chemical innovation is no longer prediction alone: it is the conversion of computational proposals into reproducible experiments” (Abstract). It calls for “validation-aware inverse design” and closed-loop autonomous labs, strongly supporting the brief’s read. - Ahmadpour, “Characterizing LLM scientific concept generation: A multi-dimensional measurement study,” PLoS ONE 21(9), 2026 — multi-dimensional framework over 9,666 LLM-generated scientific concepts; the closest thing found to a computable novelty measurement. (GALE) - Close read: This paper develops a rigorous, multi-dimensional framework to characterize, rather than evaluate, LLM-generated scientific concepts. It explicitly states it does not measure novelty, but provides a foundational methodology for doing so: “This motivates a shift away from adjudicating creativity and toward characterizing the measurable properties of model outputs” (Section 1.1). - Taskin, Xie & Lazebnik, “Knowledge integration for physics-informed symbolic regression using pre-trained large language models,” Scientific Reports 16(1), 2026 — LLM priors for symbolic regression; a concrete equation-discovery instance (AI-Feynman lineage). (EBSCO) - Moore & Tatonetti, “From prompt engineering to agent engineering…,” BioData Mining 18(1), 2025 — static prompting is limited; agents are the next step. Commentary, not an outcome study. (EBSCO) - Sisson, “The Lab of the Future Runs Itself,” Scientific American 335(1), 2026 — popular account of Berkeley’s A-Lab. Journalism, not validation. (EBSCO) - Weber, “Geometry-Informed AI for Scientific Discovery,” Daedalus 155(1/2), 2026; Tenenbaum, “Language Is Not All You Need…,” Daedalus 155(1/2), 2026 — insider limit-claims from within the special issue. (JSTOR) - Ghaderi et al., “Why biology must prioritize data reanalysis in the era of artificial intelligence,” PLoS Biology 24(7), 2026 — “scientific revolutions do not only emerge from new experiments, but often from new ways of interpreting existing data.” (GALE) - Hallsworth, Udaondo, Pedrós-Alió et al., “Scientific novelty beyond the experiment,” Microbial Biotechnology 16(6), 2023 — theory-based work produces genuine, sometimes paradigm-changing novelty; challenges lab-only automation. (EBSCO) - Close read: This review argues that “theory-based research studies also contribute novel—sometimes paradigm-changing—findings” (Abstract). It emphasizes the “intrinsic need for human engagement in scientific innovation” and cautions that AI algorithms “cannot be compared with the flexibility, dexterity, and adaptive capacity of the human mind” (Box 6), providing a strong counter-narrative to full automation. - Chin-Yee & Upshur, “Three Problems with Big Data and Artificial Intelligence in Medicine,” Perspectives in Biology and Medicine 62(2), 2019 — early, still-useful critique of data-driven medical AI epistemology. (MUSE)
Read: confirmed capability is narrow and benchmarked; flagship “AI co-scientist”/self-driving-lab claims are programmatic essays, not replicated outcome studies. Even boosters concede fragmentation, data-centrism limits, and human goal-setting — and the field is now openly building the domain-specific oracles it lacked.
4. Data-intensive science and the “fourth paradigm”
- Pietsch, “Aspects of Theory-Ladenness in Data-Intensive Science,” Philosophy of Science 82(5), 2015 — data are not theory-free; observation is a “searchlight,” not a “bucket.” (JSTOR)
- Hey & Trefethen, “Cyberinfrastructure for e-Science,” Science 308(5723), 2005; Foster, “Service-Oriented Science,” Science 308, 2005. (JSTOR)
- Close read (Hey & Trefethen): This paper outlines the vision for “e-Science,” enabled by a shared cyberinfrastructure. It envisions a future where “a community’s shared understanding is no longer documented exclusively in the scientific literature but is documented also in the various databases and programs that represent… a collective knowledge base” (p. 817), laying the technical groundwork for the fourth paradigm.
- Edwards et al., “Science friction,” SSS 41(5), 2011 — metadata/social friction limits data reuse. (JSTOR)
- Close read: This ethnographic study argues that “science friction” arises from a disconnect between metadata as a formal “product” and its actual use as an informal “process” of communication. Data sharing relies on “ad hoc, incomplete, poorly structured, mutable descriptions” (p. 673) that require constant human repair, highlighting a key social bottleneck for data reuse and automation.
- Fayyad & Smyth, JCGS 8(3), 1999 — the data flood “threatens to render traditional approaches to data analysis inadequate.” (JSTOR)
- Lenhard, Philosophy of Science 74(2), 2007 & 73(5), 2006 — simulation as a distinct epistemic mode with “epistemically opaque models.” (JSTOR)
- Krafczyk et al., Phil. Trans. A 379(2197), 2021; Kitchin & Lauriault, GeoJournal 80(4), 2015; Halford & Savage, Sociology 51(6), 2017. (JSTOR)
- Open-access: Brevini et al., Critiques of Data Colonialism (Bristol UP, 2024); Prietl & Raible, The Politics of Data Science (Amsterdam UP, 2024). (JSTOR)
Added in Pass 2: - Moore, “Machinery Hurtful to [Scientific] Commonality: Automation and Scientific Publication,” Historical Studies in the Natural Sciences 54(5), 2024 — historical analysis of automation in publishing and the scientific commons. (MUSE) - Birch, “Automated Neoliberalism? The Digital Organisation of Markets in Technoscientific Capitalism,” new formations 100–101, 2020 — places automation of technoscience inside political economy. (MUSE) - Ferrara, “The Generative AI Paradox: GenAI and the Erosion of Trust…,” Future Internet 18(1), 2026 — GenAI shifts from “deepfakes” to scalable “synthetic realities,” corroding verification. The epistemic-trust failure mode for automated pipelines. (EBSCO)
Read: the “fourth paradigm” is a cluster of practices, not a settled doctrine. Data-driven methods are not self-sufficient — theory, metadata, and verification are load-bearing.
5. Science of science — why automating research is attractive, and its incentive trap
- Fortunato et al., “Science of science,” Science 359(6379), 2018. (JSTOR)
- Close read: This review defines the “science of science” (SciSci) as a field using quantitative methods to understand science itself. It notes that “scholars are risk-averse, preferring to study topics related to their current expertise, which constrains the potential of future discoveries” (p. 1007), providing direct evidence of incentive pathologies that automation might address or exacerbate.
- Jones, “The Burden of Knowledge and the ‘Death of the Renaissance Man’,” Review of Economic Studies 76(1), 2009. (JSTOR)
- Close read: Jones provides empirical evidence that as knowledge grows, innovators must spend more time in education and become more specialized, leading to a “burden of knowledge.” This increasing burden “can explain why rapid growth in the number of R&D workers… is not associated with increased TFP growth rates” (p. 309), establishing a core economic motivation for finding more efficient discovery methods.
- Bloom, Jones, Van Reenen & Webb, “Are Ideas Getting Harder to Find?”, AER 110(4), 2020 — quantitatively falling research productivity. The core economic motivation for automating discovery. (JSTOR)
- Close read: This paper provides extensive empirical evidence that “research productivity is declining sharply” (p. 1104) across diverse sectors from semiconductors to medicine. For Moore’s Law, they find “it is around 18 times harder today to generate the exponential growth… than it was in 1971” (p. 1117). This strongly supports the brief’s claim that falling research productivity is the core economic driver for automation.
- Incentive pathologies: Nosek, Spies & Motyl, Perspectives on Psych. Science 7(6), 2012; Kapeller, AJES 69(5), 2010; Wasserstein et al., The American Statistician 73, 2019; Agafonow & Perez, Prometheus 39(2), 2023; Bozeman & Youtie, “Robotic Bureaucracy,” PAR 80(1), 2020. (JSTOR)
- Close read (Nosek et al.): This paper argues that academic incentives for “novel, positive results” are “at odds with the incentives for accurate results” (p. 616), creating a conflict of interest that inflates the rate of false findings. It directly identifies the incentive pathology at the heart of the brief’s synthesis.
- Close read (Bozeman & Youtie): This essay argues that “robotic bureaucracy” in university research administration often shifts administrative burden from administrators to researchers, creating a “true lose-lose outcome” (p. 158). It provides a concrete example of how poorly designed automation, ignoring human factors, can exacerbate incentive problems and stifle research.
- Automating scientific labor: Agrawal, Gans & Goldfarb, J. Economic Perspectives 33(2), 2019; Almeida, Naudé & Sequeira, IZA, 2024; Arenas Díaz, Piva & Vivarelli, IZA, 2025. (JSTOR)
Added in Pass 2: - Kindsiko, Rõigas & Niinemets, “Getting funded in a highly fluctuating environment: Shifting from excellence to luck and timing,” PLoS ONE 17(11), 2022 — luck is a major factor in grant allocation, with early-career researchers most exposed. A tension with meritocratic ratchet assumptions. (GALE) - Close read: This study of the Estonian grant system finds that large year-to-year funding fluctuations mean “success becomes partly uncoupled from excellence” and is “largely driven by being at a right place at a right time (RPRT)” (Introduction). This provides strong empirical evidence for the role of luck in funding, especially for early-career researchers, sharpening the brief’s point on incentives. - Shin, Kim & Kogler, “Scientific collaboration, research funding, and novelty in scientific knowledge,” PLoS ONE 17(7), 2022 — empirical SciSci linking collaboration/funding to novelty. (EBSCO) - Jing, Zhou, Cimino et al., “Development, validation, and usage of metrics to evaluate the quality of clinical research hypotheses,” BMC Medical Research Methodology 25(1), 2025 — validated metrics for hypothesis quality before investment; a real-world analog of the objective-function problem. (EBSCO) - Close read: This paper develops and validates a “brief and comprehensive versions of metrics” to “systematically, objectively, and consistently” assess the quality of clinical research hypotheses based on validity, significance, and feasibility (Abstract). It is a direct, real-world example of building an “oracle” or objective function for a scientific domain. - Rao, Kumar, Lakkaraju et al., “Detecting LLM-generated peer reviews,” PLoS ONE 20(9), 2025 — evaluators can’t reliably distinguish generated reviews; evaluation infrastructure lags the automation it polices. (EBSCO) - Close read: This paper addresses the inability to detect LLM-generated peer reviews by proposing a framework to “embed a covert watermark in the generated review” via indirect prompt injection in the manuscript PDF (Abstract). This is a concrete engineering response to the problem of evaluation infrastructure lagging automation. - Teixeira da Silva & Tsigaris, “Would AI, Like ChatGPT, Be a Good ‘Peer’ Reviewer…,” Journal of Scholarly Publishing 56(1), 2025; Lo, “Generative AI and Open Access Publishing: A New Economic Paradigm,” Library Trends 73(3), 2025; Hosier & Cantwell-Jurkovic, “AI and Library and Information Science Publishing: A Survey of Journal Editors,” Library Trends 73(3), 2025. (MUSE) - Close read (Teixeira da Silva & Tsigaris): This SWOT analysis concludes that while AI shows promise for efficiency, it has “not reached… a dependable stage yet to review articles” (p. 79), citing issues like fabricated references. It suggests a hybrid human-AI system is the likely future, qualifying claims of full automation in peer review. - Close read (Hosier & Cantwell-Jurkovic): This survey of LIS journal editors reveals deep concerns that AI may “limit ‘ideas and creativity, perpetuat[ing the] status quo’” and “problematically mask or block the creative and serendipitous aspects of doing research” (Discussion). This directly connects automation in publishing to the risk of stifling novelty and serendipity. - Berger, “Machines, Psychology, and Hypothesis Generation: Commentary on Banker et al. (2024),” American Psychologist 79(6), 2024; Petersen et al., “Causal Discovery for Observational Sciences Using Supervised Machine Learning,” Journal of Data Science 21(2), 2023; Helmy et al., “Ten simple rules for optimal and careful use of generative AI in science,” PLoS Computational Biology 21(10), 2025. (EBSCO) - Close read (Berger): This commentary argues that the key question is not whether AI can generate hypotheses, but “Why should we believe that the current (human) method of hypothesis generation is somehow ideal in the first place?” (p. 798). It suggests machines can help researchers in “a more systematic, parallel, and optimizing process for variable generation and prioritization,” shifting the focus from generation to evaluation.
Read: the “burden of knowledge” plus metric-driven incentives create both the demand for automated discovery and the risk that it optimizes the wrong thing. Pass 2 adds that funding itself is substantially luck-driven — the metric may capture luck, not quality.
Claims that did not survive close reading
- Merton & Barber’s Travels and Adventures of Serendipity (2005)
- What the brief said: The work defines discovery as “‘by chance or sagacity… not sought for’” and discusses “‘microenvironments of discovery.’”
- What the text actually says: The fetched text was a book review of The Travels and Adventures of Serendipity, not the book itself. The review, by Steven Epstein, summarizes the book’s project as tracing “the travels of serendipity in various literary traditions and publications” (p. 1302) but does not contain the phrases quoted in the brief.
- Correction: The brief’s claims are attributed to the book but were not verifiable from the fetched text, which was a secondary source. This was a retrieval issue leading to a misattribution.
Retrieval failures (not brief errors)
The v2 brief contained three entries based on incorrectly fetched source texts. The claims made in the brief for these works were based on these incorrect texts and are therefore invalid. These are retrieval failures, not errors in the brief’s synthesis of the intended works.
- Anthony (2021): The brief cited “Anthony, ASQ 66(4), 2021” on “black-boxing in knowledge work.” The fetched text was “Innovation and the Labor Market: Theory, Evidence and Challenges” by Corrocher, Moschella, Staccioli, and Vivarelli (IZA DP No. 16199, 2023), a paper on labor economics and automation patents.
- George (2025): The brief cited “George, PTJ: Physical Therapy & Rehabilitation Journal 105(11), 2025” on serendipity in AI workflows. The fetched text was “From Polyphenols to Prodrugs: Bridging the Blood–Brain Barrier with Nanomedicine and Neurotherapeutics” by Tanaka et al. (International Journal of Molecular Sciences 2026), a review of drug delivery methods.
- Hoffman (2017): The brief cited “Hoffman, STHV 42(4), 2017” on “black-boxing in knowledge work.” The fetched text was “Data-Informed Duties in AI Development” by Frank Pasquale (Texas Law Review, 2019), a legal scholarship article on AI liability and regulation.
Synthesis — what this means for a Karpathy-style autoresearch loop
- The mechanization premise is respectable, not fringe. Simon (1973, 1981), Bradshaw (1983), and Zytkow & Simon (1988) established discovery-as-heuristic-search. This lineage is affirmed by historical overviews (Waltz & Buchanan 2009), philosophical arguments for rational, deductive discovery (Zahar 1983; Shah 2008), and the framing of instrumentation as embodied knowledge (Borg 2019). The
propose → run → verify → ratchetloop sits squarely in this tradition. - The known failure mode is historical-case fitting. BACON “discovered” Boyle’s law from data it was built around (Bradshaw 1983; Simon 1981). Brannigan’s (1989) contemporary critique of this method as something less than genuine discovery is precisely what an autoresearch loop must avoid. The metric must be ungameable and the space genuinely open.
- The oracle is now engineerable, not just philosophically desirable. The v2 brief’s “oracle” is now a concrete engineering subfield. This is seen in domain-specific validation frameworks for chemistry (Zhang 2026; Zhu & Weinan 2026), multi-dimensional concept characterization (Ahmadpour 2026), knowledge-centric reasoning systems (Gomes 2026), and even infrastructure for policing AI-generated text (Rao 2025). A Karpathy loop should borrow from these rather than invent its own.
- The novelty problem has a concrete methods literature. The abstract challenge to “make novelty computable” (Krenn & Champion 2026) is being met with concrete evaluation methods. Dyport (Tyagin & Safro 2024) offers a benchmark for “discovery importance” in biomedicine, while Jing (2025) provides validated metrics for hypothesis quality in clinical research. The problem is shifting from philosophy to evaluation engineering, with a focus on prioritization and impact (Berger 2024).
- The loop’s architecture is old; its scale is new. Sparkes et al. (2010) specified and implemented the generate→design→run→test loop in their “Robot Scientist” well before the current AI wave. The core architecture is a settled concept in automated science (Waltz & Buchanan 2009; Hey & Trefethen 2005). Expect incremental, not categorical, gains from bigger models; judge the ratchet against pre-LLM baselines.
- Serendipity must be designed in, not optimized away. A strong consensus emerges that precision-targeted search starves the accidental adjacency that produces breakthroughs (Pearce 1912; Gest 1997; Barber & Fox 1958). The design literature prescribes cultivating collisions and “valuable unpredictability” (Liestman 1992; Austin et al. 2012; Yi et al. 2017). This is a function of the tools and environment (Fyfe 2015; Duff & Johnson 2002) and requires embracing “generative doubt” at a community level (Copeland 2019). An autoresearch loop must retain deliberate breadth, a point echoed by concerns that AI tools may “mask or block the creative and serendipitous aspects of doing research” (Hosier & Cantwell-Jurkovic 2025).
- Incentives are the binding constraint — and worse than v2 said. The economic demand for automation is clear: ideas are getting harder to find (Bloom et al. 2020) due to the “burden of knowledge” (Jones 2009). But the system that automation enters is deeply pathological. Incentives reward publication over truth (Nosek et al. 2012), and funding itself is substantially luck-driven, especially for early-career researchers (Kindsiko et al. 2022). Poorly designed automation exacerbates these issues, creating “robotic bureaucracy” (Bozeman & Youtie 2020) and friction that limits data reuse (Edwards et al. 2011). An automated loop will be rewarded for volume/novelty-signaling rather than truth. Slack for serendipity and a human-held objective are load-bearing, not optional.
- Precedent for skepticism. Brannigan’s 1989 “Promises versus Accomplishments” reads as a direct warning label for the 2026 Daedalus issue. Manyika’s (2026) introduction to that very issue concedes the historical precedent of hype cycles. The “knowledge acquisition bottleneck” identified in 1980s expert systems (Forsythe 1993) and the emphasis on the irreplaceable role of human theory and thought (Hallsworth 2023) provide strong grounds for skepticism about claims of full autonomy.
Net: the defensible version of an autoresearch loop is a narrow, well-instrumented ratchet on an objective, ungameable metric — with an explicit, separate novelty/validation oracle and deliberate slack for serendipity. The close readings did not overturn that; they turned it from a philosophical hope into an engineering checklist.
Method note & caveats
- Access: SBCC EZproxy via Comet cookies, no Shibboleth re-login. Four databases searched: JSTOR, Project MUSE, Gale Academic OneFile, EBSCO Academic Search Complete (
a9h). Harness:skills/sbcc-library/(search.py,auth.py). - Depth: Close readings were performed on the full text of each source, not just listing cards or snippets. Close readings were conducted via Gemini on the fetched PDF and HTML files.
- Bibliographic imperfection: Years/volumes/pages come from vendor cards and are occasionally missing or implausible (several 2025/2026-dated items are pre-prints or early online versions). Verify before formal citation.
- Retrieval Failures: As noted, four sources cited in v2 were based on incorrectly fetched texts. These have been flagged and their claims removed from the synthesis. This highlights the risk of automated fetching and the necessity of manual verification.
- Coverage limits: Only EBSCO’s default
a9hwas searched (29 other EBSCO databases behind the profile were not); ProQuest/Nexis remain unreachable by keyword search; Gale skews to open-access MDPI/PLoS content. - Vendor sessions: EBSCO/Gale run their own session on top of EZproxy and can silently expire — run
auth.py <name> --verifyif results go empty. - Positive selection: searches followed the five themes; material outside them was not pursued, so this is not a neutral sample.
- Raw captures: Pass 1 = 57 files in
/tmp/opencode/research/; Pass 2 = 41 files in/tmp/opencode/research2/.