AI Automation ·
Can an AI audit of shared clinical research code establish reproducibility for Korean institutions?
MIT News reported on November 19, 2026 that an LLM-assisted review examined code sharing in clinical prediction research.[1] The underlying paper appeared on October 9, so the news date is not the study's publication date.[2] Finding public repositories might seem to verify research. Reading the methods instead separated articles classified as sharing code, repositories characterized, and actual reproduction of results.
Is a public link sufficient handover evidence?
Consider a Korean university or hospital research-support office checking computational materials. This is an implementation proposal, not a reported deployment. Authors cannot casually disclose patient records; reviewers cannot understand every procedure from a public link. Additional documentation consumes time, but premature completion can defer expensive work until reanalysis.
Our question is whether LLM-assisted discovery followed by human review of defined execution evidence reduces omissions at an affordable review burden compared with checking links alone. This is not automated certification of clinical safety or effectiveness. Reproducing a computation and validating performance at another hospital are separate tasks.
Following the sample and denominators changed the question
The paper sampled prediction studies citing TRIPOD or TRIPOD+AI and retrievable through the PubMed Central open-access route, using citation lists as of August 11, 2025. Among 3,967 eligible articles, 482, or 12.2%, were classified as sharing code; repository characteristics were assessed for 380 repositories.[2] The classification also included supplementary-material sharing and statements linking unsupported providers. It does not mean 482 repositories were downloaded and rerun.
These figures cannot be transferred to all medical research or Korean institutions. Citation and access conditions select the sample. Automated classification remains imperfect, and agreement with human-reviewed samples cannot guarantee every individual judgment. MIT's account and the paper concern the same study, not independent replications.[1][2]
The paper examines documentation, dependencies, licensing and other characteristics, and discusses why availability alone does not establish computational reproducibility.[2] It does not prove an unavailable model is wrong or a licensed repository executes correctly. Automation can locate evidence, while executability and scientific validity still need separate assessment.
| Decision-relevant item | Verified overseas research | Verified domestic route |
|---|---|---|
| Operator and date | Sounack and colleagues, October 9, 2026; MIT account, November 19[1][2] | DataON submission guide, checked November 20[3] |
| Scope | Article sharing statements and characteristics of retrieved repositories[2] | Metadata, disclosure/licensing and files, with approval-request workflow[3] |
| Verification boundary | Sharing classification and repository assessment do not certify every model's rerun[2] | Registration instructions do not certify execution or clinical safety |
The domestic guide also links related publications and projects.[3] This changes implementation: examine attaching versioned execution evidence to existing registration and access procedures before creating another repository. We do not suggest publishing patient records on DataON or imposing one procedure on all institutions. Individual contracts, consent and ethics conditions were not reviewed.
Option A: discover automatically, review execution evidence
Limit the LLM to extracting code statements, specific links and supporting locations. Reviewers determine whether a link names a profile or a version, check execution instructions, dependencies and permissions, and examine a core computation with authorized test material in a security-reviewed isolated environment. Discovering external code is not authorization to execute it on institutional systems.
Where original records cannot be shared, examples or synthetic data may test execution structure while original-data performance checks remain in an authorized internal environment. Successful execution on examples does not reproduce the original patient-data results. Record both what was established and what was not.
A may save search time and specify omissions, but adds model usage, human error checks, secure execution and author remediation costs. Automated judgments may miss documentation context or treat malicious repository text as instructions. Proposed responsibilities are requirements for research support, environment safety for security staff, and data permissions and procedures for the principal investigator.
Option B: structure handover at submission
Authors supply linked publications, a code version, execution procedure, access conditions and expected outputs. Reviewers check omissions manually. Existing domestic metadata, permission and file categories can help, but execution-evidence fields are an institutional proposal beyond the guide.[3] B gives authors clear responsibility and may reduce extraction errors.
One form can nevertheless burden studies with diverse environments. For thousands of historical papers, waiting for authors may be impractical and A can assist prioritization. B may fit new-project handover, while A helps discover historical material needing review. Sharing prevalence alone cannot establish their relative cost effectiveness.
Compare omissions and execution, not an automatic score
Limit a pilot to authorized institutional material. Research-support staff and researchers agree one core computation and its expected outputs. An independent reviewer rechecks A and B, distinguishing missed existing links or versions from cases marked complete despite missing documentation. Preserve supporting locations rather than only model conclusions.
Proposed measures are human search time, remediation requests, false completion decisions, execution success in the permitted environment and differences from expected outputs. Record execution success separately from result agreement. Missing dependencies or inaccessible data produce an unverifiable status, not a declaration that the underlying model failed. We did not execute repositories or conduct an institutional experiment for this article.
Stop the pilot if it leads to unauthorized disclosure or unapproved code execution. Reduce extraction scope and return to B when automatic completion misses more material than the manual baseline or cannot be traced to evidence. Before using a larger model, examine whether requirements and denominators caused the disappointing result.
Be accountable for the claim actually verified
Code-sharing checks alone cannot establish reproducibility. There is a basis for testing them as assistance in finding availability and reuse gaps. Completion states should distinguish link possession, execution structure, reproduction of original results and external validation, matching only what was checked.
Evidence that could change the judgment includes human comparisons on Korean materials, remediation costs and authorized rerun records. Where should an institution invest first: broad discovery of historical work or stronger handover for new research? Distinguishing reasons for non-disclosure from unverified scientific claims is a practical starting point.