On the morning of September 21, 2026, the 189th session of the "Peking University Medical Humanities Forum," hosted by the School of Medical Humanities at Peking University, was held in Room 716, Yifu Building, Peking University Health Science Center. The lecture invited Professor Jane Kaye, Professor at the Faculty of Law, University of Oxford, and Professor at Melbourne Law School, University of Melbourne, as the speaker. Under the title "Beyond the Algorithm: Trust, Governance and the Future of Healthcare," she lectured on the applications of artificial intelligence in healthcare, the meaning of healthcare governance, UK regulatory practice, cases of governance failure, and the UK government's strategy. The lecture was chaired by Professor Cong Yali of the School of Medical Humanities, Peking University. Professor Zeng Yi of Gaoling School of Artificial Intelligence, Renmin University of China, served as the discussant. Experts, scholars, teachers, and students from Peking University and other universities attended the lecture.

Prof. Jane Kaye DPhil, LLB, Grad Dip Leg, BA is the Director of the Centre for Law, Health and Emerging Technologies (HeLEX) in the Faculty of Law at the University of Oxford and Professor of Health Law and Policy at the University of Melbourne. She was on the UN Taskforce on Health Data Privacy, appointed to the Scientific Advisory Board of the European biobanking platform BBMRI-ERIC, and was on the WP8 Advisory Group of the Joint European Action TEHDAS ‘Towards a European Health Data Space’.

At the beginning of the lecture, Professor Kaye pointed out that AI is profoundly affecting healthcare. Its main applications include imaging, pathology and diagnostic support, clinical decision support, triage and risk prediction, as well as precision medicine, drug discovery, and public health surveillance. At the same time, AI also brings erroneous decisions, algorithmic bias and hallucinations, attribution of responsibility, data privacy and security, public trust, changes in the roles and skills of healthcare professionals, and validation and regulatory challenges caused by continuous algorithm updates. She cited a survey conducted in May 2026 by King's Health Partners Academic Health Sciences Centre, Responsible AI UK, and the Policy Institute at King's College London: 15% of the public had used AI chatbots for health advice; 37% supported and 38% opposed the use of AI in clinical decision-making; 39% mainly worried about safety and accuracy; the public trusted doctors more. Most respondents demanded advance notice and the right to opt out; if AI disagreed with a doctor's diagnosis, 55% believed a second doctor should review it; 76% believed AI tools used in patient care should be officially approved and regulated. An October 2025 survey by the Nuffield Trust and the Royal College of General Practitioners showed that of 2,108 GPs, 28% had used AI in clinical practice, mainly for documentation, professional development, and administrative tasks, with lower use in clinical decision-making; doctors mainly worried about professional liability and medico-legal risk, and recommended that AI save time rather than replace clinical judgment, clarify responsibility, develop interim guidelines, and strengthen training and patient education.

Professor Kaye emphasized that AI in healthcare is not merely a technological innovation; it also reshapes clinical decision-making, institutional responsibility, and doctor-patient trust. The key question is not "Can we use AI?" but "Under what conditions should we trust AI?" Citing the World Health Organization's definition, she noted that healthcare governance includes the processes, structures, and institutions that oversee and manage a country's healthcare system, managing relationships among different actors and stakeholders, and requires strategic policy frameworks, effective oversight, coalition-building, appropriate regulation and incentives, system design, and accountability mechanisms. She systematically introduced the UK healthcare AI governance system: regulatory bodies include the Medicines and Healthcare products Regulatory Agency (MHRA), which regulates AI as a medical device (AIaMD); the Information Commissioner's Office (ICO), responsible for data protection; and the Health Research Authority (HRA), responsible for research ethics approval. Clinical deployment involves the Care Quality Commission (CQC), the National Institute for Health and Care Excellence (NICE), and guidance from professional medical bodies. Legal and policy frameworks include the EU Artificial Intelligence Act, the Medical Device Regulation, UK GDPR, the Data Protection Act, and the Clinical Trials Regulation. After Brexit, the UK still interacts closely with the EU framework. Governance also involves "carrots and sticks" compliance mechanisms: at the bottom are incentives, access to samples and information, and opportunities for collaboration; moving upward are reputational harm, reporting to an IRB, withdrawal of funding, and disciplinary action.

In the section on governance failures, Professor Kaye focused on two cases. The first was the 2025 use by doctors of unapproved "ambient voice" AI tools, which exposed a "shadow IT" governance failure: clinicians adopted consumer-grade technology faster than hospital frameworks could adapt, some vendors did not comply with data protection laws, and patient recordings were processed on external unencrypted servers. The NHS subsequently implemented strict procurement controls, requiring vendors to be on an official pre-vetted list and requiring explicit verbal consent from patients. The second was the controversy over the £330 million Federated Data Platform (FDP) contract signed between the NHS and the US data company Palantir. Because of Palantir's historical ties to military and intelligence agencies, the project triggered strong distrust among parliamentary committees, human rights groups, and doctors. The NHS later amended the contract, stipulating that Palantir acts only as a data processor, cannot merge or share NHS data with other government departments or external databases, and included parliamentary break clauses; the government is considering terminating the contract in 2027. Professor Kaye also introducedd the UK government's strategy and MHRA reform: the UK is advancing a £10 billion NHS digital overhaul, expanding AI triage and a single patient record; the MHRA is working with the newly established National Commission into the Regulation of AI in Healthcare to develop a new framework; £1.6 billion is being invested in cloud computing and data assets to accelerate drug discovery; and the HRA launched a two-year AI plan. Priorities for 2026 include establishing a new AI regulatory rulebook, shifting from one-off "pass/fail" compliance to continuous, phased monitoring; ambient voice transcription tools are being deployed in GP practices, but clinicians bear full responsibility for diagnoses; AI is used for theatre scheduling and automated cleaning, among other scenarios. The MHRA's AI Airlock regulatory sandbox allows developers to test AI medical devices in a controlled environment, identifying issues such as algorithmic bias, performance monitoring, and post-market safety tracking. The consultation found that the existing regulatory framework needs "significant reform" rather than a "complete overhaul"; AI systems require continuous post-market surveillance, and responsibility should be distributed across the entire AI lifecycle. Human oversight and clinical judgment are essential, and explainability, data access, AI literacy, incident reporting, and patient and public engagement are all key.

In the discussion session, Professor Zeng Yi pointed out that there is a fundamental difference between clinical decision support and autonomous AI decision-making. Current AI agents and foundation models often show a tendency to seek unauthorized decision-making, and technology itself does not inherently guarantee safety; legal, ethical, and governance research must clearly identify technological loopholes. He emphasized that public trust requires not only trustworthy institutions but also trustworthy and explainable technology; stakeholders should go beyond regulators to include broader actors such as disaster management, public security, and community gatekeepers. Combining this with China's "nine dragons governing water" governance structure, he introduced the multi-departmental pattern of healthcare AI governance involving the National Health Commission, the National Medical Products Administration, the Ministry of Industry and Information Technology, the National Committee on Ethics of Science and Technology, and the Cyberspace Administration of China, noting that overlapping responsibilities among multiple departments create coordination costs but may also be complementary. He also discussed Palantir data risks, informed consent and data accuracy issues in COVID trackers, dynamic consent mechanisms, and the feasibility of transnational governance of the European Health Data Space.

In the Q&A session, teachers and students exchanged views on ethical choices in AI decision-making under limited medical resources, the UK regulatory system, and the impact of carrot-and-stick mechanisms on the global mobility of scientists and AI companies. Professor Kaye responded based on UK, EU, and Australian experience.

With "Beyond the Algorithm" as its core theme, this lecture systematically demonstrated the complex relationships among trust, law, ethics, and institutional design in AI healthcare governance. Professor Kaye's talk not only presented the latest developments and controversies in UK AI healthcare governance, but also provided a comparative perspective for understanding institutional choices in China and other countries, further deepening participants' understanding of how technology can enter medical practice safely, credibly, and responsibly.

(Shuochao Jing)