Mass scanning infrastructure via AI watermarking laws and policies
Mass scanning infrastructure as a consequence of AI watermarking laws and policies results from deadlines for global AI firms to embed tamper-resistant AI signals in their content and establish the means to detect them. Detecting whether a piece of content contains these signals requires the content to be sent to scanning servers outside the user's control, even if it is fully human-generated. AI watermarking and transparency laws do nothing to prevent personal communications being scanned for this purpose.
Almost all leading western AI firms have signed legally-backed commitments that explicitly permit remote scanning servers, drive them to make detection go through centralized detection hubs, and encourage them to make detection applicable to communication platforms.
Recent developments provide grounds and incentives for tech firms to apply this scanning infrastructure to personal communications with or without prior government mandates.
Origins
[edit | edit source]Laws
[edit | edit source]As generative AI becomes more realistic and widely accessible, there has been increasing consideration of mechanisms to distinguish between human and artificial content.[1]
The European Union passed the Artificial Intelligence Act in 2024.[2] It has extraterritorial reach.[3] By 2 December 2026,[4][5] existing providers of AI models must comply with Article 50(2) by watermarking text, images, audio, and video they generate or substantially modify,[6][7] similarly to voluntary commitments taken by AI firms in 2023 under the Biden-Harris administration.[8][9] The EU law's official objective is to "enable humans to distinguish AI-generated or manipulated content",[10] citing "new risks of misinformation and manipulation at scale, fraud, impersonation and consumer deception".[11] It is one of several transparency requirements in the act.[2]
California and Washington have passed similar requirements - the Californian deadline coincides with the original EU deadline,[12][13] and the Washington deadline is within a day of a later-released industry-backed deadline for interoperability standards.[14][15] Motivations in the Californian law echo the EU's, stating action is "paramount to combating misinformation and disinformation".[16] It requires detection to be available at no financial cost,[12] and the EU-endorsed Code of Practice requires detection to be free of charge in almost all cases.[17]
On 5 May 2026 the EU published a late-2025 Technical Solutions report assessing the usability of digital watermarking. It explains detection would "need software to expose the info." "One approach is building detection into popular image viewers or social media. For example, a messaging app could scan images as they are uploaded" and that this "could have privacy implications if scanning personal media".[18] Despite the professor raising his findings, the EU took no action to prevent such use.
Six days after the report's publication, the EU released official guidelines for the law, saying actors such as "online platforms" disseminating, transmitting, or receiving AI-generated or manipulated content created by third parties are "strongly encouraged" to implement measures to "enable natural persons to detect the AI-generated or manipulated content".[19]
Code of Practice
[edit | edit source]A month later on 10 June 2026, the EU published a Code of Practice about the law's obligations.[15][20] It explicitly permits cloud-based detection solutions. It admits how "uncontrolled access to the detection solution may compromise the security of the underlying marking techniques and detection mechanisms", yet still presents local execution as a feasible option.[21]
Regarding applicability to communication platforms, the code reads: "Signatories are encouraged to collaborate with relevant actors within the ecosystem to make their detection solution directly available within distribution and communication platforms."[17]
It establishes an interoperability requirement deadlined 2 February 2027, satisfiable by options including a request routing mechanism and an "aggregated detector", while committing signatories to implement these when they emerge.[22][23] Notably, current technology dictates a watermark "aggregated detection solution" is a centralized processing server that is sent user content for scanning.[24][25]
Signatories
[edit | edit source]On 22 June 2026 the EU hosted an online information session in which they stated there are "clear benefits for signatories" including "reduced administrative burden", "legal certainty", and "if assessed as adequate can be the only way to show to any competent supervisor how you comply with the obligations" 29:50-30:25.[26] The EU AI Board later assessed the code as adequate.[27] The information session also explained how those who don't sign the code may be subject to more requests for information to demonstrate compliance with the EU's watermarking obligations 27:34-27:48,[26] and how the developers of the code see the code as setting a global example 30:25-30:55.[26]
Among others, Anthropic, OpenAI, Meta, Google, Microsoft and Mistral signed Section 1 covering AI providers' obligations.[28] Of the top five western (the rest being Chinese) frontier AI companies ranked by both model performance and US usage,[29][30] only xAI (SpaceXSI) did not sign.[28]
Development
[edit | edit source]Standards
[edit | edit source]On 7 October 2026, Google, the company behind Gemini, published a press release titled "We're making it easier to identify AI-generated content globally" presenting a centralized detection platform users world-wide may log in to and voluntarily upload images, video, and audio for watermark detection.[31] It does not yet satisfy approaching legal requirements as it does not cover text, restricts access to logged-in users, is heavily rated-limited, and does not include an API. Having partnered with OpenAI, it can detect media from the company, establishing the first centralized detection solution from major AI providers.[25][32][33]
Both companies are signatories of the Code of Practice, whose official information session raised: "The moment these [interoperability] standards or such standards become available they will sort of trigger a second phase where these standards shall be adopted" 50:50-51:20.[26]
Grounds
[edit | edit source]Below are media examples of AI content misuse involving the risks listed in motivations for EU and Californian laws. These risks also form the basis for the widely-backed Code of Practice, being derived from the EU law. The relevant region and topic is provided. Those involving citizens' private communications are highlighted in bold.
- Australia AI-generated media eroding trust on social media
- South Africa AI disinformation using text, images, video and audio on social media
- Singapore Deepfake technology used to scam business over Zoom meeting (page 8)
- United States AI-cloned audio used by scammers on phone call
- United States FBI warn criminals use gen-AI for financial fraud, including via private communications
- International Interpol: Warnings of private citizens being targeted by deepfakes in calls from family members and online relationships
- European Union Europol: "With the increasing adoption of generative AI, fraudsters can now personalise social engineering techniques" (page 8)
- United Kingdom Fraudulent AI-generated media in private messages
- United Kingdom Scammers sending ChatGPT-translated messages on iMessage and Telegram
- Indonesia Victims of romance scammers sent ChatGPT-assisted messages
- Hong Kong AI-generated audio used to scam businessman via WhatsApp DMs
Applicability to private messaging
[edit | edit source]As laws and policies across the globe establish mass scanning infrastructure, it is important to consider how it may be used. While automated labelling of AI content in personal communications is not mandated by laws such as Article 50(4),[34] there are no laws preventing tech firms proactively implementing such measures. Doing so would exfiltrate both human and artificial content in personal communications to these scanning servers.
Given that:
- AI watermarking laws and policies make demands that can only be satisfied by scanning user content on remote servers (Cause)
- To meed these demands AI and tech firms are actively establishing centralized content scanning infrastructure - with global scope (Standards)
- The risks motivating watermarking laws apply both to public and private content (Grounds)
- Awareness of these risks, including AI-amplified disinformation "targetted to the individual",[35] has been increasingly publicized by politicians[36][37][38]
- Fraud using AI content in private messages has recently received increased media attention (Grounds)
- Tech firms have previously committed to proactively detect AI content on their platforms[39][40]
- As it stands, there is no public or media awareness that these laws and policies are globally establishing mass scanning infrastructure[41]
- EU-wise, this scanning infrastructure has the same implications as widely unpopular Chat Control 2.0,[42] applying it to private messages would achieve the same scope
- Industry- and EU-backed policies push for detection that is applicable to communication platforms (Code of Practice)
- Developments in the United Kingdom have shown tech companies can implement restrictive measures on their own will,[43][44][45] falsely claim it is a legal requirement,[46][47][48] receive government backing,[49][50][51] and governments later obtain cross-party support to announce that those measures will become mandatory[52][53]
- Similar digital restrictions and requirements have recently spread fast between countries[54][55][56]
it could be considered entirely appropriate and proportionate to demand legislation in individuals' respective countries that prohibits the optional and mandatory scanning of non-public content to identify AI watermarks, to maintain the integrity of personal communications.[57]
Negative effects
[edit | edit source]Mass scanning infrastructure, including for personal communications, is not intrinsically limited to AI detection. Once infrastructure that captures confidential user content has been established, the scope of scanning can be expanded at any later date without public approval.[58] Furthermore, using robust detection rate-limiting mechanisms based on identity-backed age-verified devices/accounts would allow scanned content to be directly attributed to individuals via the age assurance method.
Fundamental understanding that scanning cannot be intrinsically restricted to just watermark detection can be demonstrated by asking North American and European politicians if they endorse sending their citizens' user content to Chinese AI firms to check for watermarks. Notably, policymakers have not requested such accommodations.
Cause
[edit | edit source]Technical realities
[edit | edit source]All functional watermark detection solutions being trialled by AI firms in preparation for compliance with laws and regulations (SynthID Detector, Meta Content Seal, OpenAI textGrain, Anthropic text detection) abide by these three technical realities. They require content to be sent to servers users don't control, they require access to the whole content for reliable results, and it can't be known a piece of content, including human content, doesn't contain their watermarks without sending it away.
Third-party scanning requirement
[edit | edit source]Article 50(2) and similar laws make demands that can only be satisfied by remote scanning servers. This is because imperceptible watermarks are trivial to remove by iterating through minimally altered versions of watermarked content, especially when guided by detection confidence scores.[59] Maintaining their integrity necessitates restricting detection requests, removing the possibility for local detection.
Raw access requirement
[edit | edit source]Detection requires access to raw content, otherwise it would reveal if only certain components contain the watermark, enabling detection to be evaded by removing those parts. In addition, not having full access to the content excludes robust watermarking methods that disperse the signal throughout content.[60]
Human content requirement
[edit | edit source]Using watermarking to label AI content means all content deemed somewhere unlabelled AI content poses a risk would need to be sent for scanning, even if it isn't AI. This is because it can only be known that a piece of content isn't watermarked once every single watermark detector has scanned it and reported back.[61] Watermarking is about to form the first wide-spread resilient AI detection mechanism. Devices can't know there are no watermarks if they don't check whether they are there.
Lack of opposition
[edit | edit source]Deriving the laws' and policies' effects requires understanding not only their contents but also current watermarking technology and digital infrastructure, a combination uncommon both generally and among political representatives. Politicians are guided by public opinion. If politicians are understandably neither aware of the negative consequence, nor informed by public opinion, it is natural they support a measure appearing to uniquely offer upsides.
Laws require detection infrastructure to be free of charge in the vast majority of cases. Without public opposition, governments gain public appeal by implementing a measure addressing an increasingly publicized issue, especially if it comes at no financial cost.
Recursively training on AI-generated content can lead to model collapse.[62] AI companies agnostic to the consequences still benefit from obtaining a tool that helps filtering out AI content scraped from the web, and are thus inclined to support it.
Future obligations able to affect messaging apps
[edit | edit source]United Kingdom
[edit | edit source]Following a cross-party UK Government report recommending "All AI-generated content should be visibly labelled",[63] government proposals for AI labelling in the UK were due fall 2026,[64][65] but were later pushed back to 2027.[66] This means final recommendations to the UK government will now happen after AI firms have established mass scanning infrastructure to meet EU and US deadlines.
California
[edit | edit source]Beginning 1 January 2027, AB-853 California AI Transparency Act (2025-2026) requires a "large online platform" including a (paid) "mass messaging platform" who exceeded "2,000,000 unique monthly users during the preceding 12 months" to detect whether "provenance data that is compliant with widely adopted specifications" is embedded in "image, video, or audio content".[67] By this date, watermarks will form "widely adopted specifications", and detecting whether they are embedded will expose user content on these platforms to mass scanning infrastructure. Despite giving users the option to be informed, the law requires platforms to detect their presence regardless.[68]
Despite "mass messaging platforms" being defined as "a direct messaging platform that allows users to distribute content to more than 100 users simultaneously", the law includes platforms meeting the criteria in their entirety without limiting the scope to the platform's mass messaging groups. The law does not apply to text (only "image, video, or audio content"),[67] but if applied to media in instant messaging, can readily be expanded to text given the infrastructure needed would already be in place.
Timing regarding Californian deadlines is unreliable. The Californian watermarking deadline of 2 August 2026 has been flouted by AI companies who appear to reason complying with the revised 2 December 2026 EU deadline is satisfactory for California, despite the US state making no apparent announcement.
References
[edit | edit source]- ↑ Dathathri, Sumanth; See, Abigail; Ghaisas, Sumedh; Huang, Po-Sen; McAdam, Rob; Welbl, Johannes; Bachani, Vandana; Kaskasoli, Alex; Stanforth, Robert; Matejovicova, Tatiana; Hayes, Jamie; Vyas, Nidhi; Merey, Majd Al; Brown-Cohen, Jonah; Bunel, Rudy (2024-10-23). "Scalable watermarking for identifying large language model outputs". Nature. 634 (8035). Nature Publishing Group: 818–823. doi:10.1038/s41586-024-08025-4. ISSN 1476-4687.
- ↑ 2.0 2.1 "REGULATION (EU) 2024/1689 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL". EUR-Lex. 2024-07-12. Retrieved 2026-11-09.
- ↑ "The EU AI Act Marks a New Era of AI Transparency". GetReal Security. 2026-08-04. Retrieved 2026-11-12.
- ↑ "REGULATION (EU) 2026/1744 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL". EUR-Lex. 2026-07-24. Amendments (39) (b). Retrieved 2026-11-09.
- ↑ "Legal Center". Mistral AI. Retrieved 2026-11-13.
- ↑ "REGULATION (EU) 2024/1689 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCI". EUR-Lex. 2024-07-12. CHAPTER IV, Article 50, 2. Retrieved 2026-11-09.
- ↑ Rankin, Jennifer (2026-07-31). "AI labels to be compulsory on authentic-looking content under EU rules". The Guardian. ISSN 0261-3077. Retrieved 2026-11-10.
- ↑ "FACT SHEET: Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI | The American Presidency Project". The American Presidency Project - University of California, Santa Barbara. Retrieved 2026-11-11.
- ↑ "Voluntary AI Commitments" (PDF). bidenwhitehouse.archives.gov. Point 5. Retrieved 2026-11-11.
- ↑ "Guidelines on transparency obligations for providers and deployers of certain AI systems". European Commission. 2026-07-20. p. 20. Retrieved 2026-11-09.
- ↑ "REGULATION (EU) 2024/1689 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL". EUR-Lex. 2024-07-12. (133). Retrieved 2026-11-10.
- ↑ 12.0 12.1 "California Code, BPC 22757.2". California Legislative Information. Retrieved 2026-11-09.
- ↑ "AB 853: California AI Transparency Act. | Digital Democracy". CalMatters. Retrieved 2026-11-10.
- ↑ "Chapter 19.435 RCW:". Washington State Legislature. Retrieved 2026-11-09.
- ↑ 15.0 15.1 "Code of Practice on Transparency of AI-generated Content". European Commission - Digital EU. Retrieved 2026-11-10.
- ↑ "SB-1000 California AI Transparency Act". California Legislative Information. SECTION 1 (a). Retrieved 2026-11-09.
- ↑ 17.0 17.1 "Code of Practice on Transparency of AI-generated Content". European Commission - Digital EU. p. 13. Retrieved 2026-11-09.
- ↑ European Commission: Directorate-General for Communications Networks, Content and Technology and Fritz, M., Technical solutions for marking and detecting AI-generated image andf video content in the context of Article 50(2) AI Act, Publications Office of the European Union, 2026, https://data.europa.eu/doi/10.2759/9763153
- ↑ "Guidelines on transparency obligations for providers and deployers of certain AI systems". European Commission. 2026-07-20. p. 7. Retrieved 2026-11-11.
- ↑ "Commission publishes Code of Practice on marking and labelling AI-generated content". European Commission. 2026-06-10. Retrieved 2026-11-09.
- ↑ "Code of Practice on Transparency of AI-generated Content". European Commission - Digital EU. p. 12-14. Retrieved 2026-11-10.
- ↑ "Code of Practice on Transparency of AI-Generated Content - Information Session". European Commission - Digital EU. 49:15-49:55. Retrieved 2026-11-10.
- ↑ "Code of Practice on Transparency of AI-generated Content". European Commission - Digital EU. pp. 18–19. Retrieved 2026-11-10.
- ↑ Aktas, Ilhan (2026-10-16). "Good intentions, bad consequences". Felix. The risks, paragraph 2. Retrieved 2026-11-10.
- ↑ 25.0 25.1 "SynthID Detector". SynthID Detector. Retrieved 2026-11-12.
- ↑ 26.0 26.1 26.2 26.3 "Code of Practice on Transparency of AI-Generated Content - Information Session". European Commission - Digital EU. 2026-06-22. Retrieved 2026-11-09.
- ↑ "Commission opinion on the assessment of the Code of Practice on Transparency of AI-generated Content | Shaping Europe's digital future". European Commission. 2026-07-09. Retrieved 2026-11-09.
- ↑ 28.0 28.1 "Strong backing for the Code of Practice on Transparency of AI-generated Content | Shaping Europe's digital future". European Commission. 2026-09-24. Signatories of section 1. Retrieved 2026-11-09.
- ↑ "Agent Arena Leaderboard: AI Agent Rankings & Benchmarks". Arena Intelligence. 2026-11-08. Retrieved 2026-11-09.
- ↑ Gottfried, Jeffrey; Bishop, William; Anderson, Monica; Faverio, Michelle; Park, Eugenie; McClain, Colleen (2026-06-17). "Americans and AI 2026: Chatbots, Smart Devices and Views on Impact". Pew Research Center. Retrieved 2026-11-09.
- ↑ "We're making it easier to identify AI-generated content globally". Google. 2026-10-07. Retrieved 2026-11-11.
- ↑ Whitwam, Ryan (2026-10-07). "Google rolls out improved SynthID AI content detector, now available globally". Ars Technica. Retrieved 2026-11-12.
- ↑ De Simone, Sergio (2026-10-21). "Google SynthID Detector Simplifies Watermark Detection and Expands Support to Other AI Providers". InfoQ - Software Development News, Trends & Best Practices. Retrieved 2026-11-12.
- ↑ "Guidelines on transparency obligations for providers and deployers of certain AI systems". European Commission. pp. 41–42. Retrieved 2026-11-09.
- ↑ "MI5 tackling more state threats than during Cold War, warns intelligence chief". BBC News. 2026-10-13. Retrieved 2026-11-11.
- ↑ Wadephul, Johann (2026-09-26). "Speech by Foreign Minister Johann Wadephul at the General Assembly of the United Nations". German Federal Foreign Office. Retrieved 2026-11-11.
- ↑ Burnham, Andy (2026-09-22). "PM speech at UNGA: 22 September 2026". GOV.UK. Retrieved 2026-11-11.
- ↑ "Joint Statement on the Protection of Humanitarian Personnel and Humanitarian Action from Misinformation, Disinformation and Harmful Narratives | Australian Minister for Foreign Affairs". Minister for Foreign Affairs - Senator the Hon Penny Wong. 2026-09-21. Retrieved 2026-11-11.
- ↑ "The Strengthened Code of Practice on Disinformation 2022". European Commission.
- ↑ "Technology industry to combat deceptive use of AI in 2024 elections". Microsoft Source. 2024-02-16. Retrieved 2026-11-11.
- ↑ "("artificial intelligence" OR "AI") AND ("watermark" OR "watermarking") AND ("content" OR "scan" OR "effect") - Google Search". Google Search - News Results. Retrieved 2026-11-13.
- ↑ "Why the EU's so-called Chat Control law has privacy experts up in arms". euronews. Archived from the original on 2026-11-05. Retrieved 2026-11-12.
- ↑ "Apple rolls out UK age checks for iPhone users". Financial Times. "App stores and mobile operating systems are not covered by the Online Safety Act". Retrieved 2026-11-11.
- ↑ Field, Matthew (2026-03-25). "Apple asks British iPhone users to prove they are over 18". Yahoo Tech - The Telegraph. Retrieved 2026-11-11.
- ↑ Raya, Adrián (2026-03-30). "Reino Unido cambia las normas: los iPhone empiezan a verificar la edad de 35 millones de usuarios y provocan el caos". El Español (in español). ¿Pasará en España?. Retrieved 2026-11-11.
- ↑ "iOS | I can't get passed this and it's driving me insane | Facebook". Facebook. 2026-03-28. Main image. Retrieved 2026-11-11.
- ↑ "Apple's New iPhone Update Is Restricting Internet Freedom in the UK". Vuink.com. Main image. Retrieved 2026-11-11.
- ↑ Raya, Adrián (2026-03-30). "Reino Unido cambia las normas: los iPhone empiezan a verificar la edad de 35 millones de usuarios y provocan el caos". El Español (in español). Main image. Retrieved 2026-11-11.
- ↑ "Apple iPhone users in the UK asked to verify age with new update". BBC News. 2026-03-25. Retrieved 2026-11-11.
- ↑ Moon, Mariella (2026-03-25). "Apple introduces age verification for iCloud accounts in the UK". Engadget. Retrieved 2026-11-13.
- ↑ "New plans to stop children taking, sharing or viewing nude images". GOV.UK. Retrieved 2026-11-11.
- ↑ "Online Safety - Hansard - UK Parliament". hansard.parliament.uk. Retrieved 2026-11-11.
- ↑ "Written questions and answers - Written questions, answers and statements - UK Parliament". questions-statements.parliament.uk. Retrieved 2026-11-11.
- ↑ Presta, Jenna (2026-04-22). "Mapping the Spread of Child Safety Rules". CEPA. Retrieved 2026-11-11.
- ↑ "Region-specific rules for managing an Apple Account – Apple Support (US)". Apple Support. Retrieved 2026-11-11.
- ↑ "Requesting assistance with age verification : r/australia". Reddit. Main image. Retrieved 2026-11-11.
- ↑ Aktas, Ilhan (2026-10-16). "Good intentions, bad consequences". Felix. A glaring blind spot, paragraph 3. Retrieved 2026-11-12.
- ↑ "Information Commissioner's Annual Report to the House of Commons - Home Affairs Committee". UK Parliament - Publications & records. 2010-12-21. sections 'FUNCTION CREEP', TRANSPARENCY AND ACCOUNTABILITY. Archived from the original on 2011-11-22. Retrieved 2026-11-12.
- ↑ Pang, Qi (2024-09-27). "No Free Lunch in LLM Watermarking: Trade-offs in Watermarking Design Choices". Machine Learning Blog | ML@CMU | Carnegie Mellon University. Retrieved 2026-11-11.
- ↑ Barni, Mauro; Campisi, Patrizio; Delp, Edward J.; Doërr, Gwenaël; Fridrich, Jessica; Memon, Nasir; Pérez-González, Fernando; Rocha, Anderson; Verdoliva, Luisa. "Information Forensics and Security". read.nxtbook.com. Digital watermarking. Retrieved 2026-11-13.
- ↑ Aktas, Ilhan (2026-10-16). "Good intentions, bad consequences". Felix. The risks, paragraph 1. Retrieved 2026-11-12.
- ↑ Shumailov, Ilia; Shumaylov, Zakhar; Zhao, Yiren; Papernot, Nicolas; Anderson, Ross; Gal, Yarin (2024-07-24). "AI models collapse when trained on recursively generated data". Nature. 631 (8022). Nature Publishing Group: 755–759. doi:10.1038/s41586-024-07566-y. ISSN 1476-4687.
- ↑ "Social media, misinformation and harmful algorithms - Second Report of Session 2024–25". UK Parliament. Retrieved 2026-11-11.
- ↑ Trendall, Sam (2026-05-05). "Government to create task force to examine labelling for AI-generated content". PublicTechnology. Retrieved 2026-11-11.
- ↑ "Written questions and answers - Written questions, answers and statements - UK Parliament". questions-statements.parliament.uk. 2026-07-14. Retrieved 2026-11-11.
- ↑ "Written questions and answers - Written questions, answers and statements - UK Parliament". questions-statements.parliament.uk. 2026-10-29. Retrieved 2026-11-11.
- ↑ 67.0 67.1 "Bill Text - AB-853 California AI Transparency Act". California Legislative Information. 2025-10-13. Archived from the original on 2026-09-29. Retrieved 2026-11-12.
- ↑ "Bill Text - AB-853 California AI Transparency Act". California Legislative Information. 2025-10-13. 22757.3.1. Retrieved 2026-11-13.