Artificial intelligence/training - Revision history

Rudxain: mention chip shortage

2026-04-27T03:21:29Z

mention chip shortage

← Older revision		Revision as of 03:21, 27 April 2026
Line 19:		Line 19:
	===Bandwidth abuse===		===Bandwidth abuse===
	Massive data needs massive bandwidth. Scraping web-pages across the entire internet requires sending millions of requests to all known servers. Some AI companies go as far as to ''repeatedly'' send requests for the same content (or several revisions of the same content) as frequent bursts in short intervals, which is indistinguishable from [[wikipedia:Denial-of-service_attack\|distributed denial-of-service (DDoS) attacks]].		Massive data needs massive bandwidth. Scraping web-pages across the entire internet requires sending millions of requests to all known servers. Some AI companies go as far as to ''repeatedly'' send requests for the same content (or several revisions of the same content) as frequent bursts in short intervals, which is indistinguishable from [[wikipedia:Denial-of-service_attack\|distributed denial-of-service (DDoS) attacks]].

			=== Chip shortage ===
			Many AI companies have pre-ordered massive amounts of computer components. So much that it doesn't even fit in their current data-centers. This is done in anticipation for ''more'' data-centers being built.{{Citation needed}} This has caused such components to become scarce, and prices to spike. The most notable being the [[wikipedia:2024–present_global_memory_supply_shortage\|increase in RAM prices]].

	==Examples==		==Examples==

Rudxain: link GPU

2026-04-27T03:07:50Z

link GPU

← Older revision		Revision as of 03:07, 27 April 2026
Line 15:		Line 15:

	===Energy use===		===Energy use===
	While [[Self-hosting\|self-hosted]] models can be trained with a single consumer-grade GPU, data-centers with hundreds or thousands of GPUs (known for being more power-hungry than CPUs) are used to train corporate-grade (or "enterprise") models. This can worsen [[wikipedia:Climate_change\|climate change]].		While [[Self-hosting\|self-hosted]] models can be trained with a single consumer-grade [[wikipedia:Graphics_processing_unit\|GPU]], data-centers with hundreds or thousands of GPUs (known for being more power-hungry than CPUs) are used to train corporate-grade (or "enterprise") models. This can worsen [[wikipedia:Climate_change\|climate change]].

	===Bandwidth abuse===		===Bandwidth abuse===

Rudxain: style-guide compliance for DDoS

2026-04-27T03:05:26Z

style-guide compliance for DDoS

← Older revision		Revision as of 03:05, 27 April 2026
Line 17:		Line 17:
	While [[Self-hosting\|self-hosted]] models can be trained with a single consumer-grade GPU, data-centers with hundreds or thousands of GPUs (known for being more power-hungry than CPUs) are used to train corporate-grade (or "enterprise") models. This can worsen [[wikipedia:Climate_change\|climate change]].		While [[Self-hosting\|self-hosted]] models can be trained with a single consumer-grade GPU, data-centers with hundreds or thousands of GPUs (known for being more power-hungry than CPUs) are used to train corporate-grade (or "enterprise") models. This can worsen [[wikipedia:Climate_change\|climate change]].

	=== Bandwidth abuse ===		===Bandwidth abuse===
	Massive data needs massive bandwidth. Scraping web-pages across the entire internet requires sending millions of requests to all known servers. Some AI companies go as far as to ''repeatedly'' send requests for the same content (or several revisions of the same content) as frequent bursts in short intervals, which is indistinguishable from [[wikipedia:Denial-of-service_attack\|DDoS attacks]].		Massive data needs massive bandwidth. Scraping web-pages across the entire internet requires sending millions of requests to all known servers. Some AI companies go as far as to ''repeatedly'' send requests for the same content (or several revisions of the same content) as frequent bursts in short intervals, which is indistinguishable from [[wikipedia:Denial-of-service_attack\|distributed denial-of-service (DDoS) attacks]].

	==Examples==		==Examples==
	While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing ~~[[wikipedia:Denial-of-service attack\|distributed denial of service~~ attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.		While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing DDoS attacks which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.

	Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow <code>[[wikipedia:robots.txt\|robots.txt]]</code>, a text file found at the root of a domain that indicates:		Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow <code>[[wikipedia:robots.txt\|robots.txt]]</code>, a text file found at the root of a domain that indicates:

Rudxain: expand why-prob

2026-04-27T03:00:34Z

expand why-prob

← Older revision		Revision as of 03:00, 27 April 2026
Line 7:		Line 7:
	==Why it is a problem==		==Why it is a problem==

	=== Intellectual property laundering ===		===Intellectual property laundering===
	Most, if not all, of the data used for training is copied indiscriminately, without even checking licenses or any copyright terms.{{Citation needed}} This is very controversial. Some people argue that it is "fair use" because AI systems learn in ways similar to animal and human brains, others claim it's more like [[wikt:parroting\|a parrot learning phrases]], others claim that it's "transformative" so it's still fair-use, others say it's akin to [[wikipedia:Tracing_(art)\|tracing images]] (this applies mostly to image models, though the analogy can work for text models).{{Citation needed\|reason=too many opinions}}		Most, if not all, of the data used for training is copied indiscriminately, without even checking licenses or any copyright terms.{{Citation needed}} This is very controversial. Some people argue that it is "fair use" because AI systems learn in ways similar to animal and human brains, others claim it's more like [[wikt:parroting\|a parrot learning phrases]], others claim that it's "transformative" so it's still fair-use, others say it's akin to [[wikipedia:Tracing_(art)\|tracing images]] (this applies mostly to image models, though the analogy can work for text models).{{Citation needed\|reason=too many opinions}}

Line 14:		Line 14:
	Some people request that, at the very least, the sources of the training data must be publicly disclosed, for the sake of [[wikipedia:Transparency_(behavior)\|transparency]] and [[wikipedia:Attribution_(copyright)\|attribution]].<ref>{{Cite web \|last=Tunney \|first=Justine \|date=2024-08-23 \|title=AI Training Shouldn't Erase Authorship \|url=https://justine.lol/history/ \|access-date=2026-04-26}}</ref>		Some people request that, at the very least, the sources of the training data must be publicly disclosed, for the sake of [[wikipedia:Transparency_(behavior)\|transparency]] and [[wikipedia:Attribution_(copyright)\|attribution]].<ref>{{Cite web \|last=Tunney \|first=Justine \|date=2024-08-23 \|title=AI Training Shouldn't Erase Authorship \|url=https://justine.lol/history/ \|access-date=2026-04-26}}</ref>

	=== ~~Ecosystem damage~~ ===		===Energy use===
	TO-DO		While [[Self-hosting\|self-hosted]] models can be trained with a single consumer-grade GPU, data-centers with hundreds or thousands of GPUs (known for being more power-hungry than CPUs) are used to train corporate-grade (or "enterprise") models. This can worsen [[wikipedia:Climate_change\|climate change]].

			=== Bandwidth abuse ===
			Massive data needs massive bandwidth. Scraping web-pages across the entire internet requires sending millions of requests to all known servers. Some AI companies go as far as to ''repeatedly'' send requests for the same content (or several revisions of the same content) as frequent bursts in short intervals, which is indistinguishable from [[wikipedia:Denial-of-service_attack\|DDoS attacks]].

	==Examples==		==Examples==

Rudxain: move notice to lone line, allowing users to edit intro

2026-04-27T02:23:10Z

move notice to lone line, allowing users to edit intro

← Older revision		Revision as of 02:23, 27 April 2026
Line 1:		Line 1:
	{{Incomplete}}'''AI training''' is a process by which data is fed into an AI model, in order to adjust its weights. This makes the output of the model closely match that of its input.		{{Incomplete}}
			'''AI training''' is a process by which data is fed into an AI model, in order to adjust its weights. This makes the output of the model closely match that of its input.

	==How it works==		==How it works==

Rudxain: populate how-works and why-prob

2026-04-27T02:22:28Z

populate how-works and why-prob

← Older revision		Revision as of 02:22, 27 April 2026
Line 1:		Line 1:
	{{Incomplete}}		{{Incomplete}}'''AI training''' is a process by which data is fed into an AI model, in order to adjust its weights. This makes the output of the model closely match that of its input.
	~~{{Ph-T-Int}}~~

	==How it works==		==How it works==
	~~{{Ph~~-T-~~HIW}}~~		There are several ways to implement AI, and even more ways to train them, the most well-known being [[wikipedia:Backpropagation\|backpropagation]]. With respect to the data-set, LLMs must be trained on massive amounts of data, which is a task that's only feasible via automation. This is in contrast to curated data-sets, in which both the data and the training is done in a more carefully controlled environment. Automated training on massive data-sets is typically done using internet web-sites as sources. The process of scraping is similar to how web-[[wikipedia:Search_engine\|search-engines]] index and [[wikipedia:Cache_(computing)\|cache]] pages.

	==Why it is a problem==		==Why it is a problem==
	{{Ph-T-~~WIIAP~~}}
			=== Intellectual property laundering ===
			Most, if not all, of the data used for training is copied indiscriminately, without even checking licenses or any copyright terms.{{Citation needed}} This is very controversial. Some people argue that it is "fair use" because AI systems learn in ways similar to animal and human brains, others claim it's more like [[wikt:parroting\|a parrot learning phrases]], others claim that it's "transformative" so it's still fair-use, others say it's akin to [[wikipedia:Tracing_(art)\|tracing images]] (this applies mostly to image models, though the analogy can work for text models).{{Citation needed\|reason=too many opinions}}

			Ultimately, it depends a lot on the technical details of how each model works, so none of those arguments are universal.

			Some people request that, at the very least, the sources of the training data must be publicly disclosed, for the sake of [[wikipedia:Transparency_(behavior)\|transparency]] and [[wikipedia:Attribution_(copyright)\|attribution]].<ref>{{Cite web \|last=Tunney \|first=Justine \|date=2024-08-23 \|title=AI Training Shouldn't Erase Authorship \|url=https://justine.lol/history/ \|access-date=2026-04-26}}</ref>

			=== Ecosystem damage ===
			TO-DO

	==Examples==		==Examples==

Rudxain: re-link Anubis: GH -> WP

2026-04-13T04:38:19Z

re-link Anubis: GH -> WP

← Older revision		Revision as of 04:38, 13 April 2026
Line 31:		Line 31:
	To protect against unethical crawlers, due to concerns of both intellectual property and service disruption, websites adopt practices that affect the experience of real users:		To protect against unethical crawlers, due to concerns of both intellectual property and service disruption, websites adopt practices that affect the experience of real users:

	*'''Bot check walls''': The user may be required to pass a security check "wall". While usually automatic for the user, this can affect legitimate bots. When a website protection service such as [[Cloudflare]] is not confident as to whether the visitor is legitimate, it may present a [[CAPTCHA]] to be manually filled out. An example is "Google Sorry", a CAPTCHA wall frequently seen when using Google Search via a VPN. An example that's popular in the FOSS community is [~~https~~:~~//github.com/TecharoHQ/anubis~~ Anubis].		*'''Bot check walls''': The user may be required to pass a security check "wall". While usually automatic for the user, this can affect legitimate bots. When a website protection service such as [[Cloudflare]] is not confident as to whether the visitor is legitimate, it may present a [[CAPTCHA]] to be manually filled out. An example is "Google Sorry", a CAPTCHA wall frequently seen when using Google Search via a VPN. An example that's popular in the FOSS community is [[wikipedia:Anubis_(software)\|Anubis]].
	*'''Login walls''': Should bots be found to pass CAPTCHA walls, the website may advance to requiring logging in to view content. A major recent example of this is [[YouTube]]'s "Sign in to confirm you're not a bot" messages.		*'''Login walls''': Should bots be found to pass CAPTCHA walls, the website may advance to requiring logging in to view content. A major recent example of this is [[YouTube]]'s "Sign in to confirm you're not a bot" messages.
	*'''[[JavaScript]] requirement''': Most websites do not need JavaScript to deliver their content. However, as many scrapers expect content to be found directly in the HTML, it is often an easy workaround to use JavaScript to "insert" the content after the page has loaded. This may reduce the responsiveness of the website, increasing points of failure, and preventing security-conscious users who disable JavaScript from viewing the website.		*'''[[JavaScript]] requirement''': Most websites do not need JavaScript to deliver their content. However, as many scrapers expect content to be found directly in the HTML, it is often an easy workaround to use JavaScript to "insert" the content after the page has loaded. This may reduce the responsiveness of the website, increasing points of failure, and preventing security-conscious users who disable JavaScript from viewing the website.

Rudxain: linkify: Chrome, Windows

2026-02-26T05:46:00Z

linkify: Chrome, Windows

← Older revision		Revision as of 05:46, 26 February 2026
Line 24:		Line 24:
	Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow <code>robots.txt</code> - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.		Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow <code>robots.txt</code> - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.

	Some AI services opt to use separate User-Agent strings, potentially also ignoring <code>robots.txt</code>, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />		Some AI services opt to use separate User-Agent strings, potentially also ignoring <code>robots.txt</code>, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard [[Google_Chrome\|Chrome]] web browser running on [[Microsoft_Windows\|Windows]]. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />

	Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).		Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).

Rudxain: style `robots.txt` as code

2026-02-26T05:38:25Z

style `robots.txt` as code

← Older revision		Revision as of 05:38, 26 February 2026
Line 11:		Line 11:
	While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.		While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.

	Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow [[wikipedia:robots.txt\|robots.txt]], a text file found at the root of a domain that indicates:		Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow <code>[[wikipedia:robots.txt\|robots.txt]]</code>, a text file found at the root of a domain that indicates:

	*Paths bots are allowed to index		*Paths bots are allowed to index
Line 20:		Line 20:
	These rules are typically configured for all bots, with minor adjustments made to individual bots as needed. Additionally, specific web pages may use the [[wikipedia:noindex\|robots meta tag]] to control use of their output.		These rules are typically configured for all bots, with minor adjustments made to individual bots as needed. Additionally, specific web pages may use the [[wikipedia:noindex\|robots meta tag]] to control use of their output.

	While it is good practice for a bot to respect robots.txt, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.		While it is good practice for a bot to respect <code>robots.txt</code>, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.

	Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow robots.txt - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.		Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow <code>robots.txt</code> - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.

	Some AI services opt to use separate User-Agent strings, potentially also ignoring robots.txt, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />		Some AI services opt to use separate User-Agent strings, potentially also ignoring <code>robots.txt</code>, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />

	Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).		Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).

Rudxain: link AI

2026-02-26T05:33:36Z

link AI

← Older revision		Revision as of 05:33, 26 February 2026
Line 22:		Line 22:
	While it is good practice for a bot to respect robots.txt, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.		While it is good practice for a bot to respect robots.txt, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.

	Unethical AI scraper bots do not follow robots.txt - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.		Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow robots.txt - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.

	Some AI services opt to use separate User-Agent strings, potentially also ignoring robots.txt, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />		Some AI services opt to use separate User-Agent strings, potentially also ignoring robots.txt, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />
Line 91:		Line 91:

	On 17 March 2025, the Git source code host SourceHut announced that the service was being disrupted by large language model crawlers. Mitigations deployed to reduce disruption involved requiring login for some areas of the service, and blocking IP ranges of cloud providers, affecting legitimate use of the website by its users.<ref>{{Cite web \|date=17 Mar 2025 \|title=LLM crawlers continue to DDoS SourceHut \|url=https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|website=sr.ht status \|url-status=live \|archive-url=http://web.archive.org/web/20251220125852/https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|archive-date=20 Dec 2025}}</ref> In response to the event, SourceHut founder Drew DeVault wrote a blog post entitled "[https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html Please stop externalizing your costs directly into my face]", discussing his frustrations with having ongoing and ever-adapting attacks that must be addressed in a timely fashion to reduce disruption to legitimate SourceHut users. DeVault estimates that between "20-100%" of his time is now spent addressing such attacks.		On 17 March 2025, the Git source code host SourceHut announced that the service was being disrupted by large language model crawlers. Mitigations deployed to reduce disruption involved requiring login for some areas of the service, and blocking IP ranges of cloud providers, affecting legitimate use of the website by its users.<ref>{{Cite web \|date=17 Mar 2025 \|title=LLM crawlers continue to DDoS SourceHut \|url=https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|website=sr.ht status \|url-status=live \|archive-url=http://web.archive.org/web/20251220125852/https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/ \|archive-date=20 Dec 2025}}</ref> In response to the event, SourceHut founder Drew DeVault wrote a blog post entitled "[https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html Please stop externalizing your costs directly into my face]", discussing his frustrations with having ongoing and ever-adapting attacks that must be addressed in a timely fashion to reduce disruption to legitimate SourceHut users. DeVault estimates that between "20-100%" of his time is now spent addressing such attacks.


	==References==		==References==

← Older revision		Revision as of 05:38, 26 February 2026
Line 11:		Line 11:
	While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.		While "mainstream" companies such as [[OpenAI]], [[Anthropic]], and [[Meta]] appear to correctly follow industry-standard practice for web crawlers, others ignore them (such as [[wikipedia:Alibaba_Group\|Alibaba]]<ref>{{Cite news \|last=Venerandi \|first=Niccolò \|title=FOSS infrastructure is under attack by AI companies \|url=https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies \|access-date=2026-02-23 \|work=LibreNews \|archive-url=http://web.archive.org/web/20260217195639/https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ \|archive-date=17 Feb 2026}}</ref>), causing [[wikipedia:Denial-of-service attack\|distributed denial of service attacks]] which damage access to freely-accessible websites. This is particularly an issue for websites that are large or contain many dynamic links.

	Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow [[wikipedia:robots.txt\|robots.txt]], a text file found at the root of a domain that indicates:		Ethical website scrapers, known as "spiders" that crawl the web, follow a certain set of minimum guidelines. Specifically, they follow <code>[[wikipedia:robots.txt\|robots.txt]]</code>, a text file found at the root of a domain that indicates:

	*Paths bots are allowed to index		*Paths bots are allowed to index
Line 20:		Line 20:
	These rules are typically configured for all bots, with minor adjustments made to individual bots as needed. Additionally, specific web pages may use the [[wikipedia:noindex\|robots meta tag]] to control use of their output.		These rules are typically configured for all bots, with minor adjustments made to individual bots as needed. Additionally, specific web pages may use the [[wikipedia:noindex\|robots meta tag]] to control use of their output.

	While it is good practice for a bot to respect robots.txt, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.		While it is good practice for a bot to respect <code>robots.txt</code>, there is no requirement for it, and there is no punishment for not following a website's wishes. It is additionally standard practice, but in no way enforced, that bots use a [[wikipedia:User-Agent header\|User-Agent header]] to uniquely identify itself. This allows a website operator to observe a bot's traffic patterns, potentially blocking the bot outright if its scraping is not desirable. The header also typically contains a URL or email address that can be used to contact the operator in case of anomalies observed in its traffic.

	Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow robots.txt - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.		Unethical [[Artificial_intelligence\|AI]] scraper bots do not follow <code>robots.txt</code> - in fact, they may not even request this file at all. They typically completely ignore it, instead opting to start from an entry point such as the root home page (<code>/</code>), working its way through an exponentially growing list of links as it finds them, with little to no delay between requests. The bots use false User-Agent header strings that would correspond to real web browsers on desktop or mobile operating systems - blocking them would also block legitimate users, or at least legitimate users on VPNs.

	Some AI services opt to use separate User-Agent strings, potentially also ignoring robots.txt, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />		Some AI services opt to use separate User-Agent strings, potentially also ignoring <code>robots.txt</code>, when a request is made through user command rather than as part of model training. For example, ChatGPT identifies itself as <code>ChatGPT-User</code> rather than its standard <code>OpenAI</code> when it uses the "search the web" command - even if searching the web was an automatic decision. In a less favorable example, Perplexity AI in this same situation falsely identifies as a standard Chrome web browser running on Windows. AI companies defend this under the belief that they are not a "spider", but rather a "user agent" (like a web browser), when called upon by a user's request.<ref name="perplexity-aws" />

	Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).		Less legitimate bots use a wide distribution of IP addresses, further reducing options for the website to protect itself. This is in a clear attempt to bypass IP-based request throttling and rate limiting the website may implement. They are also known to ignore HTTP response status codes that indicate a server error ([[wikipedia:HTTP status code#5xx server errors\|5xx]]), or warnings that the client needs to slow down ([[wikipedia:HTTP status code#429\|429 Too Many Requests]]) or has been entirely blocked ([[wikipedia:HTTP status code#403\|403 Forbidden]]).