Internet Archive: Difference between revisions

Line 27:

The Internet Archive ''used'' to hide material covered by robots.txt restrictions but that was changed on April 17, 2017.<ref>{{Cite web |last=Graham |first=Mark |date=2017-04-17 |title=Robots.txt meant for search engines don’t work well for web archives |url=https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ |url-status=live |archive-url=https://web.archive.org/web/20170417131508/http://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ |archive-date=2017-04-17 |access-date=2025-08-31 |website=Internet Archive}}</ref>

===Removal of noindex on uploaded items===

On 2023 the Internet Archive reportedly removed the ability for users to use the noindex function, which used to result in the items being hidden from internal search engines. Some users criticized the decision on the grounds that it may jeopardize users' rights, including privacy. When confronted about it, Jason Scott, who's a staffmember of the Internet Archive, reportedly responded with the following:<ref>https://old.reddit.com/r/DataHoarder/comments/156s7di/the_removal_of_noindex_from_the_internet_archive/</ref><ref>https://old.reddit.com/r/NoStupidQuestions/comments/142nm9h/internet_archive_ish/</ref>

{{tq|There is no bug or mistake in removing no-index settings for many Internet Archive items in the Community collection.}}

{{tq|At no point was the Archive contacted to arrange a situation of no-indexing (or Darking) items with an intention of later release; the no-index setting was not documented for this use, and represented a security hole that was closed. Tens of thousands of items were found, being used for encrypted files hidden from the search engine, and represented a major problem, so many items have been removed or set noindex quickly.}}

{{tq|A number of people have contacted us explaining situations where items might need to be made no-indexed, in a collection for later or timed release for example, but they've done it with communication and discussing their needs, not just uploading files under disposable accounts and then assuming the archive would keep them un-accessible in perpetuity. In some cases their requests have gotten arrangements so that community items that were noindex are noindex again, in separate collections.}}

{{tq|A situation can theoretically exist where the original uploader can e-mail us from their e-mail address and discuss arrangements, but you've indicated you intentionally obfuscated your location and have disposed your addresses. If you're able to gain access again, you can mail through those addresses.}}

{{tq|An additional situation is you can e-mail [email protected] if you want to report items at the archive (by identifier) that you believe might need to be removed from the archive; we receive a number of these requests throughout the months and respond according to policy.}}

===Data breaches (2012-2024)===

On May 19, 2017, The Archive's Development Manager made a blog post detailing that anyone who had created their account before 2012 had to change their password as the site had been breached with user's public information and lightly encrypted passwords being leaked.<ref>{{Cite web |last=Barrett |first=Katie |date=2017-05-19 |title=Re: User account breach {{!}} Internet Archive Blogs |url=https://blog.archive.org/2017/05/19/re-user-account-breach/ |url-status=live |archive-url=https://web.archive.org/web/20250520030556/https://blog.archive.org/2017/05/19/re-user-account-breach/ |archive-date=2025-05-20 |access-date=2025-08-16 |website=[[Internet Archive]]}}</ref>

@@ Line 27: / Line 27: @@
 The Internet Archive ''used'' to hide material covered by robots.txt restrictions but that was changed on April 17, 2017.<ref>{{Cite web |last=Graham |first=Mark |date=2017-04-17 |title=Robots.txt meant for search engines don’t work well for web archives |url=https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ |url-status=live |archive-url=https://web.archive.org/web/20170417131508/http://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ |archive-date=2017-04-17 |access-date=2025-08-31 |website=Internet Archive}}</ref>
+===Removal of noindex on uploaded items===
+On 2023 the Internet Archive reportedly removed the ability for users to use the noindex function, which used to result in the items being hidden from internal search engines. Some users criticized the decision on the grounds that it may jeopardize users' rights, including privacy. When confronted about it, Jason Scott, who's a staffmember of the Internet Archive, reportedly responded with the following:<ref>https://old.reddit.com/r/DataHoarder/comments/156s7di/the_removal_of_noindex_from_the_internet_archive/</ref><ref>https://old.reddit.com/r/NoStupidQuestions/comments/142nm9h/internet_archive_ish/</ref>
+{{tq|There is no bug or mistake in removing no-index settings for many Internet Archive items in the Community collection.}}
+{{tq|At no point was the Archive contacted to arrange a situation of no-indexing (or Darking) items with an intention of later release; the no-index setting was not documented for this use, and represented a security hole that was closed. Tens of thousands of items were found, being used for encrypted files hidden from the search engine, and represented a major problem, so many items have been removed or set noindex quickly.}}
+{{tq|A number of people have contacted us explaining situations where items might need to be made no-indexed, in a collection for later or timed release for example, but they've done it with communication and discussing their needs, not just uploading files under disposable accounts and then assuming the archive would keep them un-accessible in perpetuity. In some cases their requests have gotten arrangements so that community items that were noindex are noindex again, in separate collections.}}
+{{tq|A situation can theoretically exist where the original uploader can e-mail us from their e-mail address and discuss arrangements, but you've indicated you intentionally obfuscated your location and have disposed your addresses. If you're able to gain access again, you can mail through those addresses.}}
+{{tq|An additional situation is you can e-mail [email protected] if you want to report items at the archive (by identifier) that you believe might need to be removed from the archive; we receive a number of these requests throughout the months and respond according to policy.}}
 ===Data breaches (2012-2024)===
 On May 19, 2017, The Archive's Development Manager made a blog post detailing that anyone who had created their account before 2012 had to change their password as the site had been breached with user's public information and lightly encrypted passwords being leaked.<ref>{{Cite web |last=Barrett |first=Katie |date=2017-05-19 |title=Re: User account breach {{!}} Internet Archive Blogs |url=https://blog.archive.org/2017/05/19/re-user-account-breach/ |url-status=live |archive-url=https://web.archive.org/web/20250520030556/https://blog.archive.org/2017/05/19/re-user-account-breach/ |archive-date=2025-05-20 |access-date=2025-08-16 |website=[[Internet Archive]]}}</ref>