Bumpgate: Difference between revisions

Vindicator4021 (talk | contribs)
m Fixed a formatting error in the Microsoft section caused by image adding.
Vindicator4021 (talk | contribs)
Misdiagnosis/Poor Repair of Faults: Polished this section a bit; the stuff about the PS3 mods was a little wordier than it needed to be.
 
(One intermediate revision by the same user not shown)
Line 17: Line 17:


====Microsoft's Response - The "Red Ring of Death"====
====Microsoft's Response - The "Red Ring of Death"====
[[File:GPU early.jpg|alt=An image of a 90 nanometer graphics processing unit from a Microsoft Xbox 360. The die is in the center of the green square, and the eDRAM is located to the left of it. The die has dark greenish low Tg underfill under it, indicating that it is defective.|thumb|200x200px|A ''defective'' 90nm "Xenos" GPU from an '''early''' Xbox 360 (Xenon revision). Note the large die in the center of the chip, and the dark greenish underfill. The color of the underfill is one way to tell if the chip has the low T<sub>g</sub> underfill.]]
[[File:GPU early.jpg|alt=An image of a 90 nanometer graphics processing unit from a Microsoft Xbox 360. The die is in the center of the green square, and the eDRAM is located to the left of it. The die has dark greenish low Tg underfill under it, indicating that it is defective.|thumb|200x200px|A ''defective'' 90nm "Xenos" GPU from an '''early''' Xbox 360 (Xenon revision). Note the large die in the center of the chip, and the dark greenish underfill. The color of the underfill is one way to tell if the chip has the low T<sub>g</sub> underfill.]]<blockquote>''"The breakthrough came when we understood that the connections that were being broken were not located on the motherboard, but they were actually located inside the components."''
In November 2005, Microsoft released the Xbox 360 with the ATI Xenos GPU. According to leaked internal Microsoft documents from the time, 1.2 million 360s shipped out to consumers by November 30th, 2005. 3% of customers had some type of issue with their system. Out of that 3%, 19% had three flashing red rings, and 24% had freezing problems.
 
- Leo Del Castillo, member of Xbox's hardware engineering group; ''Power On: The Story of Xbox (2021)''<ref name=":3">{{Cite web |last=Xbox |date=13 Dec 2021 |title=Power On: The Story of Xbox {{!}} Chapter 5: The Red Ring of Death |url=https://www.youtube.com/watch?v=z2d6IMBS8oY |url-status=live |access-date=4 Jun 2025 |website=YouTube}}</ref></blockquote>In November 2005, Microsoft released the Xbox 360 with the ATI Xenos GPU. According to leaked internal Microsoft documents from the time, 1.2 million 360s shipped out to consumers by November 30th, 2005. 3% of customers had some type of issue with their system. Out of that 3%, 19% had three flashing red rings, and 24% had freezing problems.


Therefore, 43% of consumers shortly after launch had these familiar symptoms to retro console enthusiasts: the "Red Ring of Death"- which is what consumers around the world called the Xbox 360's Bumpgate-related fault. However, "three flashing red rings" on the Xbox 360 simply means "core digital error"- in other words, a general hardware failure. As it takes multiple power-on cycles to cause a failure in the solder bumps, it is unlikely that these systems were failing specifically because of the Bumpgate defect. 56% of systems worked on the first try, and after component reworks, that number improved to 71%. Only 200,000 consoles went to what Microsoft termed "the bonepile", a collection of broken consoles that did not work after a component rework. However, it is also worth noting that some afflicted Xbox 360s will work again after a "reflow"- which is effectively redoing all of the components on the motherboard at once. In addition, Microsoft wasn't aware that the "Red Ring of Death" was a specific widespread problem yet, so they would not have been taking major note of consoles with a GPU problem. At the height of the Bumpgate-related defect, approximately 600,000 to one million Xbox 360s were suffering from the "Red Ring of Death".
Therefore, 43% of consumers shortly after launch had these familiar symptoms to retro console enthusiasts: the "Red Ring of Death"- which is what consumers around the world called the Xbox 360's Bumpgate-related fault. However, "three flashing red rings" on the Xbox 360 simply means "core digital error"- in other words, a general hardware failure. As it takes multiple power-on cycles to cause a failure in the solder bumps, it is unlikely that these systems were failing specifically because of the Bumpgate defect. 56% of systems worked on the first try, and after component reworks, that number improved to 71%. Only 200,000 consoles went to what Microsoft termed "the bonepile", a collection of broken consoles that did not work after a component rework. However, it is also worth noting that some afflicted Xbox 360s will work again after a "reflow"- which is effectively redoing all of the components on the motherboard at once. In addition, Microsoft wasn't aware that the "Red Ring of Death" was a specific widespread problem yet, so they would not have been taking major note of consoles with a GPU problem. At the height of the Bumpgate-related defect, approximately 600,000 to one million Xbox 360s were suffering from the "Red Ring of Death".
[[File:GPU late.jpg|alt=An image of a 65 nanometer graphics processing unit from a Microsoft Xbox 360. The die is in the center of the green square, and the eDRAM is located to the left of it. The die has a light, whitish underfill- indicating that it is not one of the defective units.|thumb|204x204px|A ''non-defective'' 65nm "Xenos" GPU from a later revision of Xbox 360 (Jasper V2 on "Kronos 1" package). Note the smaller die and the light, whitish underfill.]]
[[File:GPU late.jpg|alt=An image of a 65 nanometer graphics processing unit from a Microsoft Xbox 360. The die is in the center of the green square, and the eDRAM is located to the left of it. The die has a light, whitish underfill- indicating that it is not one of the defective units.|thumb|204x204px|A ''non-defective'' 65nm "Xenos" GPU from a later revision of Xbox 360 (Jasper V2 on "Kronos 1" package). Note the smaller die and the light, whitish underfill.]]
For several months, Microsoft had consumers pay to repair their consoles. However, in 2007, they chose to extend the warranty for Xbox 360 consoles displaying an E74 error (an on-screen error also associated with "Red Ring of Death") from one year from date of purchase, to three years from date of purchase. This allowed most consumers who had consoles with the issue to get their consoles refurbished. In addition, they refunded any consumers who had paid to have their afflicted systems repaired.<!-- Felix mentions this in the 360 video (see Discussion page), but I need to find a proper source to confirm it. -->  
For several months, Microsoft had consumers pay to repair their consoles. However, in 2007, they chose to extend the warranty for Xbox 360 consoles displaying an E74 error (an on-screen error also associated with "Red Ring of Death") from one year from date of purchase, to three years from date of purchase. This allowed most consumers who had consoles with the issue to get their consoles refurbished. In addition, they refunded any consumers who had paid to have their afflicted systems repaired.<ref>{{Cite web |last=Moore |first=Peter |date=2007 |title=Open Letter From Peter Moore |url=http://xbox.com/en-ca/support/petermooreletter.htm |url-status=dead |archive-url=https://web.archive.org/web/20071023004948/http://xbox.com/en-ca/support/petermooreletter.htm |archive-date=23 Oct 2007 |access-date=4 Jun 2025 |website=Xbox}}</ref>


Microsoft primarily seemed to have done this to protect the Xbox brand. The Xbox 360 was only their second console, and the original Xbox hadn't done as well as they'd hoped it would. Microsoft confirmed this in Chapter 5 of their 2021 documentary, ''Power On: The Story of Xbox''.
Microsoft primarily seemed to have done this to protect the Xbox brand. The Xbox 360 was only their second console, and the original Xbox hadn't done as well as they'd hoped it would. Microsoft confirmed this in Chapter 5 of their 2021 documentary, ''Power On: The Story of Xbox''.<ref name=":3" />


By 2009, the cause of the issue was confirmed to be "within the components"- the low T<sub>g</sub> underfill. ATI and Microsoft completely fixed the issue in Xbox 360s made after this point.
By 2009, the cause of the issue was confirmed to be "within the components"- the low T<sub>g</sub> underfill. ATI and Microsoft completely fixed the issue in Xbox 360s made after this point.


====Sony's Response - The "Yellow Light of Death"====
====Sony's Response - The "Yellow Light of Death"====
In November 2006, Sony released the PlayStation 3. Like the Xbox 360, the early models of PlayStation 3 had what consumers believe to be Bumpgate-related issues with their Nvidia RSX GPU. Many consumers who had bought launch models- what are now known as "PlayStation 2 backwards compatible" models- found that their systems were malfunctioning. PlayStation 3 systems that were exhibiting the issue may freeze suddenly during gameplay and shut off, but all users who dealt with the defect had a common experience: when turning their PlayStation 3 on, the LED power indicator would turn green for a moment, then the system would beep three times, very briefly flash to a yellow light, then it would blink red. This was coined by consumers as the "Yellow Light of Death".
<blockquote>''"We think it is highly unfair to suggest that from an installed base of 2.5 million that the numbers you mention somehow are evidence of a 'manufacturing defect'..."''
 
- Ray Maguire, managing director and senior vice president of Sony Computer Entertainment Europe in 2009; in a letter responding to a BBC Watchdog segment covering the "Yellow Light of Death".<ref>{{Cite web |last=BBC |date=18 Sep 2009 |title=Sony rebuts BBC PlayStation claim |url=http://news.bbc.co.uk/2/hi/technology/8263063.stm |url-status=live |archive-url=https://web.archive.org/web/20250219154020/http://news.bbc.co.uk/2/hi/technology/8263063.stm |archive-date=19 Feb 2025 |access-date=4 Jun 2025 |website=BBC NEWS}}</ref></blockquote>In November 2006, Sony released the PlayStation 3. Like the Xbox 360, the early models of PlayStation 3 had what consumers believe to be Bumpgate-related issues with their Nvidia RSX GPU. Many consumers who had bought launch models- what are now known as "PlayStation 2 backwards compatible" models- found that their systems were malfunctioning. PlayStation 3 systems that were exhibiting the issue may freeze suddenly during gameplay and shut off, but all users who dealt with the defect had a common experience: when turning their PlayStation 3 on, the LED power indicator would turn green for a moment, then the system would beep three times, very briefly flash to a yellow light, then it would blink red. This was coined by consumers as the "Yellow Light of Death".


BBC Watchdog aired a segment in 2009 covering the "Yellow Light of Death".<ref name=":1">{{Cite web |last=HelpForPS3 (Reuploader) |last2=BBC |date=17 Dec 2009 |title=Sony PS3 Yellow Light of Death - BBC |url=https://www.youtube.com/watch?v=E_ef8bDQktI |url-status=live |access-date=3 Jun 2025 |website=YouTube}}</ref> The hosts stated that when viewers called Sony because their console was malfunctioning, Sony told them that it could have been for a variety of reasons, and they couldn't determine exactly what the problem was without disassembling the console having the issue. While all of this is true- the LED indicators do simply indicate a general hardware failure that requires detailed troubleshooting- it is worth noting that by August 2008, Sony was building consoles with the 65nm RSX<ref>{{Cite web |last=Miller |first=R. |date=26 Jun 2008 |title=PS3 graphics chip going 65nm this Fall |url=https://www.engadget.com/2008-06-26-ps3-graphics-chip-going-65nm-this-fall.html |url-status=live |archive-url=https://web.archive.org/web/20210924101930/https://www.engadget.com/2008-06-26-ps3-graphics-chip-going-65nm-this-fall.html |archive-date=24 Sep 2021 |access-date=3 Jun 2025 |website=Engadget}}</ref><ref>{{Cite web |title=NVIDIA Playstation 3 GPU 65nm Specs |url=https://www.techpowerup.com/gpu-specs/playstation-3-gpu-65nm.c1682 |url-status=live |access-date=3 Jun 2025 |website=TechPowerUp GPU Database}}</ref>, which has been confirmed by enthusiasts to not suffer from the likely Bumpgate-related fault.<ref name=":2" />
BBC Watchdog aired a segment in 2009 covering the "Yellow Light of Death".<ref name=":1">{{Cite web |last=HelpForPS3 (Reuploader) |last2=BBC |date=17 Dec 2009 |title=Sony PS3 Yellow Light of Death - BBC |url=https://www.youtube.com/watch?v=E_ef8bDQktI |url-status=live |access-date=3 Jun 2025 |website=YouTube}}</ref> The hosts stated that when viewers called Sony because their console was malfunctioning, Sony told them that it could have been for a variety of reasons, and they couldn't determine exactly what the problem was without disassembling the console having the issue. While all of this is true- the LED indicators do simply indicate a general hardware failure that requires detailed troubleshooting- it is worth noting that by August 2008, Sony was building consoles with the 65nm RSX<ref>{{Cite web |last=Miller |first=R. |date=26 Jun 2008 |title=PS3 graphics chip going 65nm this Fall |url=https://www.engadget.com/2008-06-26-ps3-graphics-chip-going-65nm-this-fall.html |url-status=live |archive-url=https://web.archive.org/web/20210924101930/https://www.engadget.com/2008-06-26-ps3-graphics-chip-going-65nm-this-fall.html |archive-date=24 Sep 2021 |access-date=3 Jun 2025 |website=Engadget}}</ref><ref>{{Cite web |title=NVIDIA Playstation 3 GPU 65nm Specs |url=https://www.techpowerup.com/gpu-specs/playstation-3-gpu-65nm.c1682 |url-status=live |access-date=3 Jun 2025 |website=TechPowerUp GPU Database}}</ref>, which has been confirmed by enthusiasts to not suffer from the likely Bumpgate-related fault.<ref name=":2" />
Line 37: Line 40:
Instead of admitting that there was a defect and issuing an extended warranty to consumers (like Microsoft did) Sony simply quietly fixed the defect around the same time that Microsoft did for the Xbox 360, so PS3 "slim" revisions and newer were not affected. Specifically, it was only 90nm GPUs that appeared to have low T<sub>g</sub> underfill- and hence, the defect.
Instead of admitting that there was a defect and issuing an extended warranty to consumers (like Microsoft did) Sony simply quietly fixed the defect around the same time that Microsoft did for the Xbox 360, so PS3 "slim" revisions and newer were not affected. Specifically, it was only 90nm GPUs that appeared to have low T<sub>g</sub> underfill- and hence, the defect.


As of 2025, Sony has still never made a statement confirming that "Yellow Light of Death", their likely Bumpgate-related issue, was a widespread problem in early PlayStation 3 models. The only reason that consumers know that the RSX was likely afflicted by Bumpgate is thanks to console repair and modding enthusiasts- who had to do extensive research to find out the true cause of the problem.<ref>{{Cite web |last=RIP Felix |first= |date=23 Dec 2022 |title=A PS3 Story: The Yellow Light of Death |url=https://www.youtube.com/watch?v=Za7WTNwAX0c |url-status=live |access-date=2 Jun 2025 |website=YouTube}}</ref> It was clear from the timeline- what systems have been observed to be reliable or unreliable revisions- as well as tests performed by console repair and modding enthusiasts (such as "poking" the underfill on suspected bad GPUs with a soldering iron<ref name=":2">{{Cite web |last=RIP Felix |date=15 Jan 2024 |title=A 360 Story - The RED Ring of Death & the 7th Generation Console War |url=https://www.youtube.com/watch?v=3qKtS_uxdcU |url-status=live |access-date=2 Jun 2025 |website=YouTube}}</ref>) that it was a Bumpgate-related defect.
As of 2025, Sony has still never made a statement confirming that "Yellow Light of Death", their likely Bumpgate-related issue, was a widespread problem in early PlayStation 3 models. The only reason that consumers know that the RSX was likely afflicted by Bumpgate is thanks to console repair and modding enthusiasts- who had to do extensive research to find out the true cause of the problem.<ref name=":4">{{Cite web |last=RIP Felix |first= |date=23 Dec 2022 |title=A PS3 Story: The Yellow Light of Death |url=https://www.youtube.com/watch?v=Za7WTNwAX0c |url-status=live |access-date=2 Jun 2025 |website=YouTube}}</ref> It was clear from the timeline- what systems have been observed to be reliable or unreliable revisions- as well as tests performed by console repair and modding enthusiasts (such as "poking" the underfill on suspected bad GPUs with a soldering iron<ref name=":2">{{Cite web |last=RIP Felix |date=15 Jan 2024 |title=A 360 Story - The RED Ring of Death & the 7th Generation Console War |url=https://www.youtube.com/watch?v=3qKtS_uxdcU |url-status=live |access-date=2 Jun 2025 |website=YouTube}}</ref>) that it was a Bumpgate-related defect.


====Nvidia's Response====
====Nvidia's Response====
Line 60: Line 63:


===Xbox 360/Microsoft Consumer response===
===Xbox 360/Microsoft Consumer response===
At first, consumers were angry at Microsoft. During the months that Microsoft would not acknowledge the widespread "Red Ring of Death", consumers felt as if they'd been scammed and made a bad investment. However, when Microsoft extended the warranty on Xbox 360s experiencing an E74 error to three years after purchase, consumer sentiment improved. After Microsoft's explanation and confirmation of the exact issue that caused the "Red Ring of Death" in ''Power On: The Story of Xbox'', many consumers who remembered dealing with this issue felt further validated. The release of this information also helped enthusiasts learn more about how to fix the issue not only in early Xbox 360s, but even in Sony's early PlayStation 3 systems.
At first, consumers were angry at Microsoft. During the months that Microsoft would not acknowledge the widespread "Red Ring of Death", consumers felt as if they'd been scammed and made a bad investment. However, when Microsoft extended the warranty on Xbox 360s experiencing an E74 error to three years after purchase, consumer sentiment improved. After Microsoft's explanation and confirmation of the exact issue that caused the "Red Ring of Death" in ''Power On: The Story of Xbox''<ref name=":3" />, many consumers who remembered dealing with this issue felt further validated. The release of this information also helped enthusiasts learn more about how to fix the issue not only in early Xbox 360s, but even in Sony's early PlayStation 3 systems.


===PlayStation 3/Sony Consumer Response===
===PlayStation 3/Sony Consumer Response===
Line 67: Line 70:


===Misdiagnosis/Poor Repair of Faults===
===Misdiagnosis/Poor Repair of Faults===
There was a lot of speculation among affected consumers as to why so many GPUs were failing, and theories tended to vary between communities for devices. For example, consumers who purchased an Xbox 360 thought that the issue was related to the console overheating, or it was due to RoHS requiring Microsoft to use lead-free solder balls to connect the Xenos GPU to the motherboard. Some PlayStation 3 owners were convinced for some time that the fault wasn't even related to their RSX GPU, but rather, the NEC-Tokin capacitors surrounding it. The lack of clarity from manufacturers in what was really causing the fault led many retro console and repair enthusiasts to make bad choices in how to repair their machines.
There was a lot of speculation among affected consumers as to why so many GPUs were failing, and theories tended to vary between communities for devices. For example, consumers who purchased an Xbox 360 from this era<ref>{{Cite web |last=Enever |first=Liam |date=1 Oct 2017 |title=Why has my got the red ring of death (sic) |url=https://www.ifixit.com/Answers/View/430460/Why+has+my+got+the+red+ring+of+death |url-status=live |access-date=4 Jun 2025 |website=iFixIt Answers Forum}}</ref> and even some independent repair technicians<ref>{{Cite web |date=8 Apr 2014 |title=Xbox 360 Red Ring of Death: Why Lead-Free Solder or Solder Failure Are the Problem |url=https://electronicfix.com.au/console-repairs/what-does-the-rrod-mean/ |url-status=live |access-date=4 Jun 2025 |website=The Electronic Fix}}</ref> assumed that the issue was caused by Microsoft using lead-free solder balls- used to connect the Xenos GPU to the motherboard- in order to meet new-at-the-time RoHS standards in the European Union. Many blamed it on the more brittle nature of non-leaded solder balls compared to traditional leaded ones. Others thought it was related to the solder balls' melting point, and that the console getting too hot and literally "desoldering" the GPU. These misconceptions largely faded into obscurity when Microsoft released the Jasper and Falcon revisions of the Xbox 360, and announced their extended warranty on their defective consoles. They were dispelled entirely with the release of Chapter 5 of ''Power On: The Story of Xbox''<ref name=":3" />, where hardware engineers for Xbox during this era explained the problem.
 
With the PlayStation 3, much of the same speculation happened as with the Xbox 360. However, because the defect was never acknowledged as such by Sony, it was far less clear to consumers what exactly was causing the fault in consumers' machines. Long after the PS3 was no longer supported by Sony, this led many retro console and repair enthusiasts to make poorly informed choices in how to attempt to repair their devices. By 2018, many owners were also convinced for some time that the fault wasn't even related to their RSX GPU, but rather, the NEC-Tokin capacitors surrounding it.<ref>{{Cite web |last=Naked_Snake1995 |date=15 Jul 2019 |title=(Research/Experimental) - NEC/TOKIN Capacitors Replacement - YLOD |url=https://www.psx-place.com/threads/research-experimental-nec-tokin-capacitors-replacement-ylod.25260/ |url-status=live |access-date=4 Jun 2025 |website=PSX-Place}}</ref>  The reason for the misconception was because Toshiba laptops from the same era had their NEC-Tokins fail prematurely due to a design flaw specific to those laptops. Some general hardware failures can be caused by the PS3's NEC-Tokin capacitors failing (primarily due to age), but Sony had designed the motherboard correctly when placing the NEC-Tokins- so they would not fail prematurely. It was not until 2022 that this misconception was largely corrected amongst the community, and the true defect was identified as likely to be Bumpgate-related.<ref name=":4" />
 
A common bad practice between both the Xbox 360 and the PS3 was to reflow the entire motherboard. Even alleged professional repair technicians did this, as shown in BBC's Watchdog segment for the "Yellow Light of Death".<ref name=":1" /> However, because this fault was caused by a defect in the GPU, this would only fix the problem ''temporarily'', if at all. In addition, a motherboard is not designed to go through this process more than once (specifically, at the time of manufacture), so this would damage the motherboard over time until it would be impossible to fix the system again.


A common bad practice between both the Xbox 360 and the PS3 was to reflow the entire motherboard. Ultimately, this would only fix the problem temporarily, if at all. In addition, a motherboard is not designed to go through this process more than once (at the time of manufacture), so this would damage the motherboard over time until it would be impossible to fix the system again. As this fault was caused by a defective GPU, though, the only way to fix it is to replace the faulty GPU with one that doesn't have the defect. How possible this is, however, depends much on a person's skills, tools, and how easy or difficult it is for consumers to repair and modify the system. For example, the PS3 has a modification called "Frankenstein", which makes it possible to replace a defective 90nm RSX GPU with a 65nm or 40nm one that is not defective. This modification was inspired by an "official" fix that Sony performed on some faulty PS3s. However, for a regular person to perform this repair, they would need a BGA rework station (and the skills to use it properly) and a softmodded PS3. Due to the specialized nature of this modification, most people will still recommend for consumers to find someone skilled enough to do the modification for them, buy an early PS3 that is already "Frankenstein" modded, or to simply avoid purchasing consoles impacted by Bumpgate.
The only way to permanently fix systems with a Bumpgate-affected GPU is to replace the defective GPU with one that doesn't have the defect. How possible that is, however, depends much on a person's skills, tools, and how easy or difficult it is for regular consumers to repair and modify the system. For example, the PS3 has a modification called "Frankenstein", which makes it possible to replace a defective 90nm RSX GPU with a non-defective 65nm or 40nm GPU from one of the newer PS3s. This modification was inspired by an "official" fix that Sony performed on some faulty PS3s. However, for a person to perform this repair, they would need a BGA rework station (and the skills to use it properly) and a softmodded PS3. A similar modification is also possible for early Xbox 360s, but it requires the same tools and modding experience as the PS3 does. Due to the specialized nature of such modifications, most people will still recommend for consumers to find someone skilled enough to do the modification for them, buy a system that is already modified, or to simply avoid purchasing consoles impacted by Bumpgate.


==References==
==References==