Bumpgate: Difference between revisions
m Added some headers, changed [Incidents] to "Companies involved and responses", and corrected misuses of solder "balls" to "bumps" in the intro. (yes, there's a difference) |
m →Sony's Response - The "Yellow Light of Death": Added a reference for the soldering iron "poke test". |
||
Line 27: | Line 27: | ||
In November 2006, Sony released the PlayStation 3. Like the Xbox 360, the early models of PlayStation 3 had what consumers believe to be Bumpgate-related issues with their Nvidia RSX GPU. Sony never issued a recall, or extended warranty for these systems; consumers had to pay out of their own pocket to get them repaired. Because this was due to a defect, however, many consumers who did so only ended up having their console break again some time after. They quietly fixed the issue around the same time that Microsoft did for the Xbox 360, so "slim" revisions and newer were not affected. Specifically, it was only 90nm GPUs that had low Tg underfill- and hence, the defect. | In November 2006, Sony released the PlayStation 3. Like the Xbox 360, the early models of PlayStation 3 had what consumers believe to be Bumpgate-related issues with their Nvidia RSX GPU. Sony never issued a recall, or extended warranty for these systems; consumers had to pay out of their own pocket to get them repaired. Because this was due to a defect, however, many consumers who did so only ended up having their console break again some time after. They quietly fixed the issue around the same time that Microsoft did for the Xbox 360, so "slim" revisions and newer were not affected. Specifically, it was only 90nm GPUs that had low Tg underfill- and hence, the defect. | ||
As of 2025, Sony has never made a statement confirming that "Yellow Light of Death", their likely Bumpgate-related issue, was a widespread problem in early PlayStation 3 models. The only reason that consumers know that the RSX was likely afflicted by Bumpgate is thanks to console repair and modding enthusiasts- who had to do extensive research to find out the true cause of the problem.<ref>{{Cite web |last=RIP Felix |first= |date=23 Dec 2022 |title=A PS3 Story: The Yellow Light of Death |url=https://www.youtube.com/watch?v=Za7WTNwAX0c |url-status=live |access-date=2 Jun 2025 |website=YouTube}}</ref> It was clear from the timeline- what systems have been observed to be reliable or unreliable revisions- as well as tests performed by console repair and modding enthusiasts (such as "poking" the underfill on suspected bad GPUs with a soldering iron) that it was a Bumpgate-related defect. | As of 2025, Sony has never made a statement confirming that "Yellow Light of Death", their likely Bumpgate-related issue, was a widespread problem in early PlayStation 3 models. The only reason that consumers know that the RSX was likely afflicted by Bumpgate is thanks to console repair and modding enthusiasts- who had to do extensive research to find out the true cause of the problem.<ref>{{Cite web |last=RIP Felix |first= |date=23 Dec 2022 |title=A PS3 Story: The Yellow Light of Death |url=https://www.youtube.com/watch?v=Za7WTNwAX0c |url-status=live |access-date=2 Jun 2025 |website=YouTube}}</ref> It was clear from the timeline- what systems have been observed to be reliable or unreliable revisions- as well as tests performed by console repair and modding enthusiasts (such as "poking" the underfill on suspected bad GPUs with a soldering iron<ref>{{Cite web |last=RIP Felix |date=15 Jan 2024 |title=A 360 Story - The RED Ring of Death & the 7th Generation Console War |url=https://www.youtube.com/watch?v=3qKtS_uxdcU |url-status=live |access-date=2 Jun 2025 |website=YouTube}}</ref>) that it was a Bumpgate-related defect. | ||
====Nvidia's Response==== | ====Nvidia's Response==== | ||
Line 45: | Line 45: | ||
The outcome of the suit, if any.}} | The outcome of the suit, if any.}} | ||
=== Nvidia Lawsuit === | ===Nvidia Lawsuit=== | ||
==Consumer response== | ==Consumer response== | ||
{{Placeholder box|Summary and key issues of prevailing sentiment from the consumers and commentators that can be documented via articles, emails to support, reviews and forum posts.}}[General Consumer Response (frustration at/satisfaction with Microsoft, Sony, etc.)] | {{Placeholder box|Summary and key issues of prevailing sentiment from the consumers and commentators that can be documented via articles, emails to support, reviews and forum posts.}}[General Consumer Response (frustration at/satisfaction with Microsoft, Sony, etc.)] | ||
Line 52: | Line 52: | ||
At first, consumers were angry at Microsoft. During the months that Microsoft would not acknowledge the widespread "Red Ring of Death", consumers felt as if they'd been scammed and made a bad investment. However, when Microsoft extended the warranty on Xbox 360s experiencing an E74 error to three years after purchase, consumer sentiment improved. After Microsoft's explanation and confirmation of the exact issue that caused the "Red Ring of Death" in ''Power On: The Story of Xbox'', many consumers who remembered dealing with this issue felt further validated. The release of this information also helped enthusiasts learn more about how to fix the issue not only in early Xbox 360s, but even in Sony's early PlayStation 3 systems. | At first, consumers were angry at Microsoft. During the months that Microsoft would not acknowledge the widespread "Red Ring of Death", consumers felt as if they'd been scammed and made a bad investment. However, when Microsoft extended the warranty on Xbox 360s experiencing an E74 error to three years after purchase, consumer sentiment improved. After Microsoft's explanation and confirmation of the exact issue that caused the "Red Ring of Death" in ''Power On: The Story of Xbox'', many consumers who remembered dealing with this issue felt further validated. The release of this information also helped enthusiasts learn more about how to fix the issue not only in early Xbox 360s, but even in Sony's early PlayStation 3 systems. | ||
=== PlayStation 3/Sony Consumer Response === | ===PlayStation 3/Sony Consumer Response=== | ||
=== Nvidia Consumer Response === | ===Nvidia Consumer Response=== | ||
===Misdiagnosis/Poor Repair of Faults=== | ===Misdiagnosis/Poor Repair of Faults=== |
Revision as of 00:59, 3 June 2025
Bumpgate, also known as Nvidiagate, was a scandal where Nvidia and ATI Graphics Processing Units (GPUs) were prone to high failure rates due to a design flaw that led to cracked solder bumps under the die. Despite the "Nvidiagate" name, this defect not only affected many Nvidia GPUs made from approximately 2006 to 2010, but it also affected ATI GPUs from 2006 to 2008. Among retro console enthusiasts, the defect is best known to have likely been the culprit behind the high failure rate of Nvidia GPUs in Sony's early PlayStation 3 models and ATI GPUs in Microsoft's early Xbox 360 models.
Background
Graphics Processing Unit (GPU) Faults
From approximately 2005-2010, GPU manufacturers Nvidia and ATI developed some GPUs that had a serious design flaw. They had engineered GPUs that electrically connected the silicon chip (die) to the substrate (the "green square part") using high-lead solder bumps. High-lead solder bumps were chosen in order to fit the power delivery specifications that these GPUs needed[1]. To give these solder bumps more strength (especially when operating at high temperatures), it's standard to use an epoxy with silica filler known as underfill. Underfill needs to fit certain specifications, depending on how hot the processor it's used on is expected to get. If it's too hard, the underfill will crack the die. If it's too soft, it'll crack the bumps. It also needs to still fit within the right specifications at both high and low temperatures.
In 2006, IBM and Amkor published a study that explained that use of a low Tg underfill was not acceptable with high-lead solder bumps, and high Tg would be necessary to avoid defects. Therefore, because Nvidia and ATI chose to use high-lead solder bumps, they needed a high Tg underfill. However, this study was not out at the time that GPUs from 2005 were made, and the companies ended up using low Tg underfill in these processors.
This low Tg underfill would become too soft at high, but normal operating temperatures for these GPUs. When the processor went through normal thermal changes, the solder bumps would soften under heat and harden as they cooled.[2] This would happen over and over again until they cracked under the thermal stress. When enough solder bumps cracked, it would cause a failure in the unit, hence the term "Bumpgate".
Companies involved and responses
Bumpgate was a worldwide issue for any consumer that purchased these defective graphics processors. However, it's not fully clear who was the most responsible for the incident. Building a GPU is a specialized process that requires it to go through multiple partner companies before ultimately ending up with the company that sells the unit (i.e., Microsoft, Sony, etc.). With this in mind, it is still important to note the response of each of the companies that were the public faces involved in this incident.
Microsoft's Response - The "Red Ring of Death"
In November 2005, Microsoft released the Xbox 360 with the ATI Xenos GPU. According to leaked internal Microsoft documents from the time, 1.2 million 360s shipped out to consumers by November 30th, 2005. 3% of customers had some type of issue with their system. Out of that 3%, 19% had three flashing red rings, and 24% had freezing problems.
Therefore, 43% of consumers shortly after launch had these familiar symptoms to retro console enthusiasts: the "Red Ring of Death"- which is what consumers around the world called the Xbox 360's Bumpgate-related fault. However, "three flashing red rings" on the Xbox 360 simply means "core digital error"- in other words, a general hardware failure. As it takes multiple power-on cycles to cause a failure in the solder bumps, it is unlikely that these systems were failing specifically because of the Bumpgate defect. 56% of systems worked on the first try, and after component reworks, that number improved to 71%. Only 200,000 consoles went to what Microsoft termed "the bonepile", a collection of broken consoles that did not work after a component rework. However, it is also worth noting that some afflicted Xbox 360s will work again after a "reflow"- which is effectively redoing all of the components on the motherboard at once. In addition, Microsoft wasn't aware that the "Red Ring of Death" was a specific widespread problem yet, so they would not have been taking major note of consoles with a GPU problem. At the height of the Bumpgate-related defect, approximately 600,000 to one million Xbox 360s were suffering from the "Red Ring of Death".
For several months, Microsoft had consumers pay to repair their consoles. However, in 2007, they chose to extend the warranty for Xbox 360 consoles displaying an E74 error (an on-screen error also associated with "Red Ring of Death") from one year from date of purchase, to three years from date of purchase. This allowed most consumers who had consoles with the issue to get their consoles refurbished. In addition, they refunded any consumers who had paid to have their afflicted systems repaired.
Microsoft primarily seemed to have done this to protect the Xbox brand. The Xbox 360 was only their second console, and the original Xbox hadn't done as well as they'd hoped it would. Microsoft confirmed this in Chapter 5 of their 2021 documentary, Power On: The Story of Xbox.
By 2009, the cause of the issue was confirmed to be "within the components"- the low Tg underfill. ATI and Microsoft completely fixed the issue in Xbox 360s made after this point.
Sony's Response - The "Yellow Light of Death"
In November 2006, Sony released the PlayStation 3. Like the Xbox 360, the early models of PlayStation 3 had what consumers believe to be Bumpgate-related issues with their Nvidia RSX GPU. Sony never issued a recall, or extended warranty for these systems; consumers had to pay out of their own pocket to get them repaired. Because this was due to a defect, however, many consumers who did so only ended up having their console break again some time after. They quietly fixed the issue around the same time that Microsoft did for the Xbox 360, so "slim" revisions and newer were not affected. Specifically, it was only 90nm GPUs that had low Tg underfill- and hence, the defect.
As of 2025, Sony has never made a statement confirming that "Yellow Light of Death", their likely Bumpgate-related issue, was a widespread problem in early PlayStation 3 models. The only reason that consumers know that the RSX was likely afflicted by Bumpgate is thanks to console repair and modding enthusiasts- who had to do extensive research to find out the true cause of the problem.[3] It was clear from the timeline- what systems have been observed to be reliable or unreliable revisions- as well as tests performed by console repair and modding enthusiasts (such as "poking" the underfill on suspected bad GPUs with a soldering iron[4]) that it was a Bumpgate-related defect.
Nvidia's Response
[Discuss initial response. Briefly touch on the lawsuit; there's a dedicated section below]
Lawsuit(s)
Nvidia Lawsuit
Consumer response
[General Consumer Response (frustration at/satisfaction with Microsoft, Sony, etc.)]
Xbox 360/Microsoft Consumer response
At first, consumers were angry at Microsoft. During the months that Microsoft would not acknowledge the widespread "Red Ring of Death", consumers felt as if they'd been scammed and made a bad investment. However, when Microsoft extended the warranty on Xbox 360s experiencing an E74 error to three years after purchase, consumer sentiment improved. After Microsoft's explanation and confirmation of the exact issue that caused the "Red Ring of Death" in Power On: The Story of Xbox, many consumers who remembered dealing with this issue felt further validated. The release of this information also helped enthusiasts learn more about how to fix the issue not only in early Xbox 360s, but even in Sony's early PlayStation 3 systems.
PlayStation 3/Sony Consumer Response
Nvidia Consumer Response
Misdiagnosis/Poor Repair of Faults
There was a lot of speculation among affected consumers as to why so many GPUs were failing, and theories tended to vary between communities for devices. For example, consumers who purchased an Xbox 360 thought that the issue was related to the console overheating, or it was due to RoHS requiring Microsoft to use lead-free solder balls to connect the Xenos GPU to the motherboard. Some PlayStation 3 owners were convinced for some time that the fault wasn't even related to their RSX GPU, but rather, the NEC-Tokin capacitors surrounding it. The lack of clarity from manufacturers in what was really causing the fault led many retro console and repair enthusiasts to make bad choices in how to repair their machines.
A common bad practice between both the Xbox 360 and the PS3 was to reflow the entire motherboard. Ultimately, this would only fix the problem temporarily, if at all. In addition, a motherboard is not designed to go through this process more than once (at the time of manufacture), so this would damage the motherboard over time until it would be impossible to fix the system again. As this fault was caused by a defective GPU, though, the only way to fix it is to replace the faulty GPU with one that doesn't have the defect. How possible this is, however, depends much on a person's skills, tools, and how easy or difficult it is for consumers to repair and modify the system. For example, the PS3 has a modification called "Frankenstein", which makes it possible to replace a defective 90nm RSX GPU with a 65nm or 40nm one that is not defective. This modification was inspired by an "official" fix that Sony performed on some faulty PS3s. However, for a regular person to perform this repair, they would need a BGA rework station (and the skills to use it properly) and a softmodded PS3. Due to the specialized nature of this modification, most people will still recommend for consumers to find someone skilled enough to do the modification for them, buy an early PS3 that is already "Frankenstein" modded, or to simply avoid purchasing consoles impacted by Bumpgate.
References
- ↑ Williams, Rob (29 Sep 2008). "NVIDIA at a Disadvantage Due to their Choice of Solder?". Techgage. Retrieved 1 Jun 2025.
{{cite web}}
: CS1 maint: url-status (link) - ↑ Demerjian, Charlie (1 Sep 2008). "Why Nvidia's chips are defective". The Inquirer. Archived from the original on 20 May 2009. Retrieved 1 Jun 2025.
- ↑ RIP Felix (23 Dec 2022). "A PS3 Story: The Yellow Light of Death". YouTube. Retrieved 2 Jun 2025.
{{cite web}}
: CS1 maint: url-status (link) - ↑ RIP Felix (15 Jan 2024). "A 360 Story - The RED Ring of Death & the 7th Generation Console War". YouTube. Retrieved 2 Jun 2025.
{{cite web}}
: CS1 maint: url-status (link)