Power weirdness

[Mercutio]

When I do the math on my big file server, it should have a continuous draw of just under 600W. I've gone through three Gold-rated 850W PSUs (PCP&C, Corsair HX, Seasonic - all rated for 50C operation) in the last eight months, the latest just this morning. The PSUs all fail in the same way: some of my drives just don't ever spin up no matter the pre-delay and the 5V line will show bad on my PSU tester. Needless to say, this is massively lame, but I can't really think of a good reason I've had three quality power supplies fail in the same way.
Temperatures in that room tend to be high, probably 90F (32C) in the summertime, though the issues of late occurred through fall and winter months when the room is a normal 70F (20C) or so.

I have four molex Y-splitters running to the backplane on my Norco chassis, but I've replaced each of those at least once.

I suppose that I could move to larger PSUs, but it's bothering me that this machine ran forever and a day on a 700W power supply and now I seem to have to buy $250 models on a subscription plan.

I have a multimeter and probably a good reason to use it, but for the life of me I can't think of what could be killing the PSUs that isn't hurting any of the other components. After I swap in a replacement, the machine will undoubtedly run fine for at least several months.

Any ideas or should I just give up and buy some 1200kW monster?

 

[CougTek]

Is it possible that you draw so much power on this circuit that you have brown outs? Power supplies, especially very efficient ones, don't like brown outs at all. It might be worth it to invest in an on-line UPS. They aren't cheap, but replacing 3 or 4 250$ PSU per year isn't either.

 

[Mercutio]

It's sitting on a 3000VA UPS as it is. I replaced the batteries for it in March of 2014 and I do test it periodically.

 

[ddrueding]

Does your UPS log incoming voltage? I'm with Coug on this one. If the UPS isn't an on-line model brownouts can sneak through. If you can, tighten up the UPS intervention level.

 

[CougTek]

Is it a very dry room? The fact that most failures occured during winter, when heating is on, could also point towards a too low humidity level (and therefore a very high static electricity level).

I thought about the possibility that all your drives are busting the 5V line, but modern drives are mostly feed by the 12V line and that's where the bulk of the power is directed in modern power supplies, so that's not really a plausible cause.

 

[Mercutio]

The machine is off pending a replacement PSU and so is the UPS. It's a Tripp-Lite that I've had for five or six years and I don't think it's that smart.
The electrical load in my apartment in general and in that room in particular is definitely lower now than it was a couple years ago, though.

 

[Mercutio]

Is it a very dry room? I thought about the possibility that all your drives are busting the 5V line, but modern drives are mostly feed by the 12V line and that's where the bulk of the power is directed in modern power supplies, so that's not really a plausible cause.

I think I can discount static, particularly at this time of year. Summer heat is a real problem because of my HVAC situation, but my equipment rack has been in that room for years now prior to having these problems and it's not that warm yet.

 

[sdbardwick]

Some cheap fan controllers connect both the 12v and 5v rails together (12v to +, 5v to -) to yield 7v at the fans. Some PSUs REALLY hate this, as the 5v rails are designed as current sources rather than current sinks that the 7v trick makes them.

IIRC, the newer power supplies with DC-DC conversion on the minor rails (3.3v and 5v) are the ones that don't like the 7v trick.

 

[blakerwry]

I agree with the comments that an on-line or AVR UPS might help. An AVR UPS will buck high line voltage or use the batteries to supplement low voltage to condition the power to a reasonable level and will switch completely over to battery power when needed. They also tend to provide a better approximation of a sine wave compared with some of the cheaper line-interactive UPS units (necessary for active PFC power supplies). Most larger VA UPS units are on-line, but check your manual. It's also possible that your UPS isn't working correctly.

I think it'd be worthwhile to do some spot checks using a kill-a-watt (I think I've seen these at home depot/lowes now). This should be able to tell you whether you've got high/low voltage or frequency. It will also tell you if your power draw is more or less than your estimates. In my experience, most people tend to estimate very high, but you never know. Monitoring the kill-a-watt while cycling your furnace/AC or other high draw appliances might be able to detect whether you're experiencing power sags.

 

[Mercutio]

Most larger VA UPS units are on-line, but check your manual. It's also possible that your UPS isn't working correctly.

The UPS in question is a Tripplite SmartOnline 3000 and it is definitely an online UPS. I bought it new in early 2011 and I've replaced batteries in it twice since I got it. I do run the monitor application but I really only pay attention to alerts. As far as other appliances, the only one I really have is a fridge, one other "big" desktop that probably uses under 250W continuously and a second SAS chassis (which has a 500W something-or-other in it and doesn't seem to have any problems) for the rest of the drives my file server uses.
My home has fairly regular power outages. My neighborhood is rather heavily forested, so any time there are strong winds there's a decent chance that a tree will fall somewhere or other. I have lots of crummy little 900VA UPSes, but I'm fairly proactive with the big one.

The drive cage on that machine is wired with three 1200rpm 120mm Noctua fans that each use about .5W and aren't plugged in to any sort of fan controller, though just to be sure I will check how all that stuff is plugged in since I doubt I've messed with them since I built the machine several years ago.

 

[blakerwry]

Sounds like you have it covered. About the only additional thing I can suggest is verifying the current quality and draw using a kill-a-watt (before and after the UPS). I've never used a gold rated PSU (silver at best) - so no experience as to whether they're any more picky or likely to experience failures.

 

[jtr1962]

What's the failure mode? Does it look like the electrolytic caps in the supplies are bulging? That might be symptomatic of too much heat. Incidentally, with the push to have quieter power supplies new supplies will have less airflow, and therefore higher internal temperatures, than many noisy, lower-rated older models. That might account for some of the issues you're facing. You might try going with a Platinum, or even Titanium, power supply. More efficiency equals less waste heat and lower temperatures.

I might also recommend 5400 RPM or slower drives if you're not already using them. If the server is for bulk storage, then slower, cooler drives might solve your problem without a noticeable speed decrease.

 

[Mercutio]

I don't generally open up power supplies. They normally have five or seven year warranties and I don't mind swapping them given the unit cost, though the urgency of getting a new one installed means that whatever comes back will get used elsewhere rather than back in my torture chamber. I do buy PSUs that are rated for 50C operation because I simply cannot air condition that room by order of landlord. I haven't had unusual numbers of drive failures (normal is relative but for me that means losing a drive roughly quarterly) or any issues with the Asus workstation board. I'll admit that machine kept on the ragged edge of operating temperatures all summer long but normally the motherboard or add-in cards cook off long before a power supply does.

I'm actually a bit surprised at how inexpensive the giant PSUs are. I paid around $250 for all three of the PSUs I've pulled out in the last year and I see that Newegg and Amazon both have Platinum-rated 1200W units for around the same price. I guess the mammoth power supplies aren't as exotic as they used to be.

 

[mubs]

Spot / local heat may be the issue. Since you can't cool the room, perhaps sticking a pedestal fan to blow in or suck out air might help somewhat.

Sometimes my 9 year-old APC BackUPS 1500 will switch back and forth between mains and battery non-stop. Happened this mornbing too. I stuck the only multimeter I have, a Fluke, into the socket adjacent to the one that feeds the UPS, and could see the voltage fluctuating rapidly between normal and a lower value. Because the Fluke has an LCD display that doesn't react fast like LEDs, it was difficult to catch, but the UPS was not bonkers; the voltage was dipping for a fraction of a second causing the UPS to switch t battery, then when the UPS sensed the voltage was up it switched back to mains and the loop was going on and on. I finally cut the power to the UPS for a few minutes, running on battery only, till the voltage stabilized. I second Blake's suggestion of logging power events, or you may never solve this riddle.

 

[ddrueding]

I'll go with the spot heating, too. I'm sure that case has tons of fans but I'd consider zip-tieing another to the back grille of the PSU pulling heat out. The Corsair units I've been favoring lately ship with a warning sticker explaining that the fans don't even kick on until the unit is running at some higher power draw. This is good for my application, but not for yours. I doubt another fan would be noticed.

 

[CougTek]

Sometimes my 9 year-old APC BackUPS 1500 will switch back and forth between mains and battery non-stop. Happened this mornbing too. I stuck the only multimeter I have, a Fluke, into the socket adjacent to the one that feeds the UPS, and could see the voltage fluctuating rapidly between normal and a lower value. Because the Fluke has an LCD display that doesn't react fast like LEDs, it was difficult to catch, but the UPS was not bonkers; the voltage was dipping for a fraction of a second causing the UPS to switch t battery, then when the UPS sensed the voltage was up it switched back to mains and the loop was going on and on. I finally cut the power to the UPS for a few minutes, running on battery only, till the voltage stabilized. I second Blake's suggestion of logging power events, or you may never solve this riddle.

Such power fluctuations wouldn't affect hardware connected to an on-line UPS like Mercutio already has. Everything is already powered from the batteries. If there's a hit to take, the UPS will take it. However, on a line-interactive UPS like your APC probably is, your hardware might be at risk if the UPS doesn't react fast enough to compensate the intake fluctuation.

 

[Mercutio]

I ordered a Kill-a-watt and a 1200W Seasonic PSU. I'm to see about borrowing an IR thermometer as well.
This machine is in a 13U rack inside a walk-in closet (I slide over the top of the rack when I need to mess with it). It's oriented so that the front of the PC is toward the back of the closet. There's a box fan (usually set to low speed) mounted to the wall of the closet to push air out and in the summer I add a second box fan ahead of the rack to push more air out.

It's probably not as loud as you might expect. All the fans in that machine are 1200rpm Noctuas. The box fans are kind of loud, but it's an agreeable sort of white noise that's well isolated from being in a closet.

Needless to say, the whole setup is ugly as sin, but I don't have tons of options given the space that I have, especially since I actually have someone staying in that back bedroom fairly regularly. In the winter I sleep in that room myself since it's normally 10 or 15 degrees warmer than the rest of my apartment anyway.

 

[Mercutio]

Kill-A-Watt says that machine is using 584W continuous and draws 730W at peak.

 

[time]

I'm going to guess that the waveform from your online UPS is no longer a pure sine wave. Given a high efficiency PSU and the degree to which you're loading it, sharper slopes might lead to excessive stress on the PSU.

You need a CRO or DSO (oscilloscopes).

 

[Mercutio]

Goddammitsomuch. I gave away the osciiloscope I kept in my car for YEARS just maybe 15 months ago.
The sound you hear in Australia is me sobbing as I pull a 40kg UPS out of my rack.