Nor Cal Outages - cutting over to genset power.

[Uncle Dave]

After a few days of this crap we put together our own setup to at least keep our IT going during these blackouts.

We built our own sub panel and are up and running with shitshow 1&2.
We streamlined the server room so we can run on a min compliment and have the current setup pulling about 12KW out of an available 24 so we have some over head left.

We've lost a ton of productivity over this BS and each time these guys do this we blow at least one server so being able to cut over and do it on our own terms will least us keep working and hopefully keep our stuff alive.

Newsoms worried about plastic shampoo bottles and half the states out of work and on par reliability wise with Puerto Rico - way to go pinhead...

UD

 

[Lavey29]

I thought about going with a battery wall backup but to expensive so will probably go with a generator also or maybe a Goal O solar back up. I wish there was an inexpensive way to tie in to my solar panels. Perhaps with the goal 0 I can modify something together.

 

[HydroSkreamin]

I like how you appropriately named them.

Sucks that you guys have to deal with that, good on you for coming up with a safe (albeit expensive) solution.

 

[monkeyswrench]

I'm about as technologically advanced as a sledge hammer...holy shit that's a lot of blinking lights and wires!

 

[Uncle Dave]

I'm about as technologically advanced as a sledge hammer...holy shit that's a lot of blinking lights and wires!

Yeah - so between the 2 buildings we got about 1000 machines to power up and down every time they kill and or restart power- takes 5 guys 2.5 hours to power the thing up and down safely every time - of course you need to do that before the day shift starts

PG&E mumbles out some schedule thats about as transparent as an alan greenspan speech - of when they going to cut power - then they do the same thing on a re engerize, and sometimes it randomly comes up undoing the work that 5 guys each dealing with 200 machines just did a few hours ago.

We got hundreds of man hours into this thing now -

Giving awsy 2 snow days W full pay to 200 people is a kick the balls

telling the same great 200 people that the next ones come out of their vacation bank doenst go over so well.

Sending 200 people home at noon for planned power outage that then doesnt happen for 4 more hours because somebody changed their mind and didnt communicate is laughable if weren't so expensive.

UD

 

[Uncle Dave]

I like how you appropriately named them.

Sucks that you guys have to deal with that, good on you for coming up with a safe (albeit expensive) solution.

Seems appropriate - yeah all in this was about 10K between the gensets / custom cabling and panels.

Just as of today we got this all up and running for a cut our scheduled at 1:00

Then we got word it would be closer to 3...

We just got word it'll be closer to 4 ....

UD

 

[Uncle Dave]

I thought about going with a battery wall backup but to expensive so will probably go with a generator also or maybe a Goal O solar back up. I wish there was an inexpensive way to tie in to my solar panels. Perhaps with the goal 0 I can modify something together.

Apparently at least one brand has a 2K tie in in inverter that will give you something should the grid tank

UD

 

[monkeyswrench]

My brother is the technologically advanced one, I have a wonderful selection of hammers. When he worked at Earthlink in the 90s, I went to their server...building? Warehouse?...Farm? It had two Cat power units on a tractor trailer, and another trailer was a chiller unit with ducting. Even then, power must have been a major factor.

 

[Uncle Dave]

Noisy mofos

 

[regor]

 

[BajaMike]

After a few days of this crap we put together our own setup to at least keep our IT going during these blackouts.

We built our own sub panel and are up and running with shitshow 1&2.
We streamlined the server room so we can run on a min compliment and have the current setup pulling about 12KW out of an available 24 so we have some over head left.

We've lost a ton of productivity over this BS and each time these guys do this we blow at least one server so being able to cut over and do it on our own terms will let us keep working and hopefully keep our stuff alive.

Newsoms worried about shampoo bottle and half the states out of work and on par reliability wise with puerto Rico - way to go pinhead...

UD

View attachment 810663 View attachment 810664 View attachment 810665 View attachment 810666 View attachment 810667 View attachment 810668

Nice looking, big ass server room.

I built one about half that size at my old Austin company’s corporate headquarters......never had to shut it down in 5 years.....pain in the ass if you have to.

 

[Flying_Lavey]

My brother is the technologically advanced one, I have a wonderful selection of hammers. When he worked at Earthlink in the 90s, I went to their server...building? Warehouse?...Farm? It had two Cat power units on a tractor trailer, and another trailer was a chiller unit with ducting. Even then, power must have been a major factor.

As with any technology, it's gotten more efficient and compact. I serviced the HVAC on a building in San Luis Obispo that handled all the switchgear for the tie between CenturyLink's coastal hub and the branch that ran on the floor of the Pacific. MASSIVE 2 story building. They only used about 1/8 of the square footage. Anymore. They had just designed and built a chilled water system for the building when the first major advancements in size and efficiency started happening. They now have a couple MILLION dollars worth of HVAC equipment sitting in their yard and on their roof mothballed doing absolutely nothing cause it costs too much to run. If I'm remembering correctly, each chiller was 400 tons. They had 4 of them. The equipment they were running when I left Source a little over 2 years ago only required 20 tons to keep cool. And I was told it does about 15 times the work at least of that original switchgear.

(I may not be using the proper computer terms)

Sent from my LM-G710VM using Tapatalk

 

[lIQUIDATEDdAMAGES]

When we built out our new office a couple years ago we had it set up with a shore power hook up that we can plug our 45kva generator into. Now when the power goes out we can pull it out of the warehouse, plug it in and be back online within about 15 minutes. If we hurry the IT will remain online on the UPS and we don’t have to reboot anything server wise...


Sent from my iPhone using Tapatalk

 

[Uncle Dave]

power was supposed to be cut at 1:30 been running all day on backup - still not out.

Dave

 

[Uncle Dave]

just killed sitting in the dark on a laptop writing this. (were only powering the server room) - 7 hours from when we were told it would happen - this is the kind of communication you get with PG&E .
Good thing I came in with the crew early this morning


UD

 

[Dalton]

might want to pick up some uninterruptible power supplies, those generators can make some dirty power, the UPS's smooth the power out and keep everything up till the gen kicks on, all our servers have UPS's, Automatic Transfer Switches and auto start generators, pretty seamless

 

[paradise]

What does your business do? That’s a ton of supermicro servers. Usually see DELL or HP in those numbers.

 

[Yellowboat]

I just got off the phone with a guy that has known me since I could walk. His power went out a little before 5 tonight. He is not far from Placerville.

 

[BajaMike]

PG&E sucks ass. Shutting down everyone ‘s power because they don’t maintain the lines. They suck ass.

 

[rrrr]

might want to pick up some uninterruptible power supplies, those generators can make some dirty power, the UPS's smooth the power out and keep everything up till the gen kicks on, all our servers have UPS's, Automatic Transfer Switches and auto start generators, pretty seamless

That generator has less than 5% THD, and the multiple voltage switching power supplies in servers can handle that easily. In the early 2000s, that wouldn't be the case, small generators were dirty as hell and SPS weren't able to provide a stable output from them.

 

[Dalton]

That generator has less than 5% THD, and the multiple voltage switching power supplies in servers can handle that easily. In the early 2000s, that wouldn't be the case, small generators were dirty as hell and SPS weren't able to provide a stable output from them.

I would still put some rack mounted UPS's in there to keep everything from going up and down

 

[Uncle Dave]

might want to pick up some uninterruptible power supplies, those generators can make some dirty power, the UPS's smooth the power out and keep everything up till the gen kicks on, all our servers have UPS's, Automatic Transfer Switches and auto start generators, pretty seamless

As RRR states these are pretty clean at 5% THD

These servers are all on UPS's and were on top of making sure the batts are pretty good so they are all protected.

UD

 

[Uncle Dave]

What does your business do? That’s a ton of supermicro servers. Usually see DELL or HP in those numbers.

We make TV and film gear and support about 200 people out of this building.

We've got 4 server rooms this size or better.

This room serves out email, marketing, phone and security systems.

Now that the power is out they are supposed to take between 1 and 2 hours to cut the local area over to 4 gigantic 1.4 MW gensets that have been brought in-theyll avoid etc transmission lines over the pass and energize the local stuff that got buried lines or is no risk.

UD

 

[rrrr]

Also back in the day, a lot of times small single phase UPSs wouldn't run on generators unless they were around 200% larger than the UPS load. Anything from 500 VA up to 20 kVA just said no. When line interactive UPSs came along, it solved that problem.

 

[BajaMike]

Good looking system....

 

[rrrr]

Good looking system....

That temporary power setup is some clean work.

 

[DaveC]

Holy smokes that’s a lot of storage.

I can tell you have you have UPS batteries. But those are meant as temporary backups so the servers can be shut down in case of a long term outage. The batteries are meant so you don’t blow up drives in the event of sudden power outages or Brown outs.

This sucks.

Living in the third world guess we gotta start looking at permanent backup power which is costly.

But at least the weather looks nice there.

 

[Uncle Dave]

Holy smokes that’s a lot of storage.

I can tell you have you have UPS batteries. But those are meant as temporary backups so the servers can be shut down in case of a long term outage. The batteries are meant so you don’t blow up drives in the event of sudden power outages or Brown outs.

This sucks.

Living in the third world guess we gotta start looking at permanent backup power which is costly.

But at least the weather looks nice there.

Yeah we do and we needed every minute of time they bought us yesterday

They've got us on a microgrid right now - but it burps sometimes and went down Sunday. I fucked up - I trusted them.

From the minute we get shut down we've got exactly 20 minutes to the switch to geneses before we die- without an isolation xfer switch on the wall you have to move cords and its a pain

It takes one guy about 20 minutes to do it and you always lose something - or two guys 10 - so during the day its easy we've got the whole staff - at night we cant risk it so we switchover to the gensets and run till morning.

We caught it Sunday just in time - but there is no more trust in these guys and their communication sucks,

Prior to the genset addition we had taken the room up and down pre emptively 5 times in the last 3 weeks and we lost something every time - people have no idea how hard it is to spin up and shutdown all that stuff - so we just dont let the room come down and keep it all under our control.

This is the new normal - Id get a UPS if I were thinking I may need one.

UD