What is HA/HD you may ask.
It’s our way of describing two of Liberty’s most important attributes.
HA = High Availability. We’ve built redundant systems to ensure that our customers enjoy a resilient environment that can withstand power outages, carrier outages, hardware failures, and other types of problems that could potentially take a data center down.
Our original facility design had us using one huge generator per data room. And while having back-up generator capacity is extremely important, it’s more important that your back-up generator starts every time you need it. Take the case of 365 Main, a data center in San Francisco that hosted for CraigsList, Technocrati, Yelp and others. In 2007, their generators failed to start during a Pacific Gas & Electric power outage and they and their customers were down for several hours. The cause? The generators didn’t start. See the article here: http://www.datacenterknowledge.com/archives/2007/07/24/generator-failures-caused-365-main-outage/
So, we said, let’s not trust the availability of power to our customers to a starter on a single generator, so we decided to bring two in. So we mitigate the risk of a starter failure by having two – and one generator by itself can support the entire facility. By the way, if you weren’t around when we did bring in the generators, here’s a photo you might like:
Yes, that is a flying generator.
And having the back-up generator plant is only part of the story. What’s the benefit of having redundancy if it’s not maintained? Pat Turner, our CTO spent the better part of last week working on our semi-annual electrical systems maintenance and part of that maintenance was to use our own load bank to perform the National Fire Prevention Association schedule used by hospitals and other high availability entities to ensure that not only will the generators start, but that they will support load. And because we have our own load bank, we perform this test whenever we want, not just when we have the budget.
HD stands for High Density. It was important for us to be in a position to say yes to customers, not no. We had heard “no” a lot when we asked to get more power in our racks at other hosting facilities over the years. Many leading commercial facilities across the country only allow a customer to put 4 kW in a rack. So what do they do with a customer with a blade enclosure that’s only 21u deep and draws 8 or 9 kW? They charge them for more space – more than they need. We don’t like charging for vapor, we’d prefer to provide value, so we built an environmental infrastructure that can support more – a lot more. In fact, we have one customer running 19 kW in a single rack.
Now not everyone is ready for high density computing today, so as we go forward with Data Room B we are looking at our past experience and comparing that to future projections of where IT delivery systems. Predicting the future is part art and part science and we’re going to take our best shot. Perhaps you can help. If you haven’t virtualized your environment yet, do you plan to do so? Tell us more about where you think you’re heading – fewer, more powerful servers? What would lead you to downsize your hardware but to increase your hardware investment? We’re interested in your input, anytime.
How can you say that one generator can run the entire data center? Do you guys know what you’re talking about?
If you have a 750KW generator with a 650KVA UPS? In the middle of the summer, you need to allow for an operating PUE of 2.0 or higher to handle the loss in the UPSs & HVAC load – which means that you need 1300 KVA to operate the data center. Sounds like you need both generators to run the data center at load. That’s N, not N+1 on the generators.
Well, John, my wife has often asked me the same thing!
But actually you raise a great question.
First, your PUE assumption is absolutely correct – our PUE today hovers around 2.0 and if our UPS usage was at maximum, yes, we would need a generator solution of 1300 kVA.
However, there are a couple of assumptions that you didn’t take into account.
First, counting the total load on our redundant UPS systems and the building mechanical load, we are below 500 kVA in the total power to run the facility and our present data room floor is approaching 100% occupancy. At today’s load capacity, one of our two 750 kVA generators will easily maintain power to the entire facility. So, with the expected power requirements to support our customers, we are actually more like 2N+ in available back up power.
Second, when we built the facility, we designed in scalable components that would allow us to maintain the N+1 generator approach. So we already have the paralleling switch gear sized to accommodate four total generators with all the panel boards, breakers, and conduit infrastructure in place.
This gear is a little unusual in a commercial facility, but it allows us to run multiple generators as a single utility feed to the facility and the Kohler systems provides us with a very clean interface to manage their operation.
And we’ve already prepped the pads and conduit in the generator yard to accommodate the additional generators we’ll be adding to the facility.
And as an extra added bonus, the Kohler switch gear controls our on-site load bank which allows us to test these generators (no matter how many we have) with varying load without having to bring in expensive portable load banks or (horrors!) put the data room on generator just to test their ability to handle load (yes, we did hear that someone was doing that).
So, as we look at going forward, one of our main planning issues is not determining if, but when we need to bring in the additional generators to support increasing load in the facility. If growth dictates, we’ll end up with 3000 kW of generator capacity which would support a UPS load of about 1300-1500 kVA, depending upon the PUE and other factors as you rightly point out. But suffice it to say, regardless of how our customer power needs grow, we’ll never get ourselves in any less than an N+1 situation.