CIT 470: Advanced Network and System Administration Slide #1

CIT 470: Advanced Network and System Administration

Data Centers

CIT 470: Advanced Network and System Administration Slide #2

Topics

Data Center: A facility for housing a large amount of computer or communications equipment.

1. Environment

2. Power

3. Racks

4. Blackbox

CIT 470: Advanced Network and System Administration Slide #3

Environmental Requirements

Temperature: 64-72F (17-22C)Temperature inside case around 40F higher.

Chips and hard disks fail around 120F.

Humidity 35-65%Too low: static discharges.

Too high: water condenses, short circuits.

PowerReliable, conditioned power.

Physical SecurityPrevent accidents, hacking, theft, vandalism.

CIT 470: Advanced Network and System Administration Slide #4

Environment Features

HVAC: Heating, Ventilation, Air Cond

UPS: Power conditioning, <1hr battery

Generator: For long term power outages.

Accessibility: For large equipment.

Card Lock: SAs only, records entrances/exits.

Fire Suppression System: Not water based.

Humidity/Water Detection

CIT 470: Advanced Network and System Administration Slide #5

HVAC

HVAC systems can fail due toPower loss.

Coolant leakage.

Mechanical failure.

Detect HVAC failures early.Many HVACs have sensors to report failure.

Temperature monitor in data center otherwise.

HVAC maintenancen+1 redundancy to avoid immediate disaster.

Usually handled by external contractor.

CIT 470: Advanced Network and System Administration Slide #6

Space

AislesMust be wide enough to move equipment.

Hot spotsResult from poor air flow.

Servers can overheat when avg room temp low.

Work spaceA place for SAs to work on servers.

Desk space, tools, etc.

CapacityEnsure that you have enough room to grow.

CIT 470: Advanced Network and System Administration Slide #7

Power

• UPS provides power.• Generator provides

auxiliary power.• ATS switches to

generator if building power fails.

Liebert UPS and battery bank.

CIT 470: Advanced Network and System Administration Slide #8

Power Distribution

Under floor power– Susceptible to water; requires water sensors.– Overhead power bus is preferrable.

Overhead Power Bus– Drop power down to each rack individually.– Don’t run power cords between racks.– Each rack will have its own PDU.

CIT 470: Advanced Network and System Administration Slide #9

Power Distribution Units (PDU)

• Different power sockets can be on different circuits.

• Individual outlet control (pwr cycle.)

• Current monitoring and alarms.

• Network management (web or SNMP.)

APC Power Distribution Unit

CIT 470: Advanced Network and System Administration Slide #10

The Power Problem

• Servers require much more power today.

• Blades allow more servers per square ft.

• Upgrades may have to wait for electricity.

• Power is a major data center cost– $5.8 billion for server power in 2005.– $3.5 billion for server cooling in 2005.– $20.5 billion for purchasing hardware in 2005.– Google, MS building centers near cheap power.

CIT 470: Advanced Network and System Administration Slide #11

Physical Security

• Keycard + code to restrict who can access.

• Biometrics are becoming popular alternative.

• Electronically log all accesses.

• Surveillance cameras.

• No windows.

• No floor/ceiling access.

CIT 470: Advanced Network and System Administration Slide #12

Racks

• 19” rack standard– EIA-310D– Other standard numbers.

• NEBS 21” racks– Telecom equipment.

• 2-post or 4-post• 1U = 1.75 in• Air circulation (fans)• Cable management• Doors or open

CIT 470: Advanced Network and System Administration Slide #13

Rack Purposes

Organize equipment– Increase density with vertical stacking.

Cooling– Internal airflow within rack cools servers.– Airflow determined by arrangement of racks.

Wiring– Cable guides keep cables within racks.

Power infrastructure– PDUs in racks distribute power to servers.

CIT 470: Advanced Network and System Administration Slide #14

Buying a Rack

Buy the right size– Space for servers.

– + space for power, patch panel, etc.

Be sure it fits your servers.– Appropriate mounting rails.

– Shelves for non-rack mountable servers.

Physical/environment security– Locking front and back doors if needed.

– Sufficient power and cooling.

– Power/environment monitors if needed.

CIT 470: Advanced Network and System Administration Slide #15

Wiring

It’s important to be neat.– So you can get the cable for the right server.

– Label both ends of each cable.

Prewiring– Run power, net, serial cables when rack setup.

– Greatly reduces time to setup a new server.

Hiding messiness isn’t being neat.– Raised floors can hide cables.

– Some sites don’t remove old cables to avoid accidentally taking down the wrong server.

CIT 470: Advanced Network and System Administration Slide #16

Project Blackbox

• Data Center in a shipping container.– Google– Sun

• PB of storage.

CIT 470: Advanced Network and System Administration Slide #17

References1. “19-inch Rack”, http://en.wikipedia.org/wiki/19-inch_rack, 2007.2. 42u, “Top 5 Things to Keep in Mind When Buying a 19” Rack,”

http://www.42u.com/server-rack-article.htm, 2007.3. Mark Burgess, Principles of System and Network Administration, Wiley,

2000.4. Data Center Journal, http://www.datacenterjournal.com/, 2007.5. “Data Center,” http://en.wikipedia.org/wiki/Data_center, 2007.6. Aeleen Frisch, Essential System Administration, 3rd edition, O’Reilly, 2002.7. Dan Goodin, “IT confronts the Data Center Power Crisis,”

http://www.infoworld.com/article/06/10/06/41FEgreen_1.html, 2006.8. Hurgh, “Rack Layout,”

http://www.hurgh.org/articles.php?article=rack_layout, 2007.9. “Project Blackbox,” http://en.wikipedia.org/wiki/Project_Blackbox, 2007.10. Thomas A. Limoncelli and Christine Hogan, The Practice of System and

Network Administration, Addison-Wesley, 2002.11. Evi Nemeth et al, UNIX System Administration Handbook, 3rd edition,

Prentice Hall, 2001.