Datacenter

Selection

Adam Rothschild <asr@packet.net>

Open-IX - Americas Interconnection Summit

San Diego, CA - April 2015

DISCLAIMERS

○ Personal experiences and opinions only, not those of the Open-IX Association, working

groups, or any employer.

○ Not intended as legal advice. Datacenter contracts are extremely complex and require

review by counsel.

○ Datacenters presented are completely fictitious. Any resemblances, living or dead, are

coincidental.

WHY DATACENTERS

MATTER○ Significant operational expenditure, often exceeding cost of IP transit or

interconnection.

○ Often selected by the “network person” or “vendor negotiator” (hence, this audience)

○ Can be a major source of downtime if planned poorly.

○ Power, Plumbing, and Ping is not an “attractive” topic like “cloud” or “SDN”, though it

underpins it; therefore not often discussed around these parts.

○ Lack of best-practice sharing among operators. Not a germane topic for a NANOG,

etc.

WHEN YOU’RE NOT

ALIGNED

CENSORED

WHEN YOU’RE NOT

ALIGNED: SANDY EDITION

Force majeure, or poor site selection by the

customer?

WHEN YOU’RE ALIGNED

VENDOR POSITIONING

RETAIL WHOLESALE

CHARACTERISTICS OF RETAIL

DATACENTER OR MMR

WHOLESALE

○ Power is billed at a fixed MRC, based on breakered (or capped) capacity.

○ Common increments of space are RUs, racks, or private cages.

○ Diverse meet-me room, with access to dozens of IP, lit services, dark fiber providers.

○ Cross-connection is a large profit center for the operator, and may carry significant MRC.

○ Access to 24 x7 remote hands for basic troubleshooting, rack & stack, managed services.

○ Limited options for power redundancy beyond N or N+1; so called “primary” and “redundant” power

feeds may terminate on common infrastructure.

CHARACTERISTICS OF

WHOLESALE DATACENTER

WHOLESALE

○ Power is metered and billed based upon consumption (KW/hr or mean KW/month).

○ Common increments of space are private cages or partitioned suites.

○ Limited providers on-site, but should connect to one or more dark fiber provider(s) with entrance

diversity. Tethering is common. May also offer extended cross-connect service to nearby MMR(s).

○ Cross-connection is not a profit center for the operator; commonly NRC to cover time & materials.

○ Access to remote hands may be limited. Clients typically provide their own operational staffing.

○ Greater options for power diversity (e.g. 2N[+n] designs). Some customization on power, cooling, and

rack designs to meet customer’s needs.

SCOPE OF THIS TALK

WHOLESALE

○ Most relevant to wholesale colocation, in the hundreds of KWs and larger.

○ Some elements are applicable to retail colocation, e.g. a cage, though you’re probably wasting your

and your account rep’s time if you’re using this method to quote out a single cabinet.

○ Some elements are applicable to larger wholesale suites/pods in the MWs range, however further due

diligence may be helpful.

○ We’re not discussing greenfield construction, though the TCO comparatives may help.

RFP: INTRO AND BASICS

WHOLESALE

○ RFP should be short and sweet, while explaining your requirements. Should not exceed several

pages of core requirements.

○ Should be conducive to vendor providing concise, binary, responses, without the hassle of flowery

marketing narratives.

○ Should clearly state nature of customer’s business and use case; establish expectations for a

response and go-live date.

○ Should clearly establish search radius, leave no room for interpretation as to whether or not a

particular location is in-scope for consideration.

RFP: BUILD AND POWER

WHOLESALE

○ Explain clearly what type of equipment racks or cabinets you need to place, and quantity. Floor plan

and rack layout can be helpful as attachments. Establish provider’s “sweet spot” for density (W/ft^2 or

KW/cabinet) so you can tweak your layouts appropriately.

○ Establish “day one” and future (12mo, 24mo, …) power requirements, in terms of both base commit

and ceiling. Negotiate a ramp beneficial to both parties, if you’re not using all power initially.

○ Establish how power is metered and invoiced; base commitments vs. variable.

○ Establish mean Power Usage Effectiveness (PUE) for facility, and its impact on how power is billed.

○ Establish costs for installing incremental cabinets, NEMA receptacles.

RFP: COMMERCIAL TERMS

WHOLESALE

○ Establish term length. Long terms (>= 5 years) ensure stability and cost predictability for the

purchaser, as well as certainty for the vendor.

○ Establish YoY escalations, if applicable. Solidify language for additional pricing increases, e.g. notice

period, commensurate to increased costs from power utility.

○ Establish “look back” period for incorrect billings. Ensure it’s bilateral; same period for billing disputes

initiated by customer, and billing corrections issued by vendor…

RFP: CABLING

WHOLESALE

○ Establish rates (NRC and MRC) for all common types of cross-connections: Cat5e, SMF; are

increases permitted during the contract term?

○ Establish policies for running your own data connections inside your datacenter space, using your own

materials and labor.

○ Establish facility’s policy on unionized labor. Is there any type of non-electrical work (e.g. bolting of

cabinets, intra-suite data cabling) that you’re not allowed to perform by yourself?

○ Develop with the facility a mutually agreeable plan for wire tray placement, rack placement, cable

plant, etc.

RFP: COMMUNICATIONS

WHOLESALE

○ “Provider ABC is on-net” is, itself, not a complete answer:

○ Develop list of on-site IP and telco providers you’re looking to purchase from.

○ Drill down to determine which building entrance[s] are utilized; underlying dark fiber providers and

routes are into the street. Also look for convergence points further away.

○ Vendor KMZ files overlayed into Google Earth are your friends. Some vendors have commissioned

reports to help with this planning, also.

○ Discuss relative strengths and weaknesses of aerial vs. buried fiber.

RFP: OIX-2 CERTIFICATION

WHOLESALE

○ My employer (Packet) asks this of all RFP respondents.

○ We view OIX-2 as a good indicator of transparency in operations, willingness to promote a diverse

carrier/interconnection ecosystem.

○ We understand that certifying every single site in a provider’s portfolio won’t happen overnight,

however [un]willingness to certify a particular site is a telling conversation.

RFP: OTHER IDEAS

WHOLESALE

○ Review history of planned maintenance events for past 12+ months, and upcoming 3+ months.

○ Review history of unplanned site outages for past 24+ months.

○ Establish access methods/controls into the datacenter, and private cage or suite; multiple “factors”.

○ Review policy on CCTV placement, retention, customer access. Discuss placement of your own

cameras, to augment the provider’s systems.

○ Confirm that temperature and RH will be maintained in accordance with (e.g. ASHRAE) norms.

○ Review SLA threshold and remedies for loss of power or cooling.

FLOOD MAPS

Source: FEMA MSC

SUBJECTIVE ANALYSIS

WHOLESALE

SUBJECTIVE ANALYSIS (2)

WHOLESALE

Documents are here:

RFP: http://baremet.al/packet-dc-

rfp

TCO Model:

http://baremet.al/packet-dc-tco

Questions or comments:

asr@packet.net