PowerPoint Presentation

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#1SPARC T5 Servers Deep Dive NDA Part 1Insert Presenters Name HereInsert Presenters Title Here

0.20

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#2The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Scales to 8 sockets using directoryMinimizes latency Avoids congestionMaximize bandwidth

Double cores and cacheBalance single thread and throughput Dynamically thread

Oracle workloads Engineered SystemsExtends on-chip crypto acceleration RAS

Maximizes peak performanceManages thermal and current loadsScales elastically

OptimizeSystemsMultiply Performance SPARC T5Scale EfficientlyAdvance Power ManagementDesign Objectives Achieved Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

Next Generation SPARC T5 ServersFaster. Optimized. Secure.SPARC T5-1B

SPARC T5-8SPARC T5-4

SPARC T5-2

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#55SPARC T5-1BSPARC T5-2SPARC T5-4SPARC T5-8ProcessorSPARC T5 3.6GHzSPARC T5 3.6GHzSPARC T5 3.6GHzSPARC T5 3.6GHzMax Processor Chips1248Max Cores/Threads16, 12832, 25664, 512128, 1024DIMM Slots163264128Max Memory256GB512GB2TB4TBDrive Bays2688I/O Slots2 x PCIe 3.0 EM, 2 NEM,1 FEM slots8 LP x8 PCIe 3.0, 4 x 10GbE ports16 LP x8 PCIe 3.0,4 x 10GbE ports16 LP x8 PCIe 3.0,4 x 10GbE portsForm Factor/RUBladeRack 3RURack 5RURack 8RUSPARC T5 ServersProduct Line Overview

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#If customers are familiar with the SPARC T4 line of products, then the actual SPARC T5 systems should seem very familiar. The main change between the 2 generations is the processor and motherboard. 6

New l SPARC T5-1B Blade ServerCompute1x SPARC T5 16-core CPU16x DDR3 DIMMs, up to 256GB memoryI/O and Storage2x hot-plug PCIe 2.0 x8 Express Modules2x 2.5 SAS HDD or SSD drivesAvailability and ManagementBuilt-in RAID 0, 1Hot plug disksIntegrates with Sun Blade 6000 network architectureOracle ILOM service processor

Next Generation 1-socket SPARC Blade Server

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#While 32GB DIMMs will work on the blade, only the 16GB DIMM is being qualified. This is to address cost issues.

T5-1B BoardT5 CPU16 DDR3DIMMsServiceProcessorREMBoBsPCIeSwitchCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

New l SPARC T5-2 Enterprise Server

Compute2x SPARC T5 16-core CPU32x DDR3 DIMMs, 256GB or 512GB memoryI/O and Storage8x PCIe 3.0 x8 slots4x 10G-baseT ports6x 2.5 SAS HDD or SSD drivesAvailability and ManagementBuilt-in RAID 0, 1Hot-plug disksHot-swap and redundant fans and power suppliesOracle ILOM service processor

Next Generation 2-socket SPARC Server

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#8GB and 16GB DIMMs will be support. Support for 32GB DIMMs is TBD.

T5-2 Chassis

T5 CPUServiceProcessorMemoryRisersCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

T5-2 FrontDISK 1DISK 0DISK 3DISK 2DISK 5DISK 4RFID/SerialNumber2x USB 3.0PortsHD-15 VGA PortDVDLocatorLED/ButtonFault LEDPowerButtonStatusLEDOver Temp IndicatorStatus LEDsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

T5-2 Rear2x USB 3.0HD-15 VGA Port4x 10GbE PortsSP SerialSP Network 10/100PCIe 1PCIe 2PCIe 3PCIe 4PCIe 5PCIe 6PCIe 7PCIe 8System RearIndicatorsAC0AC1PSU 0PSU 1Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#New l SPARC T5-4 Enterprise ServerCompute4x SPARC T5 16-core CPU64x DDR3 DIMMs, 1TB or 2TB memoryI/O and Storage16x PCIe 3.0 x8 slots, w/ carriers4x 10G-baseT ports8x 2.5 SAS HDD or SSD drivesAvailability and ManagementBuilt-in RAID 0, 1Hot-plug disks, PCI cardsHot-swap and redundant fans and power suppliesOracle ILOM service processor

Next Generation 4-socket SPARC Server

FrontRear

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#The Processor Modules (PM) are designed for hot-plug, but that functionality will be delivered post-RR.16GB and 32GB DIMMs are supported.

SPARC T5-4 Front PanelDISK 0DISK 1DISK 2DISK 3DISK 4DISK 5DISK 7DISK 6PSU 0PSU 1

LocatorLED/ButtonOver Temp IndicatorProcessor ModuleFault LEDPowerButtonStatus LEDProcessor Module Status LEDsMain Module(Entire Board)Dual USB 3.0PortsSP Serial PortRFID/SerialNumberPSU StatusLEDsMain Module &SP Status LEDsHD-15 VGA PortRear Fan/EM IndicatorPM 0PM 1MainModuleCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#14

AC3AC0SPARC T5-4 Rear Panel

Fan Module 0Fan Module 2Fan Module 3Fan Module 4PCIe 1PCIe 2PCIe 3PCIe 4PCIe 5PCIe 6PCIe 7PCIe 8PCIe 9PCIe 10PCIe 11PCIe 12PCIe 13PCIe 14PCIe 15PCIe 16Rear I/OModule (RIO)AC3 OKLEDSystem RearIndicatorsPCIe CarrierHot-Plug Button, LEDsFan Module 1HD-15 VGA PortAC0 OK LEDSP SerialSP Network 10/1002x USB 3.04x 10GbE PortsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#15New l SPARC T5-8 Enterprise Server

Compute8x SPARC T5 16-core CPU128x DDR3 DIMMs, 2TB or 4TB memoryI/O and Storage16x PCIe 3.0 x8 slots, w/ carriers4x 10G-baseT ports8x 2.5 SAS HDD or SSD drivesAvailability and ManagementBuilt-in RAID 0, 1Hot-plug disks, PCI cardsHot-swap and redundant fans and power suppliesOracle ILOM service processor

Next Generation 8-socket SPARC Server

FrontRear

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#The Processor Modules (PM) are designed for hot-plug, but that functionality will be delivered post-RR.16GB and 32GB DIMMs are supported.

Comparing T3, T4, and T5SPARC T3SPARC T4SPARC T5Processor Clock1.65GHz2.85GHz, 3.0GHz3.6GHz# of cores16816Core ArchitectureS2S3# of memory controllers24DIMMs/BoB2 BoBs/memory controller4 DIMMs/BoB4 BoBs/CPU socket2 BoBs/memory controller2 DIMMs/BoB8 BoBs/CPU socketPCIe Gen support2.03.0PCIe Card Form FactorBlade: Express Modules1 and 2 socket rack server: LP4 socket server: Express ModulesBlade: Express Modules1-2 socket rack server: LP4-8 socket rack server: LP on carrier cardCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SPARC T4-4 Comparison SPARC to T5-4

FeatureSPARC T5-4SPARC T4-4Form Factor5RU, 28 deep5RU, 28 deepCPU4x SPARC T53.6 GHz (512 threads)4x SPARC T43.0 GHz (256 threads)MemoryDDR3, 2TB MAX64x SlotsDDR3, 2TB MAX64x slotsNetwork4x 10GbE 4 x 1GbE + 8x 10GbE (XAUI) Requires 2 Separate QSFP ConnectorsInternal StorageUp to 8 x 2.5 SAS 3.0 or SSD, hot-plugUp to 8 x 2.5 SAS 2.0, can use up to 4x SATA SSDs, hot-plugRemovable Media1x DVD-RW (via rKVMS; not local)1x DVD-RW (via rKVMS; not local)Serial1x RS-232, 4x USB, 1x VGA1x RS-232, 4x USB, 1x VGAPCI Express slots16x x8 slots (Hot-Plug Low Profile slotswith carrier card, ver 3.0)16x x8 slots (Hot-Plug Express Module slots,ver 2.0)Power Supply2 x 3000 Watt AC, N+NRedundant/Hot-Swap4 x 2060 Watt AC, N+NRedundant/Hot-SwapFans5 x Redundant Hot-Swap5 x Redundant Hot-Swap

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#18T5-4 and T5-8 Processor Module

32x DIMMs16x per T5 T5 Processor 0T5 Processor 1Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#C0C1C2C3C4C5C6C7SSI, Present, 12C(FPGA connectivity)FPGA-T5-48C0C1C2C3C4C5C6C7SSI, Present, 12C(FPGA connectivity)FPGA-T5-48C0C1C2C3C4C5C6C7SSI, Present, 12C(FPGA connectivity)FPGA-T5-48C0C1C2C3C4C5C6C7SSI, Present, 12C(FPGA connectivity)FPGA-T5-4866666666555555554444444433333333222222221111111100000000PM3PM2PM1PM0C6C7C4C5C2C1C0C1666666555555444444333333222222111111000000PM3PM1PM0C6C7C2C1C0C16666555544443333222211110000PM3PM0C6C7C0C166554433221100PM0C0C1PFM not connected (required for airflow/EMI onlyCL routingCL routingCL routingCL routing1 CL between the 8 nodes1 CL between the 6 nodes2 CL between the 4 nodes2 CL between the 2 nodesT5-8 (8P option)T5-8 (6P option)T5-8 (4P option)T5-8 (2P option)C001C101PMOC201C301PM1C401C501PM2C601C701PM3Part #Sw 0Partition #Switch 1Partition #Switch 2Partition #Switch 2Part #Sw 4MP8?PCIe switches on MB in commonT5-4/8 Main ModulePCIe upstream routing (PSR)0000112221103331C001C101PMOC201C301PM1PFM2C601C701PM3Part #Sw 0Partition #Switch 1Partition #Switch 2Partition #Switch 2Part #Sw 4MP8?PCIe switches on MB in commonT5-4/8 Main ModulePCIe upstream routing (PSR)000011223331C001C101PMOPFM1PFM2C601C701PM3Part #Sw 0Partition #Switch 1Partition #Switch 2Partition #Switch 2Part #Sw 4MP8?PCIe switches on MB in commonT5-4/8 Main ModulePCIe upstream routing (PSR)00003331C001C101PMOPFM1PFM2PFM3Part #Sw 0Partition #Switch 1Partition #Switch 2Partition #Switch 2Part #Sw 4MP8?PCIe switches on MB in commonT5-4/8 Main ModulePCIe upstream routing (PSR)0000PFM1PFM (not required)Or Empty PM slotPFM (not required)Or Empty PM slotNote: the T5-8 2P option isnot a shipping configurationPFM1PFM2PFM2T5-8 Processor ConnectivityCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#C0C1C2C3C4C5C6C7SSI, Present, 12C(FPGA connectivity)FPGA-T5-48C0C1C2C3C4C5C6C7SSI, Present, 12C(FPGA connectivity)FPGA-T5-486666555544443333222211110000PM1PM0C6C7C0C166554433221100PM0C0C1CL routingCL routing2 CL between the 4 nodes2 CL between the 2 nodesT5-8 (4P option)T5-8 (2P option)C001C101PMOC601C701PM3Part #Sw 0Partition #Switch 1Partition #Switch 2Partition #Switch 2Part #Sw 4MP4PCIe switches on MB in commonT5-4/8 Main ModulePCIe upstream routing (PSR)00003331C001C101PMOPFM1Part #Sw 0Partition #Switch 1Partition #Switch 2Partition #Switch 2Part #Sw 4MP4PCIe switches on MB in commonT5-4/8 Main ModulePCIe upstream routing (PSR)0000PFM1T5-4 Processor ConnectivityCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#x2x8x8LSI SASx1x1Disk1Disk2x4NEM0NEM1PCI-EM1x8x8FEM0 FEM1 Nalia Niantic 1x4? hereUSB 3.0Host CtrlUSB 1.0Hub CtrlSideband MgmtUSB 2.0 EnetStorageEmulexPilot 3VGA(HD15)Front UCP (Dongle Cable)Serial(RJ45)Ethernet Mgmt(to CMM)NEM0(4:7)

BoB

BoB

BoB

BoB

BoB

BoB

BoB

BoBDIMMDIMMDIMMDIMMDIMMDIMMDIMMDIMMT5CPU0LPC12CFPGAHost &Data FlashTPMDual GigE10/100/1000x8NEM0(0:3)NEM1(0:3)NEM1(4:7)1x41x41x4PCIeSwitch 0PCIeSwitch 1CPUDC/DCsCPUDebugPortPCIex8PCIe Gen3PCIe Gen2DBGSP ModuleUSB 3.0USB 2.0USB MIDPLANET5-1B Block DiagramUSB 1.1 Keyboard/MousePCI-EM0USB 01Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#T5-0CPUDC/DCsCPUDebugPortPCIeUSB 2.0 Storageget rid of all inside small boxesEnetMgmt10/100SerialMgmt

BoB

BoBDIMMDIMMT5-1CPUDC/DCsCPUDebugPortPCIe

BoB

BoBDIMMDIMM

BoB

BoBDIMMDIMM

BoB

BoBDIMMDIMM

BoB

BoBDIMMDIMM

BoB

BoBDIMMDIMMHost &Data FlashTPMDBGService ProcessorSP ModuleFRUIDDRAMSPIFlashNANDUSB 3.0 HostVGADB15Sideband MgmtUSB0USB1VGAUSB 2.0 HubUSBPCIeSwitch 0PCIeSwitch 1Slot 1 (8)Slot 2 (8)Slot 3 (8)Slot 4 (8)Slot 5 (8)Slot 6 (8)Slot 7 (8)Slot 8 (8)x8x8x8x8x8x8x8x8x4x1x1HDD0HDD0HDD0HDD0HDD0HDD0SATA DVDUSB 3.0HubUSBVGASAS/SATAIO ControllerSAS/SATAIO ControllerQuad 10Gig Enet

BoB

BoBDIMMDIMM

BoB

BoBDIMMDIMMx4x4FPGAUSB2USB3

REAR IOFAN BOARD

x4T5-2 Block DiagramInternal USBREAR IO BoardUSB 1.1 Keyboard MouseVGA0101Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#DBG = System debug assist port of FPGA, for bring up of serial or USB port23SAS 2308SAS1SW6DC-DCCONVERTERSCLOCK SYNTH.&BUFFERSSW1SW2SW3SW0Debug Conn4-DISK BACKPLANEDISKS 4-7SBP1SBP04-DISK BACKPLANEDISKS 0-3VIDEOMUXSerialMUXSERVICE PROCESSORSPFPGA, TOD, TPM10GB NIC 010GB NIC 1RIO Rear IO Module[VGA/DB15][Serial Mgmt][Enet Mgmt 10/100]Quad 10Gig Enet (10GBase-T Copper)1x USB (Gen2)TO RIOTO RIOEnet MgmtNC-SI2x USB (Gen3)[VGA/DB15][Serial Mgmt]HDD[4:7]HDD[0:3]MP4, MP8T5-4 or T5-8MidplaneCPUPCIE PORTSCPUPCIE PORTSCPUPCIE PORTSCPUPCIE PORTS1234567891011121314CPUPCIE PORTS1516SW4EBPCI Express BackplaneMONITOR &CONTROLMMMain ModuleMBMotherboardT5-4 & T5-8 I/O Block DiagramRIONet 0REAR USBMOST CTRLLR.2x USB (Gen3)SAS 2308SAS0FRONT USBHOST CTRLLR.FIO + VGAFRONT IOVGA, SERIAL MGMT, USBx2 USB PortsDebug ConnSW5Net 1Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#CLR0CLR0Memory Processor Module Block Diagram (T5-8 and T5-4)L0FSR0L1FSR1L0FSR2L1FSR3L0FSR4L1FSR5L0FSR6L1FSR7T5 (CM1)MCU0MCU1MCU2MCU3IOS0IOS1CLR1CLR2CLR3CLR4CLR5

BoB0C C0 1

BoB2C C1 0

BoB4C C0 1

BoB6C C0 1

BoB1C C0 1

BoB3C C1 0

BoB5C C1 0

BoB7C C1 0CLR6CMO / MCU0 / L0 = FSR0 = BOB0CMO / MCU0 / L1 = FSR1 = BOB1CMO / MCU1 / L0 = FSR2 = BOB2CMO / MCU1 / L1 = FSR3 = BOB3CMO / MCU2 / L0 = FSR4 = BOB4CMO / MCU2 / L1 = FSR5 = BOB5L0FSR0L1FSR1L0FSR2L1FSR3L0FSR4L1FSR5L0FSR6L1FSR7T5 (CM0)MCU0MCU1MCU2MCU3IOS0IOS1CLR1CLR2CLR3CLR4CLR5

BoB2C C0 1

BoB1C C1 0

BoB4C C0 1

BoB5C C0 1

BoB3C C0 1

BoB0C C1 0

BoB7C C1 0

BoB6C C1 0CLR6CMO / MCU0 / L0 = FSR0 = BOB0CMO / MCU0 / L1 = FSR1 = BOB1CMO / MCU1 / L0 = FSR2 = BOB2CMO / MCU1 / L1 = FSR3 = BOB3CMO / MCU2 / L0 = FSR4 = BOB4CMO / MCU2 / L1 = FSR5 = BOB5Xcede HD 6-Row Midplane ConnectorCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#25T5 PCIe Subsystem Dual x8 PCI Express Gen 3 ports provide 32 GB/s peak b/w Supports Atomic Fetch-and-Add, Unconditional-Swap and Compare-and-Swap operationsAccelerates virtualized I/O with Oracle Solaris VMs128k virtual function address spaces ensure direct SR-IOV access for all logical domains 64-bit DVMA space reduces IO mapping overhead, improving network performanceGuarantees fault and performance isolation among guest OS instancesSupports PCI Express Power ManagementCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

T5 PCIe Progression T4 T5PCI Express revisionGen 2 (dual x8 ports)Gen 3 (dual x8 ports)Throughput full duplex16 GBs32 GBsData Management UnitSingle shared unit for both x8 PCIe portsTwo independent units one for each x8 PCIe portPhysical Address Support44 bit48 bitTransaction Id Identification on MSI and MSI-XNoYesPCIe Atomic TransactionsNoYesTLP Processing HintsNoYes, directs data to L3 cachePCIe 2.0 compliance (ECN Internal Error Reporting)Signaled via MSI interruptSignaled via PCIe messageCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#T5 and M5 PCIe Carrier Card

x16 Connector(x8 electrical)PCIe Retimer

Air FlowCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Retimer improves PCIe signal integrity

28CPU0CPU10101PM0x8 up.Part. 0x8 up.Part. 0x8 up.Part. 0x8 up.Part. 0x8 up.Part. 0NET1 x8SAS0 x8Front USB x1Debug Slot x1SP VGA x1Rear USB x1SAS1 x8NET0 x8Debug Slot x11x8c02x8c04x8c03x8c09x8c110x8c111x8c112x8c15x8c06x8c07x8c18x8c113x8c114x8c115x8xx16x8xxMPMPEBPCI-ExpressLow ProfileHot Plug SlotsDotted linedevicesresideon RIOT5-4/8 Native 2-Socket Configuration with One Root DomainSingle non-redundant DomainSecond level Switch 6 is partitioned differently from other configsBlock fill color identifies Root Domain ownershipBlock outline color identifies association to PMSwitch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassisx8 up.Part. 0x8 up.Part. 1x8 up.Part. 0x8 up.Part. 1{Switch 0Switch 1Switch 2Switch 3Switch 4Switch 5Switch 6Native Config(T5-4 only)7ConfigIDPM1PFMx8 up.Part. 1T5-4: 1 PM PCIe ConnectivitySlot #8 lanesCPU #Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#This also hold true for a T5-8 with a single PM. A T5-8 will not be offered with a single PM, however, this diagram applies to a T5-8 with a single functional PM and the second PM has failed.PCIe slots 15 and 16 can not be used as there is no path to the switch from a CPU.

29CPU0CPU10101PM0CPU2CPU30101PM1x8 up.Part. 0x8 up.Part. 0x8 up.Part. 3x8 up.Part. 0x8 up.Part. 3x8 up.Part. 0x8 up.Part. 3x8 up.Part. 1NET1 x8SAS0 x8Front USB x1Debug Slot x1SP VGA x1Rear USB x1SAS1 x8NET0 x8Debug Slot x11x8c02x8c04x8c03x8c09x8c110x8c111x8c112x8c15x8c26x8c27x8c28x8c213x8c314x8c315x8c316x8c3MPMPEBPCI-ExpressLow ProfileHot Plug Slotsx8 up.Part. 0x8 up.Part. 1x8 up.Part. 0x8 up.Part. 1{Switch 0Switch 1Switch 2Switch 3Switch 4Switch 5Switch 6Native Config6ConfigIDT5-4: 2 PM PCIe ConnectivityT5-4/8 Native 4-Socket Configuration with One Root DomainSingle non-redundant DomainSecond level Switch 6 is partitioned differently from other configsBlock fill color identifies Root Domain ownershipBlock outline color identifies association to PMSwitch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassisDotted linedevicesresideon RIOSlot #8 lanesCPU #Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#All White outlined PCIe slots are driven by PM0.All Black outlined PCIe slots are driven by PM1.

30CPU0CPU10101PM0PM1PM2CPU6CPU70101PM3x8 up.Part. 0x8 up.Part. 0x8 up.Part. 3x8 up.Part. 0x8 up.Part. 3x8 up.Part. 0x8 up.Part. 3x8 up.Part. 1NET1 x8SAS0 x8Front USB x1Debug Slot x1SP VGA x1Rear USB x1SAS1 x8NET0 x8Debug Slot x11x8c02x8c04x8c03x8c0MPMPEBPCI-ExpressLow ProfileHot Plug Slotsx8 up.Part. 0x8 up.Part. 1x8 up.Part. 0x8 up.Part. 1{Switch 0Switch 1Switch 2Switch 3Switch 4Switch 5Switch 6Native Config6ConfigIDPFMPFMT5-8: 2 PM PCIe ConnectivityT5-8 Native 4-Socket Configuration with Two Root DomainsTwo slot per Root ComplexTwo Redundant Path Root DomainsDomains survive dynamic removal of a PM with connectivity lossBlock fill color identifies Root Domain ownershipBlock outline color identifies association to PMSwitch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassis9x8c110x8c111x8c112x8c15x8c66x8c67x8c68x8c613x8c714x8c715x8c716x8c7Dotted linedevicesresideon RIOSlot #8 lanesCPU #Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#All White outlined PCIe slots are driven by PM0.All Black outlined PCIe slots are driven by PM3.PFM is a Processor Module Filler Board, or Processor Filler Module.

31CPU0CPU10101PM0CPU2CPU30101PM1PM2CPU6CPU70101PM3x8 up.Part. 0x8 up.Part. 1x8 up.Part. 0x8 up.Part. 1x8 up.Part. 3x8 up.Part. 0x8 up.Part. 2x8 up.Part. 3x8 up.Part. 0x8 up.Part. 2x8 up.Part. 3x8 up.Part. 1NET1 x8SAS0 x8Front USB x1Debug Slot x1SP VGA x1Rear USB x1SAS1 x8NET0 x8Debug Slot x11x8c02x8c24x8c23x8c09x8c110x8c111x8c112x8c15x8c66x8c67x8c38x8c613x8c314x8c715x8c716x8c7MPMPEBPCI-ExpressLow ProfileHot Plug Slotsx8 up.Part. 0x8 up.Part. 1x8 up.Part. 0x8 up.Part. 1{Switch 0Switch 1Switch 2Switch 3Switch 4Switch 5Switch 6Native Config2ConfigIDPFMT5-8: 3 PM PCIe ConnectivityT5-8 Native 6-Socket Configuration with Two Root DomainsTwo slot per Root ComplexTwo Redundant Path Root DomainsDomains survive dynamic removal of a PM with connectivity lossBlock fill color identifies Root Domain ownershipBlock outline color identifies association to PMSwitch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassisDotted linedevicesresideon RIOSlot #8 lanesCPU #Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#CPU0CPU10101PM0CPU2CPU30101PM1CPU4CPU50101PM2CPU6CPU70101PM3x8 up.Part. 0x8 up.Part. 1x8 up.Part. 0x8 up.Part. 1x8 up.Part. 2x8 up.Part. 3x8 up.Part. 0x8 up.Part. 1x8 up.Part. 2x8 up.Part. 3x8 up.Part. 0x8 up.Part. 1x8 up.Part. 2x8 up.Part. 3x8 up.Part. 0x8 up.Part. 1NET1 x8SAS0 x8Front USB x1Debug Slot x1SP VGA x1Rear USB x1SAS1 x8NET0 x8Debug Slot x11x8c02x8c24x8c23x8c09x8c110x8c311x8c112x8c35x8c46x8c67x8c48x8c613x8c514x8c715x8c516x8c7MPMPEBPCI-ExpressLow ProfileHot Plug Slotsx8 up.Part. 0x8 up.Part. 1x8 up.Part. 0x8 up.Part. 1{Switch 0Switch 1Switch 2Switch 3Switch 4Switch 5Switch 6Native Config 00ConfigIDT5-8: 4 PM PCIe ConnectivityT5-8 Native 8-Socket Configuration with Two Root DomainsTwo slot per Root ComplexTwo Redundant Path Root DomainsDomains survive dynamic removal of a PM with connectivity lossBlock fill color identifies Root Domain ownershipBlock outline color identifies association to PMSwitch 2 Slots crossed to maintain consistent Slot population order Slots drawn in order from left to right as in the actual chassisDotted linedevicesresideon RIOSlot #8 lanesCPU #Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#33Zakim (ZK) M5/T5 Systems Memory InterfaceAlso referred to as a BoB (Buffer-on-Board)Features and Technology:4 ZKs per M5 memory controller2 ZKs per T5 memory controllerSupports DDR3 DIMMsMemory Link to DDR Interface2 Memory Link Ports, 2 DDR ports8 Write FIFOsPass through commands to DIMMsCore frequency: 1066 MHzCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#T5-4 and T5-8 Memory Config RulesAll DIMMs must have the same Oracle Part Number on a PM (Processor Module).All PM's must be half populated (16 DIMMs) or fully populated (32 DIMMs).If a PM is half populated DIMMs must be in Channel 0 only, see Processor Module Block Diagram for DIMM Channel locations.Note that the above rules imply that both T5 (cpu) nodes on a PM are configured the same.All PM's in the system must be configured the same (ATO specific rule)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#T5-4 and T5-8 Memory Upgrade RulesAll DIMMs must have the same Oracle Part Number on a PM (Processor Module).All PM's must be half populated (16 DIMMs) or fully populated (32 DIMMs).If a PM is half populated DIMMs must be in Channel 0 only, see Processor Module Block Diagram for DIMM Channel locations.Note that the above rules imply that both T5 (cpu) nodes on a PM are configured the same.For T5-8 (8P config): PM0 and PM3 must be configured the same and PM1 and PM2 must be configured the same.For T5-8 (6P config): All PM's (PM0, PM1 and PM3) must be configured the same.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#T5-4 and T5-8 Airflow/Cooling

T5-8(top 3U)T5-4(5U)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SPARC T5 Disk DrivesDisk drives are 2.5 form factorChassis supports SAS HDD and SATA SSDsSAS-2 HDDs300GB @ 10K RPM600GB @ 10K RPM900GB @ 10K RPMSATA SSDs100GB300GB

Ready to RemoveFaultStatusDisk LED's

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#38SPARC T5 Disk ControllerT5 has Dual LSI SAS2008 8port SAS3/SATA2 controllersSupport for RAID 0 (striping) and RAID 1 (mirroring) using 'raidctl'

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#39The SPARC T5 ProcessorCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SPARC T5 ProcessorFeatures16 S3 cores, 16-128 Strands @ 3.6GhzSingle or multi-threaded operation per coreSystem scalability to 8 socketsSPARC Core S31-8 Strand Dynamically Threaded PipelineISA-based Crypto-acceleration8MB Shared L3$Integrated I/ODouble I/O bandwidth over T42 x8 Lane PCIe 3.0 @ 8GT/sSystem Scalability7 Coherence Ports for scalability to 8SPower ManagementDynamic Voltage Frequency ScalingDownclock, Overclock

Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SPARC T5 CPU Block Diagram2 x 8 PCIe 3.0 @ 8 GBps16 GBps each direction8 threads per CoreIOSubsystem

BoB

BoB

BoB

BoB

BoB

BoB

BoBDDR3 1066 MHzMemory ControlCoherency 4x4 SwitchMemory ControlMemory ControlMemory Control8 x 9 Crossbar (~1TBps bandwidth)C0C1C2C3C4C5C6C7C8C9C10C11C12C13C14C15L3$ B0L3$ B01MB,16-wayL3$ B01MB,16-wayL3$ B21MB,16-wayL3$ B11MB,16-wayL3$ B31MB,16-wayL3$ B0L3$ B01MB,16-wayL3$ B41MB,16-wayL3$ B61MB,16-wayL3$ B51MB,16-wayL3$ B71MB,16-way128 KB L2$16 KB L1I$16 KB L1D$ FGUCryptoSPARC S3CoreBoBDDR3 1066 MHzDDR3 1066 MHzDDR3 1066 MHzCoherence UnitCoherence UnitCoherence UnitCoherence Unit

Link 0

Link 1

Link 2

Link 3

Link 4

Link 5

Link 6Coherency Links 12.8 Gbps per lane - 12 lanes per linkCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#BoB = Buffer on BoardCLU = Coherency Link UnitThe S3 core is the same used on T4. This means all T4 optimizations also apply to the T5 (and M5/M6).The seven coherency links are how the T5 (and M5/M6) are able to talk with 7 other CPUs directly, with no hub.The 4 memory controller mean the T5 as approx. 60% higher memory bandwidth that the T4 with 2 memory controllers.The on-chip 10GbE was removed. This allowed for the additional cores, memory controllers, extra links.42T5 Processor Overview16 S3 cores @ 3.6GHz8MB shared L3 Cache 8 DDR3 BL8 Schedulers providing 80 GB/s BW8-way 1-hop glueless scalability Integrated 2x8 PCIe Gen 3Advanced Power Management with DVFS

SPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreSPARC CoreCross BarMCUCoherenceL3L3L3L3L3L3L3L3SerDesSerDesSerDesMI/OSerDesSerDesMCUMCUMCUPCIe Gen3PwrCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#16 S3 cores @ 3.6GhzSingle or multi-threaded operation per core16-128 total strands8MB Shared L3$Memory Controllers8 DDR3 BL8 Schedulers80 GB/s Sustained memory bandwidthIntegrated I/O - 2 x8 Lane PCI-Express Gen3System Scalability to 8 sockets7 Coherency PortsData forwardingPower ManagementDynamic Voltage Frequency ScalingCore Pair Cycle SkippingLink ScalingMCU Low power states

43SPARC T5: Processor Key FeaturesSPARC T5 based on SPARC T4: Same S3 core as T416 S3 cores, dual PCI Express 3.0 root complexesUp to eight T5 processors per system, 16 cores x 8 threads 128 (on T5-1B) or up to 512 threads (on T5-8)Clock frequency is 3.6 GHzEach SPARC T4 core contains:2 Integer pipelines1 FGU pipeline (consisting of 3 physical sub-pipelines):FPX pipelineFGX pipelineFPD pipeline1 Load-Store (Memory) pipelineCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#44SPARC T5: Processor Key FeaturesEach Core has two Level 1 cache memories, one for data and one for instructions, each 16KB in sizeEach Core has a Level 2 unified cache, 128KB in sizeCaches are all inclusive: L3 inclusive of L2; L2 inclusive of L1 (in this context, inclusive refers to the fact a cached entry is always present in the next higher level of cache)Each core on SPARC T5 is capable of OoO execution, dual-issue of instructions but in order commit.Each core on SPARC T5 also includes cryptographic acceleration hardware, accessible via user-level instructions.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#45SPARC T5: Processor Key FeaturesMemory systems based upon SPARC T5 use DDR3 Quad Rank DIMMsDIMMs used are Registered DIMMS (RDIMMs)16 DIMM slots per Processor memory capacity depends on size of DIMM chosen for system8 GB, 16 GB DIMM ECC DIMMs required to be used at 1066 MT/sNewly designed BoB between MCU and DIMMsFour MCUs per processor; each maintains a memory link speed of 12.8 GbpsProtocol between each MCU and its two BoBs is proprietary in natureCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#46S3 Core Recap28nm port from 40nm T4Out-of-order, dual-issueHigh frequency achieved with 3.6GHz 16 stage integer pipelineDynamically threaded, one to eight strandsAccelerates 16 encryption algorithms and random number generationCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

SPARC T5/M5 Leads in On-Chip Encryption AccelerationProcessor / MechanismsSPARC T4, T5, and M5IBM Power7IBM Power7+Intel Westmere/ SandybridgeOperational ModelUserland unprivileged access to on-core cryptographic functionsnone3 accelerators sharedacross 8 coresUserlandAsymmetric /Public Key EncryptionRSA, DH, DSA, ECCnoneRSA, ECCRSA, ECCSymmetric Key / Bulk EncryptionAES, DES, 3DES, Camellia, KasuminoneAESAESMessage Digest / Hash Functions CRC32c, MD5, Sha-1, SHA-224, SHA-256, SHA-384, SHA-512noneMD5, SHA-1, SHA-256, SHA-512noneRandom Number GenerationSupportednoneSupportednoneAPI SupportPKCS#11, Ucrypto APIs, JCEnonePKCS#11Intel IPP librariesVirtualization SupportSolaris ZonesOracle VM for SPARCnone??Intel VTCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#For crypto savvy customersProvides extra security against side channel attacks On-chip encryption resists software tampering Extensible cryptographic chaining supports future changes (generally only useful to those developing their own crypto algorithms like DoD)Oracles advantage is the crypto per core design. This means the T5 has 16 accelerators per chip.

SPARC Core RoadmapSPARC64 VII/VII+ CoreM-SeriesT3 Servers M5 ServersT4 ServersM6 ServersT5 ServersS2 CoreS3 CoreS4 CoreCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#S3 Core Overview8-way threaded, dual-issue, OoO execution, in order commitDynamically threaded with hardware-optimized resource sharingSupport for Critical ThreadsDeep pipeline for high frequency operation (3 GHz in 40 nm)Balanced single-thread and multi-thread performance5X better single-thread than SPARC T3 with equivalent multi-thread performanceEnhanced instruction set to accelerate Oracle SW stackPAUSE, fused compare-branchIntegrated user-level cryptographic accelerationDES/3DES, AES, Kasumi, Camellia, MD5, SHA-1, SHA-224/256/384/512,RSA, DSA, CRC32cFoundation core for future technology / product nodesCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Oracle SPARC S3 CoreEnables T4, T5, M5Design GoalsDevelop a common replacement core for T-series processorsSignificantly improve the single strand performance of the T3 processorSignificantly improve the throughput performance of the M3 processor Improve the RAS and power management capabilitiesMaintain backward ISA, Solaris and OVM for SPARC compatibilityCore S1: M2, M3Core S2: T2, T2+, T3Core S3: T4, T5, M5Frequency2.4 3.0 GHz1.4 1.65 GHzT4: 2.85 3.0 GHzT5, M5: 3.6 GHzL1 Instruction Cache64KB16KB16KBL1 Data Cache64KB8KB16KBL2 Cache--128KB# of Pipelines121# of Threads per Pipeline241-8 DynamicInstructions per Thread4 per cycle1 per cycle2 per cycleOut of Order IssueYesNoYes (36 instr window)Cryptography AccelerationNoneSPUISA BasedOVM for SPARC CompatibleNoYesYesSPARC V9 ISA CompatibleYesYesYesCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#S3 Core: Dynamic ThreadingWhile software can activate up to 8 strands on each core at a time, hardware dynamically and seamlessly allocates core resources such as instruction, data, and L2 caches and TLBs, and out-of-order execution resources such as the 128-entry re-order buffer in the core among the active strands. Software activates strands by sending an interrupt to a HALTed strand. Software deactivates strands by executing a HALT instruction on each strand it wants to deactivate. No strand has special hardware characteristics; all strands have identical hardware capabilities.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#S3 Core: Dynamic ThreadingSince the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivate. If software effectively halts all strands except one on a core via Critical Thread Optimization, the core devotes all of its resources to the sole running strand. Thus, that strand will run as quickly as possibleCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#S3 Core: Critical Thread OptimizationThe S3 core, starting with Oracle Solaris 10 8/11, has the capability to have a so-called Critical Thread declared by softwareIf so declared, this Critical Thread will be assigned exclusively to a given core, i.e., no other threads will be assigned to that core, given that the following condition is met:The system is not over-committed, defined as more runnable threads than available CPUsAll threads are equal: no thread has special hardware properties, all threads have identical hardware properties that potentially allow any thread to access any hardware resource that it needsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#S3 Core: Critical Thread OptimizationSolaris Critical Threads optimization for S3 core, tries to provide exclusive access of certain hardware resources to certain application threadsSince the core dynamically allocates resources among the active strands, there is no explicit "single-thread mode" or "multi-thread mode" for software to activate or deactivateThere is no new API for declaring threads as critical; that would require significant changes to source codeRather, to invoke Critical Thread Optimization, use the following CLI or system calls to flag a thread as critical by raising its priority to 60:priocntl(1)priocntl(2)priocntlset(2)Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#S3 Core: Critical Thread OptimizationStarting with Oracle Solaris 10 8/11, a thread is declared to be critical if raised to priority 60; the thread can be in any scheduling classIn Oracle Solaris 11, to be considered critical by the scheduler, a thread must be:in the FX (Fixed Priority) or RT (Real-Time) scheduling classesbe raised to priority 60 by one of the previously mentioned mechanismsIn either of the above instances, this one thread will run as quickly as possible as it has exclusive access to all core resourcesCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Critical Threads for key applicationsApplicabilityOpportunityCurrent StatusDatabaseLogwriter, LMSUp to 30% improvement in efficiencyLMS is already CT ready. LGWR planned for 12cJAVA (JVM)Compiler threads, GC and priority mapping supportUp to 2x improvement for app startup, Smooth GC Support for JVM and JAVA apps to be CT aware is integrated in JDK7U4CoherencePacket writer, service thread Up to 20% improvement in throughputIntegrated in Coherence version 3.7.1 Patch 1SolarisS11U1 / S10U11Improve CT perf to be within 10% of best case (hand optimized)Optimizations for decayed PG util and stealing being integrated in S11U1 Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#57Logwriter is a throughput limiter in Oracle 11g R2. All transactions are logged for recovery purposes. Customers today typically statically bind the LGWR=Logwriter to a core. CT can assign LGWR much more dynamically.

LMS is the lock manager in RAC's Cache Fusion. RAC is notorious for lack of scalability and primarily due to LMS. So assigning this to a core all on its own is a win.

GC is garbage collection in Java.

Packet Writer is actually I think Packet Publisher in Coherence. Much like the log writer in DB it is a throughput limiter that benefits from assignment to a core on its own with CT.

PG is Processor Group is an abstraction used by the operating system to represent the CPUs that share performance relevant hardware such as the execution pipelines, caches, and so forth.

1. Stealing related changes :

When a runnable thread is stolen by an idle CPU, check if the load average of the PG (core in this case) is already high. That may indicate that you have a Critical Thread running on that core. In that case, try not to steal work.

This helps CT get more exclusive access to the core.

2. Decayed PG Utilization :

When a CT goes off the CPU, do not reduce the load average to normal level drastically or immediately.. but do it gradually that way, it gives CT some more chance to expect the core to be more idle when it gets back on the CPU in that core in a reasonable time.

It basically minimizes the utilization of the Core on which CT ran before, when it goes off-cpu so that if CT comes back on CPU in that core then it gets a better / clean core..

Internode Coherency OverviewGlueless 1-hop scaling to eight socketsA precise directory tracks all L3s in the systemstriped across all processors stored in on-chip SRAMsflexible for different socket countsHigher BW efficiency than snoop-based protocols enables better scaling50% more effective bandwidth than comparable snoopy implementationT-T5T5T5T5T5T5T5T5Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#The T5 (and M5) is the first true 8-way glueless architecture processor.The 4-way T3 and T4 used snoopy-based communication between CPUs, but this slows performance down for larger 8-way system. Using a crossbar switch would have allowed for point-to-point communications, but would have added latency and cost to the server. The best design is providing 7 links per CPU so the server could grow to 8-way glueless.

58Internode Coherency FabricEach link is 14 lanes wide and runs up to 15Gbps per laneDirectly connected links minimize latencyTrunked links achieve more bandwidth in smaller configurationsSupports single lane failover T5T5T5T5T5T5Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#The practical limits of the interconnect:T5-8 is 12.6T5-4 is 11.2T5-8 with 2 PM is 10.7T5-2 is TBD

59

Internode Performance Optimizations Speculative memory reads prior to cache line serialization in the directoryCache-to-cache line transfers between nodesDynamic congestion avoidance routes inter-node data around congested linksCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#T5 System Interconnects

1-WaySingle Socket2-WayDual Socket4-Way6-Way8-WayCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#

All-to-All InterconnectWhere the Node-to-Node Fabric is 14 diff pairs per link in each direction.

At 12.8Gb/sec: ~22.4GB/sec/directionBisection Bandwidth7 Links x 22.4GB/sec/Link

X 2 directions = 313GB/sec DDR3-1066 Memory Bandwidth16 DDR channels x 1066Gbps x 8 Bytes/DDR channel = 128GB/sec

~1024GB/sec for T5-8PCIe Gen3 Bandwidth8 diff pairs per portsAt 16Gb/sec ~16GB/sec/direction

~16Gb/sec/lane X 8 lanes = 64 Gb/sX 2 directions X 2 Ports/chip= 512 Gb/s/chip8 Socket Local Coherency & Data Interconnect

DIMMSM5/T5M5/T5M5/T5M5/T5M5/T5M5/T5M5/T5M5/T5

DIMMS

DIMMS

DIMMS

DIMMS

DIMMS

DIMMS

DIMMSPOINT-TO-POINTLOCALINTERCONNECTCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Latency for T5T5-2T5-4T5-8Local Memory136nsRemote Memory209nsCache to Cache127ns146ns155nsCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Scalability of T4 vs T5T4 Snoopy BasedCoherence ProtocolT5 Directory BasedCoherence ProtocolBy numbers8 node snoops will consume 25% Link B/W & increases linearly w/more nodes8 node directory based will consume 5% of Link B/WFeatureAddress serialization is done at Home Node. Home Node broadcasts snoop request to all nodes. All nodes except the requesting node require to participate the snoop operation and provide snoop response back to requesting node.Address serialization is done at Directory Node. Directory Node keeps track of which node hold each cache line. Eliminating the need for broadcasting, and relieve the L3$ from unnecessary foreign snoop operation.Link BandwidthMessage broadcast and response consume a lot of link bandwidth.Directory filter the snoops sent to the share nodes. Allow link bandwidth to be used more efficiently.L3$ PerformanceL3$ need to participate every snoop request from any other node. The L3$ performance can be dropped due to lots of foreign snoop requests.Only the L3$ from the selective node require to participate the foreign snoop operation. L3$ has less distraction from foreign snoop request.ScalabilityLimited to small scale of system.Easy to scale to large number of processor environment.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Memory Controller of T4 vs T5T4 MCUT5 MCUMemory Link Speed6.4 Gb/s12.8 Gb/sMemory Link ProtocolLegacy Intel FBDIMM2 ProtocolAdvanced In-house Link ProtocolMemory Link Low Power FeatureNot SupportedL0s, L1Memory BufferIntel Milbrook2 MBAdvanced In-house MBDDR3 ProtocolBurst length of 4Burst length of 8DDR3 Speed800/10661066DDR3 Device1Gb/2Gb2Gb/4GbCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#RASCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Definition of TermsHot-plug:refers to the fact that a component can be plugged and unplugged without powering down the platform. It applies to both hot swap and hot service.Hot service:refers to the ability to perform hot-plug operations, with the additional necessity of some operator actions (invocation of a CLI or actuating a hot service button on the component to be removed).The system will notify the user when it is safe to remove the component. Typical examples would be PCIe Express modules.Hot swap: refers to an operation where you walk up to the box, yank something out, replace it, and you are done. Typical examples here are a single RAID disk or a power supplies.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#SPARC T5 System RAS OverviewDesigned to minimize part count and operating temperature to enhance reliabilityEnd-to-end data protection detecting and correcting errors throughout server ECC everywhereProcessor and Memory protection CPU core and thread off-lining Memory with ECC, x4/x8 DRAM Extended ECC, page retirement, and lane failoverMajor components redundant & hot-replaceable Fan, Power Supply, and internal disks RAID capability for internal disksFMA support on ILOMCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#6868The previous T4 servers only had FMA when the domain was up and running. FMA is now part of the ILOM. This is true for T5 and M5.

T5/M5 Systems RAS

S11 FMA Diagnosis engine on SP Auto reconfigure on failure Soft Error Rate Discrimination (SERD) Bad page retirement OS and SP watchdogs FMA Component hot-upgradeable

T5/M5 Processor L1$ Tag, Status $ Data Parity protection Retry on error L2$/L3$ Data SEC/DED protection Cache-line Sparing L2$/L3$ Tags SEC/DED protection Inline Correction Cache-line Sparing L2$/L3$ Status & Directory SEC/DED protection Inline Correction Architectural RegistersL2 Cache SEC/DED protection Precise Trap andHypervisor Correction and RetrySystem Redundant SPs with automatic failover Redundant clock boards Diagnosis to the FRU level on first fault Hot-plug processor/memory**

Power and Cooling Advanced Power Management Redundant hot-swap fans Redundant hot-swap AC/DC Dual grid power

System I/O PCI-Express end-to-end CRC PCI Express link retry Hot-plug low profile PCI Express cards Redundant, hot-plug boot disks Alternate connections between M5 and IOcontrollersMemory SDRAM Soft Errors ECC Protection and Correction Extended ECC Protection 4-bit Correction Pin Steering Channel Interconnect CRC protection/Message Retry Lane SparingHypervisor Enables software partitioning (LDoms)virtualization and failure containment Processor support for error clearing, correction and collectionCentral Directory and Switch SEC/DED protection with in line correction Physical domain isolation CRC protected System Interconnect with message retry and lane sparing Deconfigurable directory chips, no loss of functionality, minimized bandwidth loss Redundant Scalability Switch BoardsUnique to M5** Post-RRCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#T5 RAS is the same as for M5, except for the items in Red.69Fault ManagementKnowledge Articles in MOSILOM fdd DiagnosisFaults and AlertsNo ALOM CompatibilityILOM FMA Captive ShellSideband Service Processor Network ConnectionNew ILOM Fault Notification (SNMP Trap)ASR SupportFMA on M5 ILOM also applies to T5 ILOM, except for M5 specific featuresCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#70FMA restricted shell in SPSHTo get FMA details in SPSHstart -script /SP/faultmgmt/shellReturns a faultmgmtsp> promptAvailable build-in commands:echo - Display information to user.Typical use: echo $?helpExit exits restricted shellExternal commands:fmadm - Administers the fault management servicefmdump - Displays contents of the fault and ereport/error logsfmstat - Displays statistics on fault management operationsetcd - ereport injectorCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#Fault Management on T5 systemsT5 CPU and Memory faults are now diagnosed by ILOMFMA's Fault Proxy is used to keep ILOM's fault manager in sync with Solaris' fault manager. Both will display the sum of all faults in the system.Faults can be repaired from either side.Fault Proxy communicates via the Ethernet Over USB connection.IO faults are still diagnosed by Solaris.Disabled Database (DDB) owned by ILOMFor faults which diagnose resources as unusable, ILOM will add those resources to the DDB. Resources excluded on next host reset.When faults are repaired, ILOM automatically updates the DDB. Bringing components back online requires a host reset.Extended SP-POST (Power on Self Test)Runs at SP boot. Tests devices on the SP FRU and its Ethernet port.Status stored and converted to ereports after ILOM boots.Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#72Fault proxyIO ereports are forwarded from the SP to the control domain, and then on to any relevant IO domainFaults are proxied between the SP, the control domain and any IO domains to provide a single view of faults in the system.Non-servicable faults such as memory faults are not proxied.The SP and the control domain can view and manage all faults in the system.An IO domain can only view and manage faults local to the domain.Control DomainIO DomainLDCLDCTCP/IPSPhostdFETDip-transprtETMETMETMETMETMip-transportereportsereportsfaultsfaultsLDCCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#73Degraded HW Configurations T5-8If a CPU has no configurable memory, i.e., all memory links are unconfigurable, then the CPU itself is unconfigurableA T5-8 will not operate with only 5 or 7 CPUs configuredOne more CPU(s) must be chosen to be deconfiguredFor an 8-way, if we fault a CPU, we will offline the other CPU on the same PMFor a 6-way, if we fault a CPU, we will also offline the other CPU on the same PMCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#ASR SupportSPARC T5 servers will be supported by ASR (Automatic Service Request) at releaseContinues use of sunHwTrapFaultDiagnosed SNMP notificationTelemetry for ILOM fdd diagnosisSupports platform and FRU identitySupports multi-suspect listCopyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#75Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#76Copyright 2012, Oracle and/or its affiliates. All rights reserved.Confidential Oracle Internal#77