7 eti pres

transcript

ETI SCC Baremetal FrameworkBandwidth and Power Findings

Rishi Khan3/30/11

alOutline

• SCC Framework Overview• Bandwidth Findings• Power Findings• Software Access

alSCC Framework Overview

alMessaging Goals

• Asynchronous Communications• Single Threaded• Possibly Long Latency until data is received• Maximize bandwidth• Handle big and small messages• Extensible layer that supports MPI, BSD

sockets, etc

alDesign Choices

• One channel per core-pair per direction• Large window size (up to 1MB/channel)• Fast polling of incoming data (use MPB)• Circular buffer with 16 slots and read/write pointers• Poll local pointers, signal remote pointers• Use separate cache lines to avoid locking

2 cache lines * 48 channels = 3K per core

• Double map read and write pages Read – L2 cache enabled Write – L2 cache disabled (write back)

alCircular Buffer Example

Core 0 (reader)

Channel->local_read

Channel->mpb_write

Core 1 (writer)

Channel->local_write

Channel->mpb_read

DRAMChannel->body[]

Is there space?

Write the data (with length as first 2 bytes)

Read data

UpdateRead

Pointer

alSocket API

• int stream_recv(int nid, void *buf, size_t len, int nb);• int stream_send(int nid, const void *buf, size_t len);

Intel RCCE

ETI Streams (DRAM, Blocking)

ETI Streams (MPBs, Blocking)

ETI Streams (MPBs, Non-Blocking)

Message Size (bytes)

Linux MPI (Intel, Blocking, TCP)

Baremetal MPI (ETI, Blocking)

Baremetal MPI (ETI, Non-blocking)

RCKMPI

Message Size (bytes)

alPower Goals

• External monitoring of voltage and current• Backend Power API

Update time functions with frequency changesKeep chip under safe conditions!!

• Internal synchronization of clocks• External synchronization of host and SCC

alExternal Monitoring

• Read /opt/sccKit/systemSettings.ini• Telnet BMC 5010• Request Status / Parse Data• Store timestamps

alBackend Power API

• power_session scc_open_power(heap h);• void scc_close_power(power_session ps);• int scc_set_freq(power_session ps, u32 requested_frequency);• int scc_set_voltage(power_session ps, u32 requested_millivolts);• char* scc_error_string(status_code code);

100 106 114 123 133 145 160 178 200 266 320 400 533 800

0.7 X X X X X X X X X X X X

0.8 X X X X X X X X X X X X X

1.1 X X X X X X X X X X X X X X

Allowable Frequency

alInternal Synchronization

• Cores come out of sccReset in 20ms intervals• Each core’s clock starts at cycle 0 at reset• Each core’s frequency may be different• Solution:

Set all cores to 400MHz Barrier After Barrier, set internal integrator to 0

alFormulas for Time

• Use this formula for time: count = scc_cycle_count() - _integral_cycle; ns = _integral_time_ns +count*_current_ns_in_cycles;

• Use this for frequency change: _integral_time_ns += (scc_cycle_count() - _integral_cycle) *_current_ns_in_cycles; _integral_cycle = current_time; _current_ns_in_cycles = 1.0e9/((double)_global_clock/

(double)freq_divider);

scc_cycle_count()

_integral_cycle

alSyncing Front/Back

• Change voltage from 0.7 to 1.1 every 1 second• Measure changes on frontend• Cannot get better than 0.5 seconds

0 2 4 6 8 10 12 14 16 18 2020

0 2 4 6 8 10 12 14 16 18 200

Residuals

alBug in BMC Voltage Readings

• 3 power islands• Drop voltage from 1.2 to 0.7 immediately• Raise Voltage after 20 seconds

0 5 10 15 20 25 30 35 400.7

Voltage

0 5 10 15 20 25 30 35 4017

20.5 Seconds0.6 Seconds

alOther SCC issues

• If more than 24 cores pound on one MPB, contention overtakes system.Sleep required between polling

• Allowable Voltage/freq are chip specific• BMC telnet response is > 100ms

alFuture Work

• DARPA UHPC: Study how voltage/freq affect power dissipation

• Allan Snavely (UCSD)Systematically study loops over a number of

parameters to find the best voltage/freq.Create formulas to approximate good power

settings for unknown loops

alAccess to Software

• Email scc-support@etinternational.com • Beta available• Considering open sourcing SCC-specific

portions of our work for others to test/learn/improve

alAcknowledgements

• Mark Deazley (ETI)• Eric Hoffman (ETI)• Allan Snavely (UCSD)• Intel:

Tim MattsonTed KubaskaRob NoradkiWilf Pinfold, Shekhar Borkar (UHPC)

7 eti pres

Documents