Cluster Components

• Compute Server• Disk Storage• Image Server

Compute Server:Stanford Research Computing Facility (RCF)

• Accessible only to Stanford faculty and authorized sub users for their research

• 128 nodes provided by the provost– 8 cores each (16 threads), 64GB RAM, Infiniband card– Ivy Bridge V2 with 1866GHz memory – 1024 cores, 2048 threads, 8TB RAM total in the initial

cluster– Cluster re-evaluated every 2-3 years for

upgrade/expansion

Compute Server:Stanford Research Computing Facility (RCF)• 1PB of scratch space (not backed up)

– Must push data to the local storage prior to running jobs– Home directories allocated to individual users (~10-15GB/user),

but may be combined among users within a group– Expansion units will be available for purchase later on

• Two additional nodes with 1.5TB RAM each– Jobs running on these machines must meet minimum RAM

requirements or else they will be terminated and sent to the general cluster

• Infiniband 56Gbits/sec Ethernet lines

Purchasing into the RCF: Hardware

• Purchasers will always have access to the main cluster• Base system purchase

– 8 cores (16 threads), 64GB RAM (4GB/thread), Infiniband card, $5000

• Custom systems can be purchased– i.e., 8 cores (16 threads), 128GB RAM (8GB/thread), ~$7000

• Flexible configurations can be made to purchased machines– i.e., 4 active threads (12 “wasted” threads, 128GB RAM with

32GB/thread)– Can be changed as long as DIMM configuration remains optimal

Purchasing into the RCF: Administration

• Can assign priority of jobs among authorized users that have access to a purchased machine.

• Server administration is included in purchase price. No annual fees.

• Purchase announcements made ~3 times per year.– May have to wait a couple months before purchased

machine is available.• Other users’ access to purchased machines that are

idle is TBD– Compliance/audit issues remain

Loading Software on the RCF cluster

• Can request to have software packages on the general cluster– More likely to be installed if widely used– Matlab is on the list to be added, but is currently a

source of frustration• Can install software on a purchased machine– Recommended for software that is not widely used

(i.e., offline.recon)– Need to evaluate current licenses for transfer to

purchased machines (i.e., COMSOL, SEMCAD, etc.)

RCF Timeline

• February– Designated “friendly” users to begin testing– One each from Rutt group and McNab group

• March– Cluster becomes available for general use

• April-June?– 1st round of purchased machines available

RCF Considerations• May be advantageous to purchase early due

to our high RAM requirements• Purchase costs may increase next year• Unclear if purchased machines are available to

other users• Job priority rules are still being decided

Compute Server Action Items

• Leaning towards RCF purchase of 2 x 128GB systems to be used as a mini cluster.– Jennifer’s group gets top priority on one machine and

Brian’s group gets top priority on other.– Users from both groups are authorized to use both

machines.

• ASAP: Designate friendly users for RCF testing. One each from Brian’s group and Jennifer’s group.– Jason/Simone + Grant Yang

Disk Storage

• Synology desktop NAS or rack-mounted• RAID6 configuration– Main Unit: 48TB capacity 40TB actual storage– Expansion Units: 48TB capacity 48TB actual storage

• Must decide maximum expansion capacity prior to purchasing main unit– Main + 2xExpansion = 144TB system 136TB RAID6– Main + 1xExpansion = 96TB system 88TB RAID6

Desktop vs. Rack Storage vs. Max Capacity

DESKTOP NAS SYSTEM 88TB RAID6 136TB RAID6 RACK MOUNTED SYSTEM 88TB RAID6 136TB RAID6       Main Unit DS2413+ DS3612xs Main Unit RS2212+ RS3614xs+44TB Capacity (36TB RAID 6) $1,900.00 $3,000.00 44TB Capacity (36TB RAID 6) $1,900.00 $5,000.00        Main Unit Drives WD Red 4TB WD Red 4TB Main Unit Drives WD Red 4TB WD Red 4TB12 Drives @ $189 each $2,268.00 $2,268.00 12 Drives @ $189 each $2,268.00 $2,268.00        MAIN UNIT INITIAL COST $4,168.00 $5,268.00 MAIN UNIT INITIAL COST $4,168.00 $7,268.00

Expansion Unit DX1211 DX1211 Expansion Unit RX1214 RX121444TB Capacity $1,300 $1,300 44TB Capacity $1,700 $1,700        Expansion Drives WD Red 4TB WD Red 4TB Expansion Drives WD Red 4TB WD Red 4TB12 Drives @ $189 each $2,268.00 $2,268.00 12 Drives @ $189 each $2,268.00 $2,268.00        EXPANSION UNIT COST $3,568.00 $3,568.00 EXPANSION UNIT COST $3,968.00 $3,968.00

Maximum Expansion Units 1 2 Maximum Expansion Units 1 2MAX EXPANSION TOTAL COST $7,736.00 $12,404.00

MAX EXPANSION TOTAL COST $8,136.00 $15,204.00

COST PER TB $87.91 $91.21 COST PER TB $92.45 $111.79

Image Server

• Want a multi-user DICOM server and database– Need a way for people to be able to view and pull

DICOM images remotely– Adding all these users as recognized DICOM nodes

is cumbersome– A web interface is an effective way to accomplish

this

OsiriX

• Has a fairly basic web layout– Rudimentary search– Organized by albums (which cannot be managed through the

website, I think)– Excellent image viewer– Can be customized to our heart’s content though

• Advanced user management with customized access rights for each individual

• Try it out: http://171.65.72.243:3333/– Just press login

OsiriX – Main Page

OsiriX – Album Page

• Download options– “OsiriX remote transfer”– Zip of DICOMs– Weasis remote view

OsiriX – Exam Page

OsiriX – Series View