Post on 01-Jan-2016
description
transcript
1CHEP 2000, 10.02.2000 Roberto Barbera
Roberto Barbera
Meeting INFN-Grid, 18.10.2000
“Farming” with VENUS
2Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
Goals of VENUS• Simplify administration of heterogeneous Unix/NT clusters• Increase efficiency of administration• Increase productivity of users• Providing a single system image• Simplify software integration and installation• Central point of administration• Integration of Unix with WindowsNT• Considering special needs in complex engineering and scientific computing environments
3Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
Design of today‘s clusters
HP
SGI
IBM
SUN
Compaq
DEC
Compute Servers File Servers
Backup Servers
Firewall
WEB
Printer
Printer
Pre-, postprocessingCAD/CAE Simulations
Batch, JobflowmanagementBackupArchivPDM
PlotprocessingDatabases
Communication
4Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
Today‘s problems in system management I
Complexity of structure: grouping of hosts, users with different working environments
Complexity of data and filesystemstructures: Many projects,databases, centralized and decentralized filesystems
Maintaining consistency in login behaviour: user management
Fast changes in hard and software
5Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
Today‘s problems in system management II
Complexity of software:
• many software packages used: graphical (CAD, Pre/postprocessing)• sometimes big packages (>500 MB)• many software releases per year, patch management• licensing using different mechanisms (flexlm, netls,...)• integration into the desktop (CDE, KDE, ...)• integration into the filesystem• integration into the printer environment
6Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
Today‘s needs in system management
• Methods to install software and patches to be fast and reliable• User management to be simple, providing best user environment for users• Well defined filesystem structure to provide a global file name space• Well defined operating system status (which OS, which patches) to assure quality and reliability• Independence of special knowlegde of system administrators• Quality driven system management instead of „quick“ but undocumented solutions• Flexibility to solve daily problems methodically
7Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
Single system image (SSI) I
Definition: For users, an ideal cluster presents itself as a single system image, i.e. a heterogeneous workstation server cluster appears like a single big computer with many (intelligent) terminals attached.
• Single point of authentification• Same password on any node in the cluster• Unique access rights on each node• Unique file space : Global file name space
8Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
Single system image (SSI) II
• Common login environment on each node• common (architecture dependent) variables (e.g. $path)• shell aliases• desktop configuration (CDE, KDE, GNOME,...)
• Considering needs of users and goal to simplify the login environment • Access to all printers, peripherals available for the user or user group
9Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
Overview of VENUS functionalities
User- and file systemmanagement
Uniform view of thefile system
/home/$useradd
sc del usrch grouplist
/$map/$keyadd
sc del keychlist
SoftwareManagement
• Software depot server• Full documented
software packages in VENUS format
• Pre/post/install/remove methods and procedures
• VENUS tools to create VENUS software packagesscsoftpkg, scsoftfilescsoftinfo
• sc{add|ch|del|list}pkg
ConfigurationManagement
• central storage of allconfiguration differencesto original OS
• As global as possibleas specific as necessary
common architecture groups hosts• VENUS methods
to configure subsystems
10Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
User management I
scadduser, scchuser, scdeluser• modify passwd map and automounter maps or corresponding information bases on NT• Create, move, or delete user homes on target nodes
scadduser -u calvin -h schoolschool: /v/school/home/calvin
scchuser -u calvin +h playgroundmoves calvin from school to playground
scdeluser -u calvinremoves user calvin
11Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
User management II
Unix
Unix
NT/DFS Server
File Server
Unix
UnixPDC,samba
NT
NT
NT
Local file system
Local file system
NFS
SMB
VENUS Master
Unix: passwd mapAutomounter map
WindowsNT: PDC configurationDFS configuration
12Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
Configuration management II
Methods properties
• Methods are tasks to be executed on a client to configure a client subsystem• Methods are (usually) shellscripts• subsystems are: tcpip, nis, nfs, mail, resolver, xwin,cde, printer, etc.• Methods are „dataless“, i.e. they are automatically configured in their specific context• Methods are able to communicate with the configuration depot on the VENUS master• Methods are able to install VENUS software packages• Methods use the VENUS lookup hierarchy: host > groups > architecture> common• Methods have an „install“ and an „uninstall“ section• Methods can be modified, extended and administrators can develop their own methods
13Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS architecture
scrsh
rshd
scinstall
scvenusinetd
venusd
MethodsContext
Software
Depots
VENUScommands
Accesscontrol
VENUS master VENUS client
14Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS software management I
Goals:
• Installing Software by a simple command including
configuration, licensing, pre and postinstallation
• Automatic maintenance of a software inventory
• Parallel installation of software
• Installing the correct software version (architecture,
i.e. OS)
• Ability to create software packages from any 3rd
party vendor
• Easy removal of software from a client
15Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS software management II
VENUS Software packages consist of:• All files to be installed• pre/post installation procedures and/or VENUS methods• filelist, version, architecture, author
Application integrationIntegration using wrapper scripts:
• setting path and environment• creating symbolic links (in installation method !)• wrappers are located in the standard path
(e.g. /usr/local/bin)
16Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000DEC
VENUS software management III
VENUS master
MethodsContext
Software
Depots
SGI
IBM
SUN
Compaq
SoftwareDepot
HP
SGI
IBM
SUN
Compaq
DEC
HP
Parallelized SoftwareInstallation !
17Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS software management IV
Commands for software management
scaddpkg lsf lsfhosts
This command will install LSF on each node contained in thehostgroup „lsfhosts“. Each client asks the VENUS master to providethe proper software package
• for the correct operation system type• and the latest version
The software package will be transfered from the „nearest“ software depot to each client. This makes it possible to parallelize the softwareinstallation process. If the installation is finished successfully, VENUS makes an entry into the software inventory.The installation of the configuration files is done automatically in the installation method.
18Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS software management V
Commands for software management
scdelpkg lsf lsfhosts
This command will remove LSF on each node contained in the hostgroup „lsfhosts“. If done successfully, the entry in the software inventory is also removed.
sclistpkg hostname
This command creates a list of software installed on „hostname“and displays the current version of the software.
19Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS references I
DaimlerChrysler: 1600 CAD Workstations in car design(SGI, HP, IBM,SUN)Main appl.: CATIA
300 CAE Workstations in engineering(SGI, IBM, HP, SUN, CRAY, NEC)100 different applications
80 CAE Workstations in truck engineering(SGI, IBM, HP)
200 Workstations in DaimlerChryslerResearch Center Stuttgart (SGI, Linux, IBM, HP)
100 Workstations in bus design and engineering (EVOBUS)(SGI,HP,IBM)
20Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS references II
DaimlerChrysler: Details in CAE
• 5 departments with different tasks: Structuremechanics, crash, engine development,various applications (> 100)
• High end Unix workstations and > 30 compute servers (HP, SGI Origin, NEC SX4, IBM SP)
Using VENUS specials:• Managing all printers with original VENUS printer method• Installing complex Pre/postprocessors using VENUS methods
e.g. Medina from debis, abaqus, etc.• Configuring batch (LSF, NQS), Jobstarter and batchsystem using
VENUS depots
21Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS references III
BMW: 1600 CAD Workstations in car design(SGI, HP, IBM,SUN)Main appl.: CATIA
400 CAE Workstations in engineering(SGI, IBM, HP, SUN, CRAY)Compute Center with
CRAY SV1, T90, OriginIBM SP, SUN HPC
100 different applications50 Workstations at BMW Technik Gmbh
(Designers of Z1)
22Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS references IV
300 CAD/CAE Workstations in engineering and design(SGI, HP, IBM,SUN)
30 Workstations at BOSCH Research
23Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS references V
330 CAD Workstations in ECAD design (SUN)
planned: 1200 Workstation (Autumn)
24Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS references VI
Details
ECAD at Infineon in the memory products division:• >250 workstations using cadence, mentor software• Big fileserver, used as file and codeserver
All nodes are completely managed using VENUS methods:• standard VENUS NIS, NFS, but using SUNs automounter• TCPIP, DNS setup• printer method modified: using lpngr • Installing backup client software (legato networker) using VENUS• Installing and configuring LSF using VENUS• security method, SUN patch method, method to create CAD-Admin-users
25Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS references VII
Details cont´d
Benefits:• Reducing costs by increasing the efficiency for administrators:
• > 50 Workstations managed by one administrator• Parallel execution of tasks (methods)• documented and well defined status of any clients
• Quality driven administration in contrast to manually changing configurations• Increasing efficiency of users by high stability and uptime of clients• Avoiding conflicts between different administrators by VENUS
multi-administrator ability: Locking of methods and central filese.g. NIS sources, passwd.Error prevention !
26Roberto Barbera
Dipartimento di Fisica dell’Università di Catania and INFN Catania - Italy
ALICE Collaboration
Meeting INFN-Grid, 18.10.2000
VENUS references VIII
Details cont´d
Benefits: ROI (Return on investment)
Service costsbefore using
VENUS Service costs using VENUS
Costs to introduce VENUSand licensing
0 3 months Time
Break even