1

Basic Concepts Basic Concepts of FEM of FEM Framework & APIFramework & API

Gunavardhan Gunavardhan KakulapatiKakulapati

((kakulapa@cs.uiuc.edukakulapa@cs.uiuc.edu))

2

IntroductionIntroduction FEM Framework overviewFEM Framework overview Using the framework APIUsing the framework API InstallationInstallation Conversion ExampleConversion Example

3

MotivationMotivation

Finite Element Method Finite Element Method Used extensivelyUsed extensively Computationally intensiveComputationally intensive

Do it in parallel !!Do it in parallel !!

4

FEM ProgramsFEM Programs

FEM programs manipulate FEM programs manipulate elements and nodeselements and nodes

Element is a portion of problem Element is a portion of problem domain, surrounded by nodesdomain, surrounded by nodes

Element computation: based on Element computation: based on fieldsfields of surrounding nodes of surrounding nodes

Elements contribute to Elements contribute to fieldsfields of of surrounding nodessurrounding nodes

5

FEM MeshFEM Mesh

ElemenElementt

Surrounding Surrounding NodesNodes

E1E1 N1N1 N3N3 N4N4

E2E2 N1N1 N2N2 N4N4

E3E3 N2N2 N4N4 N5N5

6

FEM Program StructureFEM Program Structure

read mesh, connectivity, boundary conditionstime loop           element loop- Element deformation applies forces to surrounding nodes           node loop- Forces and boundary conditions change node positions end time loopwrite out mesh data for postprocessing

7

ParallelizationParallelization

Partition the FEM Mesh into Partition the FEM Mesh into multiple chunksmultiple chunks

Distribute elements, replicate Distribute elements, replicate shared nodesshared nodes

Shared nodes = Communication !!Shared nodes = Communication !! Partition so that communication is Partition so that communication is

minimizedminimized

8

PartitioningPartitioning

ElemenElementt

Surrounding Surrounding NodesNodes

E1E1 N1N1 N3N3 N4N4

E2E2 N1N1 N2N2 N3N3

ElemenElementt

Surrounding Surrounding NodesNodes

E1E1 N1N1 N2N2 N3N3

Shared NodesShared Nodes

AA BB

N2N2 N1N1

N4N4 N3N3

9

Parallel FEM ProgramParallel FEM Program

read/get chunk mesh data, connectivity, shared nodeschunk time loop           element loop- Element deformation applies forces to surrounding nodes           <update forces on shared nodes>           node loop- Forces and boundary conditions change node positions end time loopwrite chunk mesh data for postprocessing

>>

10

FEM Framework: GoalsFEM Framework: Goals

Separate parallel implementation Separate parallel implementation from numerical algorithmsfrom numerical algorithms

Parallel version to closely resemble Parallel version to closely resemble the serial programthe serial program

Allow features like load balancing, Allow features like load balancing, visualization.visualization.

11

FEM Framework: FEM Framework: ResponsibilitiesResponsibilities

Charm++(Dynamic Load Balancing, Communication)

FEM Framework(Update of Nodal properties, Reductions over nodes or partitions)

FEM Application(Initialize, Registration of Nodal Attributes, Loops Over Elements, Finalize)

METIS I/O

Partitioner Combiner

12

FEM Framework ProgramFEM Framework Program

May contain user-written, library-called May contain user-written, library-called subroutines:subroutines: initinit driverdriver mesh_updatedmesh_updated (more on this later…) (more on this later…)

init init andand mesh_updatedmesh_updated are called on are called on processor 0processor 0

driverdriver is called on every chunk is called on every chunk

13

Structure of an FEM Structure of an FEM ApplicationApplication

init()

Update Update Update

driver driver driver

Shared Nodes Shared Nodes

14

initinit

subroutine init           read the serial mesh and configuration data inform the framework about the mesh end subroutine

15

driverdriver

subroutine driver           get local mesh chunk           time loop                FEM computations                update shared node fields                more FEM computations           end time loop end subroutine

16

Data outputData output

Parallel output at the end of driverParallel output at the end of driver Possible to visualize the data rather Possible to visualize the data rather

than printing it out (netfem).than printing it out (netfem). Framework calls like FEM_Mesh_updated. Framework calls like FEM_Mesh_updated.

(more on this later …)(more on this later …)

>>>>

17

Framework CallsFramework Calls

FEM_Set_*FEM_Set_* Called from initialization to set the serial meshCalled from initialization to set the serial mesh Framework partitions mesh into chunksFramework partitions mesh into chunks

FEM_Create_fieldFEM_Create_field Registers a node data field with the framework, Registers a node data field with the framework,

supports user data typessupports user data types FEM_Update_fieldFEM_Update_field

Updates node data field across all processorsUpdates node data field across all processors Handles all parallel communicationHandles all parallel communication

Other parallel calls (Reductions, etc.)Other parallel calls (Reductions, etc.)

18

Framework Calls: MeshFramework Calls: Mesh

FEM_Set_*FEM_Set_* From From initinit, these routines describe the , these routines describe the

mesh to the FEM Frameworkmesh to the FEM Framework Framework calls the partitionerFramework calls the partitioner

FEM_Get_*FEM_Get_* From each chunk, the local portion of From each chunk, the local portion of

the mesh is obtained by the the mesh is obtained by the driverdriver

19

FEM_*_elemFEM_*_elemFEM_Set_elem(int elType,int nEl,

int doublePerEl,int nodePerEl);

subroutine FEM_Set_elem(elType,nEl,doublePerEl,nodePerEl) integer, intent(in) :: elType,nEl,doublePerEl,nodePerEl

FEM_Get_elem(int elType,int* nEl, int* doublePerEl,int* nodePerEl);

subroutine FEM_Get_elem(elType,nEl,doublePerEl,nodePerEl) integer, intent(in) :: elType integer, intent(out) :: nEl,doublePerEl,nodePerEl

20

FEM_*_nodeFEM_*_node

subroutine FEM_Set_node(nNode,doublePerNode) integer, intent(in) :: nNode,doublePerNode

subroutine FEM_Get_node(nNode,doublePerNode) integer, intent(out) :: nNode,doublePerNode

21

Element ConnectivityElement Connectivitysubroutine FEM_Set_Elem_Conn_r(elType,conn) integer, intent(in) :: elType integer, intent(in), dimension(nodePerEl,nEl) :: conn

subroutine FEM_Get_Elem_Conn_r(elType,conn) integer, intent(in) :: elType integer, intent(out), dimension(nodePerEl,nEl) :: conn

subroutine FEM_Set_Elem_Conn_c(elType,conn) integer, intent(in) :: elType integer, intent(in), dimension(nEl,nodePerEl) :: conn

subroutine FEM_Get_Elem_Conn_c(elType,conn) integer, intent(in) :: elType integer, intent(out), dimension(nEl,nodePerEl) :: conn

22

Additional Data for Nodes Additional Data for Nodes and Elementsand Elements

subroutine FEM_Set_node_data_r(data) REAL*8, intent(in), dimension(doublePerNode,nNode) :: data

subroutine FEM_Get_node_data_r(data) REAL*8, intent(out), dimension(doublePerNode,nNode) :: data

subroutine FEM_Set_elem_data_r(data) REAL*8, intent(in), dimension(doublePerElem,nElem) :: data

subroutine FEM_Get_elem_data_r(data) REAL*8, intent(out), dimension(doublePerElem,nElem) :: data

23

Node FieldsNode Fields

Framework handles combining data for Framework handles combining data for shared nodes and keeps them in syncshared nodes and keeps them in sync

Framework does not understand Framework does not understand meaning of node fields, only their meaning of node fields, only their location and typeslocation and types

Framework needs to be informed of Framework needs to be informed of locations and types of fieldslocations and types of fields

Create_fieldCreate_field once, once, Update_fieldUpdate_field every every timesteptimestep

24

FEM_Create_fieldFEM_Create_field

To handle the updating of shared To handle the updating of shared node values.node values.

Tell the framework where the Tell the framework where the shared data items of each node shared data items of each node are.are.

Creates a “field” and pass the field Creates a “field” and pass the field ID for updating shared nodal ID for updating shared nodal values.values.

25

FEM_Create_simple_fieldFEM_Create_simple_field

function integer :: FEM_Create_simple_field( base_type, vec_len) integer, intent(in) :: base_type, vec_len

Base_type

•FEM_BYTE- INTEGER*1, or CHARACTER*1 •FEM_INT- INTEGER*4 •FEM_REAL- REAL*4 •FEM_DOUBLE- DOUBLE PRECISION, or REAL*8

26

Create_simple_field Create_simple_field ExampleExample

! 3D Force for each node! stored as 3*n real*8 array

REAL*8 ALLOCATABLE, DIMENSION(:) :: nodeForce INTEGER :: fid

... allocate nodeForce as 3*n_nodes...

fid = FEM_Create_simple_field(FEM_DOUBLE,3)

27

FEM_Create_fieldFEM_Create_field

function integer :: FEM_Create_Field(base_type, vec_len, offset, dist)

integer, intent(in) :: base_type, vec_len, offset, dist

28

Node FieldsNode Fields

29

Create_field ExampleCreate_field Example

! 3D force is contained as fXYZ variable! in a user-defined type node_type

TYPE(node_type), ALLOCATABLE, DIMENSION(:) :: nodes INTEGER :: fid

...allocate nodes array as n_nodes...

fid = FEM_Create_Field(FEM_DOUBLE,3,          offsetof(nodes(1), nodes(1)%fXYZ),          offsetof(nodes(1), nodes(2)) )

30

Update and Reduce FieldUpdate and Reduce Fieldsubroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes

subroutine FEM_Reduce_Field(fid,nodes,outVal,op) integer, intent(in) :: fid,op varies, intent(in) :: nodes varies, intent(out) :: outVal op is

•FEM_SUM •FEM_MIN •FEM_MAX

31

UtilityUtility

function integer :: FEM_Num_Partitions()

function integer :: FEM_My_Partition()

function double precision :: FEM_Timer()

subroutine FEM_Print_Partition()

subroutine FEM_Print(str) character*, intent(in) :: str

32

Advanced FEM callsAdvanced FEM calls

FEM_Update_meshFEM_Update_mesh Reassembles chunks of the meshReassembles chunks of the mesh

FEM_Add_nodeFEM_Add_node Adds a new node into the meshAdds a new node into the mesh

33

FEM_Update_meshFEM_Update_mesh Reassembles all the chunksReassembles all the chunks Can be called only from driver Can be called only from driver Must be called from all chunksMust be called from all chunks Useful scenariosUseful scenarios

Giving out data as the simulation runsGiving out data as the simulation runs Repartition the meshRepartition the mesh Serial output when simulation finishesSerial output when simulation finishes

34

FEM_Update_meshFEM_Update_mesh

C call:C call:

void FEM_Update_mesh(int callMeshUpdated, int doWhat)

Fortran call:Fortran call:subroutine FEM_Update_mesh(callMeshUpdated,doWhat)

integer, intent(in) :: callMeshUpdated, doWhat

35

FEM_Update_mesh FEM_Update_mesh parametersparameters

callMeshUpdatedcallMeshUpdated is non-zero => is non-zero => call call mesh_updated mesh_updated (callMeshUpdated)(callMeshUpdated)

doWhat:doWhat:doWhatdoWhat RepartitionRepartition FEM_Update_meshFEM_Update_mesh

00 NoNo Non-blocking Non-blocking mesh_updatedmesh_updated

11 YesYes Blocks for repartitioningBlocks for repartitioning

22 NoNo Blocking mesh_updatedBlocking mesh_updated

36

FEM_Update_mesh FEM_Update_mesh examplesexamples

FEM_Update_meshFEM_Update_mesh(k,0) (k,0) Call mesh_updated(k) on assembled Call mesh_updated(k) on assembled

mesh, while the driver continuesmesh, while the driver continues FEM_Update_meshFEM_Update_mesh(k,1) (k,1)

Repartition after mesh_updatedRepartition after mesh_updated FEM_Update_meshFEM_Update_mesh(k,2)(k,2)

Block driver routines till Block driver routines till mesh_updated(k)mesh_updated(k)

37

Adding NodesAdding Nodes

One can add new nodes, and One can add new nodes, and update connectivity in driverupdate connectivity in driver Use Use FEM_Set_*FEM_Set_* subroutines subroutines New nodes are considered privateNew nodes are considered private

Framework can repartition the Framework can repartition the meshmesh Optionally calls user’s Optionally calls user’s mesh_updatedmesh_updated

subroutinesubroutine

38

FEM_Add_nodeFEM_Add_node

C call:C call:

void FEM_Add_node(int localIdx, int nBetween, int * betweenNodes)

Fortran call:Fortran call:

subroutine FEM_Add_node(localIdx,nBetween,betweenNodes)

integer, intent(in) :: localIdx,nBetween integer, intent(in) :: betweenNodes(nBetween)

>>

39

Installing FEM Installing FEM FrameworkFramework

40

Where to Get It ?Where to Get It ?FEM Framework is included in Charm++ distribution, available under CVSCSH:setenv CVSROOT ":pserver:checkout@charm.cs.uiuc.edu:/cvsroot"Or BASH:export CVSROOT=":pserver:checkout@charm.cs.uiuc.edu:/cvsroot"

You should now be able to do a

> cvs login(no password needed, just type [Enter] at prompt)

and then

> cvs co -P charm

to get the entire Charm++ source.

41

How to Build It ?How to Build It ?

> cd charm

and do

> ./build FEM net-linux -O

This will make a net-linux directory, with bin, include, lib etc subdirectories.

Platforms: net-sol, mpi-origin, mpi-linux etc.

42

How to Write Programs ?How to Write Programs ?

Write from scratchWrite from scratch Concepts and API discussed earlierConcepts and API discussed earlier Read the FEM-Framework Manual Read the FEM-Framework Manual

http://charm.cs.uiuc.edu/manuals/femhttp://charm.cs.uiuc.edu/manuals/fem

Convert existing programConvert existing program To be covered in next sectionTo be covered in next section

43

How to Compile & Link ?How to Compile & Link ?

Use “Use “charmccharmc”: available under bin”: available under bin a multi-lingual compiler driver, understands a multi-lingual compiler driver, understands

f90f90 Knows where modules and libraries areKnows where modules and libraries are Portable across machines and compilersPortable across machines and compilers

LinkingLinking use use “-language femf” “-language femf” : for F90: for F90 Use Use “–language fem”“–language fem” : for C/C++ : for C/C++

See example MakefilesSee example Makefiles pgms/charm++/fem/…pgms/charm++/fem/…

44

How to Run ?How to Run ?

Just run it! (net- versions only)Just run it! (net- versions only) Serial, but nice for debugging/testingSerial, but nice for debugging/testing

Use CharmrunUse Charmrun A portable parallel job execution scriptA portable parallel job execution script Specify number of processors: Specify number of processors: +pN+pN Special “Special “nodelistnodelist” file for net-* versions” file for net-* versions Multiple chunks per processor: use Multiple chunks per processor: use +vpM+vpM

45

Charmrun ExampleCharmrun Example

>>>>

./charmrun pgm +p4 +vp8

Nodelist File: $(HOME)/.nodelist

group main host tur0001.cs.uiuc.edu host tur0002.cs.uiuc.edu host tur0003.cs.uiuc.eduetc…

46

FEM Framework FEM Framework Conversion Conversion exampleexample

47

A Serial ProgramA Serial Program

Input

Processing

Output

48

A Parallel Framework A Parallel Framework ProgramProgram

InputParallelProcessing

Output ParallelInfrastructure

49

Real Names of PiecesReal Names of Pieces

InitDriver

Finalize orMesh_updated Charm++ FEM

Framework

50

Serial Example ProgramSerial Example Program

F90 exampleF90 example Reads input mesh in “Triangle” formatReads input mesh in “Triangle” format Does simple explicit mechanics Does simple explicit mechanics

computation (CST triangles)computation (CST triangles) Writes Tecplot outputWrites Tecplot output

Not a toy example (by Philippe Not a toy example (by Philippe Geubelle)Geubelle) Reads a parameter fileReads a parameter file Applies boundary conditionsApplies boundary conditions Uses real mechanicsUses real mechanics

51

FEM Framework VersionFEM Framework Version

Uses all the same code as serial Uses all the same code as serial versionversion

Main program becomes init subroutineMain program becomes init subroutine Split up init and driverSplit up init and driver

Copy entire mesh into framework Copy entire mesh into framework (FEM_Set)(FEM_Set)

Copy portion of mesh out (FEM_Get)Copy portion of mesh out (FEM_Get) Changes to time loop are minimalChanges to time loop are minimal Output poses an interesting problemOutput poses an interesting problem

52

Splitting the programSplitting the program

Declare variablesDeclare variables

call readTriGlobalscall readTriGlobals InitInit

Read mesh conn. DataRead mesh conn. Data

Dynamics loopDynamics loop

Calc forcesCalc forces

Calc accl,velCalc accl,vel

Apply bound. conditionsApply bound. conditions DriverDriver

Update displacementsUpdate displacements

Output current dataOutput current data

53

Init – Basic differencesInit – Basic differences

Serial versionSerial version

Declare variablesDeclare variables

. . .. . .

call readTriGlobalscall readTriGlobals

. . .. . .Read mesh conn. dataRead mesh conn. data

Parallel versionParallel version

Declare variablesDeclare variables

. . .. . .

call readTriGlobalscall readTriGlobals

. . .. . .Read mesh conn. dataRead mesh conn. data

. . .. . .Call the set methodsCall the set methods

54

DriverDriver

Serial versionSerial version

Dynamics loopDynamics loop

Calc forcesCalc forces

Calc accl,velCalc accl,vel

Apply bound. CondApply bound. Cond

Update dispUpdate disp

Output current Output current datadata

Parallel versionParallel version

Call get methodsCall get methods

Create fieldCreate field

Dynamics loopDynamics loop

Calc forcesCalc forces

Update fieldUpdate field

Calc accl,velCalc accl,vel

Apply bound. CondApply bound. Cond

Update dispUpdate disp

Visualize current Visualize current datadata

55

More information: More information: Read the manual and FEM framework Read the manual and FEM framework

details at details at http://charm.cs.uiuc.eduhttp://charm.cs.uiuc.edu