Moncrief-O’Donnell Chair, UTA Research Institute (UTARI) The … 01 coop ctrl of AC-DC mic… ·...

Moncrief-O’Donnell Chair, UTA Research Institute (UTARI)The University of Texas at Arlington, USA

F.L. Lewis, NAI

Talk available online at http://www.UTA.edu/UTARI/acs

��

Distributed Synchronization Control of Microgrids:Optimal Synchronization on

Sparse Communication GraphsWork with Dr. A. Davoudi

F.L. Lewis, H. Zhang, A. Das, K. Hengster-Movric,Cooperative Control of Multi-Agent Systems:Optimal Design and Adaptive Control, Springer-Verlag, 2013

Key Point

Lyapunov Functions and Performance IndicesMust depend on graph topology

Hongwei Zhang, F.L. Lewis, and Abhijit Das“Optimal design for synchronization of cooperative systems: state feedback, observer and outputfeedback,” IEEE Trans. Automatic Control, vol. 56, no. 8, pp. 1948‐1952, August 2011.

H. Zhang, F.L. Lewis, and Z. Qu, "Lyapunov, Adaptive, and Optimal Design Techniques forCooperative Systems on Directed Communication Graphs," IEEE Trans. Industrial Electronics, vol.59, no. 7, pp. 3026‐3041, July 2012.

6

A. Bidram, V. Nasirian, A. Davoudi, and F.L. Lewis,Cooperative Synchronization inDistributed Microgrid Control, Springer-Verlag, Berlin, 2017.

7

A. Bidram, F.L. Lewis, and A. Davoudi, “Distributed Control Systems for Small‐scale Power Networks Using Multi‐agent Cooperative Control Theory,” IEEE Control Systems Magazine, pp. 56‐77, December 2014 (Featured cover article).

New Research ResultsDistributed Cooperative Control on GraphsReinforcement Learning for Online Optimal ControlMulti‐Player Games on Communication Graphs Output Synchronization of Heterogeneous MAS

Applications to:Building HVAC BalancingAC MicrogridDC Microgrid

The Power of Synchronization Coupled OscillatorsDiurnal Rhythm

1

2

3

4

56

Diameter= length of longest path between two nodesVolume = sum of in-degrees

1

N

ii

Vol d

Spanning treeRoot node

Strongly connected if for all nodes i and j there is a path from i to j.

Tree- every node has in-degree=1Leader or root node

Followers

Synchronization on Communication Graph

State at node i is ( )ix tSynchronization problem

( ) ( ) 0i jx t x t

Strongly Connected implies exists Spanning Tree

Communication Graph1

2

3

4

56

N nodes

[ ]ijA a

0 ( , )ij j i

i

a if v v E

if j N

oN1

Noi ji

jd a

Out-neighbors of node iCol sum= out-degree

42a

Adjacency matrix

0 0 1 0 0 01 0 0 0 0 11 1 0 0 0 00 1 0 0 0 00 0 1 0 0 00 0 0 1 1 0

A

iN1

N

i ijj

d a

In-neighbors of node iRow sum= in-degreei

(V,E)

i

Algebraic Graph Theory

Dynamic Graph- the Distributed Structure of ControlEach node has an associated state i ix u

Standard local voting protocol ( )i

i ij j ij N

u a x x

1

1i i

i i ij ij j i i i iNj N j N

N

xu x a a x d x a a

x

( )u Dx Ax D A x Lx L=D-A = graph Laplacian matrix

x Lx

If x is an n-vector then ( )nx L I x

x

1

N

uu

u

1

N

dD

d

Closed-loop dynamics

i

j

[ ]ijA a

Global Form

Distributed Controlled Consensus: Cooperative Tracker

Node state i ix u

Distributed Local voting protocol

0( ) ( )i

i ij j i i ij N

u a x x g x x

control node

Synchronization problem

( ) ( ) 0i jx t x t

( )ix t

Theorem. Let graph have a spanning treeand for at least one root node. Then the distributed protocol makes

0ig ( ) ( ) 0i jx t x t

Sparse Communication Graph Topology

Highly efficient fast algorithmsScalable to any nodesLow communication overhead

iN is the set of immediate neighbors of agent i

0( ) 1x L G x G x {g }iG diag

0x

i i

i i ij i ij j ij N j N

u g a x a x g v

20

Balancing Building HVAC Ventilation Systems

SIMTech 5th floor temperature distribution

Work with SIMTech – Singapore Inst. Manufacturing Technology

Automated HVAC control system

AHUFan

C 1 C 2

CWRCWS

Air Flow

Diffuser outlets

VSD

Control Panel

Control stationVAV box

Room thermostat

Air diffuser

LEGENDS

Extra WSN temp. sensors

SIMTech

( 1) ( ) ( ) ( )i i i ix k x k f x u k 1( ) ( ) ( ( ) ( ))

1i

i i ij j ij Ni

u k k a x k x kn

( 1) ( ) ( ) ( )i i i ix k x k f x u k

1( ) ( ) ( ( ) ( ))1

i

i i ij j ij Ni

u k k a x k x kn

1 12 4( ) 1, , ,...i k

Control damper position based on local voting protocol

Temperature dynamics

Unknown fi(x)

Under mild conditions this converges to steady-state desired temp. distribution

Adjust Dampers for desired Temperature distributionSIMTech

Open Research Topic - HVAC Flow and Pressure control

23

Cooperative Control forAC Microgrids

What is a micro‐grid?• Micro-grid is a small-scale power system that provides the power for

a group of consumers.• Micro-grid enables

local power support for local and critical loads.

• Micro-grid has the ability to work in both grid-connected and islanded modes.

• Micro-grid facilitates the integration of Distributed EnergyResources (DER).

Photo from: http://www.horizonenergygroup.com

AC Microgrid Frequency and Voltage Synchronization

• The main building block of smart-grids• Rural plants• Business buildings, hospitals,and factories

Smart‐grid photo from: http://www.sustainable‐sphere.com

An introduction to micro‐grids: Micro‐grid applications

Distributed Generators (DG)Distributed Energy Resources (DER)

• Non-renewables Internal combustion engine Micro-turbines Fuel cells

• Renewables Photovoltaic Wind Hydroelectric Biomass

26

Micro‐grid Advantages

• Micro-grid provides high quality and reliable power to the criticalconsumers

• During main grid disturbances, micro-grid can quickly disconnectform the main grid and provide reliable power for its local loads

• DGs can be simply installed close to the loads which significantlyreduces the power transmission line losses

• By using renewable energy resources, a micro-grid reduces CO2emissions

27

AC vs. DC Microgrids

• Voltage and frequency synchronization for both grid-connected and islanded operating modes

• Proper load sharing and DG coordination• Power flow control between the microgrid and the main grid• Optimizing the microgrid operating cost

Hierarchical control structure28

An introduction to AC micro‐grids: Micro‐grid Objectives

Micro‐grid Hierarchical Control Structure

Tertiary Control-Economic dispatch

modesOptimal operation in both operating

modes

Secondary Control

Primary Control

MicrogridTie

Power flow control in grid-tied mode

Voltage deviation mitigationFrequency deviation alleviation

Voltage stability provision

preservingFrequency stability

preservingPlug and play capability for DGs

Main grid

Do coop. ctrl. here toSynchronize frequencyand voltage

Bidram, A., & Davoudi, A. (2012). Hierarchical structure of microgrids control system. IEEE Transactions on Smart Grid, vol. 3, pp. 1963‐1976, Dec 2012.

Maintains Stabilityof individual DGs

Secondary control: The secondary control restores the voltage andfrequency of the micro-grid to their nominal value after islanding.

Conventional Secondary control implementation: Centralized structure

Low reliability – single point of failure Requires a Central control authority Requires too many communication links Not scalable to many DGs

We want to develop a new Distributed Control structure Highly reliable Uses sparse communication network

31

( ) ( )

( ) ( )

n PE ref mag IE ref mag

n P ref I ref

V K v v K v v dt

K K dt

Standard Micro‐grid secondary control

33

Microgrid

DG 1DG 2 DG 3

DG 4

DG 5

DG 6DG 7

DG 8

DG 1DG 2 DG 3

DG 4

DG 5

DG 6

DG 8

DG 7

Communication link

Cybercommunication

framework

Micro‐grid secondary control:New Distributed CPS structure

Physical LayerThe interconnect structure of the power grid

Primary Control

Cyber layerA sparse, efficient communication network to allow

cooperative control for synchronization ofvoltage and frequency

Secondary Control

Work of Ali BidramWith Dr. A. Davoudi

Cyber Physical System (CPS)

34

Microgrid

DG 1DG 2 DG 3

DG 4

DG 5

DG 6DG 7

DG 8

DG 1DG 2 DG 3

DG 4

DG 5

DG 6

DG 8

DG 7

Communication link

Cybercommunication

framework

The Importance of the Communication Network ‐Interactions Between Communication and Control





Local controller design must take into account the Graph Topology

Synchronization on Good Graphs

Chris Elliott fast video

65

34

2

1

1

2 3

4 5 6

Regular mesh

Synchronization Speed depends on communication topology

Synchronization on Gossip Rings

Chris Elliott weird video

12

3

4 5

6

Graph Laplacian L has complex eigenvalues

40

1. Distributed secondary frequency control of micro-grids2. Distributed secondary voltage control of micro-grids


Synchronization in AC Microgrid of Interconnected DG

DER 8 DER 6

DER 4

Rline1 Lline1

Pload1+jQload1

Rline2 Lline2Rline3 Lline3

Rc4Lc4

Rc3Lc3

Rc2Lc2

Rc1Lc1

vo4vo3vo2vo1

Pload2+jQload2

DER 3DER 2DER 1

DER 5DER 7

Pload3+jQload3Pload4+jQload4

Rline7 Lline7 Rline6 Lline6 Rline5 Lline5

Rline4

Lline4

vo5vo6vo7vo8

Lc5Lc6Lc7Lc8Rc8 Rc7 Rc6 Rc5

DG 1 DG 2 DG 3 DG 4

DG 8 DG 7 DG 6 DG 5

Voltage synchronization (per unit)

i i ni Pi iy m P Frequency synchronization

Voltage synchronization

odi ni Qi iE v V n Q

Dynamical model of a DG

43

vo io

VSC

vod*

iLd*

Currentcontroller

Voltagecontroller

LC filteriL

Power controller

vb

, voq*

vod , voq

iod , ioq

, iLq*

ω

ωn Vn

Outputconnector

Rc Lc

abc/dq

iLd , iLq

Rf Lf Cf

VSC‐ Voltage source converter Power electronics

Renewable DERProvides DC voltage

Primary ControlDroop control is here

MicrogirdNetworkLoad disturbances

Given load conditions ‐ pick using Droop to maintain stability0 0,v i * *, ,od oqv v

Primary Control Structure

iv

Voltage synchronization

Frequency synchronization


Power controller dynamics:

44

( ) ( ) ( )( )

i i i i i i i i i

i i i i i

uy h d u

x f x k x D g xx

13i x

abc/dqvoi

ioi

vodivoqiiodiioqi

vodi iodi + voqi ioqi

voqi iodi - vodi ioqi

Low-passfilter

ωni - mPi PiPi

Low-passfilter

Vni - nQi Qi

Qi

ωi

vodi*

voqi*0

ωni

Vni

[ ]Ti i i i di qi di qi ldi lqi odi oqi odi oqiP Q i i v v i i x

i i com

( )i ci i ci odi odi oqi oqiP P v i v i

( )i ci i ci oqi odi odi oqiQ Q v i v i

Pogaku, N., Prodanovic, M., & Green, T. C. (2007). Modeling, analysis and testing of autonomous operation of an inverter‐based microgrid. IEEE Transactions on Power Electronics, 22(2), 613–625.

Droop control is here

Heterogeneous agent dynamics


Voltage controller dynamics

45

Σ

Σ

vodi*

voqi*

vodi

voqi

KPViKIVi

+ s+

+

_

_

KPViKIVi

+ s

ωbCfi

ωbCfi

Fi

Fi

Σ

Σ

vodi

voqi

+

+

_Σ

Σ

+

+

iodi

ioqi

+

++

ildi*

ilqi*

* ,di odi odiv v

* ,qi oqi oqiv v

* *( ) ,ldi i odi b fi oqi PVi odi odi IVi dii F i C v K v v K

* *( ) ,lqi i oqi b fi odi PVi oqi oqi IVi qii Fi C v K v v K


Current controller dynamics

46

*di ldi ldii i

*qi lqi lqii i

* *( )idi b fi lqi PCi ldi ldi ICi div L i K i i K

* *( )iqi b fi ldi PCi lqi lqi ICi qiv L i K i i K

Σ

Σ

vidi*

viqi*

KPCiKICi

+ s+

+

_

_

KPCiKICi

+ s

ωbLfi

ωbLfi

Σ

Σ

+

+

_

ilqiildi

+

ildi*

ilqi*

ildi

ilqi


Output filter dynamics

47

Output connector dynamics

1 1fildi ldi i lqi idi odi

fi fi fi

Ri i i v v

L L L

1 1filqi lqi i ldi iqi oqi

fi fi fi

Ri i i v v

L L L

1 1odi i oqi ldi odi

fi fiv v i i

C C

1 1oqi i odi lqi oqi

fi fiv v i i

C C

1 1ciodi odi i oqi odi bdi

ci ci ci

Ri i i v vL L L

1 1cioqi oqi i odi oqi bqi

ci ci ci

Ri i i v vL L L

Depends on microgrid conditions and loads

Voltage disturbances

48

,1 ,2

,2 ,6 ,8 ,7 ,9 ,2

,3 ,6 ,9 ,7 ,8 ,3

,3 ,6,4 ,4 ,5

,7,5 ,5 ,4

,4 ,8,6 ,7

,7

( )( )

i ni Pi i com

i ci i i i i i

i ci i i i i i

fi ni Qi i ii i com i

fi fi

fi ii i com i

fi fi

i ii com i

fi

i

x m xx x x x x xx x x x x x

r V n x xx x x

L L

r xx x x

L L

x xx x

C

x

,5 ,9,6

,6,8 ,8 ,9

,7,9 ,9 ,8

i icom i

fi

i bdicii i com i

ci ci

i bqicii i com i

ci ci

x xx

C

x vrx x x

L Lx vr

x x xL L

The nonlinear dynamics of the ith DG, while neglecting the fast dynamics of voltage and current controllers

[ ] .Ti i i i Ldi Lqi odi oqi odi oqix P Q i i v v i i

From adaptive voltage ctrl‐ Trans CST paper

1. For secondary frequency control: synchronize

2. For secondary voltage control: synchronize

50

( ) ( ) ( )( )

i i i i i i i i i

i i i i i

uy h d u

x f x k x D g xx

13i x

i oiy v

i niu V

i i ni Pi iy m P

i niu

0id

0id

DG Microgrid Model and Synchronization Control Objectives

Heterogeneous Agent Dynamics – different dynamics


Droop Control in Primary Loop

i

odiv

Micro‐grid Primary Control

Primary control: The primary control maintains voltage andfrequency stability

Conventional primary control: Droop techniques

n P

od n Q

m PE v V n Q

Power calculator

vo

io

Q

P

ω

E E

ω

*Reference

voltage

Esin(ωt)

vo

P

Q

2

1 max1 maxP PN Nm P m P

1 max1 maxQ QN Nn Q n Q

Microgrid load conditions Resulting

Power

Droop Control

Required voltage and frequencyTo maintain stability

How to Synchronize? Look at Power controller dynamics

Power Controller

i

iP

ni

maxP i

Pim

New Secondary Control Input for Frequency Synchronization

i ni Pi im P

Change

To synchronize

ni

i

Secondary Control input

1. Secondary Frequency Control

Existing power conditions in the microgrid

Prescribed frequencye.g. 60 Hz

Primary Droop Control


53

i ni Pi im P

i ni Pi i im P u

i i iu c e

( ) ( )i

i ij i j i i refj N

e a g

Theorem . Let the digraph of the communication network have a spanning tree and the pinning gain be nonzero for at least one DG placed on a root node.

Let the auxiliary control be chosen as above.

Then, the global neighborhood error is asymptotically stable. Moreover, the DG frequencies synchronize to

iu

ref

Droop control relationship

Using input-output feedback linearization

( ) ( )i

ni Pi i i Pi i i ij i j i i refj N

m P u m P c a g

Then


54

ref

ij N

( ) ( )ij i j i i refj

a g ie iu nii

ix

pim

1s

( ) ( )( )

i i i i i ii i i i i

uy h x du

x f x g xic

j

calculating iP

Restores Frequency Synchronization after islanding

i ni Pi i im P u

Feedback Linearization Inner Loop

Distributed Cooperative Tracker

1. Secondary frequency control

DER 8 DER 6

DER 4

Rline1 Lline1

Pload1+jQload1


Rc4Lc4

Rc3Lc3

Rc2Lc2

Rc1Lc1

vo4vo3vo2vo1

Pload2+jQload2

DER 3DER 2DER 1

DER 5DER 7



Rline4

Lline4

vo5vo6vo7vo8


DER 1DER 2DER 3DER 4 LeaderDER 5DER 6DER 7DER 8

DG 1 DG 2 DG 3 DG 4

DG 8 DG 7 DG 6 DG 5

DG 5DG 6DG 7DG 8 DG 4 DG 3 DG 2 DG 1

Simulation Example

Physical MicrogridNetwork

Cyber communication network‐ sparse

1. Secondary frequency control

56

0 0.5 1 1.5 2 2.5 3

49.6

49.8

50

50.2

t (s)

f (H

z)

DER1DER2DER3DER4DER5DER6DER7DER8

DG 1

DG 2

DG 3

DG 4

DG 5

DG 6

DG 7

DG 8

Islanding Turn onCoop secondary control

Ref. FrequencyIs 50 Hz

Secondary frequency and power control

Guarantees that (( )( ) ) 0.ref Pc L G Lm P

( ( ) ( ) ( ))i i

i ij i j i i ref ij Pi i Pj jj N j N

u c a g a m P m P

The local neighborhood tracking error control

There is another relation between power and frequency in the microgrid DGs

sin( ) sin( ),oi bii i i i

ci

v vP h

X Write the output active power as

So that approximately ( ),i i i refP h

( ),refP h In global form

Therefore at steady state all frequencies synchronize to the reference frequency

( ) 0ref PG Lm P So that

This does not guarantee synchronization of freq. and power separately

Secondary frequency and power control

60

Σ DG iωni

Piui

aij ( ωi -ωj )+gi (ωi -ωref)j

Nij∈_

Σωj ωi

s1

ωref

c

aij ( mPiPi -mPjPj )j

Nij∈

ΣPj

_

TWO CONTROL OBJECTIVES WITH ONE CONTROL INPUT

Cooperative tracker

Cooperative regulator

ni i Pi i im P u

( ( ) ( ) ( ))i i

i ij i j i i ref ij Pi i Pj jj N j N

u c a g a m P m P

Frequency synchronization Power consensus

Simulation results

61



Simulation results

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6311

312

313

314

315

316

ω

0.2 0.4 0.6 0.8 1 1.2 1.4 1.60

10

20

30

40

time (s)

P (k

W)

DG1DG2DG3DG4

(a)

(b)

(rad

/s)


Ref. FrequencyIs 50 Hz

63

2. Secondary Voltage ControlMicrogrid of Interconnected DG

DER 8 DER 6

DER 4

Rline1 Lline1

Pload1+jQload1


Rc4Lc4

Rc3Lc3

Rc2Lc2

Rc1Lc1

vo4vo3vo2vo1

Pload2+jQload2

DER 3DER 2DER 1

DER 5DER 7



Rline4

Lline4

vo5vo6vo7vo8


DG 1 DG 2 DG 3 DG 4

DG 8 DG 7 DG 6 DG 5

2 2,o magi odi oqiv v v

2. Voltage synchronization (per unit)

i i ni Pi iy m P 1. Frequency synchronization


Synchronize per‐unitvoltages

mag n QE v V n Q

odiv

iQ

niV

maxQ i

Qin

New Secondary Control Input for Voltage Synchronization

Change

To change

To synchronize

niV

odiv

Secondary Control input

2. Secondary Voltage Control

Existing power conditions in the microgrid

Prescribed voltage


oiv

Primary Droop Control

2. Secondary Voltage Control

67

If , there is no direct relationship between the output and

input .

i oiy v

i niu V

Input-output feedback linearization for heterogeneous nonlinear agents

( ) 1i i i

r r ri i i iy L h L L h u F g F

1i i i

r ri i i iv L h L L h u F g F

1 1( ) ( )i i i

r ri i i iu L L h L h v g F F

( ) ,ri iy v i

,1

,1 ,2

, 1

,

i i

i i

i r i

y yy y

i

y v

( ) ( ) ( )( )

i i i i i i i i i

i i i

uy h

x f x k x D g xx

( ) ( ) ( )i i i i i i i F x f x k x D

, ,i i iv i BA

Assume relative degree r is the same for all agentsZero dynamics can be different, but assume they are stable

Must use Lie derivatives

0id

( , ),ii i iW i Internal Dynamics

2. Secondary voltage control

68

0 1 0 0 00 0 1 0 00 0 0 1 0

0 0 0 0 10 0 0 0 0 0 r r

A

1[0,0, ,1]TrB

Leader node dynamics

The synchronization problem is to find a distributed such that iv

0 0 0

0 0 0

( )( )y h

x f xx

, ,i i iv i BA ,1 , 1[ ]Ti i i i ry y y

( )0 0 0 ,ry BY AY ( 1)

0 0 0 0[ ]r Ty y y Y

0, .i i Y

Assumption. The vector is bounded so that , with a finite but generally unknown bound.

( ) ( )0 0 ,r r

N y r y 1 ( )0r r

MYy

DG Agent Dynamics

( , ),ii i iW i Internal dynamics

Theorem. Let the digraph of the multi-agent system have a spanning tree and the pinning gain be nonzero for at least one root node.

Let all agents have stable zero dynamics

Let the auxiliary control be chosen as

is the first eigenvalue of


i iv cK e

where is the coupling gain, and is the feedback control gain.

Then, are cooperative UUB with respect to and all nodes synchronize to if is chosen as

c R 1 rK R

1 rK R 1

1,TK R P B 1

1 1 1 1 0.T TP P Q P R P BA BA

andmin

1 ,2

c

min min ( )i iRe i L G

0( ) ( )i

iiN

ii jj ij

a g

e Y

i 0Y

0Y

Zhang, H., Lewis, F. L., & Das, A. (2011).Optimal design for synchronization of cooperativesystems: State feedback, observer, and output feedback. IEEE Transactions on AutomaticControl, 56(8), 1948–1952.

(0, ),i i iW i

Secondary voltage control

70

Σ DG i

Mi (xi)

cKVni

xi-vi

_aij ( yi -yj )+gi ( yi -y0)

j

Nij∈

ei_Σ 1

Ni

y0 =vref

0

vodjvodj

yj =

vodivodi

yi =

(2) 2 1i i ii i i iy L h L L h u F g F

2 1i i ii i i iv L h L L h u F g F

1 1 2( ) ( )i i ii i i iu L L h L h v g F F

( ), .i i i

nii

v MV i

N

x

Synchronizes Output voltages after Islanding

iu

Feedback Linearization Inner Loop


DER 8 DER 6

DER 4

Rline1 Lline1

Pload1+jQload1


Rc4Lc4

Rc3Lc3

Rc2Lc2

Rc1Lc1

vo4vo3vo2vo1

Pload2+jQload2

DER 3DER 2DER 1

DER 5DER 7



Rline4

Lline4

vo5vo6vo7vo8


DER 1DER 2DER 3DER 4 LeaderDER 5DER 6DER 7DER 8

DG 1 DG 2 DG 3 DG 4

DG 8 DG 7 DG 6 DG 5

DG 5DG 6DG 7DG 8 DG 4 DG 3 DG 2 DG 1

Simulation Example



Simulation results

72

1 1.2 1.4 1.6 1.8 2350

360

370

380

390

t (s)

v o,m

ag (V

)

DER1DER2DER3DER4DER5DER6DER7DER8

DG 1

DG 2

DG 3

DG 4

DG 5

DG 6

DG 7

DG 8


Ref. Per‐unitVoltageIs 380 V


Adaptive Voltage Control

Σ

Vni

__

ωni

Adaptivesecondary

voltage control

voi ioivoqdi*

Voltage and currentcontroller

Power controller

vb

Rci LciRfi Lfi Cfi

Primarypowersource

iL

ωi

calculatorri _cYi =[vodi vodi]

T

Y-i =[vod(-i) vod(-i)]T

di+bi

1

di+bi

1

ˆ î i i i i iif f f f f fW F r F Wφ κ= −�

î

TfW

ifφ

î

TgW

igφ ˆ î i i i i ig g g i g g gW F r F Wφ κ= −�

Using Neural Network to compensate for unknown nonlinear dynamics

Load changePrimary alone!

Secondary

( ), ( )i i i iM x N x

1 1 2( ) ( )i i ii i i iu L L h L h v g F F

Multiobjective Distributed Secondary Control

Frequency Control

Voltage Control

Synchronizes frequencyCooperative tracker

Active load sharingCooperative regulator

Restore voltagesCooperative tracker

Reactive load sharingCooperative regulator

75

Microgrid

DG 1DG 2 DG 3

DG 4

DG 5

DG 6DG 7

DG 8

DG 1DG 2 DG 3

DG 4

DG 5

DG 6

DG 8

DG 7

Communication link

Cybercommunication

framework

Micro‐grid secondary control:New Distributed CPS structure


Primary Control



Secondary Control



77

Work of Vahidreza Nasirian with Ali Davoudi

Game-theoretic Control for DC Microgrids

AC Microgrid:

1) Complex synchronization procedure for grid-tied operation (frequency, magnitude, and phase match is required)

2) Complex control circuitry (voltage, frequency, and active/reactive power control)

3) Unwanted transmission loss due to reactive power exchange

4) Redundant dc-ac-dc conversions for integration of renewable sources, loads, and storage units

5) Harmonic current management and phase unbalances

DC Microgrid:

1) Only voltage and power control is needed2) No reactive power flow and, thus, an

improved overall efficiency3) Converted renewable energies are

basically dc and, thus, a dc distribution is more effective for integration of these sources

4) No harmonic current or phase unbalance issue

78

Advantages of DC Microgrids

Cooperative Game-theoretic Control of Active Loads in DC Microgrids Ling-ling Fan, Vahidreza Nasirian,

Hamidreza Modares, Frank L. Lewis, Yong-duan Song, and Ali Davoudi,

3t

2t

1t

3t

2t

1t

e

outp

inp

e

outp

3t

2t

1t

inp

Power buffer operation during a step change in power demand.

Supplies excess power needed during load changes until sources can respond

18r

48r

58r

59r

47r

27r

67r

69r

39r

iv i

p

s1vs1r

iu

ie

Power buffers in Microgrid Network

Background Work of Wayne Weaver

2

,i

i ii

i i

ve p

rr u

ìïïï = -ïíïï =ïïî

Active Load Power Buffer

Stored energyInput impedanceBus voltage Control input Output power = a disturbance

ieir

iviu

ip

Vahid Nasirian

Nonlinear dynamicsNot obvious how to handle ip

2

0

d , 1, , ,i

i j ij j i ij N

J u t i M M Nr¥

Î

æ ö÷ç ÷ç= + = + +÷ç ÷ç ÷çè øåò x Q xT

Define coupled performance indices

( )

2q q

1( )q

0 00 2 1

1 00 0 0

0 10 0 0

2 0 ,

0

i i i ii

i ii ii i

i i i i

i i

M N

ij jj M i

i

e ei i

r r u w

p p

r

i

g

g+

= + ¹

é ùé ù é ù é ù é ù- -ê úê ú ê ú ê ú ê úê úê ú ê ú ê ú ê úê ú= + + +ê ú ê ú ê ú ê úê úê ú ê ú ê ú ê úê úê ú ê ú ê ú ê úê úë û ë û ë û ë ûë û

é ùê úê úê úê ú+ ê úê úê úê úë û

å

x x B DA

1, , ,i M M N= + +

Solve for bus voltage to get coupled agent dynamics

Define Communication GraphSparse efficient topologyOptimal design provides Resilience

and disturbance rejection

Vahid NasirianReza Modares

Dr. Ali DavoudiLinearize.Add as a state.Formulate as H‐infinity Problem.

ip

Coupling terms

82

Optimal Cooperative Control as a Dynamic Graphical Game

82

Minimize the performance function for active loads

Ji x jTQijx j

jNi

iui2

dt

0

Let’s define the neighborhood state vector as xi xiT, x j

T jNi T

The optimal solution is in a general form of

With such solutions, the performance function Ji is quadratic in x:

ui kixi

Ji (xi ) xiTPixi

which helps to find the optimal solution by solving an algebraic Riccati equation

ui* Bii

TPixi i1

Graphical Game

83

Optimal Cooperative Control: Policy Iteration finds Optimal Solutions

83

• Substituting the optimal solution in Bellman equations leads to the following coupled Algebraic Riccati Equations (ARE)

• Policy iteration (a class of reinforcement learning) is used to solve ARE and find Pi and the optimal control input

• Policy evaluation: the performance of a given control policy, ui, is evaluated using the Bellman equation, and Pi are found.

• Policy improvement: an improved control policy, ui, is found for each agent, using Pi found in the first step.

• Policy evaluation and improvement are repeated until no improvement in control policies, ui, of any agent is observed.

Hi xiTQixi

T +i ui* 2

+xiTPi Aixi Biui

* Diwi (xi ) + Aixi Biui

* Diwi (xi ) TPixi =0

ui* ui

*, uj* jNi

T

ui* Bii

TPixi i1

(a) DC microgrid system(b) Active load(c) Communication network

84

Controller Implementation

Microgrid Setup and Cooperative Controller

Controller Performance with Load Change

85

(a) microgrid bus voltages at the load terminals, (b) Output voltage of the power buffers, (c) output voltage across theresistive loads, (d) Source currents, (e) Stored energies in power buffers, (f) Input impedance of the power buffers, (g)Output of the active loads, (h) energy-impedance trajectory of power buffers during the load transient.

Load change in bus 5; Buffers 4 & 5 assisting Load change in bus 4; Multiple assistive buffers

86

Real‐Time Optimal Cooperative Control:Reinforcement Learning

Optimal Control is Effective for:Aircraft AutopilotsVehicle engine controlAerospace VehiclesShip ControlIndustrial Process Control

Multi-player Games Occur in:Networked Systems Bandwidth AssignmentEconomicsControl Theory disturbance rejectionTeam gamesInternational politicsSports strategy

But, optimal control and game solutions are found byOffline solution of Matrix Design equationsA full dynamical model of the system is needed

Optimality and Games

88

t

T

t

dtRuuxQdtuxrtxV ))((),())((

Nonlinear System dynamics

Cost/value

( , ) ( ) ( )x f x u f x g x u

The Importance of Optimal Control

Formulate an Optimal Control Problem

Then you can always learn the optimal solution online using data measured in real time

By using Integral Reinforcement Learning

DDO‐ Data‐Driven Optimization

xux Ax Bu

SystemControlK

PBPBRQPAPA TT 10

1 TK R B P

On-line real-timeControl Loop

Off-line Design LoopUsing ARE

Optimal Control- The Linear Quadratic Regulator (LQR)

An Offline Design Procedurethat requires Knowledge of system dynamics model (A,B)

System modeling is expensive, time consuming, and inaccurate

( , )Q R

User prescribed optimization criterion ( ( )) ( )T T

t

V x t x Qx u Ru d

Adaptive Control is online and works for unknown systems.Generally not Optimal

Optimal Control is off-line, and needs to know the system dynamics to solve design eqs.

Reinforcement Learning turns out to be the key to this!

We want to find optimal control solutions Online in real-time Using adaptive control techniquesWithout knowing the full dynamics

For nonlinear systems and general performance indices

Bring together Optimal Control and Adaptive Control

D. Vrabie, K. Vamvoudakis, and F.L. Lewis,Optimal Adaptive Control and DifferentialGames by Reinforcement LearningPrinciples, IET Press,2012.

BooksF.L. Lewis, D. Vrabie, and V. Syrmos,Optimal Control, third edition, John Wiley andSons, New York, 2012.New Chapters on:

Reinforcement LearningDifferential Games

t

T

t

dtRuuxQdtuxrtxV ))((),())((

( , , ) ( , ) ( , ) ( ) ( ) ( , ) 0T TV V VH x u V r x u x r x u f x g x u r x u

x x x

112( ) ( )T Vu h x R g x

x

dxdVggR

dxdVxQf

dxdV T

TT *1

*

41

*

)(0

, (0) 0V

Nonlinear System dynamics

Cost/value

Bellman Equation, in terms of the Hamiltonian function

Stationary Control Policy

HJB equation

CT Systems‐ Derivation of Nonlinear Optimal Regulator

Off‐line solutionHJB hard to solve. May not have smooth solution.Dynamics must be known

Stationarity condition 0Hu

( , ) ( ) ( )x f x u f x g x u

Leibniz gives Differential equivalent

To find online methods for optimal control Focus on these two equations

Problem‐ System dynamics shows up in Hamiltonian

),,(),(),(0 uxVxHuxruxf

xV T

0 ( , ( )) ( , ( ))T

jj j

Vf x h x r x h x

x

(0) 0jV

1121( ) ( ) jT

j

Vh x R g x

x

CT Policy Iteration – a Reinforcement Learning Technique

• Convergence proved by Leake and Liu 1967, Saridis 1979 if Lyapunov eq. solved exactly

• Beard & Saridis used Galerkin Integrals to solve Lyapunov eq.• Abu Khalaf & Lewis used NN to approx. V for nonlinear systems and proved convergence

RuuxQuxr T )(),(Utility

The cost is given by solving the CT Bellman equation

Policy Iteration Solution

Pick stabilizing initial control policy

Policy Evaluation ‐ Find cost, Bellman eq.

Policy improvement ‐ Update control Full system dynamics must be knownOff‐line solution

dxdVggR

dxdVxQf

dxdV T

TT *1

*

41

*

)(0

Scalar equation

M. Abu-Khalaf, F.L. Lewis, and J. Huang, “Policyiterations on the Hamilton-Jacobi-Isaacs equation for H-infinity state feedback control with input saturation,”IEEE Trans. Automatic Control, vol. 51, no. 12, pp.1989-1995, Dec. 2006.

( ) ( )u x h xGiven any admissible policy

0 ( )h x

Converges to solution of HJB

LQR Policy iteration = Kleinman algorithm

1. For a given control policy solve for the cost:

2. Improve policy:

If started with a stabilizing control policy the matrix monotonically converges to the unique positive definite solution of the Riccati equation.

Every iteration step will return a stabilizing controller. The system has to be known.

ju K x

0 T Tj j j j j jA P P A Q K RK

11

Tj jK R B P

j jA A BK

0K jP

Kleinman 1968

Bellman eq. = Lyapunov eq.

OFF‐LINE DESIGNMUST SOLVE LYAPUNOV EQUATION AT EACH STEP.

Matrix equation

Lemma 1 – Draguna Vrabie

Solves Bellman equation without knowing f(x,u)

( ( )) ( , ) ( ( )), (0) 0t T

t

V x t r x u d V x t T V

0 ( , ) ( , ) ( , , ), (0) 0TV Vf x u r x u H x u V

x x

Allows definition of temporal difference error for CT systems

( ) ( ( )) ( , ) ( ( ))t T

t

e t V x t r x u d V x t T

Integral reinf. form (IRL) for the CT Bellman eq.Is equivalent to

( ( )) ( , ) ( , ) ( , )t T

t t t T

V x t r x u d r x u d r x u d

value

Key Idea- US Patent

Work of Draguna Vrabie 2009Integral Reinforcement Learning

Bad Bellman Equation

Good Bellman Equation

( ( )) ( , ) ( ( ))t T

k k kt

V x t r x u dt V x t T

IRL Policy iteration

Initial stabilizing control is needed

Cost update

Control gain update

f(x) and g(x) do not appear

g(x) needed for control update

Policy evaluation‐ IRL Bellman Equation

Policy improvement11

21 1( ) ( )T kk k

Vu h x R g xx

),,(),(),(0 uxVxHuxruxf

xV T

Equivalent to

Solves Bellman eq. (nonlinear Lyapunov eq.) without knowing system dynamics

CT Bellman eq.

Integral Reinforcement Learning (IRL)- Draguna Vrabie

D. Vrabie proved convergence to the optimal value and controlAutomatica 2009, Neural Networks 2009

Converges to solution to HJB eq.dx

dVggRdx

dVxQfdx

dV TTT *

1*

41

*

)(0

( ( )) ( ) ( ( ))t T

Tk k k k

t

V x t Q x u Ru dt V x t T

Approximate value by Weierstrass Approximator Network ( )TV W x

( ( )) ( ) ( ( ))t T

T T Tk k k k

t

W x t Q x u Ru dt W x t T

( ( )) ( ( )) ( )t T

T Tk k k

t

W x t x t T Q x u Ru dt

regression vector Reinforcement on time interval [t, t+T]

kWNow use RLS along the trajectory to get new weights

Then find updated FB1 11 1

2 21 1( ( ))( ) ( ) ( )

( )

TT Tk

k k kV x tu h x R g x R g x Wx x t

Nonlinear Case- Approximate Dynamic Programming

Direct Optimal Adaptive Control for Partially Unknown CT Systems

Value Function Approximation (VFA) to Solve Bellman Equation– Paul Werbos (ADP), Dimitri Bertsekas (NDP)

Scalar algebraic equation with vector unknowns

Same form as standard System ID problems in Adaptive Control

Optimal ControlandAdaptive Controlcome togetherOn this slide.Because of RL

Adaptive Critic structure Reinforcement learning

Two Learning NetworksTune them Simultaneously

Synchronous Online Solution of Optimal Control for Nonlinear Systems

A new form of Adaptive Control with TWO tunable networks

A new structure of adaptive controllers

K.G. Vamvoudakis and F.L. Lewis, “Online actor-critic algorithm to solve the continuous-time infinitehorizon optimal control problem,” Automatica, vol. 46, no. 5, pp. 878-888, May 2010.

11 1 1 2 22

( ( )) 1ˆ ˆ ˆ ˆ( ( )) ( )41 ( ( )) ( ( ))

tT T

Tt T

x tW a x t W Q x W D W dx t x t

1 12 2 2 2 1 1 2 2 14 2( ( ))ˆ ˆ ˆ ˆ ˆ( ( )) ( )

1 ( ( )) ( ( ))

TT

T

x tW a F W F x t W a D x W Wx t x t

A New Class of Adaptive Control

Plantcontrol output

Identify the Controller-Direct Adaptive

Identify the system model-Indirect Adaptive

Identify the performance value-Optimal Adaptive

)()( xWxV T

Sun Tz bin fa孙子兵法

Games on Communication Graphs

500 BC

Multi‐player Game SolutionsIEEE Control Systems Magazine,February 2017

Multi‐player Differential Games

F.L. Lewis, H. Zhang, A. Das, K. Hengster-Movric, Cooperative Control of Multi-Agent Systems: Optimal Design and Adaptive Control, Springer-Verlag, 2013

Key Point

Lyapunov Functions and Performance IndicesMust depend on graph topology

Hongwei Zhang, F.L. Lewis, and Abhijit Das“Optimal design for synchronization of cooperative systems: state feedback, observer and output feedback,”IEEE Trans. Automatic Control, vol. 56, no. 8, pp. 1948-1952, August 2011.

H. Zhang, F.L. Lewis, and Z. Qu, "Lyapunov, Adaptive, and Optimal Design Techniques for Cooperative Systems on Directed Communication Graphs," IEEE Trans. Industrial Electronics, vol. 59, no. 7, pp. 3026‐3041, July 2012.

,i i i ix Ax B u

0 0x Ax

0( ) ( ),ix t x t i

0( ) ( ),i

i ij i j i ij N

e x x g x x

( ) ,nix t ( ) im

iu t

Graphical GamesSynchronization‐ Cooperative Tracker Problem

Node dynamics

Target generator dynamics

Synchronization problem

Local neighborhood tracking error (Lihua Xie)

x0(t)

( )i

i i i i i i ij j jj N

A d g B u e B u

12

0

( (0), , ) ( )i

T T Ti i i i i ii i i ii i j ij j

j N

J u u Q u R u u R u dt

12

0

( ( ), ( ), ( ))i i i iL t u t u t dt

Local nbhd. tracking error dynamics

Define Local nbhd. Optimal performance index

Local agent dynamics driven by neighbors’ controls

Values driven by neighbors’ controls

K.G. Vamvoudakis, F.L. Lewis, and G.R. Hudas, “Multi-Agent Differential Graphical Games: online adaptive learning solution for synchronization with optimality,” Automatica, vol. 48, no. 8, pp. 1598-1611, Aug. 2012.

M. Abouheaf, K. Vamvoudakis, F.L. Lewis, S. Haesaert, and R. Babuska, “Multi-Agent Discrete-Time GraphicalGames and Reinforcement Learning Solutions,” Automatica, Vol. 50, no. 12, pp. 3038-3053, 2014.

1u

2u

iu Control action of player i

Value function of player i

New Differential Graphical GameDISTRIBUTED ALGORITHMS- SCALABLE

( )i

i i i i i i ij j jj N

A d g B u e B u

State dynamics of agent i

Local DynamicsLocal Value Function

Only depends on graph neighbors

12

0

( (0), , ) ( )i

T T Ti i i i i ii i i ii i j ij j

j N

J u u Q u R u u R u dt

1

N

i ii

z Az B u

12

10

( (0), , ) ( )N

T Ti i i j ij j

j

J z u u z Qz u R u dt

1u

2u

iu Control action of player i

Central Dynamics

Value function of player i

Standard Multi-Agent Differential Game

Central DynamicsLocal Value Functiondepends on ALL

other control actions

Def. Local Best response.is said to be agent i’s local best response to fixed policies of its neighbors if

* * *( , ) ( , ), ,i j G j i j G jJ u u J u u i j N

A restriction on what sorts of performance indices can be selected in multi‐player graph games.

A condition on the reaction curves (Basar and Olsder) of the agents

This rules out the disconnected counterexample.

*( , ) ( , ),i i i i i i iJ u u J u u u

*iu iu

New Definition of Nash Equilibrium for Graphical Games

* * *1 2, ,...,u u u

* * * *( , ) ( , ),i i i G i i i G iJ J u u J u u i N

Def: Interactive Nash equilibrium

are in Interactive Nash equilibrium if

2. There exists a policy such that ju

1.

That is, every player can find a policy that changes the value of every other player.

i.e. they are in Nash equilibrium

1 1 12 2 2( , , , ) ( ) 0

i i

TT T Ti i

i i i i i i i i i ij j j i ii i i ii i j ij ji i j N j N

V VH u u A d g B u e B u Q u R u u R u

10 ( ) Ti ii i i ii i

i i

H Vu d g R B

u

2 1 2 1 11 1 12 2 2( ) ( ) 0,

i

TT Tj jc T T Ti i i

i i ii i i i i ii i j j j jj ij jj ji i i j jj N

V VV V VA Q d g B R B d g B R R R B i N

2 1 1( ) ( ) ,i

jc T Tii i i i i ii i ij j j j jj j

i jj N

VVA A d g B R B e d g B R B i N

* *( , , , ) 0ii i i i

i

VH u u

12( ( )) ( )

i

T T Ti i i ii i i ii i j ij j

j Nt

V t Q u R u u R u dt

Value function

Differential equivalent (Leibniz formula) is Bellman’s Equation

Stationarity Condition

1. Coupled HJ equations

where

Graphical Game Solution Equations

Now use Synchronous PI to learn optimal Nash policies online in real‐time as players interact

Distributed Multi‐Agent Learning Proofs

Online Solution of Graphical Games

Use Reinforcement Learning Convergence Results

POLICY ITERATION

Kyriakos VamvoudakisMulti‐agent Learning Convergence proofs

121

Date post:	14-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Moncrief-O’Donnell Chair, UTA Research Institute (UTARI) The … 01 coop ctrl of AC-DC mic… ·...

Documents