Herding

Dr. Daisy Tang

http://www.youtube.com/watch?v=wJUVULU7GDM

Objectives

� Understand how herding behaviors can be generated

� Understand engineered solutions vs. learned solutions

� Understand line-of-sight herding

Introduction to Herding

� What is herding?

� Has different meanings to different researchers:

� Common uses of the term “herding”:

� Herding = coming together as a group(often, to avoid predators)

� Herd of African Antelope

� Herding = surrounding targets and moving them to some destination

� Herding sheep

� Often predator/prey scenarios involve:

� Use of herding to surround targets, then declaring victory (prey are eaten)

Engineered vs. Learned Solutions

� Two approaches to designing complex cooperative behavior:

� Engineer solution “by hand”

� Create learning environment where cooperative behavior can be learned or evolved

� How do you decide which to use?

Evolution of Herding Behavior in Artificial Animals

Gregory M. Werner and Michael G. Dyer

AI Lab @ UCLA, 1993

Abstract

� Simulation of evolution of herding behavior in prey animals

� A population of preys were placed with a population of predators in an environment

� Biot is controlled by a neural networkspecified by its individual genome

� Evolving herding behavior in simple artificial animals (“biots”) that do not originally herd

Herding, Flocking and Schooling

� Many types of animals seek the company of others

� safety from predators

� access to mates

� help in finding food

Motivations for Herding

� Herding provides safety for individual� increases vigilance of individual

� no need to have everyone watch out; less likely a danger will be overlooked

� confuses or intimidates a predator� musk oxen form a circle

� baboons mob leopards

� makes it less likely for predators to find prey

� selfish-herding

� desire to remain in the center vs. greater resources at the periphery

� increases feeding

� brings together mates

� etc.

This Approach

� Objective: simulate the evolution of herding behavior in prey animals that do not originally herd

� A bottom-up exploration of the evolution of cooperative behavior and communication

� BioLand contains information relevant to the life of simple animals (biots)

� The biots should evolve cooperative skills similar to those in nature

Approach: Environment (1)

� 2-D environment with 2 species “deer” and “wolves” and 2 non-animal objects “plants” and “trees”� Deer increase energy by eating plants

� Wolves increase energy by eating deer

� Both expend energy in eating, mating, existence, etc.

� Wrap-around simulation map� Animals that go up through the top appear at the

bottom

� Effectively no boundaries

Approach: Environment (2)

� Populate with trees and edible plants

� Trees cannot be eaten, simply add complexity to map

� Plants tend to cluster, encouraging aggregation in “deer”

� Supply with initial population of plants to provide energy

� Plants added at constant rate

� They are placed near other existing plants

Approach: Environment (3)

� Each object has a unique “smell”� Smell strength decays exponentially with

distance

� Can be sensed by a biot

� Non-animal entities (trees, plants) produce smells of constant strength

� Animals have characteristic “sounds”� Volume of sound associated with movement is

proportional to speed

� Mating, eating have different sounds (frequency, louderness)

� Sounds are used as signals to others animals

Costs and Gain for Each Action

Parameters need to be reasonably set

Modeling Biots (1)

� Behavior of each biot is determined by a neural network� Specific sensory neurons detect “smells” and

“sounds” (wolf, plant, tree, etc.), as well as other indicators (energy level, being eating, etc.)

� Motor neurons control movement, mating impulse, eating, etc.

� The above are predetermined

� Connections, hidden neurons are allowed to evolve

Modeling Biots (2)

� Sensory neurons placed on two distinctly located antennae

� One on right, one on left

� Difference between the two allows biots to determine directional gradients, locatingobjects by sound or smell

Modeling Biots (3)

� Genetic algorithm determines neural network architecture for individual biots

� Genetic string encodes synaptic strength, source and destination

� When animals mate, the genomes of parents are combined using crossover and mutation, which is translated into the neural network that controls the biot

Experiment

� Initial population:

� 8,000 genetically random wolves

� 8,000 genetically random deer

� 1,000 by 1,000 length environment, with initial clusters of plants

plant on left

plant on right

deer on left

deer on right

female deer on left

female deer on right

forward

backward

go left

go right

……

……

Randomly connected initially

Experiment

� Behaviors of populations checked throughout run

� Genome sample taken every 20 generations

� Genomic sample analyzed to determine what biot behavior is implemented in the network

Experimental Results

� In early generations, biot behavior is random

� Most biots die off, unable to find food

� Population drops to around 400

� Remainder have the few necessary sensory/motor connections to eat and mate

Results, Cont.

� At around 30 generations, biots evolve to home toward food, away from predators

� This tendency causes an elementary aggregation around food sources, safe locations

Results, Cont.

� Elementary aggregation around food sources and safety is not herding

� Biots being pushed together by environmental factors, not yet seeking out members of their own species and moving toward them

Results, Cont.

� Biots do begin herding

� At around 80-100 generations, biots evolve to home toward members of same species

� Neural connections that produce this behavior have evolved in biots’ neural network

3 Pressures

� 3 Pressures cause biots to begin herding� Pressure to find a mate

� Herding confuses predators’ sense of gradient toward prey

� Finding food� herd as a distributed sensor network where

nodes implicitly communicate locations of food sources by moving towards them, or by sound of eating

Why is This Model Successful?

� Plenty of food to begin with (8000 plants)

� Massive death of initial biot population leaves plenty of food for the rest

� Biots do not die through aging

� The input neuron, “always on”, represents a kind of internal “drive”

� Biots don’t have to have perfect connections to survive

Learned Solution to Herding

� Set up neural-net framework for learning:

� Inputs are “percepts”:

� Plant on left, Deer on left, Sound #4 on right, …

� Outputs are “actions”:

� forward, backward, turn left, eat, …

� Note that these inputs and outputs are pre-defined

� To learn: connections and weights between inputs and outputs

Learned Solution to Herding (con’t.)

� Approach:� Two types of populations: predator and prey

� Use Genetic Algorithms to evolve populations of agents

� Members of population are controlled by randomly generated connections between inputs and outputs

� “Fitness” of population members determined by energy levels

� Different costs and gains for predator and prey

� Results:� Prey avoid predators; predators approach prey

� Clumping with like-animals

� Move toward like-animals – herding� Happens due to 1) need to find mate, 2) pressure from

predators, 3) pressure to find food

Engineered Solution to Herding

� Exercise:

� Assume

� 2 sets of robots – sheep and dogs

� The sheep just want to wander around

� The dogs want to get the sheep into a corral

� How would you write (non-learning) code to generate the sheep herding behavior?

One Suggested Approach to Engineered Herding Behavior…

� Similar to ideas we discussed earlier in flocking/schooling

� Recall Mataric’s paper on basis behaviors:

� Proposed method for generating herding:

Heterogeneous Mobile Sensor Net Deployment Using Robot Herding and Line-of-Sight Formations

by Parker, et al., IROS’03

Presented by Kenny Iraheta

Introduction

� Address issues of robot team heterogeneity in context of mobile sensor net deployment

� Heterogeneous team:

� Some robots are highly capable

� Some are less capable

� A typical dispersion algorithm does not work

Herding in Heterogeneous Team

� Mobile sensor node

� Leader helper

� Follower helper

� Herding process

Approach Overview

� A plan for how to deploy entire set of sensor nodes is generated

� Leader helper follows deployment path

� All robots within a herd move into their formation

� Maintain visibility

� The herd moves together

� Sensor nodes deployed via teleoperation through follower helper

Robot Detection

� Follower helper needs to:

� Identify sensor nodes

� Identify leader helper

� Color cylindrical marker

� Extract ID, orientation, angle, distance

Final-Version of Color Marker

Line-Of-Sight Formation Keeping

� Generate a formation that satisfies the line-of-sight constraints

� Φx(n-1) ≤ θ

Pictures & Videos

� http://dilab.eecs.utk.edu/DILpicmovie.html

Shepherding Deployment Simulation

Deployment

Important Points Regarding Herding

� Complex cooperative behaviors can either be engineered or learned

� One approach to learning herding involves two populations simultaneously, with different goals

� Herding involves complex interaction between two types of agents with different goals

� Herding requires animals of similar types work together to be successful

Student Presentation

� Multi-Robot Caravanning

� Presented by Fariba Derakhshan