Intelligent Distributed Human Motion Simulation inHuman-Robot Collaboration EnvironmentsAndré Antakli, Erik Hermann, Ingo Zinnikus, Han Du, Klaus Fischer
German Research Center for Artificial Intelligence (DFKI) GmbHSaarbruecken, Germany
ABSTRACTThe manufacturing industry has been putting considerable researchand development work into the implementation of hybrid teamsconsisting of workers, robots and assistance systems for severalyears. The production environment of the future will consist ofhybrid teams of humans closely collaborating with robots and vir-tual agents. Hybrid teams require adaptive control, coordination ofinteractions between team members as well as reactive behaviorfor adapting production processes just in time and on demand e.g.when facing unexpected obstacles. In this paper we present a frame-work for 3D simulation of hybrid teams in production scenariosbased on an agent framework and motion generation which can beused for evaluating critical properties of the planned productionsetting and dynamical assignment of tasks to team members. Weillustrate the approach with a use case in which a human-robotteam collaborates in a car manufacturing scenario.
CCS CONCEPTS• Computing methodologies → Modeling and simulation;Knowledge representation and reasoning; Planning and scheduling;Physical simulation; Motion processing;
KEYWORDSHuman-Robot Collaboration, Worker Support, Motion Synthesis,Multi-Agent System
ACM Reference Format:André Antakli, Erik Hermann, Ingo Zinnikus, Han Du, Klaus Fischer. 2018.Intelligent Distributed Human Motion Simulation in Human-Robot Col-laboration Environments. In IVA ’18: International Conference on IntelligentVirtual Agents (IVA ’18), November 5–8, 2018, Sydney, NSW, Australia. ACM,New York, NY, USA, 6 pages. https://doi.org/10.1145/3267851.3267867
1 INTRODUCTIONThe manufacturing industry has been putting considerable researchand development work into the implementation of hybrid teamsconsisting of workers and robots for several years. The main aim ofthese efforts is to relieve the strain on people in production lines, butalso to combine the strengths of the involved hybrid team members.
In addition to the specific assembly planning, the establishmentof hybrid teams asks for optimal orchestration of individual ac-tors with fundamentally different characteristics. Configurationsof such teams and their internal work distribution and interactionshave to be validated in advance in order to reduce costs and timein real production and to avoid injuries as far as possible. For thisreason, a highly configurable simulation environment to modeland validate such configurations, is needed. 3D simulations canbe used to show spatial relationships, time-dependencies, but theyare also intuitive for the end-user, e.g. manufacturing planners, interms of understanding and interaction. Currently tools for plan-ning, simulation and validation of complex production processesand flows of materials are avilable, but these only allow model-ing plain action sequences of actors, so they are not in a positionto adapt themselves to changes that are unforeseen. Furthermore,these solutions look at each actor separately, which means theydo not interact directly with each other, making a simulation ofcollaborative behavior very hard or even virtually impossible. Incontrast, simulated production processes involving hybrid teamsshould be highly configurable and the simulated actors have tobe—to a certain degree—autonomous and must interact directlywith each other. Simulating dynamic and adaptive behavior of indi-vidual team members requires a paradigm that allows to intuitivelymodel autonomous entities that can independently decide on howto achieve a given goal depending on the simulation state, exploringthe space of possible (inter-)action sequences. If autonomous enti-ties are to be simulated, agent systems are usually used to modeland execute their behavior and interactions. If independent actorsare not only to be visualized abstractly—e.g. with pre-modeled ani-mations to evaluate ergonomic and spatial feasibility—approachesare needed that dynamically synthesize motions that are as closeto reality as possible, especially for the simulation of human actors.
In the following we present our approach to a highly config-urable simulation environment that should support end-users, suchas manufacturing planners, to optimally prepare collaboration ofhybrid teams in production lines. Beside of a visualization environ-ment, in which the end-user can flexibly manipulate the sceneryobjects at runtime and validate time constraints, we use approachesto dynamically generate human motions based on real data and toenable intuitive modeling of agent behavior for orchestration andcontrol of simulated hybrid teams in such a dynamic environment.
This paper is structured as follows. After pointing out the RelatedWork in Section 2, Section 3 shows our framework and the interplayof the different approaches. In section 4 we introduce the AJANagent service and our motion synthesis system. In section 5, a use-case scenario in the context of hybrid teams in manufacturing ispresented. Finally, we conclude in section 6.
IVA ’18, November 5–8, 2018, Sydney, NSW, Australia A. Antakli et al.
2 RELATEDWORK2.1 Agent Behavior PlanningA well-established paradigm in the field of virtual human simula-tion are agent technologies, used for example in [1, 20]. [23] definesan agent as an autonomous software entity that can perceive andinteract with its environment to reach goals. An established archi-tecture to model agents is the Belief Desire Intention (BDI) [18]paradigm. Other behavior languages for agents, mostly used in thegame industry, are Finite State Machines (FSM) [22] or the BehaviorTree (BT) [17] paradigm. In [20], enhanced BTs are used to controlhuman character interactions. In this approach, however, the in-teraction between agents must be pre-modeled, i.e. they are notgenerated dynamically. As in our approach, very often automaticreasoning methods are combined to implement e.g. more adaptivesystems. Deductive reasoning based examples are [13] and [17]in which BDI respectively BT agents are extended with classicalplanning. [2] presents a BDI-POMDP (Partially Observable MarkovDecision Process) approach to realize planning under uncertainty.
2.2 Motion SynthesisMotion synthesis methods can be divided into procedural and data-driven approaches.We restrict the related work to data-drivenmeth-ods –which can create realistic motions by reusing motion capturedata– because we are interested in realistic animations. Data-drivenapproaches can be divided into four categories: Motion Graphs con-struct a graph of poses from a set of motion clips by identifyingtransitions based on pose similarity [10, 19]; Interpolation-basedmethods create new variations by interpolating between temporallyaligned structurally similar example motions [4, 6]; Statistical Mod-els methods learn a latent parameter space from example motionsand learn a statistical model of the distribution of the examples[15, 16]. New variations can then be generated by optimizing inthe latent parameter space using the distribution as prior. Recently,deep neural networks, which have the ability to learn arbitraryfunctions given training data, have also been applied to the fieldof motion synthesis, for example in [8, 21]. We have chosen toapply a statistical motion synthesis method based on [15] becauseit can interpolate inside of the captured variation and the graph ofstatistical motion models can be easily extended with new actions.
2.3 Industrial Simulation FrameworksSeveral commercial systems allowing the configuration and 3Dsimulation of target processes in production environments in theshop-floor context, e.g. Tecnomatix1, FlexSim2, visTABLEtouch3,ema4 or SIMUL85. DELMIA6 for example, is a tool which allowsadditionally the validation of ‘produced‘ products and the evalu-ation of manufacturing processes. DELMIA is also used in [9] forprototyping CPPS environments. The main deficit of these indus-trial frameworks is the limited support for specifying dynamicaland mutual dependencies between the actors which is required forevaluating team-oriented behavior especially in critical situations.1Tecnomatix: plm.automation.siemens.com/Tecnomatix2FlexSim: www.FlexSim.com/FlexSim3visTABLEtouch: www.vistable.de/visTABLEtouch-software4ema: https://www.imk-ema.com/homepage.html5SIMUL8: www.SIMUL8.com6DELMIA: www.transcat-plm.com/software/ds-software/delmia
Figure 1: Overview of layered cycle simulation approach
3 FRAMEWORK OVERVIEWTo plan and simulate hybrid teams in which each team memberreacts collaboratively to environmental changes and adapt theirmotions dynamically, we present a three-layered cyclic approach(see Fig. 1). This approach consists of the environment to be simu-lated, which contains all environmental objects and their states; theagent-based deliberation, which autonomously carries out actionsfor each simulated team member based on the world state; and theexecution of atomic actions of the simulated members, e.g. humanavatars, for which generated human-like motions are carried outand visualized in the simulated environment.
3.1 Simulated EnvironmentThe simulated three-dimensional environment is visualized in anplanning-editor and contains all objects to be displayed with theirstates, such as the avatars and rigid bodies for robots and workers ofthe hybrid team and the 3D objects and properties of the productionunits, which can be manipulated at runtime by the end-user. Thewidespread game engine Unity7 is used as the basis for the imple-mentation of this environment. The other two layers of the cycleaccess and manipulate the Unity managed objects while simulatinga manufacturing configuration.
3.2 Agent Based Behavior PlanningFor the individual control of the simulated actors of a hybrid team,AJAN agents are used in our approach. AJAN is a multi-agentWeb service developed for the intelligent orchestration of LinkedData8 (LD) resources and was already used for various humansimulations in 3D worlds. With AJAN several deliberative agentscan be created and executed. Each agent has its own knowledge base(KB) and several behaviors and is able to communicate with others.The RDF9-based KB contains among other things the current stateof the agent with the perceived domain information. This KB isaccessed by the behaviors to perform context-based atomic actionsin the agent environment. Enhanced Behavior Trees (BT) are usedas behavior model, e.g. for behavioral planning of hybrid teammembers and for implementing inter-agent communication. The7Unity: https://unity3d.com8Linked Data: https://www.w3.org/standards/semanticweb/data9RDF: https://www.w3.org/RDF/
320
Intelligent Distributed Human Motion Simulation IVA ’18, November 5–8, 2018, Sydney, NSW, Australia
BTs used access the KB of the agent via SPARQL10, extend it viarule-based reasoning or are synthesized via classical planning. Toenable intuitive modeling of agents, AJAN offers an editor withwhich behavior can be modeled using graphical elements.
3.3 Motion Generation for Simulated WorkersTo generate realistic human animations in our simulation, we usea data-driven approach based on motion capture data. The mo-tion data is manually segmented into actions and automaticallyprocessed into a directed graph of parameterized motion modelsbased on the approach presented by Min et al. [15]. The resultingdata structure enables the generation of any number of randomvariations of the captured motions. For controlled motion synthesis,a graph search based on different kinds of constraints is performedand the low dimensional parameters of the motion models are op-timized numerically. In order to stay in a natural motion spacedefined by the captured data, the objective is defined as a maximuma posteriori estimation problem using the motion model as prior. Tointegrate the approach with the real time agent simulation of AJAN,we use a stateful animation server that fills an animation bufferahead of time. Furthermore, we use acceleration data structures forthe evaluation of the motion models according to [7].
4 COMPONENTS4.1 Distributed Avatar Control with AJAN
4.1.1 SPARQL-BT: AJAN Behavior Model. For modeling agentbehavior, AJAN uses the BT paradigm extended with SPARQLqueries –called SPARQL-BT– to operate on RDF datasets. BTs orig-inate from the game industry and were primarily developed tointuitively model Non-Player Character (NPC) behavior in a modu-lar and reusable way. This paradigmwith loops, sequences, parallelsand an implicit knowledge base is often described as a combina-tion of decision trees with state machines [12]. Typically, a BT isexecuted repeatedly in a depth-first procedure, which ensures thereactivity of the agent. Goals are implicitly defined and their pri-ority is specified by the hierarchical structure of the tree. Thereare four basic node types from which a BT can be built: the rootnode; composite nodes; decorator nodes; and leaf nodes which in-teract directly with the agent environment. Decorators have onlyone child, the node they are decorating, whereas composites canhave many, but must have at least one child. Child nodes can againbe decorators, composites or leafs like action or condition nodes.Composites decide the execution order of their child nodes, e.g. insequence, or in parallel. Decorator nodes are used to decide howtheir child node has to be executed, e.g. in a loop, or how its statushas to be propagated up to the tree. Leaf nodes are using the agentknowledge base (KB) to execute internal (e.g. belief updates) orexternal actions and to check if a defined condition holds. Each treenode can be seen as a stand-alone behavior, allowing higher-qualityBTs to be integrated into new BTs.
The SPARQL-BTs approach defines three main new leaf nodetypes to work on the RDF-based agent KB using SPARQL queries:the SPARQL-BT Condition leaf node makes a binary statement aboutthe presence of a graph-pattern in a RDF dataset to check the agent
state; the SPARQL-BT Update leaf node uses the SPARQL-Updatelanguage to create, delete or update RDF data in the agent KB; anda SPARQL-BT Action leaf node sends messages to external resourcewhere an RDF dataset is sent to a defined URI endpoint, e.g. forexecuting domain actions or for inter-agent communication.
4.1.2 AJAN Architecture Overview. Fig. 2 shows an overview ofthe AJAN architecture. AJAN consists of several independent Webservices: The Execution Service (yellow); a plug-in system (orange);the agents domain (blue) with accessible resources (green); a TripleStore (red); and a graphical editor (violet).
Triple Store: RDF Triple Stores maintain various models andmanage the knowledge of the agents. Four defined triple data bases(TDB) are available for this: the Template TDB contains differentagent models (see Section 4.1.3), which are used for the initializa-tion of an agent; predefined SPARQL-BT behaviors are stored inthe Behavior TDB respectively the plan library; the Domain TDBcontains the domain model with ontologies and rules; finally, aKnowledge Base (KB) TDB is available for each agent.
Domain: Domains, for which AJAN was primarily developedare LD environments. In these, the interaction takes place via HTTPwith resources that describe their status and actions that can beperformed in RDF. If available, AJAN can directly access a functionaland logical description of these actions in order to execute them,but also for reasoning purposes, e.g. for action planning. Thus, theirpreconditions and effects are included in these descriptions11.
Execution Service: This is the core of AJAN, with which agentscan be created, deleted, managed and executed via variousHTTP/RDFinterfaces. For example, to create an agent, the agent model to beused and its initial knowledge must referenced or defined in a RDFdataset. This service also offers a plug-in system in which furtherAI technologies can be integrated for agent behavior modeling.
AJAN Editor: The AJAN Editor is a web service, available tothe user for modeling an AJAN agent. It can be used to createSPARQL-BTs, define agent templates, and edit the domain model.To implement the editor, the behavior3editor12 is extended formodeling SPARQL-BTs and AJAN agents.
11The description of resource actions respectively affordances is oriented to the actionlanguage A defined in [5].12behavior3editor: https://github.com/behavior3/behavior3editor
321
IVA ’18, November 5–8, 2018, Sydney, NSW, Australia A. Antakli et al.
Figure 3: Agent model overview
4.1.3 AJAN Agent Model. Fig. 3 shows the AJAN agent modelwith its interactions. An AJAN agent has one or more behaviors,each executed in a single thread and consisting of a SPARQL-BTand a corresponding local RDF database; one agent specific KB,storing inter-behavior knowledge like the agent status; one or moreevents, each holding RDF data in the form of named graphs forbehaviors; and one or more HTTP endpoints. These endpoints arethe agent’s interfaces to its domain and forward incoming RDFmessages as named graphs in form of events. Behaviors can belinked to these events. If an event occurs, the behaviors linked to itare executed. While executing a SPARQL-BT, it can access specialincoming event data by querying its named graph. Each Behaviorcan also create events to trigger other behaviors. Beside of theSPARQL-BT nodes presented, further SPARQL-based nodes wererealized, e.g. to dynamically choose and execute AJAN behaviorsor to send messages to AJAN-Agents. By using the AJAN plug-insystem, AI methods can be integrated into the system. By nowAJAN has two plug-ins for: GraphPlan-based action planning inSTRIPS or PDDL domains, see [14]; and a rule-based inferencingengine for deductive reasoning. For this, rules are defined using theSPIN-rule13 language based on SPARQL-construct queries and anappropriate SPIN-engine. Both plug-ins are available as SPARQL-BTnodes for behavioral modeling. When such an extended SPARQL-BT is executed, the data required for the plug-in is converted fromRDF to the required format. As a result, for example, a SPARQL-BTcan be synthesized during action planning or, in the case of the ruleengine, the knowledge base of the agent can be extended.
4.2 Motion Synthesis using Statistical ModelsWe use a statistical motion synthesis method according to Minet al. [15]. Offline, we process motion capture data into a graphof low dimensional motion models. On runtime, new motions aresynthesized by generating a graph walk and optimizing the latentparameters of each step to fit spatial and time constraints. The speedof the synthesis is improved by building search data structures onthe latent motion space of each motion model to find a good initialguess for optimization [7].
4.2.1 Model Construction. We process motion capture data intoa graph of generative motion primitive models. A motion primitiveis defined as a short motion segment of a more complex action,13SPIN - SPARQL Inferencing Notation: http://spinrdf.org
such as left stance and right stance for the action walk. In orderto construct a motion primitive model, dimension reduction andstatistical modeling is performed on example motion data.
We represent motion data as sequence of skeleton poses consist-ing of the root translation and the relative orientation of the joints,each represented as a quaternion. In order to project the examplesinto low dimensional space, we align the motion clips spatially andtemporally to a reference. Similar to [3], we reduce the dimensional-ity by creating a functional representation of each motion by fittingcubic B-splines to the aligned clips and time warping functions. Wethen apply Principal Component Analysis on the coefficients of thealigned motion splines and time splines separately to project themotions into a low dimensional space that captures the implicitvariation of the example motions. To create a continuous distri-bution we apply the Expectation-Maximization algorithm to fit aGaussian Mixture Model (GMM). Using this statistical model, newvariations of a motion primitive can be sampled by back projectingthe low dimensional parameters into a motion spline.
In order to accelerate the search for a sample fitting given con-straints, we construct a space partitioning tree structure on thelatent motion space according to [7]. We construct the tree by re-cursively applying the k-Means algorithm on random samples ofthe motion model so that each node in the tree stores the meanlatent parameters of its children. The resulting tree can be used tosearch for a close initial guess for further numerical optimization.
4.2.2 Synthesis. To generate a constrained motion starting froman initial motion primitive, we sequentially evaluate the possibletransitions defined in the graph and select the best transition foreach step. To evaluate a transition, we perform a search in the k-Means tree of the motion primitive using (1) as objective function.
O(s,C) =N∑i=0
√(f (M(s)) −Ci )2 (1)
Here s is the latent motion parameter, C is a set of constraints, Nis the number of constraints, f is the forward kinematics functionandM(s) is the projection from latent space to motion space. Thetree search algorithm evaluates the mean of each child node usingthe objective function to decide which branch to traverse. To avoidending up in a local minimum, parallel candidates traverse the treeusing different branches. The motion primitive parameters foundvia this graph walk generation are used as initial guess for furthernumerical optimization. The parameters of multiple steps can beoptimized at once by stacking the parameter vectors into one vector.Equation 2 shows the objective function that is optimized using theLevenberg-Marquardt algorithm [11].
arg mins1, ...,sT
T∑i=1
O(si ,Ci ) − ln(pri (si )) (2)
Here si is the latent motion parameter for step i and pr gives thelikelihood from the GMM of the motion primitive. Ci are the con-straints for step i andO is Equation 1. The resulting graph walk canbe back projected into a motion spline for display and optionallyfurther modified using inverse kinematics to reach constraints thatare outside of the range of the motion models. To generate smoothtransitions between motion primitives, transition constraints onhand and feet are applied and a standard smoothing method is used.
322
Intelligent Distributed Human Motion Simulation IVA ’18, November 5–8, 2018, Sydney, NSW, Australia
Figure 4: Scenario with two workers: Worker 1 (blue overall)andWorker 2 (white overall); and one commissioning robot
5 HYBRID TEAM USE CASEUsing the presented simulation platform, an end-user, e.g. an assem-bly planner, can model the following human-robot collaborationscenario. In this scenario, a paced assembly system for mountingof gearbox bearings in the engine compartment of a vehicle is con-sidered. To install the bearing within a production cycle, severalsteps are necessary: first the bearing must be transported from astock to the installation site; then it must be placed in the enginecompartment of a vehicle; and finally the required screws mustbe inserted into the provided screw holes and then tightened. Toaccomplish this task we assume a hybrid team consisting of twoworkers and one mobile robot with two gripper arms.
Fig. 4 shows a screenshot of the scenario visualized in Unity. Thetask of the robot is to transport the gearbox bearing to a trolleynearby the installation site. Worker 1 (W1) is responsible for pickinga bearing from a trolley and placing it into the engine compartmentand worker 2 (W2) then tightens it. If W1 recognizes that the robotfailed, he asks W2 for help and plans autonomously how to get thebearing by himself. The scenario under consideration is dynamicso that the user of the framework can reposition objects (such asstorage items) and manipulate filling levels to evaluate walkingpaths of the involved actors and time-constraints, or initiate robotfailures and exchange tasks (using the AJAN Editor) within thehybrid team during runtime, to display behavioral changes and totest inter-actor collaboration. If the commissioning robot fails, oneof the workers must react to the failure and take over its tasks sothat a given cycle can be maintained. Even without the direct in-tervention of the user, the simulated actors must react dynamicallyand to a certain degree adaptively to changes in their environment,especially when actions depend on the condition of another teammember, as it is the case when passing the screws from W1 to W2.
5.1 Scenario realization5.1.1 Visualization Engine. With the visualization component,
three-dimensional production environments can be modeled andmanipulated through an editor and the corresponding static anddynamic objects are managed and displayed. While simulating apossible manufacturing configuration, the end-user can manipulatethe simulated process with a user interface for evaluation purposes,e.g. by inducing failures, to display resulting dynamic behavioralchanges of the simulated actors and their effects to time-constraints.The Unity-based visualization component combines the other twomain services of our approach: the agent system as well as themotion synthesis system. Via HTTP interfaces, AJAN agents per-ceive the current state of the scenery simulated in the visualization
environment and transmit action commands to be displayed for thesimulated entities. Since AJAN is optimized for orchestration of LDresources, all objects of the simulation environment with whichAJAN agents interact are externally accessible via URIs and arerepresented as RDF resources. The output of the behavior planningresulting from AJAN agents consists of actions that are supposed tobe performed by a human or robot. Scene objects can contain com-ponents specifying constraints for the action, e.g. the action to "fixscrews" has to be associated with a scene object that contains infor-mation on the positions of the screw holes. In the context of workersimulations, this information is converted into constraints for themotion synthesis algorithm. Additionally, the navigation planningof the Unity engine is used to generate walk trajectories to reachthe scene objects from the current position of the avatar. When anaction has been completed, a completion event is transmitted tothe AJAN agent system. For interaction visualization of simulatedrobots, inverse kinematics and manually created animations areused instead of motion synthesis.
5.1.2 Behavior Planning. In order for an agent to perceive in-formation from its LD domain, the agent can query it directly viaHTTP requests or receive it via his own HTTP endpoints. If infor-mation about these endpoints is received, the agent’s environmenthas changed accordingly, to which the agent must react. Therefore,the agent behavior is executed. Information that AJAN agents canreceive and process via BTs are RDF based datasets. To performactions in the agent domain, an AJAN agent sends action-specificRDF datasets to the LD resource via HTTP requests. In order for anagent to decide whether an action has been successfully executed,the corresponding LD resource propagates the action result. Thesame mechanism is used for communication between agents, sinceAJAN agents themselves are LD resources. For communication, anagent sends an RDF record to the receiving agent. The message isthen processed by a protocol implemented with SPARQL-BTs.
For the use-case scenario, three AJAN agents are used, two forcontrolling the worker avatars and one that controls the simulatedrobot. Besides the factual knowledge about the scene, every agenthas a SPIN rule-based domain ontology. Further, each agent hastwo SPARQL-BTs, one for inter-agent communication and one fordomain-action execution. As an example behavior, Fig. 5 shows thedomain-action SPARQL-BT of W1. If this figure is viewed from leftto right and from top to bottom - the execution sequence of thedisplayed SPARQL-BT - the first sequence sub-tree (arrow) checks
323
IVA ’18, November 5–8, 2018, Sydney, NSW, Australia A. Antakli et al.
via a SPARQL-BT Condition (blue leaf) whether a bearing is requiredfor the vehicle. Then a selector sub-tree (indicated by ’?’) executesits child nodes one by one until one is successful. The first stepis to check if W1 already has a bearing in his hand. If true twoSPARQL-BT Actions (yellow leafs) are carried out one after the otherwith which the avatar transports the bearing to the vehicle andinserts it. If W1 does not have a bearing, none is occupied on thetrolley and if the agent detects that the robot has failed, the actionsof the last sequence are executed. In this sub-behavior two specificnodes are executed in parallel (double arrow): first, W1 sends amessage to inform W2 about the robotic problem by triggering acommunication behavior with an event (green leaf); and second,W1 uses a GraphPlan-node (white leaf) to plan and execute anaction sequence to pick the needed bearing by himself. If the robotdid not fail W1 waits until the bearing is at place.
5.1.3 Human Motion Generation. In the use case, the humanworkers need to perform four basic actions: walk, pick, place andfix screws. To model the walk action, we used a marker-based Op-tiTrack system to capture 29 minutes of walk data. On the otherhand, the stationary actions pick, place and fix screws, were cap-tured using a marker-less system by The Captury14 and treatedas single motion primitives. The recorded data was modeled intostatistical motion models and organized into a graph structure. Toenable the interactive control of agents we integrate the motionsynthesis method into an animation server that manages the stateof the animation and transitions between motion primitives basedon constraints. Using a TCP-connection, the server sends individualposes to the visualization client. In each scene update the currentpose containing the root translation and the local orientations ofeach joint is applied directly on the transformation hierarchy of theskeleton in the visualization environment. Each pose update alsoincludes additional information consisting of the name of the ac-tion that is performed. Optionally, the information contains a list ofevents for scene manipulation that are executed in the visualizationenvironment when the pose is applied on the avatar. Using scenemanipulation events, scene objects can be attached to and detachedfrom the skeleton of the avatar when a pick and place action isperformed. To control the motion, the server expects a messagethat describes a single action with an optional walk trajectory andan optional list of constraints on joints of the skeleton at specificlabeled keyframes. To keep the server interface responsive inde-pendent of the synthesis speed and duration of the task, the motionsynthesis is executed in a separate thread to fill an animation bufferahead of time. If new input is provided before an action is complete,the buffer is cleared and the synthesis thread is restarted based onthe current state and the new constraints. This way it is possible toreact to changes in the scene using a continuous motion.
6 CONCLUSIONWe presented a framework for the 3D simulation of hybrid teamsin production scenarios. Using an agent framework for modelingdynamic behavior andmotion synthesis for the animation of humanworker behavior we developed a platform that enables end-users(e.g. manufacturing planners) to specify and coordinate team-based
14The Captury: http://thecaptury.com/
production processes where critical situations can be created andevaluated. The modular architecture allows incorporating externalservices, as e.g. a ROS-based robot control software to ensure anappropriate simulation of the behavior of the robots involved. Ourapproach has been applied to a use case in a car assembly line.
ACKNOWLEDGMENTThe work described in this paper has been partially funded bythe German Federal Ministry of Education and Research (BMBF)through the projects Hybr-iT under the grant 01IS16026A, andREACT under the grant 01/W17003.
REFERENCES[1] J. E. Almeida, R. JF Rosseti, and A. L. Coelho. 2013. Crowd simulation modeling
applied to emergency and evacuation simulations using multi-agent systems.arXiv preprint arXiv:1303.4692 (2013).
[2] K. Bauters, K. McAreavey, J. Hong, Y. Chen, W. Liu, L. Godo, and C. Sierra.2016. Probabilistic Planning in AgentSpeak using the POMDP framework. InCombinations of Intelligent Methods and Applications. Springer, 19–37.
[3] H. Du, S. Hosseini, M.Manns, E. Herrmann, and K. Fischer. 2016. Scaled functionalprincipal component analysis for human motion synthesis. In Proceedings of the9th International Conference on Motion in Games. ACM, 139–144.
[4] L. Feng, C. Xu, B. Jin, F. Chen, and Z. Yin. 2011. Underlying Semantic AnnotationMethod for Human Motion Capture Data. Information Technology Journal 10, 10(2011), 1957–1963.
[5] M. Gelfond and V. Lifschitz. 1998. Action languages. Electronic Transactions onAI 3 (1998).
[6] R. Heck and M. Gleicher. 2007. Parametric motion graphs. In Proceedings of the2007 symposium on Interactive 3D graphics and games. ACM, 129–136.
[7] E. Herrmann, M. Manns, H. Du, S. Hosseini, and K. Fischer. 2017. Acceleratingstatistical human motion synthesis using space partitioning data structures.Computer Animation and Virtual Worlds 28, 3-4 (2017).
[8] D. Holden, J. Saito, and T. Komura. 2016. A deep learning framework for charactermotion synthesis and editing. Vol. 35. ACM. 138 pages.
[9] A. Kashevnik, N. Teslya, E. Yablochnikov, V. Arckhipov, and K. Kipriianov. 2016.Development of a prototype Cyber Physical Production System with help ofSmart-M3. In IECON 2016 - 42nd Annual Conference of the IEEE Industrial Elec-tronics Society. 4890–4895. https://doi.org/10.1109/IECON.2016.7793468
[10] J. Lee, J. Chai, P. S. A. Reitsma, J. K. Hodgins, and N. S. Pollard. 2002. InteractiveControl of Avatars Animated with Human Motion Data. In Proceedings of the 29thAnnual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’02). ACM, New York, NY, USA, 491–500. https://doi.org/10.1145/566570.566607
[11] D. W. Marquardt. 1963. An algorithm for least-squares estimation of nonlinearparameters. Journal of the society for Industrial and Applied Mathematics 11, 2(1963), 431–441.
[12] C. Martens, E. Butler, and J. C. Osborn. 2018. A Resourceful Reframing of BehaviorTrees. (March 2018). http://arxiv.org/abs/1803.09099
[13] F. Meneguzzi and M. Luck. 2013. Declarative planning in procedural agentarchitectures. Expert Systems with Applications 40, 16 (2013), 6508–6520.
[14] F. Meneguzzi, A. Zorzo, and M. Da Costa M Ora. 2004. Propositional Planning inBDI Agents. (Sept. 2004).
[15] J. Min and J. Chai. 2012. Motion Graphs++: A Compact Generative Model forSemantic Motion Analysis and Synthesis. ACM Trans. Graph. 31, 6, Article 153(Nov. 2012), 12 pages. https://doi.org/10.1145/2366145.2366172
[16] J. Min, Y. Chen, and J. Chai. 2009. Interactive generation of human animation withdeformable motion models. Vol. 29. ACM. 9 pages.
[17] S. Rabin. 2017. Game Ai Pro 3: Collected Wisdom of Game Ai Professionals. AKPeters/CRC Press.
[18] A. S. Rao, M. P. Georgeff, et al. 1995. BDI agents: from theory to practice.. InICMAS, Vol. 95. 312–319.
[19] C. Ren, L. Zhao, and A. Safonova. 2010. Human Motion Synthesis withOptimization-based Graphs. In Computer Graphics Forum, Vol. 29. Wiley OnlineLibrary, 545–554.
[20] A. Shoulson, N. Marshak, M. Kapadia, and N. I. Badler. 2014. Adapt: the agentdevelopmentand prototyping testbed. IEEE Transactions on Visualization andComputer Graphics 20, 7 (2014), 1035–1047.
[21] G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler. 2010. Convolutional learningof spatio-temporal features. In European conference on computer vision. Springer,140–153.
[22] F. Wagner, R. Schmuki, T. Wagner, and P. Wolstenholme. 2006. Modeling softwarewith finite state machines: a practical approach. CRC.
[23] M. Wooldridge and N. R. Jennings. 1995. Intelligent agents: Theory and practice.The knowledge engineering review 10, 2 (1995), 115–152.
![Journal of Membrane Science - static.tongtianta.sitestatic.tongtianta.site/paper_pdf/0a57a758-4c7b-11e9-9d98-00163e08… · dustrially important mixtures [2–4]. Desirably, zeolite](https://static.documents.pub/doc/80x56/603bf825c4442d6995662594/journal-of-membrane-science-dustrially-important-mixtures-2a4-desirably.jpg)
