+ All Categories
Home > Documents > Response Group Application

Response Group Application

Date post: 28-Nov-2014
Category:
Upload: andrea-candian
View: 163 times
Download: 1 times
Share this document with a friend
Popular Tags:
50
Transcript
Page 1: Response Group Application
Page 2: Response Group Application

This document is provided “as-is”. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it.

Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

Copyright © 2011 Microsoft Corporation. All rights reserved.

Microsoft, Active Directory, Lync, Windows, and Windows PowerShell are trademarks of the Microsoft group of companies. All other trademarks are property of their respective owners.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 2

Page 3: Response Group Application

This chapter is part of the Microsoft Lync Server 2010 Resource Kit book that is currently being developed. Chapters will be available for download while this book is being completed. To help us improve it, we need your feedback. You can contact us at [email protected].

For information about the continuing release of chapters, check the DrRez blog, http://go.microsoft.com/fwlink/?LinkId=204593.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 3

Page 4: Response Group Application

Contributors Content Architect: Rui Maximo

Lead Writer: Stéphane Cavin

Contributing Writers: Marta Luis Burguete, Mike Sneeringer, Sam Gantner

Sidebar Contributor: Ahmet Gyger

Technical Reviewers: Ajit Kalele, Frédéric Dubut

Project Manager: Susan S. Bradley

Editorial Staff: Janet Lowen, Senior Technical Editor; Jim Bradley, Senior Art Manager; Kelly Fuller Blue, Senior Technical Editor

Cover: Jim Bradley, Senior Art Manager

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 4

Page 5: Response Group Application

Table of ContentsContributors..........................................................................................................................................4

Introduction...........................................................................................................................................6

Scenarios...............................................................................................................................................6

Queue Timeout..................................................................................................................................6

Queue Overflow................................................................................................................................8

Business Hours and Holidays.............................................................................................................9

Agent Anonymity.............................................................................................................................10

Announcements..............................................................................................................................12

Internals..............................................................................................................................................12

Components and Data Stores..........................................................................................................13

Agent Selection................................................................................................................................14

Direct from the Source........................................................................................................................21

The Influence of Presence on the Response Group Application..........................................................21

Privacy Configuration..........................................................................................................................21

Presence Policy....................................................................................................................................22

Conclusion...........................................................................................................................................22

Queue Timeout and Overflow.........................................................................................................25

Resiliency.........................................................................................................................................27

Response Group Web Component..................................................................................................29

Agent Anonymity.............................................................................................................................33

Announcement Application.............................................................................................................41

Summary.............................................................................................................................................44

Additional Resources...........................................................................................................................45

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 5

Page 6: Response Group Application

IntroductionThis chapter covers the Response Group application. First introduced in Microsoft Office Communication Server 2007 R2, Microsoft® Lync™ Server 2010 communications software brings new features as well as a new management interface. This chapter presents a deep dive into the call flows used by the Response Group application as well as a description of the Response Group Web component and the Announcement application.

ScenariosThe Lync Server Response Group application runs on a Front End Server and provides supporting functionality for the Response Group application. The Response Group application routes and queues incoming calls to a group of agents.

Instead of assigning a number to an individual, the Response Group application assigns a number to a workflow. A workflow can be a simple hunt group that plays a greeting to callers, and then places the call in a queue. Or, the workflow can be an interactive workflow that gives callers a menu to select from, and then puts the call into one of multiple queues.

A queue routes the call to a group (or multiple groups) of agents. The group matches the caller to an agent by using a variety of methods such as longest idle routing or round robin routing.

In the scenarios presented in the following sections, Contoso uses a response group to handle calls for the main office line. Contoso uses a basic hunt group workflow to route the calls to two receptionist agents that use the Microsoft Lync 2010 Attendant.

Queue TimeoutA queue timeout is an option that you can configure to re-route a call that has not been answered within a specific period of time. Calls that exceed the queue timeout threshold can be disconnected, forwarded to voice mail, forwarded to a telephone number, forwarded to a SIP address, or forwarded to another queue. These options are shown in Figure 14-1.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 6

Page 7: Response Group Application

Figure 14-1. Options for a queue timeout

By using the Lync Server Management Shell, you can configure a custom message that is played to the caller when the call is forwarded. The following Windows PowerShell® command-line interface script creates this configuration. “Custom message” is the message to be customized.

// To add a custom prompt to a queue timeout, the following commands must be run:$queue = Get-CSRgsQueue –identity {Application Server Identity} –name {Queue Name}$target = Get-CSRgsQueue –identity {Application Server Identity} –name {Queue Name}$action = New-CsRgsCallAction –prompt “Custom message” –target TransferToQueue –TargetQueueID $target.identity$queue.TimeoutAction = $actionSet-CSRgsQueue $queue

Note. The identity {Application Server Identity} parameter is not required. However, not specifying this parameter will cause the command to run slower.

Figure 14-2 illustrates how a call to a response group appears to the caller while the call is in the queue. You can configure the Main Line in the display name property of the workflow by using the Response Group Configuration Tool (for details, see http://go.microsoft.com/fwlink/?LinkId=210762) or by using the following Windows PowerShell script.

// Change the display name of a Response Group:$workflow = Get-CsRgsWorkflow -Name {Queue Name}$workflow.Name = “New Display Name”Set-CsRgsWorkflow $workflow

Figure 14-2. User view of call in a queue

Although the call window won’t indicate any change, when the queue timeout is exceeded, a prompt is played to the caller and the call is moved to another queue.

Queue OverflowYou can use queue overflow to forward a call if a queue has too many concurrent callers waiting. You can configure this option to forward the newest call or the oldest call.

In this scenario, Contoso configured calls to the main line to have a queue overflow that routes calls to a specific lobby phone. You might want to configure overflow so that the oldest call is forwarded. For example, the caller that has been waiting the longest is the first one to have their call answered, even if it is answered by the lobby phone and not the main line queue. Or, you might want to configure overflow so that the newest call is forwarded. For example, callers that are already in the queue for the main line will still get their call routed as they expect, and new calls will be sent directly to the lobby phone.

In Contoso’s example (Figure 14-3), the call will be forwarded to a SIP address.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 7

Page 8: Response Group Application

Figure 14-3. Queue overflow to a SIP address

From the caller perspective, this call looks the same as the queue timeout example while the caller is in queue.

However, because the call is forwarded to another SIP address, a second call window (Figure 14-4) will appear indicating that the call has been transferred after the queue timeout is exceeded.

Figure 14-4. Caller view for transferred call

Business Hours and HolidaysFor each response group you can configure hours of operation, including business hours and holidays. Business hours can be configured by using the Microsoft Lync Server 2010 Control Panel; however, holiday hours must be created in Management Shell.

In Contoso’s example, normal business hours are 9:00 A.M. to 5:00 P.M. as shown in Figure 14-5. Contoso also has a holiday on Christmas, which is configured by using the Windows PowerShell script that follows Figure 14-5.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 8

Page 9: Response Group Application

Figure 14-5. Operating hours in Lync Server Control Panel

// To create a holiday set, the following commands must be run:$christmas = Get-CSRgsHoliday –StartDate “12/25/2010” –EndDate “12/25/2010” –Name “Christmas”$holidays = New-CsRgsHolidaySet –Parent {Application Server Identity} –name “2010 Holidays” –Holidays($christmas)Set-CsRgsHolidaySet $holidaysYou can configure a custom greeting to play when a call is received outside of business hours. You can use text-to-speech or upload a custom audio recording. After the greeting is played, the normal call forwarding options are available.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 9

Page 10: Response Group Application

In this example, Contoso selected the options to play a message to callers outside of business hours by using text-to-speech and then disconnect the call (Figure 14-6).

Figure 14-6. Options for handling calls outside business hours

Agent AnonymityBy default, when a Response Group agent answers a call, the caller will see the name of the agent. However, in Microsoft Lync Server 2010, calls to a response group can be configured to preserve the anonymity of the agent.

From a call flow perspective, the caller places a call to the response group and is put on hold while the Match Making component finds an available agent. If you enable agent anonymity, when an agent is located, the Response Group application establishes a back-to-back SIP signaling session between the agent and the caller. Although the media between the caller and the agent is peer-to-peer, the signaling continues to route through the Response Group application.

In this scenario, Contoso configured the main line response group for agent anonymity. You can configure agent anonymity by using the following Windows PowerShell script or by using the Response Group Configuration Tool. For details, see http://go.microsoft.com/fwlink/?LinkId=210762.

// In order to convert a response group to anonymous, the following commands must be run:$rg = Get-CSRgsWorkflow –identity {Application Server Identity} –name {Response Group Name}$rg.Anonymized = $trueSet-CSRgsWorkflow $rgBy default, when a call is placed to a response group and answered by an agent, the call will appear as a transfer to the agent and the caller will see the agent’s identity as shown in Figure 14-7.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 10

Page 11: Response Group Application

Figure 14-7. Agent’s identity not anonymous

When the response group is configured for agent anonymity, the agent’s name will not be visible to the caller. Even if the caller initiates an instant message conversation, the identity of the agent remains concealed as shown in Figure 14-8.

Figure 14-8. Agent is anonymous even in an instant messaging (IM) session

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 11

Page 12: Response Group Application

AnnouncementsA new feature in Lync Server 2010 is the Announcement application. You can configure the Announcement application to play a message if a caller dials a number that is within a specified range. The message can either be an audio file that has been uploaded to the server or a text-to-speech translation.

In Contoso’s example, the Announcement application plays a message using text-to-speech when an invalid number has been dialed.

There are two steps to configuring the Announcement application:

1. Create an announcement by using the following Windows PowerShell script. 2. Configure an unassigned number range in the Lync Server Control Panel (Figure

14-9).

// To create an announcement, the following commands must be run:New-CsAnnouncement –Identity {Application Server Identity} –name {Announcement Name} –PromptTts {Text-to-speech message} –Language en-US

Figure 14-9. Configure an unassigned number range

If a caller attempts to dial a number within the range, 9990-9999, the call is answered and the message, “An invalid number has been dialed,” is played to the caller.

InternalsThe internal workings of the Response Group application require several components and data stores that work together to accomplish things such as selecting agents, handling queue timeouts and overflow, and maintaining resiliency. The following sections in this chapter:

Define these internals. Explain how the Response Group application components work together to

connect callers to agents.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 12

Page 13: Response Group Application

Describe how the Response Group application internals work in case of queue timeout or overflow.

Describe how the Response Group application behaves in a pool that has multiple servers.

Describe the Response Group Web component internals. Introduce the two main new features in the Response Group application: agent

anonymity and the Announcement application.

Components and Data StoresThe Response Group application has the following logical components:

Hosting component: Bootstraps the Response Group application and communicates with the Windows® service controller.

Workflow runtime component: Manages the audio interaction between the Response Group application and callers. Workflows are built on Windows Workflow Foundation.

Note. The workflow runtime component also manages workflows for unassigned numbers.

Match Making component: Performs the following tasks:

o Tracks presence of agents and determines which agents are available to service calls.

o Queues incoming calls.

Call control component: Connects calls to the available agents or transfers calls to alternate destinations.

In addition to these components, the Response Group application has a Web component that runs on Internet Information Services (IIS) instead of the Response Group service. The Response Group Web component includes the Web service as well as two webpages. The Response Group Web component does the following:

Enables agents to access information about the response groups they are serving. Enables agents to sign in or sign out of the agent groups they are members of. Enables administrators to manage response groups from a Web interface.

The Response Group application uses the following data stores:

Active Directory: Call routing is done by using contact objects that are created in Active Directory® Domain Services.

Rgsconfig: Stores the configuration settings of the response groups. This database is installed on the SQL Server back-end in the case of a Front End pool or on the local SQL Server Express in the case of a Standard Edition server.

Rgsdyn: Stores dynamic data, such as records of calls that are currently being handled by the system and the FQDN of the Response Group application that is hosting the active Match Making component. This data is used to provide resiliency in case a Front End Server fails. This database is installed on the SQL back-end in the case of a Front End pool or on the local SQL Server Express in the case of a Standard Edition server.

Network share: Created on the file store to store audio files. The Response Group application uses one folder per response group. It is located in the MyShare\Application Server Identity\AppServerFiles\Rgs\Instances directory.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 13

Page 14: Response Group Application

Figure 14-10 shows the logical component architecture for the Response Group application.

Figure 14-10. Components that constitute the Response Group application

Important. The Response Group application settings are specific to a particular pool. Those settings, such as an agent group, cannot be shared across Response Group applications.

Agent SelectionThe main purpose of the Response Group application is to match incoming voice calls with an available agent. Therefore, selecting the appropriate agent is the most important task of the application. When selecting an agent, the application takes into account the following criteria:

The configuration of agents in groups and queues.

Note. Group of agents are referenced as an “agent group”. Agent groups are not the same as response groups. A response group includes the whole caller experience; from the time the phone rings to the time somebody answers the call.

The agent’s presence status. The routing method used by the agent’s group.

The following section describes how the different Response Group application components work together to match incoming callers with available agents.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 14

Page 15: Response Group Application

WorkflowThe Response Group application uses the Windows Workflow Foundation to handle incoming calls and interact with the caller. When Lync Server routes a call to the Response Group application, a workflow is started to handle the call. Meaning that prompts are played, on-hold music is played, and interactions with the other service components begin. This workflow process explained in the following paragraphs is illustrated in Figure 14-11.

1. The caller is greeted by a message (optional).2. The holiday and business hours logic runs (holidays take precedence over

business hours). The result depends on the day and time of day.

a. If the business is closed, the caller hears an optional message before the Response Group application transfers the caller to a configured target or disconnects the call.

b. If the business is open, the caller goes to the next step of the workflow.

3. The caller is put in the appropriate queue.

a. If the administrator has configured only one queue, the caller is automatically added to it. This workflow behaves like a simple hunt group.

b. If there are multiple queues, the caller interacts with the workflow through an interactive voice response (IVR) component. The caller answers one or more questions. Based on the caller’s answers, the workflow logic determines which queue to add the caller to.

Note. The IVR component can be as large as you want. However, we recommend that you do not use more than three levels, and that you set a maximum of five answers per question. More complex IVRs are difficult to use.

If the caller takes more than 30 seconds to answer a question three times in a row or inputs three invalid answers, the call is disconnected.

4. The caller listens to music, while waiting for the Match Making component to locate an available agent.

a. After an agent is located, the workflow component asks the call control component to attempt a transfer. If the transfer fails, the Match Making component searches for a new agent. The caller remains on hold with music playing while waiting to be connected to another agent.

b. If a queue timeout or a queue overflow occurs, the workflow transfers the caller to a target or disconnects the call, depending on how the administrator configured the queue. If the transfer to the target fails, the call is disconnected.

5. The workflow is ended. The media connection is ended.

a. For a response group that is not anonymous, the SIP channel is ended as well.

b. For an anonymous response group, the caller is still connected through the signaling channel with the Response Group application, as described in the section titled “Agent Anonymity” later in this chapter.

At any time during the preceding process, the caller can disconnect the call. In that case, the workflow is automatically ended.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 15

Page 16: Response Group Application

Figure 14-11. Workflow process

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 16

Page 17: Response Group Application

Components InteractionsAs explained earlier, the workflow component defines what happens to a call. It interacts with the Match Making component and the call control component. The Match Making component gives the workflow a list of available agents. The call control component handles all SIP related operations (for example, transferring a call to an agent). The interactions happen in an asynchronous fashion that is always triggered by the workflow component.

The interactions that take place during the establishment of a successful call are the following:

1. The workflow component registers a call with the Match Making component. 2. The Match Making component finds available agents as described in the section

titled “Match Making Process” later in this chapter.3. The workflow component requests a transfer to the call control component.4. The call control component executes the transfer and reports the result to the

workflow component.5. The workflow component notifies the Match Making component that the transfer

is complete. The Match Making component uses the information to determine if a new call can be transferred to the agent. The result of the transfer, success or failure, is not required by the Match Making component.

Figure 14-12 describes these interactions.

Figure 14-12. Successful call establishment

If the transfer fails or the agent does not answer the call, the workflow component notifies the Match Making component and registers the call to get a new agent, as illustrated in Figure 14-13. This action does not reset the queue timeout.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 17

Page 18: Response Group Application

Figure 14-13. Workflow component registers a call after a failed transfer

Match Making ProcessAs described in the Lync Server 2010 product documentation, a queue can contain several agent groups and each agent group can contain several agents. The Match Making component must navigate through this structure to select an agent. The agent selection logic depends on the routing method of the agent group.

The Match Making component tracks the following information about each agent:

Sign-in state Presence status If the agent is currently prompted for a call. Meaning that the workflow

component is in the process of trying to transfer another caller to the agent. If the agent has already declined the current call in the last 60 seconds.

Note. You can configure the call-back duration by modifying the AgentRingbackGracePeriod parameter. For details, see http://go.microsoft.com/fwlink/?LinkId=210763.

An agent is eligible for a call only if the following is true:

1. The agent is signed-in to their agent groups.2. The agent is available.3. The agent is not currently receiving another response group call.4. The agent has not declined the call within the last 60 seconds.

If you have selected Attendant routing when configuring your routing method, criteria two and three in the previous list are ignored. The agent must still be signed into Microsoft Lync 2010.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 18

Page 19: Response Group Application

For any routing method, if the agent has declined a call in the last 60 seconds, the agent will not receive the same call again during the next 60 seconds. This is the agent grace period.

After a call is registered with the Match Making component, the Match Making component cycles through the list of agent groups that are assigned to the queue sequentially as specified by the administrator’s configuration. In each agent group, the Match Making component searches for available agents. Figure 14-14 illustrates this process.

Figure 14-14. Match making process

If during the match making process, no available agent is found, the call is queued until an agent becomes available.

When an agent becomes available, the Match Making component cycles through the list of agent groups that the agent is a member of. For each agent group that the agent is a member of, the application searches the list of queues that each agent group is part of to determine which call queue the agent should respond to first. Figure 14-15 illustrates one example of how an agent (A) is matched to a queue (Q3). Agent A is member of agent groups, G1-G3. The Match Making component iterates through this list of agent groups sequentially. Starting with G1, the Match Making component checks the queues, Q3 and Q4, to which the agent group is assigned. Because there is a caller waiting on hold in Q3, the call control component connects the caller in Q3 to the available agent (A). The call control component performs every SIP related operation. You cannot configure the agent group order or the queue order. The order is determined by the load order at startup. In

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 19

Page 20: Response Group Application

Figure 14-15, at service startup, the group order is G1, G2, and G3, and the queue order is Q1, Q2, Q3, and Q4. The caller from Q3 is transferred to agent A.

Figure 14-15.  Example of process of agent being matched to a queue

Direct from the SourceThe Influence of Presence on the Response Group Application

Ahmet Gyger

Software Development Engineer in Test

The Response Group application relies on the presence information of agents to make routing decisions for customer calls to agents. The following two situations can affect how the Routing Group application routes calls:

If the agent configures privacy mode, this can prevent the Response Group application from receiving presence notifications from agents.

If too many users subscribe to an agent’s presence information, it can result in Response Group application receiving presence information that is “stale,” meaning not up to date.

To prevent Response Group application from getting stale presence information from agents, there are two configuration settings that response group administrators should be aware of. These are described in the following sections.

Privacy ConfigurationAn administrator (by default, RTCUniversalServerAdministrator) can enable privacy mode by using the Windows PowerShell cmdlet Set-CsPrivacyConfiguration. In privacy mode, only people or voice applications (for example, Response Group) that are in the user’s contact list are allowed to view their presence information. This means that if the agent does not add the response group to their contact list, the Response Group application will not be aware of the agent’s presence changes and will not send call notifications to the agent.

To avoid the conflict between privacy mode and the Response Group application, you can do any one of the following:

Update the routing method of the agent groups to use the attendant routing method by configuring Set-CsRgsAgentGroup. This method routes the incoming calls to all agents no matter what their presence is (unless the agent’s presence is set to Do Not Disturb).

Agents can add RGS Presence Watcher to their contact list in Lync 2010 by typing RGS Presence Watcher into the Find a contact box.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 20

Page 21: Response Group Application

Agents can update their Lync 2010 or Lync 2010 Attendant settings by clicking the options button, clicking Status, and then selecting I want everyone to be able to see my presence.

Presence PolicySetting the presence policy helps limit bandwidth costs that are associated with presence information. You can configure this setting by using Set-CsPresencePolicy. The only property that can impact routing for an agent group is MaxCategorySubscription (default value is 1000).

Each client that subscribes to a user’s presence consumes five state category instances from the user’s available presence subscription quota. When the MaxCategorySubscription limit is reached, no more users (or voice applications) will receive presence notifications from this agent. Therefore, if more than 200 clients subscribe to an agent’s presence, the agent will reach their category subscription limit.

The effect on the Response Group application is that the presence of the agent will not be updated after the last presence update that was published when the agent’s client stopped publishing.

Note. By default, the presence notify expire time is one day.

To avoid this situation, make one of the following two configuration changes:

Increase the limit of MaxCategorySubscription. Use the attendant routing method as the routing method for your agent group.

ConclusionWhen deploying Lync Server 2010, some configuration settings may conflict with your Response Group application. Reconciling those conflicts will prevent the Response Group application from routing calls based on stale presence information. The simplest solution is to use the attendant routing method for your agent group.

Understanding the Call Flow—Routing a Call to an Agent To better understand the logic of how Response Group routes a caller to an agent, we will examine a call flow from the point where the Response Group application attempts to route a caller to a set of agents. Figure 14-16 illustrates this call flow.

1. Call is made to the response group: The caller establishes an audio call with a response group. This is done when a caller places an audio call from a client application to the SIP URI or phone number of the response group (INVITE-1).

2. Call goes to an interactive voice response: After the audio call is established between the caller and the Response Group application, the caller goes through an IVR.

Note. This is only the case if it is not a hunt group; in a hunt group scenario, the logic proceeds directly with the next step.

This interaction is happening through the audio channel, which is the first real-time transport protocol (RTP) channel.

3. Search for an agent: The call is placed in a queue and the workflow component requests that the Match Making component searches for an agent to which to route the call (or a set of agents in the case of a parallel routing scenario). If no agent is available, the call is placed on music-on-hold until an agent becomes available or until the queue timeout is reached. The on-hold music is played through the audio channel (first RTP channel).

4. Alert the agent: When available agents are found, the workflow component notifies the call control component to initiate calls to the set of agents. These are consultative calls (INVITE-2 and INVITE-3). The consultative call results in a desktop alert that is displayed to the agent. The desktop alert shows which response group the call is coming from and who the caller is. Impersonation is

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 21

Page 22: Response Group Application

used to communicate the identity of the caller to the agent. The agent can accept or decline the consultative call. If the agent declines the consultative call, a new agent is searched for and alerted. After an agent accepts the consultative call, the call control component cancels the other consultative calls with the identity of the agent who picked up the call.

5. Caller and agent are connected: After the agent accepts the consultative call, the agent is connected to the caller. The connection is made through INVITE-4, which has a replacement header that indicates that this new dialog replaced the dialog established by INVITE-1. When the call transfer is successfully completed, the caller drops the connection with the Response Group application, and the call control component disconnects the consultative call with the agent.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 22

Page 23: Response Group Application

Figure 14-15. Sample call flow of a response group call

Unsupported RequestsSIP is a very flexible protocol. However, the Response Group application supports only a limited subset of SIP requests. This section describes the behavior of the Response Group application when it receives valid but unsupported SIP requests that are used by Lync Server clients, as well as a few invalid SIP requests.

The Response Group application supports the following SIP requests:

Caller can put the call on hold and retrieve it. Caller and agent can terminate the call. Agent can redirect an incoming call to their cell phone (or any configured

telephone number).

Table 14-1 describes how requests are declined and the reasons they are declined.

Table 14-1. Declined SIP requests

Description Response Reason

INVITE without support for replaces.

421 Extension Required+Require header for Replaces+ Ms-diagnostic: 26010;Reason=“UA must support Replaces”

The call flow used by the Response Group application requires the use of the replaces header. If an endpoint does not support this header, the Response Group application cannot connect the caller with an agent.

INVITE without Ms-Application-Aor header

404 Not Found+Ms-diagnostic: 26011;Reason= “Ms-Application-Aor missing”

This header is added by the Front End when routing calls to the Response Group application. If this header is not present, the request is not valid.

INVITE for conference 486 Busy Here+Ms-diagnostic: 24093;Reason=”Conference invitation is not supported by this endpoint.”OrMs-diagnostic: 24018;Reason=”Conference escalation is not supported by application”

The Response Group application does not support conferencing.

INVITE for team call (Ms-Routing-Scenario:Team header present)

488 Not Acceptable Here+Ms-diagnostic: 26013;reason= “Team call not supported”

The Response Group application cannot be used as a target for a team call.

INVITE with Ms-Sensitivity header but without Referred-By

488 Not Acceptable Here +Ms-diagnostic: 26012;Reason= “Cannot accept call with Ms-Sensitivity header set”

The Response Group application cannot guarantee that the caller will not be transferred. For example, when there is a queue timeout, the caller may be transferred to a voice mail.

INVITE for IM 488 Not Acceptable Here++Ms-diagnostic: 26009;Reason= “Media type not available”

The Response Group application does not support the IM modality.

INVITE for desktop sharing

451 Unsupported Media Type+

Ms-diagnostic:24078;Reason=”Unsupported media types”

The Response Group application does not support the application sharing modality.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 23

Page 24: Response Group Application

INVITE w/ replaces (caller side)

420 Bad Extension + Unsupported: Replaces +Ms-diagnostic: 26008;Reason= “Supervised transfer not supported”

The Response Group application does not allow a caller to accept or transfer calls.

Queue Timeout and OverflowAs described in the previous section, the workflow component puts the call on hold while the Match Making component searches for an available agent to take the call. At the same time, a queue timer is started. The timer is not stopped when the Match Making component returns a list of agents to the workflow component.

If the timer expires while a call transfer is in progress by the call control component, the queue timeout action is not run immediately. The queue timeout action is run when the workflow component re-registers the call with the Match Making component after the agent declines the call or when the agent ring timeout expires as shown in Figure 14-17.

Figure 14-17. Interaction of components when a queue timer expires

A similar process occurs when the queue overflows. You can configure the overflow action to run on the newest or oldest call in the queue (the oldest call is the call waiting for the longest time). If the call control component transfers the oldest call, the overflow action is performed after the agent declines the call or when the agent ring timeout expires.

Note. You can configure the queue timeout value and the queue overflow by using the graphical user interface in Lync Server. (For details, see http://go.microsoft.com/fwlink/?LinkId=210764.) Or, you can use Windows PowerShell cmdlets. (For details, see http://go.microsoft.com/fwlink/?LinkId=210765.)

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 24

Page 25: Response Group Application

Understanding the Call Flow—Queue TimeoutThis section explains a sample call flow in a scenario where a queue timeout occurs, and the queue timeout action is configured to transfer the caller to the receptionist, Mindy.

1. Call is established: The caller is connected to the Response Group application and is listening to music while on hold.

2. Queue timeout occurs: The amount of time configured for the queue timeout expires. The caller hears a prompt.

3. Call is transferred: After the prompt is played, the caller is transferred. If the transfer target is a phone number, the Outbound Routing component on the Lync Server will apply authorization based on voice policies. If the original caller is an enterprise user, their voice policy will be used to validate the outbound call. If the caller is not an enterprise user, the voice policy of the response group that is identified in the Referred-By header is used.

4. Call is connected: The caller is connected to the transfer target. If the transfer fails (for example, the target declines), the call between the caller and the Response Group application is ended. The Response Group application then adds the following ms-diagnostic header in the BYE SIP message:

Ms-diagnostic: 26005;reason=“The Response Group application was unable to transfer the call to the configured destination and no fallback exists.”

Figure 14-18 shows the call flow for the queue timeout scenario.

Figure 14-18. Sample call flow of a transfer on queue timeout

ResiliencyBecause the Response Group application is automatically installed on all Front End Servers in a pool, it takes advantage of the high availability and resiliency that is built into the Enterprise Edition of Lync Server. Calls directed to a response group are load

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 25

Page 26: Response Group Application

balanced across the Front End of the pool by a hardware load balancer or by DNS load balancing.

To ensure that calls are handled in a first in, first out (FIFO) fashion, only one Front End Server in the pool does the match making. This ensures that an agent receives only one response group call at a time. The Front End Server that has this role is the active match making server.

One Front End Server (active) handles the match making and the remaining Front End Servers (passive) contact the active Front End Server to request matches. This communication is performed through a secure remote call by using Windows Communication Foundation (WCF). This is illustrated in Figure 14-19.

Figure 14-19. Active-passive match making

When the active Front End Server becomes unresponsive, any passive Front End Server can become active. This failover mechanism is described in the following section.

Failover MechanismIn the Rgsdyn database, the table ActiveFEInstance determines which Front End Server hosts the active server that is doing the match making. The table contains only one record. This record contains the FQDN of the active Front End Server (along with the port and the date of the last time the record was updated). At startup, each Response Group application attempts to write a record that uses its own FQDN. The stored procedure (SPROC) that is called returns the FQDN of the active Front End Server. Two possible outcomes are:

The returned FQDN is the same as the Front End Server’s FQDN. If this happens, the Front End Server becomes the active match making server. The Front End Server that is the active match making server writes its FQDN in the database every seven seconds.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 26

Page 27: Response Group Application

The returned FQDN is different from the Front End Server’s FQDN. If this happens, the Front End Server becomes a passive match making server and establishes a WCF connection with the active Front End Server. To ensure that a failure of the active Front End Server is detected, the passive Front End Servers attempt to write their FQDN to the database every 12 seconds.

This scheme gives a minimum of a five-second buffer to the active Front End Server to ensure that it remains the active match making server. To force a Front End Server to lose the active role, you can drain the Front End Server. In draining mode, the Response Group application declines new calls and lets current calls end. After all active calls end, the Response Group application stops and another Front End Server takes over the active role.

Response Group Web ComponentThe Response Group application Web component provides the main interface between the Response Group application and agents. This section describes the responsibilities of the Response Group Web component as well as the mechanism used to gather agent group and workflow membership in a site. Understanding these concepts is important for troubleshooting issues.

The Response Group Web component has two main responsibilities:

Provide a webpage that allows agents to verify their agent group membership and sign in or sign out of agent groups that have a formal participation policy. For details about agent group participation policies, see http://go.microsoft.com/fwlink/?LinkId=210766.

Provide the Web service that is used by Lync 2010 and Lync 2010 Attendant to query workflow membership of agents to allow calls on behalf of a response group. This feature is described in detail in the section titled “Outbound Anonymous Calls” later in this chapter.

The first goal is achieved through the webpage that lists all the agent groups as shown in Figure 14-20.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 27

Page 28: Response Group Application

Figure 14-20. Agent groups

Agents can access this webpage in Lync 2010 by using the Tools menu and selecting Response Group Settings. Lync will open the following URL, https://<web_pool FQDN>/RgsClients/Tab.aspx. This page shows all response groups the agent is a member of. The page supports Integrated Windows Authentication so agents who are logged on to Windows by using their Windows domain credentials are not prompted for their username and password when connecting to this page; otherwise, Internet Explorer prompts the agent for their credentials. The agent groups page automatically refreshes every 60 seconds so that agents always have up-to-date information. This feature requires JavaScript to be enabled on the browser.

The second goal of the Response Group Web component is to provide the Web service that is used by Lync 2010 and Lync 2010 Attendant for the call on-behalf-of feature. This goal is achieved by providing a WCF service that clients can query for information about response group membership. The metadata exchange endpoint that exposes metadata that describes the service (this data is used, for example, by Svcutil.exe to generate client code for accessing the service) is located at the following URL, https://<web_pool_FQDN>/RgsClients/AgentService.svc/mex?wsdl.

Even though the Web service provides several types of information, Lync 2010 and Lync 2010 Attendant perform only one query. This query calls the GetActiveResponseGroups() method. This method returns all response groups (meaning, workflows) the agent is a member of.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 28

Page 29: Response Group Application

Note. A response group is not directly linked to agents. The relationship between a response group and an agent has several levels: an agent is part of one or more groups; a group is part of one or more queues; and queues are part of response groups. This means that the actual result of GetActiveResponseGroups() is the list of response groups that have queues that contain groups of which the agent is a member.

Gathering Data in a SiteIn a complex topology, response groups, queues, groups, and agents can be homed on different pools. The Response Group Web component installed on each pool is responsible for merging agent information across all the pools in a site to provide a single interface to agents. This section describes how this information is gathered.

In the topology shown in Figure 14-21, the following response groups are defined:

The workflow “IT Helpdesk” is stored on pool 1. The workflow has one queue that contains one agent group named “IT Helpdesk group.”

The workflow “Support” is stored on pool 2. The workflow has one queue that contains one agent group named “Support group.”

The workflow “Hotline” is stored on pool 4. The workflow has one queue that contains one agent group named “Hotline group.”

Each Response Group Web component caches the entire content of the Rgsconfig database data for each Response Group application database in a network site. This cache is refreshed every 2 minutes. Figure 14-21 illustrates that the Response Group Web component on pool 1 maintains a cache of Rgsconfig data for Application Databases 1, 2, and 3. The Response Group Web component on pool 2 also maintains a cache of Rgsconfig data for Application Databases 1, 2, and 3. The same is true for the Web component on pool 3. However, the Response Group Web component on pool 4 maintains a cache only for Application Database 4 because it belongs to a different site. Caching the databases of all the sites impedes performance, so only the current site is cached.

Note. The Response Group application uses two databases to store data. The first database, Rgsconfig, stores all static data related to workflow configuration, queues, agent groups, and agents. The second database, Rgsdyn, stores information that must be frequently updated. Examples of this include information regarding the active match making instance and the list of calls that are waiting in a queue.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 29

Page 30: Response Group Application

Figure 14-21. Response group topology with 4 pools split across 2 sites

When an agent accesses the agent groups page, the Response Group Web component uses the cached database information of each application database in the network site to retrieve the agent’s agent group membership. In the case of formal agent groups, the active match making server of the pool that is hosting the agent group is also queried for the agent’s sign-in state.

Figure 14-22 gives an example of two agents, Bob and Mindy who are agents in the topology shown in Figure 14-21. Figure 14-22 illustrates the interactions between agents and the Resonse Group Web component.

When Bob accesses the Agent Groups page in pool 1, the corresponding Web component queries each cache for Bob’s group membership and merges the resulting information. At this point, the active match making instance of each of the resulting agent groups is queried for Bob’s sign-in state. The results are then rendered on the webpage. Note that Bob could access the Agent Groups page on any one of the pools in Site A and the resulting page would be the same.

The situation for Mindy is more complex because she is an agent in two different sites. Because the Web component merges only data that is within sites, Mindy has to open one Agent Groups webpage for each site that she belongs to. Lync 2010 and Lync 2010 Attendant will query only the Response Group Web component WCF service of the pool where the agent is homed. Therefore, Mindy’s Lync 2010 will query the Web component on pool 3. This means than an agent can only make calls on-behalf-of response groups that are homed in the same site where the agent itself is homed. This is because Lync 2010 has no knowledge of response groups that are homed in different sites than the user.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 30

Page 31: Response Group Application

Figure 14-22. Response Group Web component retrieving data in a complex topology

Agent AnonymityAgent anonymity is a new feature in the Response Group application that allows agents to remain anonymous to the caller.

Understanding the Call Flow—Call to an Anonymous AgentAgent anonymity is implemented by using a signaling back-to-back user agent (B2BUA) architecture. As shown in Figure 14-23, the only difference between a standard response group and a response group in which the agent is anonymous is step 3. Instead of connecting a caller directly to an agent, the signaling traffic goes through the Response Group application. However, the media flows directly between the caller and the agent. This connects the call but makes the agent’s identity anonymous to the caller.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 31

Page 32: Response Group Application

Figure 14-23. Signaling back-to-back user agent

Compared to the steps described in the section titled “Understanding the Call Flow—Routing a Call to an Agent” earlier in this chapter, the only step that is different for agent anonymity is step 5. Instead of referring the agent to the caller, the Response Group application refers the agent back to the Response Group application to establish a B2BUA call. Figure 14-24 illustrates the call flow for a call connected to an anonymous agent. The INVITE-3 and INVITE-4 establishes the back-to-back session.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 32

Page 33: Response Group Application

Figure 14-24. Sample call flow of an anonymous call

Outbound Anonymous CallsIn addition to accepting calls as anonymous agents, agents can also make outbound calls on behalf of a response group. To make sure that agents remain anonymous, calls made on behalf of a response group must be made through the Response Group application. The supported clients (Lync 2010 and Lync 2010 Attendant) retrieve the list of available response groups from the Response Group application web service when the agent signs in. This list is updated every eight hours, which is the same time that the registration information is refreshed. For the call-on-behalf-of feature, the voice policy of the response group is used, not the voice policy of the agent.

As an example, when Mindy makes an outbound call on behalf of a response group, the following happens:

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 33

Page 34: Response Group Application

1. Call is established: Mindy starts a call with the Response Group application (INVITE-1). To indicate that this call is a call on behalf of a response group, a special header, P-Agent-On-Behalf-Of, is added. Lync 2010 or Lync 2010 Attendant adds this header in the SIP INVITE that is sent to the Response Group application. The value of this header is the address of the response group that Mindy wants to call on behalf of. The following is an example of this header.

INVITE sip:[email protected];gruu;opaque=user:epid:qIIWS2j5AVeDHxnQdxmlwAA SIP/2.0From: sip:[email protected];epid=02020202;tag=02020202To: sip:[email protected];Call-Id: f0ec9c595c1f412ca6b71318beb599bbVia: SIP/2.0/TLS 192.0.2.1:27221;branch=z9hG4bKa8d4Cseq: 4 INVITEP-Agent-On-Behalf-Of: sip:[email protected]

2. Authentication is checked: After the call request is received, the Response Group application validates that Mindy is an agent of the response group that was specified in the P-Agent-On-Behalf-Of header. The application also validates that the response group is an anonymous response group.

3. Call is connected: The Response Group application establishes the call with Bob (INVITE-2), the person Mindy wants to call on behalf of the response group.

Figure 14-25 shows the call flow for the call on behalf of a response group.

Figure 14-25. Call on behalf of a response group

In the previous example, if Mindy is not part of the response group that she wants to make a call on behalf of, the call is declined with a “403 Forbidden” message from the Response Group application. The following diagnostic code is used with the SIP response:

Ms-diagnostic: 26014;reason=“User not authorized to make outbound call on behalf of the Response Group”

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 34

Page 35: Response Group Application

IM Transfer of an Anonymous CallAfter an anonymous call is established, the caller or the agent can add IM to the call. All messages will transit through the Response Group application to ensure anonymity of the agent. This section explains a sample call flow.

1. Audio call is established: Bob and Mindy are talking. Media is flowing directly between Mindy and Bob.

2. IM session is added: Mindy adds IM to the call. An IM dialog is established (INVITE-1) to the Response Group application. The Response Group application creates a back-to-back agent targeted at Bob (INVITE-2).

3. IM messages are transmitted: Mindy is typing IMs. Messages are transmitted through the back-to-back session (Message-1 and Message-2).

Figure 14-26 illustrates the call flow for this scenario.

Figure 14-26.  Adding IM to an anonymous call

Call Transfer of an Anonymous CallIn addition to adding IM to the call, an agent can also perform the following:

Blind transfer of the call Consultative transfer of the call Park the call

Blind Transfer

To keep anonymity during blind transfer, the Response Group application uses the same call flow as described in the section titled “Understanding the Call Flow—Call to an Anonymous Agent” earlier in this chapter. In essence, a consultative transfer is used to connect the transfer target to the caller. The following is the process for a blind transfer:

1. Call is established: The caller is connected to an agent through the Response Group application.

2. Agent transfers the call: The agent issues a call transfer request (REFER-1).

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 35

Page 36: Response Group Application

3. Consultative call is placed: The Response Group application makes a consultative call to the target to guarantee that the target will accept the call (INVITE-1).

Note. If the transfer target is another response group in the enterprise, the REFER-1 is proxied to the caller. This ends the connection between the caller and the agent. The caller establishes a new dialog with the new response group.

4. Call is connected: After the transfer target accepts the call, the caller is connected to the new agent (REFER-2, then INVITE-2 and 3 to create the new back-to-back session).

Figure 14-27 illustrates the call flow for a blind transfer of a call.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 36

Page 37: Response Group Application

Figure 14-27. Agent blind transfers a call

In a blind transfer scenario, if the transfer target fails to answer the call, the agent cannot cancel the transfer. While the call transfer is ringing, the caller will not hear anything. Note that the forwarding rules of the transfer target are ignored. To ensure a better experience for the users calling the system, the agent should use a consultative transfer instead of a blind transfer.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 37

Page 38: Response Group Application

Consultative Transfer

A consultative transfer is similar to the blind transfer described in the preceding section. To make a consultative transfer, the agent first establishes a new conversation with the transfer target. After the transfer target answers the call, the agent connects the caller to the transfer target. The following is the process for a blind transfer:

1. Call is established: The caller is connected to an agent through the Response Group application.

2. Consultative call is placed: The agent makes a consultative call to the transfer target (INVITE-1).

3. Call is connected: After the transfer target accepts the call, the agent connects the caller to the new agent (REFER-1, then INVITE-2 and 3 to create the new back-to-back session).

Note. Depending on which conversation the agent uses to connect the caller and the transfer target, a slightly different call flow is used. In that variant, the agent sends the REFER-1 above to the Response Group application. From that point on, the call flow illustrated on Figure 14-27 is used. The only difference is the presence of a “Replaces” header in the INVITE-1. Figure 14-28 illustrates the call flow for a consultative transfer of a call.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 38

Page 39: Response Group Application

Figure 14-28. Agent makes a consultative transfer

Call Park

Parking a call is very similar to a consultative transfer. The only difference is the nature of the consultative call (INVITE-1 on Figure 14-28). Instead of establishing an audio call, the dialog is used only for exchanging information about the status of the park request.

Announcement ApplicationThe Announcement application is a new feature provided by the Response Group application. This application is used when unassigned numbers are called. When a call to an unassigned number is received, there are three actions that the Announcement application can perform:

Play an announcement Transfer the call Disconnect the call

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 39

Page 40: Response Group Application

The combination of these three actions provides several ways for dealing with calls to the unassigned numbers. The Announcement application can play an announcement that informs the user that they have reached an unassigned number or direct transfer the call and make this process transparent for the caller.

The call flow is exactly the same as the one described in the section titled “Understanding the Call Flow—Queue Timeout” earlier in this chapter. The exception is that the call is transferred or disconnected right after the optional message is played.

The routing to the Announcement application is explained in Chapter 8 “Enterprise Voice Overview.” The Announcement application does not perform any routing by itself. Figure 14-29 illustrates the internal logic of the workflow that handles a call to an unassigned number inside the Response Group application. This process is as follows:

1. Call is established.2. The Announcement application checks if an announcement must be played. If yes,

Response Group retrieves the announcement from the Central Management Store and it is played.

Note. You must configure the announcement by using Windows PowerShell cmdlets.

3. The Announcement application checks if the call must be transferred. If yes, the Announcement application transfers the call to the transfer target.

4. The Announcement application disconnects the call. The Announcement application logs the appropriate error messages if the call transfer failed.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 40

Page 41: Response Group Application

Figure 14-29. Announcement workflow

Troubleshooting the Announcement ApplicationThe Announcement application is the default SIP endpoint of the Response Group application. All requests that are not targeted at a known Response Group SIP URI are handled by the Announcement application. 

Some requests are calls to unassigned number, but other requests may be invalid. To identify exactly why a call is declined, several ms-diagnostic headers have been added to help the administrator. Table 14-2 lists these diagnostic headers.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 41

Page 42: Response Group Application

Table 14-2. ms-diagnostic headers

Ms-Diagnostics Code Reason Explanation

26000 Ms-App header missing INVITE without ms-app header

26001 Wrong application type INVITE with ms-app header but invalid application

26002 Invalid parameter INVITE with ms-app header and correct application type but wrong parameter (i.e., ID=GUID is not the first parameter)

26003 Invalid announcement ID INVITE with ms-app header and correct application type but unknown announcement ID

26004 Invalid announcement configuration. The announcement has no TTS prompt, no wav file, and no transfer

In Table 14-2, codes 26003 and 26004 indicate that the administrator might have configured the system incorrectly, and the troubleshooting suggestions in the following paragraphs could help.

The code ms-diagnostic 26003 indicates that an unassigned number range was configured incorrectly and is pointing to an announcement that does not exist. This might be caused by an announcement being deleted although it was still being used. As a result, the Announcement application cannot find the announcement. To fix the issue, the administrator must edit the unassigned number range and associate it with a valid announcement.

Ms-diagnostic 26004 indicates that the administrator configured an invalid announcement. Although, validation methods to enforce appropriate configuration exist, direct manipulation of the database can leave the announcement objects in an incorrect state. For an announcement object to be valid, it must contain one of these elements:

An audio file: Audio files are stored together with the other Response Group application files, under the OcsFileStore, in the appropriate service-named folder. If the announcement points to an audio file and it was not found, check that the audio file exists.

A text-to-speech prompt: If the language that you selected for the text-to-speech prompt is invalid, the Announcement application will use the default language.

A transfer URI: The Announcement application uses the Response Group service activities to transfer a call. Therefore, if a transfer fails, the information in the ms-diagnostic headers and in the event logs is the same as a Response Group application transfer that did not use the Announcement application.

SummaryThe Response Group application provides hunt group and IVR functionality. By using queue timeout and overflow, calls can be handled even if no agent is available to service calls. By using the anonymity feature, the identity of a Response Group agent is preserved for incoming and outgoing calls. With Lync Server 2010, a new Announcement application has been introduced to handle calls to unassigned numbers.

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 42

Page 43: Response Group Application

Additional ResourcesFor more information, see the following:

Create Response Group Workflows, http://go.microsoft.com/fwlink/?LinkId=210762

Set-CsRgsConfiguration, http://go.microsoft.com/fwlink/?LinkId=210763 Create a Response Group Queue, http://go.microsoft.com/fwlink/?LinkId=210764 Response Group Application Cmdlets, http://go.microsoft.com/fwlink/?

LinkId=210765 Create an Agent Group, http://go.microsoft.com/fwlink/?LinkId=210766 Configuring Response Group, http://go.microsoft.com/fwlink/?LinkId=205574 Configuring Announcements for Unassigned Numbers,

http://go.microsoft.com/fwlink/?LinkId=210767 Microsoft Lync Server 2010 Resource Kit Tool: Response Group Agent Live,

http://go.microsoft.com/fwlink/?LinkId=210768

Microsoft Lync Server 2010 Resource Kit Response Group Application Page 43


Recommended