Bounded Regret for Finitely Parameterized Multi-Armed Bandits

Kishan Panaganti and Dileep Kalathil

Abstract— We consider the problem of finitely parameterizedmulti-armed bandits where the model of the underlying stochas-tic environment can be characterized based on a commonunknown parameter. The true parameter is unknown to thelearning agent. However, the set of possible parameters, whichis finite, is known a priori. We propose an algorithm thatis simple and easy to implement, which we call FP-UCBalgorithm, which uses the information about the underlyingparameter set for faster learning. In particular, we show thatthe FP-UCB algorithm achieves a bounded regret under somestructural condition on the underlying parameter set. We alsoshow that, if the underlying parameter set does not satisfy thenecessary structural condition, the FP-UCB algorithm achievesa logarithmic regret, but with a smaller preceding constantcompared to the standard UCB algorithm. We also validatethe superior performance of the FP-UCB algorithm throughextensive numerical simulations.

Index Terms— Multi-Armed Bandits, Reinforcement Learn-ing Learning, Sequential Decision Making

I. INTRODUCTION

Multi-Armed Bandits (MAB) problems are the canonicalformalism for studying how an agent learns to take optimalactions by repeated interactions with a stochastic environ-ment. The learning agent receives a reward at each timestep, and it will depend on the action of the agent as well asthe stochastic uncertainty associated with the environment.The goal of the learning agent is to take actions in such away to maximize the cumulative reward. When the stochasticmodel of the environment is perfectly known, computing theoptimal action is a straightforward optimization problem. Thechallenge, as in the case of most real-world problems, is thatagent does not know the stochastic model of environmenta priori, and it has to be learned using the sequentialobservations. The learning agent needs to do exploration,i.e., take various actions sequentially to gather information,in order to estimate the stochastic model of the system. At thesame time, the learning agent needs to do exploitation of theavailable information at any given time for maximizing thecumulative reward. This exploration vs. exploitation trade-offis at the core of the multi-armed bandits problems.

Multi-armed bandits problems have been studied exten-sively in the literature. Lai and Robbins in their seminalpaper [1] formulated the non-Bayesian stochastic, indepen-dent and identically distributed (i.i.d.), multi-armed banditsproblem and characterized the performance of a learningalgorithm using the metric of regret. They showed that no

Authors are with the Department of Electrical and Computer Engineer-ing at Texas A&M University, College Station, TX, USA. Email:kpb,dileep.kalathil@tamu.edu

This work was supported in part by the National Science FoundationGrant CRII:CPS-1850206

learning algorithm will be able to achieve a regret betterthan O(log T ). They also proposed a learning algorithmthat achieves an asymptotic logarithmic regret, matching thefundamental lower bound. Ananthram et al. later extendedthis to the more general setting of Markovian rewards andmultiple plays [2], [3]. A simple index based algorithmcalled UCB algorithm was introduced in [4] which achievesthe order optimal regret in a non-asymptotic manner. Thisapproach led to the development of a number of interestingalgorithms, like linear bandits [5], contextual bandits [6],combinatorial bandits [7], and decentralized and multi-playerbandits [8].

Thompson (Posterior) Sampling is another class of algo-rithms that gives superior numerical performance in multi-armed bandits problems. Posterior sampling heuristic wasfirst introduced by Thompson [9], but the first rigorousperformance guarantee, an O(log T ) regret, was given in[10]. Thompson sampling idea has been used to developalgorithms for bandits with multiple plays [11], contextualbandits [12], general online learning problem [13], andreinforcement learning [14]. Both classes of algorithms havebeen used in a number of practical applications, like com-munication networks [15], smart grids [16], and recommen-dation systems [17].

Our contribution: We consider a class of MAB problemswhere the model of the underlying stochastic environmentcan be characterized based on a common unknown pa-rameter. In particular, we consider the setting where thecardinality of the set of possible parameters is finite. Thisis inspired by many real-world applications. For example,in recommendation systems and e-commerce applications(Amazon, Netflix), it is typical to assume that each user hasa certain ‘type’ parameter, and the set of possible parametersis finite. The preferences of the user is characterized by hertype (for example, prefer science books over fiction books).The set of all possible types and the preferences of each typemay be known a priori, but the type of a new user may beunknown. So, instead of learning the preferences of this userover all possible choices, it may be easier to learn the typeparameter of this user from a few observations. In this work,we propose an algorithm that explicitly uses the availabilityof such structural information about the underlying parameterset which enables a faster learning.

We show that, the proposed FP-UCB algorithm canachieve a bounded regret (O(1)) under some structuralcondition on the underlying parameter set. This is in sharpcontrast to the increasing (O(log T )) regret of the standardmulti-armed bandits algorithms. We also show that, if theunderlying parameter set does not satisfy the necessary

arX

iv:2

003.

0132

8v3

[cs

.LG

] 1

5 M

ar 2

020

structural condition, the FP-UCB algorithm achieves a regretof O(log T ), but with a smaller preceding constant comparedto the standard UCB algorithm. The regret achieved by ouralgorithm also matches with the fundamental lower boundgiven by [18]. One remarkable aspect of our algorithm isthat, it is oblivious to the fact if the underlying parameter setsatisfies the necessary condition or not, and thus avoiding re-tuning of the algorithm depending on the problem instance.Instead, it achieves the best possible performance given theproblem instance.

Related work: Finitely parameterized multi-armed ban-dits problem was first studied by Agrawal et al. [18]. Theyproposed an algorithm that achieves a bounded regret whenthe parameter set satisfies some necessary condition, anda logarithmic regret otherwise. However, their algorithm israther complicated which limits its practical implementationsand extension to other settings. The regret analysis is alsoinvolved and asymptotic in nature, different from the recentsimpler index-based bandits algorithms and their finite timeanalysis. [18] also provided a fundamental lower bound forthis class of problems. Compared to this work, our FP-UCB algorithm is simple, easy to implement, and easy toanalyze, while providing non-asymptotic performance guar-antees, which matches the lower bound.

There are many recent work on exploiting the availablestructure of the MAB problem for getting tighter regretbounds. In particular, [19] [20] [21] [22] consider the prob-lem setting similar to this paper where the mean reward ofeach arm is parameterized by a single unknown parameter.[19] assumes that the reward functions are continuous inthe global parameter and gives a bounded regret result. [20]gives specific conditions on the mean reward to achieve abounded regret. [21] considers a latent bandit problem wherethe reward distributions are partitioned into a number ofclusters and indexed by a latent parameter corresponding tothe cluster. [22] characterizes the minimal rates at which sub-optimal arms have to be explored depending on the structuralinformation, and proposes an algorithm which achieves theserates. [23] exploits a different structural information whereit is shown that if the mean value of the best arm andthe second best arm (but not the identity of the arms) areknown, then a bounded regret can be achieved. [24] [25]also address problems with similar structural information.There are also work on bandits algorithms that try to exploitthe side information [26] [27], and recently in the context ofcontextual bandits [28]. Our problem formulation, algorithm,and analysis are very different from these works.

II. PROBLEM FORMULATION

We consider the following sequential decision makingproblem. In each time step t ∈ 1, 2, . . . , T, the agentselects an arm (action) from the set of L possible arms,denoted as, a(t) ∈ [L] = 1, . . . , L. Each arm i, whenselected, yields a random real-valued reward. More precisely,let Xi(τ) be the random reward from arm i in its τ thselection. We assume that Xi(τ) is drawn according to aprobability distribution Pi(·; θo) with a mean µi(θ

o). Here

θo is the (true) parameter that determines the distribution ofthe stochastic rewards. The agent doesn’t know θo or the cor-responding mean values µi(θo). The random reward obtainedfrom playing an arm repeatedly are i.i.d. and independent ofthe plays of the other arms. We assume that rewards arebounded with support in [0, 1]. The goal of the agent isto select a sequence of actions (a∗(t), t = 1, . . . , T ) thatmaximizes the expected cumulative reward, i.e., (a∗(t), t =1, . . . , T ) = arg max(a(t),t=1,...,T ) E[

∑Tt=1 µa(t)(θ

o))].Clearly, the optimal choice is to select the best arm (the

arm with the highest mean value) all the time, i.e., a∗(t) =a∗(θo),∀t, where a∗(θo) = arg maxi∈[L] µi(θ

o). However,the agent will be able to make this optimal decision onlyif she knows the parameter θo or the corresponding meanvalues µi(θo) for all i. The goal of a MAB algorithm isto learn to make the optimal sequence of decisions withoutknowing the true parameter θo a priori.

We consider the setting where the agent knows the set ofpossible parameters Θ. We assume that Θ is finite. If the trueparameter were θ ∈ Θ, then agent selecting arm i will geta random reward drawn according to a distribution Pi(·; θ)with a mean µi(θ). We assume that for each θ ∈ Θ, theagent knows Pi(·; θ) and µi(θ) for all i ∈ [L]. The optimalarm corresponding to the parameter θ is denoted as a∗(θ) =arg maxi∈[L] µi(θ). We emphasize that agent doesn’t knowthe true parameter θo (and hence the optimal action a∗(θo))except the fact that it is in the finite set Θ.

In the multi-armed bandits literature, it is standard tocharacterize the performance of an online learning algorithmusing the metric of regret. Regret is defined as the perfor-mance loss of the algorithm as compared to the optimalalgorithm with complete information. Since a∗(t) = a∗(θo),the expected cumulative regret of a multi-armed banditsalgorithm after T time steps is defined as

E[R(T )] := E

[T∑t=1

(µa∗(θo)(θo)− µa(t)(θo))

]. (1)

The goal of a multi-armed bandits learning algorithm isto select actions sequentially in order to minimize E[R(T )].

III. UCB ALGORITHM FOR FINITELY PARAMETERIZEDMULTI-ARMED BANDITS

In this section, we present our algorithm for finitely pa-rameterized multi-armed bandits and the main theorem. Wefirst introduce a few notations for presenting the algorithmand the results succinctly.

Let ni(t) be the number of times arm i has been selectedby the algorithm until time t, i.e., ni(t) =

∑tτ=1 1a(τ) =

i. Here 1. is an indicator function. Define the empiricalmean corresponding to arm i at time t as,

µi(t) :=1

ni(t)

ni(t)∑τ=1

Xi(τ). (2)

Define the set A := a∗(θ) : θ ∈ Θ, which is thecollection of optimal arms corresponding to all parametersin Θ. Intuitively, a learning agent can restrict to selecting the

arms from the set A. Clearly, A ⊂ [L] and this reduction canbe useful when |A| is much smaller than L.

Our Finitely Parameterized Upper Confidence Bound (FP-UCB) Algorithm is given in Algorithm 1. Figure 1 gives anillustration of the episodes and time slots of the FP-UCBalgorithm.

For stating the main result, we introduce a few morenotations. We define the confusion set B(θo) and C(θo) as,

B(θo) := θ ∈ Θ : a∗(θ) 6= a∗(θo) andµa∗(θo)(θ

o) = µa∗(θo)(θ),C(θo) := a∗(θ) : θ ∈ B(θo).

Intuitively, B(θo) is the set of parameters that can beconfused with the true parameter θo. If B(θo) is non-empty,selecting the arm a∗(θo) and estimating the empirical mean isnot sufficient to identify the true parameter because the samemean reward can result from other parameters in B(θo). So,if B(θo) is non-empty, more exploration (i.e., selecting sub-optimal actions other than a∗(θo)) is necessary to identify thetrue parameter. This exploration will contribute to the regret.On the other hand, if B(θo) is empty, optimal parameter canbe identified with much less exploration, which results in abounded regret. C(θo) is the corresponding set of arms thatneeds to be explored sufficiently for identifying the optimalparameter.

We make the following assumption.

Assumption 1 (Unique best action). For all θ ∈ Θ, theoptimal action, a∗(θ), is unique.

We note that this is a standard assumption in the literature.This assumption can be removed at the expense of morenotations. We define ∆i as,

∆i := µa∗(θo)(θo)− µi(θo), (3)

which is the difference between the mean value of theoptimal arm and the mean value of arm i for the trueparameter θo. This is the standard optimality gap notion usedin the MAB literature [4]. Without loss of generality assumenatural logarithms.

For each arm in i ∈ C(θo), we define,

βi := minθ:θ∈B(θo),a∗(θ)=i

|µi(θo)− µi(θ)|. (4)

We use the following Lemma to compare our result withclassical MAB result. The proof for this lemma is given inthe appendix.

Lemma 1. Let ∆i and βi be as defined in (3) and (4)respectively. Then, for each i ∈ C(θo), βi > 0. Moreover,βi > ∆i.

We now present the finite time performance guarantee forour FP-UCB algorithm.

Algorithm 1 FP-UCB

1: Initialization: Select each arm in the set A once2: Initialize episode number k = 1, time step t = |A|+ 13: while t ≤ T do4: tk = t− 15: Compute the set

Ak =

a∗(θ), θ ∈ Θ : ∀i ∈ A,

|µi(tk)− µi(θ)| ≤√

3 log(k)ni(tk)

6: if |Ak| 6= 0 then7: Select each arm in the set Ak once8: t← t+ |Ak|9: else

10: Select each arm in the set A once11: t← t+ |A|12: end if13: k ← k + 114: end while

b b b btk

episode 1 episode k

1 + t1 1 + tk1 |A| = t1b

A1

Ak

episode 2

A2

1 + t2b b

t2

b b b

Fig. 1: An illustration of the episodes and time slots of theFP-UCB algorithm.

Theorem 1. Under the FP-UCB algorithm,

E[R(T )] ≤ D1, if B(θo) empty, and, (5)

E[R(T )] ≤ D2 + 12 log(T )∑

i∈C(θo)

∆i

β2i

, if B(θo) non-empty,

(6)

where D1, D2 are problem dependent constants that does notdepend on T .

Remark 1 (Comparison with the classical MAB results).Both UCB type algorithms and Thompson Sampling typealgorithms give a problem dependent regret bound O(log T ).More precisely, assuming that the optimal arm is arm 1, theregret of the UCB algorithm, E[RUCB(T )], is given by [4]

E[RUCB(T )] = O

(L∑i=2

1

∆ilog T

).

On the other hand, FP-UCB algorithm achieves the regret

E[RFP-UCB(T )] = O(1), if B(θo) empty, and,

O

∑i∈C(θo)

∆i

β2i

log T

, if B(θo) non-empty.

Clearly, for some MAB problems, FP-UCB algorithmachieves a bounded regret (O(1)) as opposed to the increas-ing regret (O(log T )) of the standard UCB algorithm. Evenin the cases where FP-UCB algorithm incurs an increasing

regret (O(log T )), the preceding constant (∆i/β2i ) is smaller

than the preceding constant (1/∆i) of the standard UCBalgorithm because βi > ∆i.

We now give the asymptotic lower bound for the finitelyparameterized multi-armed bandits problem from [18], forcomparing the performance of our FP-UCB algorithm.

Theorem 2 (Lower bound [18]). For any uniformly goodcontrol scheme under the parameter θo,

lim infT→∞

E[R(T )]

log(T )≥ maxθ∈B(θo)

µa∗(θo)(θo)− µa∗(θ)(θo)

Da∗(θ)(θo‖θ).

where Da∗(θ)(θo‖θ) is the KL-divergence between the dis-

tributions Pa∗(θ)(·; θo) and Pa∗(θ)(·; θ).

Remark 2 (Optimality of the FP-UCB algorithm). FromTheorem 2, the achievable regret of any multi-armed banditslearning algorithm is lower bounded by Ω(1) when B(θo)is empty, and Ω(log T ) when B(θo) is non-empty. Our FP-UCB algorithm achieves these bounds and hence achievesthe order optimal performance.

IV. ANALYSIS OF THE FP-UCB ALGORITHM

In this section, we give the proof of Theorem 1. Forreducing the notation, without loss of generality we assumethat the true optimal arm is arm 1, i.e., a∗ = a∗(θo) = 1.We will also denote µj(θo) as µoj , for any j ∈ A.

Now, we can rewrite the expected regret from (1) as

E[R(T )] = E

[T∑t=1

(µo1 − µoa(t))]

=

L∑i=2

∆i E

[T∑t=1

1a(t) = i]

=

L∑i=2

∆i E [ni(T )] .

Since the algorithm selects arms only from the set A, thiscan be written as

E[R(T )] =∑i∈A

∆i E [ni(T )] . (7)

We first prove the following important propositions.

Proposition 1. For all i ∈ A \C(θo), i 6= 1, under FP-UCBalgorithm,

E [ni(T )] ≤ Ci, (8)

where Ci is a problem dependent constant that does notdepend on T .

Proof. Consider an arm i ∈ A \ C(θo), i 6= 1. Then, bydefinition, there exists a θ ∈ Θ such that a∗(θ) = i. Fix a θwhich satisfies this condition. Define

α1(θ) := |µ1(θo)− µ1(θ)|.It is straightforward to note that when i ∈ A \ C(θo), thenthe θ which we considered above is not in B(θo). Hence, bydefinition, α1(θ) > 0.

For notational convenience, we will denote µj(θ) simplyas µj , for any j ∈ A. Notice that the algorithm picks ith

arm once in t ∈ 1, . . . , |A|. Define KT (note that this is a

random variable) to be the total number of episodes in timehorizon T for the FP-UCB algorithm. It is straightforwardthat KT ≤ T . Now,

E[ni(T )] = 1 + E

T∑t=|A|+1

1a(t) = i

(a)= 1 + E

[KT∑k=1

(1i ∈ Ak+ 1Ak = ∅)]

≤ 1 +

T∑k=1

(P (i ∈ Ak) + P (Ak = ∅)) (9)

= 1 +

T∑k=1

(P (i ∈ Ak, 1 ∈ Ak)

+P (i ∈ Ak, 1 /∈ Ak) + P (Ak = ∅))

≤1 +

T∑k=1

(P (i ∈ Ak, 1 ∈ Ak)

+P (i ∈ Ak, 1 /∈ Ak) + P (i /∈ Ak, 1 /∈ Ak))

≤ 1 +

T∑k=1

(P(i ∈ Ak, 1 ∈ Ak) + P(1 /∈ Ak)). (10)

Here (a) follows from the algorithm definition.We will first analyze the second summation term in (10).

First observe that, we can write nj(tk) = 1 +∑k−1τ=1(1j ∈

Aτ + 1Aτ = ∅) for any j ∈ A and episode k. Thus,nj(tk) lies between 1 and k. Now,

T∑k=1

P(1 /∈ Ak)

(a)=

T∑k=1

P

⋃j∈A

|µj(tk)− µoj | >

√3 log k

nj(tk)

(b)

≤T∑k=1

∑j∈A

P

(|µj(tk)− µoj | >

√3 log k

nj(tk)

)

(c)=

T∑k=1

∑j∈A

P

∣∣∣∣∣∣ 1

nj(tk)

nj(tk)∑τ=1

Xj(τ)− µoj

∣∣∣∣∣∣ >√

3 log k

nj(tk)

(d)

≤T∑k=1

∑j∈A

k∑m=1

P

(∣∣∣∣∣ 1

m

m∑τ=1

Xj(τ)− µoj

∣∣∣∣∣ >√

3 log k

m

)(e)

≤T∑k=1

∑j∈A

k∑m=1

2 exp

(−2m

3 log k

m

)

=

T∑k=1

∑j∈A

2k−5 ≤ 4|A|. (11)

Here (a) follows from algorithm definition, (b) from theunion bound, and (c) from the definition in (2). Inequality(d) follows by conditioning the random variable nj(tk) thatlies between 1 and k for any j ∈ A and episode k. Inequality(e) follows from Hoeffding’s inequality [29, Theorem 2.2.6].

For analyzing the first summation term in (10), definethe event Ek :=

n1(tk) < 12 log k/α2

1(θ). Denote the

complement of this event as Eck. Now the first summationterm in (10) can be written as

T∑k=1

P(i ∈ Ak, 1 ∈ Ak)

=

T∑k=1

P(i ∈ Ak, 1 ∈ Ak, Eck) (12)

+

T∑k=1

P(i ∈ Ak, 1 ∈ Ak, Ek). (13)

Analyzing (12), we get,

P(i ∈ Ak, 1 ∈ Ak, Eck)

= P

∩j∈A|µj(tk)− µoj | <√

3 log knj(tk)

,∩j∈A|µj(tk)− µj | <

√3 log knj(tk)

, Eck

≤ P

|µ1(tk)− µo1| <√

3 log kn1(tk)

,|µ1(tk)− µ1| <

√3 log kn1(tk)

, Eck

= 0. (14)

This is because the events |µ1(tk) − µo1| <√

3 log kn1(tk)

and

|µ1(tk) − µ1| <√

3 log kn1(tk)

are disjoint under Eck, that is,when n1(tk) ≥ 12 log(k)/α2

1(θ). To see this, notice that|µ1(tk)− µo1| <

√3 log k

n1(tk)

⊆|µ1(tk)− µo1| <

α1(θ)

2

,

|µ1(tk)− µ1| <√

3 log k

n1(tk)

⊆|µ1(tk)− µ1| <

α1(θ)

2

,

for n1(tk) ≥ 12 log k/α21(θ). Moreover, since |µo1 − µ1| =

α1(θ), |µ1(tk) − µo1| < α1(θ)/2 and |µ1(tk) − µ1| <α1(θ)/2 are disjoint sets. Hence, their subsets are alsodisjoint.

For analyzing (13), define n′1(tk) := 1 +∑k−1τ=1 11 ∈

Aτ. Note that, according to the FP-UCB algorithm, arm 1can be selected if Aτ is empty as well, so n′1(tk) ≤ n1(tk).Define ki(θ) and m(k) as,

ki(θ) := mink : k ≥ 3, k > d12 log(k)/α2

1(θ)e, (15)

m(k) := max1, k − d12 log(k)/α21(θ)e. (16)

Note that ki(θ) is a problem dependent constant and doesnot depend on T . Also, m(k) = k − d12 log(k)/α2

1(θ)e forall k ≥ ki(θ). We claim that for all k ≥ ki(θ),

n′1(tk) < 12 log(k)/α21(θ)

⊆ 1 /∈ Aτ , for some τ,m(k) ≤ τ ≤ k − 1 . (17)

To see this, suppose there exists no τ, m(k) ≤ τ ≤ k − 1,such that 1 /∈ Aτ . Then, 1 ∈ Aτ for all τ, where m(k) ≤τ ≤ k − 1. So, by definition n′1(tk) ≥ (k − m(k)) =d12 log(k)/α2

1(θ)e for k ≥ ki(θ). So, the complement ofthe RHS of (17) is a subset of the complement of the LHSof (17). Hence the claim follows.

Now,

T∑k=1

P(i ∈ Ak, 1 ∈ Ak, Ek) ≤T∑k=1

P(Ek)

(a)

≤T∑k=1

P(n′1(tk) < 12 log(k)/α2

1(θ))

(b)

≤ ki(θ) +

T∑k=ki(θ)

P(n′1(tk) < 12 log(k)/α21(θ))

(c)

≤ ki(θ) +

T∑k=ki(θ)

P (1 /∈ Aτ , for some τ,m(k) ≤ τ ≤ k − 1)

(d)= ki(θ)+

T∑k=ki(θ)

P

k−1⋃τ=m(k)

⋃j∈A|µj(τ)− µoj | >

√3 log τ

nj(tτ )

≤ ki(θ) +

T∑k=ki(θ)

k−1∑τ=m(k)

∑j∈A

P

(|µj(τ)− µoj | >

√3 log τ

nj(tτ )

)(e)

≤ ki(θ) +

T∑k=ki(θ)

k−1∑τ=m(k)

2|A|τ5

(18)

≤ ki(θ) +

T∑k=ki(θ)

2|A|k(m(k))5

= ki(θ) +

T∑k=ki(θ)

2|A|k(k −

⌈12 log(k)α2

1(θ)

⌉)5

(f)

≤ ki(θ) +Ki(θ), (19)

where Ki(θ) is a problem dependent constant that does notdepend on T .

In the above analysis, (a) follows from the definition ofEk and the observation that n′1(tk) ≤ n1(tk). Considering Tto be greater than or equal to ki(θ)|A|, equality (b) follows;note that this is an artifact of the proof technique and doesnot affect the theorem statement since E[ni(T

′)], for anyT ′ less than ki(θ)|A|, can be trivially upper bounded byE[ni(T )]. Inequality (c) follows from (17), (d) by the FP-UCB algorithm, (e) is similar to the analysis in (11), and(f) follows from the fact that k > d12 log(k)/α2

1(θ)e for allk ≥ ki(θ).

Now, using (19) and (14) in (12) and (13), we get,

T∑k=1

P(i ∈ Ak, 1 ∈ Ak) ≤ ki(θ) +Ki(θ). (20)

Using (20) and (11) in (10), we get,

E[ni(T )] ≤ Ci,

where Ci = 1 + 4|A|+ minθ:a∗(θ)=i(ki(θ) +Ki(θ)), whichis a problem dependent constant that does not depend on T .This concludes the proof.

Proposition 2. For any i ∈ C(θo), under FP-UCB algo-rithm,

E [ni(T )] ≤ 2 + 4|A|+ 12 log(T )

β2i

. (21)

Proof. Fix an i ∈ C(θo). Then there exists a θ ∈ B(θo) suchthat a∗(θ) = i. Fix a θ which satisfies this condition. Definethe event F (t) :=

ni(t− 1) < 12 log T/β2

i

. Now,

E[ni(T )] = 1 + E

T∑t=|A|+1

1a(t) = i

= 1 + E

T∑t=|A|+1

1a(t) = i, F (t)

+ E

T∑t=|A|+1

1a(t) = i, F c(t)

. (22)

Analyzing the first summation term in (22) we get,

E

T∑t=|A|+1

1a(t) = i, F (t)

= E

T∑t=|A|+1

1a(t) = i1ni(t− 1) < 12 log T/β2

i

≤ 1 + 12 log T/β2

i . (23)

We use the same decomposition as in the proof of Propo-sition 1 for the second summation term in (22). Thus weget,

E

T∑t=|A|+1

1a(t) = i, F c(t)

=

E

[KT∑k=1

1i ∈ Ak, F c(tk + 1)+ 1Ak = ∅, F c(tk + 1)]

≤T∑k=1

P(i ∈ Ak, 1 ∈ Ak, F c(tk + 1))+ (24)

T∑k=1

P(1 /∈ Ak, F c(tk + 1)), (25)

following the analysis in (10). First, consider (25). From theanalysis in (11) we haveT∑k=1

P(1 /∈ Ak, F c(tk + 1)) ≤T∑k=1

P(1 /∈ Ak) ≤ 4|A|.

(26)For any i ∈ A and episode k under event F c(tk + 1), wehave

ni(tk) ≥ 12 log T

β2i

≥ 12 log tkβ2i

≥ 12 log k

β2i

since tk satisfies k ≤ tk ≤ T . From (4), it further followsthat √

3 log k

nj(tk)≤ βi

2≤ |µi(θ

o)− µi(θ)|2

.

So, following the analysis in (14) for (24), we get

P(i ∈ Ak, 1 ∈ Ak, F c(tk + 1))

= P

∩j∈A|µj(tk)− µj(θo)| <√

3 log knj(tk)

,∩j∈A|µj(tk)− µj(θ)| <

√3 log knj(tk)

, F c(tk + 1)

≤ P

|µi(tk)− µi(θo)| <√

3 log kni(tk)

,|µi(tk)− µi(θ)| <

√3 log kni(tk)

, F c(tk + 1)

= 0.

(27)

Using equations (23), (26), and (27) in (22), we get

E[ni(T )] ≤ 2 + 4|A|+ 12 log(T )

β2i

.

This completes the proof.

We now give the proof of our main theorem.

Proof. (of Theorem 1)

From (7),

E[R(T )] =∑i∈A

∆iE[ni(T )]

=∑

i∈A\C(θo)

∆iE[ni(T )] +∑

i∈C(θo)

∆iE[ni(T )].

(28)

Whenever B(θo) is empty, notice that C(θo) is empty. So,using Proposition 1, (28) becomes

E[R(T )] =∑i∈A

∆iE[ni(T )] ≤∑i∈A

∆iCi ≤ |A|maxi∈A

∆iCi.

Whenever B(θo) is non-empty, C(θo) is non-empty. An-alyzing (28), we get,

E[R(T )] =∑

i∈A\C(θo)

∆iE[ni(T )] +∑

i∈C(θo)

∆iE[ni(T )]

(a)

≤∑

i∈A\C(θo)

∆iCi +∑

i∈C(θo)

∆iE[ni(T )]

(b)

≤∑

i∈A\C(θo)

∆iCi +∑

i∈C(θo)

∆i

(2 + 4|A|+ 12 log(T )

β2i

)≤ |A|max

i∈A∆i(2 + Ci + 4|A|) + 12 log(T )

∑i∈C(θo)

∆i

β2i

.

Here (a) follows from Proposition 1 and (b) from Proposition2. Setting

D1 := |A|maxi∈A

∆iCi and

D2 := |A|maxi∈A

∆i(2 + Ci + 4|A|)

proves the regret bounds in (5) and (6) of the theorem.

V. SIMULATIONS

In this section, we present detailed numerical simulationto illustrate the performance of FP-UCB algorithm comparedto the other standard multi-armed bandits algorithms.

We first consider a simple setting to illustrate intu-ition behind FP-UCB algorithm. Consider Θ = θ1, θ2with [µ1(θ1), µ2(θ1)] = [0.9, 0.5] and [µ1(θ2), µ2(θ2)] =[0.2, 0.5]. Consider the reward distributions Pi, i = 1, 2 tobe Bernoulli. Clearly, a∗(θ1) = 1 and a∗(θ2) = 2.

Suppose the true parameter is θ1, i.e., θo = θ1. Then, itis easy to note that, in this case B(θo) is empty, and henceC(θo) is empty. So, according to Theorem 1, FP-UCB willachieve an O(1) regret. The performance of the algorithmfor this setting is shown in Fig. 2. Indeed, the regret doesn’tincrease after some time steps, which shows the boundedregret property. We note that in all the figures, the regretis averaged over 10 runs, with the thick line showing theaverage regret and the band around shows the ±1 standarddeviation.

Now, suppose the true parameter is θ2, i.e., θo = θ2. In thiscase B(θo) is non-empty. In fact, B(θo) = θ1 and C(θo) =1. So, according to Theorem 1, FP-UCB will achieve anO(log T ) regret. The performance of the algorithm shown inFig. 3 suggests the same. Fig. 4 plots the regret scaled bylog t, and the curve converges to a constant value, confirmingthe O(log T ) regret performance.

We consider a problem with 4 arms where the meanvalues for the arms (corresponding to the true parameterθo) are µ(θo) = [0.6, 0.4, 0.3, 0.2]. Consider the parameterset Θ such that µ(θ) for any θ is a permutation of µ(θo).Note that the cardinality of the parameter set, |Θ| = 24,in this case. It is straightforward to show that B(θo) isempty for this case. We compare the performance of FP-UCB algorithm for this case with two standard multi-armedbandits algorithms. Fig. 5 shows the performance of standardUCB algorithm and that of FP-UCB algorithm. Fig. 6compares the performance of standard Thompson samplingalgorithm with that of FP-UCB algorithm. The standardbandits algorithm incurs an increasing regret, while FP-UCBachieves a bounded regret. For µ(θ′) = [0.4, 0.6, 0.3, 0.2],we have a∗(θ′) = 2. Now we give a typical value for thek2(θ′), defined in (15), used in the proof. For this θ′ wehave k2(θ′) = min

k : k ≥ 3, k > d12 log(k)/α2

1(θ′)e

=min

k : k ≥ 3, k > d12 log(k)/0.22e

= 2326 since

α1(θ′) = 0.2. When the reward distributions are not neces-sarily Bernoulli, note that ki(θ) is 3 for any θ with a∗(θ) = isatisfying α1(θ) > 2

√3/e.

As before assume that µ(θo) = [0.6, 0.4, 0.3, 0.2]. Butconsider a larger parameter set Θ such that for any θ ∈ Θ,µ(θ) ∈ 0.6, 0.4, 0.3, 0.24. Note that, due to repetitions inthe mean rewards for the arms, definition of a∗(θ) needs tobe updated, and the algorithmic way is to pick the minimumarm index out of which are having the same mean rewards.For example, consider µ(θ) = [0.5, 0.6, 0.6, 0.2], and so asper our new definition, a∗(θ) = 2. Even in this scenario, wehave B(θo) to be empty. Thus, FP-UCB achieves an O(1)

regret rather than O(log(T )) as opposed to standard UCBalgorithm and Thompson sampling algorithm.

We now consider a case where FP-UCB incurs an increas-ing regret. We again consider a problem with 4 arms wherethe mean values for the arms are µ(θo) = [0.4, 0.3, 0.2, 0.2].But consider a larger parameter set Θ such that for anyθ ∈ Θ, µ(θ) ∈ 0.6, 0.4, 0.3, 0.24. Note that the cardinalityof Θ, |Θ| = 44 in this case. It is easy to observe thatB(θo) is non-empty, for instance θ with mean arm values[0.4, 0.6, 0.3, 0.2] is in B(θo). Fig. 7 compares the perfor-mance of standard UCB and FP-UCB algorithms for thiscase. We see FP-UCB incurring O(log(T )) regret here. Alsonote that the performance of the FP-UCB in this case alsois superior to the standard UCB algorithm.

VI. CONCLUSION AND FUTURE WORK

We proposed an algorithm for finitely parameterized multi-armed bandits. Our proposed FP-UCB algorithm achievesbounded regret if the parameter set satisfies some necessarycondition, and logarithmic regret in other cases. In bothcases, the theoretical performance guarantees for our algo-rithm is superior to the standard UCB algorithm for multi-armed bandits. Our algorithm also shows superior numericalperformance.

In the future, we will extend this approach to linear banditsand contextual bandits. Reinforcement learning problemswhere the underlying MDP is finitely parametrized is anotherresearch direction we plan to explore. We will also developsimilar algorithms using Thompson sampling approaches.

REFERENCES

[1] T. L. Lai and H. Robbins, “Asymptotically efficient adaptive allocationrules,” Advances in applied mathematics, vol. 6, no. 1, pp. 4–22, 1985.

[2] V. Anantharam, P. Varaiya, and J. Walrand, “Asymptotically efficientallocation rules for the multiarmed bandit problem with multiple plays-part i: Iid rewards,” IEEE Transactions on Automatic Control, vol. 32,no. 11, pp. 968–976, 1987.

[3] V. Anantharam, P. Varaiya, and J. Walrand, “Asymptotically efficientallocation rules for the multiarmed bandit problem with multiple plays-part ii: Markovian rewards,” IEEE Transactions on Automatic Control,vol. 32, no. 11, pp. 977–982, 1987.

[4] P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis ofthe multiarmed bandit problem,” Machine learning, vol. 47, no. 2-3,pp. 235–256, 2002.

[5] V. Dani, T. P. Hayes, and S. M. Kakade, “Stochastic linear optimizationunder bandit feedback,” 2008.

[6] W. Chu, L. Li, L. Reyzin, and R. Schapire, “Contextual bandits withlinear payoff functions,” in Proceedings of the Fourteenth InternationalConference on Artificial Intelligence and Statistics, pp. 208–214, 2011.

[7] N. Cesa-Bianchi and G. Lugosi, “Combinatorial bandits,” Journal ofComputer and System Sciences, vol. 78, no. 5, pp. 1404–1422, 2012.

[8] D. Kalathil, N. Nayyar, and R. Jain, “Decentralized learning formultiplayer multiarmed bandits,” IEEE Transactions on InformationTheory, vol. 60, no. 4, pp. 2331–2345, 2014.

[9] W. R. Thompson, “On the likelihood that one unknown probabilityexceeds another in view of the evidence of two samples,” Biometrika,vol. 25, no. 3/4, pp. 285–294, 1933.

[10] S. Agrawal and N. Goyal, “Analysis of thompson sampling for themulti-armed bandit problem,” in Conference on Learning Theory,pp. 39–1, 2012.

[11] J. Komiyama, J. Honda, and H. Nakagawa, “Optimal regret analysisof thompson sampling in stochastic multi-armed bandit problem withmultiple plays,” in International Conference on Machine Learning,pp. 1152–1161, 2015.

0 2000 4000 6000 8000 10000Time

2

0

2

4

6

8

10

Cum

ulat

ive

Regr

et

Fig. 2

0 2000 4000 6000 8000 10000Time

0

2

4

6

8

10

Cum

ulat

ive

Regr

et

Fig. 3

0 2000 4000 6000 8000 10000Time

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

R[t]/

log(

t)

Fig. 4

0 20000 40000 60000 80000 100000Time

0

20

40

60

80

100

120

140

Cum

ulat

ive

Regr

et

FP-UCBStandard UCB

Fig. 5

0 20000 40000 60000 80000 100000Time

0

5

10

15

20

25

30

Cum

ulat

ive

Regr

et

FP-UCBStandard TS

Fig. 6

0 200000 400000 600000 800000 1000000Time

0

100

200

300

400

Cum

ulat

ive

Regr

et

FP-UCBStandard UCB

Fig. 7

[12] S. Agrawal and N. Goyal, “Thompson sampling for contextual banditswith linear payoffs,” in International Conference on Machine Learn-ing, pp. 127–135, 2013.

[13] A. Gopalan, S. Mannor, and Y. Mansour, “Thompson sampling forcomplex online problems,” in International Conference on MachineLearning, pp. 100–108, 2014.

[14] I. Osband, D. Russo, and B. Van Roy, “(more) efficient reinforcementlearning via posterior sampling,” in Advances in Neural InformationProcessing Systems, pp. 3003–3011, 2013.

[15] C. Tekin and M. Liu, “Approximately optimal adaptive learning in op-portunistic spectrum access,” in 2012 Proceedings IEEE INFOCOM,pp. 1548–1556, IEEE, 2012.

[16] D. Kalathil and R. Rajagopal, “Online learning for demand response,”in 2015 53rd Annual Allerton Conference on Communication, Control,and Computing (Allerton), pp. 218–222, IEEE, 2015.

[17] S. Zong, H. Ni, K. Sung, N. R. Ke, Z. Wen, and B. Kveton, “Cascadingbandits for large-scale recommendation problems,” in Proceedings ofthe Thirty-Second Conference on Uncertainty in Artificial Intelligence,pp. 835–844, AUAI Press, 2016.

[18] R. Agrawal, D. Teneketzis, and V. Anantharam, “Asymptoticallyefficient adaptive allocation schemes for controlled iid processes:Finite parameter space,” IEEE Transactions on Automatic Control,vol. 34, no. 3, 1989.

[19] O. Atan, C. Tekin, and M. Schaar, “Global multi-armed bandits withholder continuity,” in Artificial Intelligence and Statistics, pp. 28–36,2015.

[20] T. Lattimore and R. Munos, “Bounded regret for finite-armed struc-tured bandits,” in Advances in Neural Information Processing Systems,pp. 550–558, 2014.

[21] O.-A. Maillard and S. Mannor, “Latent bandits.,” in InternationalConference on Machine Learning, pp. 136–144, 2014.

[22] R. Combes, S. Magureanu, and A. Proutiere, “Minimal explorationin structured stochastic bandits,” in Advances in Neural InformationProcessing Systems, pp. 1763–1771, 2017.

[23] S. Bubeck, V. Perchet, and P. Rigollet, “Bounded regret in stochasticmulti-armed bandits,” in Conference on Learning Theory, pp. 122–134, 2013.

[24] S. Bubeck and C.-Y. Liu, “Prior-free and prior-dependent regretbounds for thompson sampling,” in Advances in Neural InformationProcessing Systems, pp. 638–646, 2013.

[25] S. Vakili and Q. Zhao, “Achieving complete learning in multi-armed

bandit problems,” in 2013 Asilomar Conference on Signals, Systemsand Computers, pp. 1778–1782, IEEE, 2013.

[26] C.-C. Wang, S. R. Kulkarni, and H. V. Poor, “Bandit problems withside observations,” IEEE Transactions on Automatic Control, vol. 50,no. 3, pp. 338–355, 2005.

[27] S. Caron, B. Kveton, M. Lelarge, and S. Bhagat, “Leveraging sideobservations in stochastic bandits,” Conference on Uncertainty inArtificial Intelligence, 2012.

[28] H. Bastani, M. Bayati, and K. Khosravi, “Mostly exploration-freealgorithms for contextual bandits,” arXiv preprint arXiv:1704.09011,2017.

[29] R. Vershynin, “High dimensional probability; an introduction withapplications in data sciences,” 2018.

APPENDIX

A. Proof of Lemma 1

Proof. Fix an i ∈ C(θo). Then there exists a θ ∈ B(θo) suchthat a∗(θ) = i. For this θ, by the definition of B(θo), wehave

µ1(θo) = µ1(θ). (29)

Using Assumption 1, it follows that

µi(θ) = µa∗(θ)(θ) > µ1(θ) = µ1(θo) = µa∗(θo)(θo) > µi(θ

o).

Thus, βi = minθ:θ∈B(θo),a∗(θ)=i |µi(θo)− µi(θ)| > 0.Now, for any given θ considered above, suppose |µi(θ)−

µi(θo)| ≤ ∆i. Since ∆i > 0 by definition, this implies that

µa∗(θ)(θ) = µi(θ) ≤ ∆i + µi(θo)

(a)= µ1(θo)− µi(θo) + µi(θ

o) = µ1(θo) (b)= µ1(θ),

where (a) follows from definition of ∆i and (b) from (29).This is a contradiction because µa∗(θ)(θ) > µ1(θ).

Thus, |µi(θ)− µi(θo)| > ∆i for any θ ∈ B(θo) such thata∗(θ) = i. So, βi > ∆i.