Date post: | 25-May-2015 |
Category: |
Technology |
Upload: | vasil-remeniuk |
View: | 3,178 times |
Download: | 2 times |
a million bots can't be wrong @remeniuk, Viaden Media #ScalaSBP, 18-05-2012
In Viaden our aim is to make the best poker ever
we know that performance tests should be
the first-class citizens
and kill 2 birds with one stone, using bots for testing
#1 we can emulate 50k players using just one medium EC2 instance #2 bots are interactive, so client teams can use them in development, and QA for testing
everyone knows Akka, huh?
why Scala and Akka is a perfect choice for making bots?
actors are !(interactive)straightforward remotingsimple scalability/clustering~30 minutes to make a DSL and CLI with Scala and SBT
..., but, I have to warn you ...
4 dead simple tips for keeping your sanity
when you do asynch with Akka 2.0
tip #1: live fast, die young
typical approach in Akka 1.x
lobby
desk
botlogin lin
k
tourney
Akka 1.x: actors have a long lifecycle
lobby
desk
bot
botplay game
link
linkun
link
tourney
lobby
desk
tourney
bot
bot
botplay tournament linkun
link
Akka 1.x: actors have a long lifecycle
in Akka 2.0 the world has changed
actors
paths
props
newsupervision
now, you're forced to do "the right thing" (c)
lobby
tournament desk
desk
IdleBot
DeskBot
DeskBot
all actors are supervised
lobby
desk login
easy come
lobby
desk
IdleBotplay game
easy go (when supervisor to be changed)
lobby
desk
IdleBotDeskBot
dies
borns
class Lobby extends Actor {
case Login(username, password) => context.actorOf(Props(new IdlePokerBot(...)))
case JoinAnyDesk(props) => findDesk ! JoinDesk(props) } class Desk extends Actor {
case JoinDesk(botProps)=> context.actorOf(Props(new DeskPokerBot(botProps)))
} class IdlePokerBot(props: BotProps) extends Actor {
case PlayNow => context.parent ! JoinAnyDesk(props); context.stop(self)
}
Props Pattern - "the soul" of an actor
IdleBot
DeskBot
BotProps
BotProps
Props remains alive between actor "reincarnations"
case class BotProperties(id: Long, login: String, script: Seq[BotEvent], aggressiveness: Int, sessionId: Protocol.SessionId) class IdlePokerBot(val botProperties: BotProperties) extends Bot[Poker] class DeskPokerBot(val botProperties: BotProperties) extends Bot[Poker]
tip #2: think beyond
when you know, who's supervising, life's simple
akka://gpdev/user/lobby/player1234 akka://gpdev/user/lobby/desk1/player1234 akka://gpdev/user/lobby/tournament1/desk1/player1234
ActorRegistry, actor UUID
but what should I do, now, when I don't know, where to look for my bot?
were removed from Akka
Bad news
you can make your own registry (using Extensions, backed with a distributed data structure)...
or, use the full power of location transparency
lobby
desk
IdleBot
DeskBot
ProjectionManager
Projectionvar location:
ActorPath
/lobby/desk123/player123/projection/player123
class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) ! msg } } class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg } } projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")
class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) ! msg } } class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg } } projectionManager ! Add(actorRef)system.actorFor(projectionManager.path.child("actor" + i)) ! "ping"
class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorSelection("../*/" + path) ! msg } } val projectionManager = system.actorOf(Props[ProjectionManagerRoutee]
.withRouter(RoundRobinRouter(resizer = Some(DefaultResizer(lowerBound = 10, upperBound = 20)))), "projection")
projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")
case class CustomRouter(n: Int, routerDispatcher: String = DefaultDispatcherId, supervisorStrategy: SupervisorStrategy = defaultStrategy) extends RouterConfig { def createRoute(props: Props, provider: RouteeProvider) = { provider.registerRoutees((1 to n).map(i => provider.context.actorOf(Props[ProjectionManager], i.toString))) def destination(sender: ActorRef, path: String) = List(Destination(sender, provider.routees(abs(path.hashCode) % n))) { case m@(sender, Add(actor)) ⇒ destination(sender, actor.path.name) case m@(sender, Forward(_, name)) ⇒ destination(sender, name) } } }
tip #3: don't do anything stupid
you've tried all the options, system load is fine, only 1/10 of the heap is used, but you still can start not more than 1k bots!?
ulimit -n <proper value>
your actor is lacking of throughput? wait before adding poolsshare responsibility!one fine-grained actor is enough in 99% of the cases
100-300 threads are serving 300 bots!?
spawn futures, backed with standalone [bounded] pools, for blocking operations
class ThirdPartyWrapper extends Actor { case F(x) => sender ! thirdPartyService.f(x) // call to a function that takes a lot of time to // complete } class ThirdPartyWrapper extends Actor { case F(x) => val _sender = sender Future(thirdPartyService.f(x)).map(_sender ! _) // ugly, but safe, and perfectly right}
use separate dispatchers
lobby
desk
DeskBot
ProjectionManager
Projection
projection-manager-dispatcherBalancingDispatcher
projection-dispatcherDispatcher
lobby-dispatcherPinnedDispatcher
container-dispatcherDispatcher
desk-bot-dispatcherDispatcher
GOTCHA: Akka successfully bootstraps, even if your dispatcher is not configured, or the config is wrong Always check the logs to make sure that dispatchers are used!
[WARN][gpdev-akka.actor.default-dispatcher-1] [Dispatchers] Dispatcher [bot-system.container-dispatcher] not configured, using default-dispatcher[WARN][gpdev-bot-system.container-dispatcher-1] [PinnedDispatcherConfigurator] PinnedDispatcher [bot-system.lobby-dispatcher] not configured to use ThreadPoolExecutor, falling back to default config. [DEBUG][gpdev-akka.actor.default-dispatcher-24] [akka://gpdev/user/poker/lobby] logged in[DEBUG][gpdev-akka.actor.default-dispatcher-14] [akka://gpdev/user/poker/projeciton/$g/player20013] starting projection...
tip #4: analyze that
how to measure? Metrics - pushes various collected metrics to GraphiteCarbon and Graphite - gather metrics, and expose them via web interface
object BotMetrics { val loggedInCount = new Counter(Metrics.newCounter(classOf[Lobby[_]], "logged-in-count")) GraphiteReporter.enable(1, TimeUnit.MINUTES, "localhost", 2003) } class Lobby extends Actor { case Login(username, pass) => BotMetrics.loggedInCount += 1 }
1. add logged-in user counter 2. update it3. enable reporting to Graphite4. build a graph in Grtaphite
1.
2.
3.
4.
what to measure? - mailbox size1
- throughput- time, before the message is processed (both in actor and future)2
- time to process a message- count of threads- actor pool size- heap size
1 requires implementation of a custom mailbox that can expose mailbox size2 every message should be stuffed with a timestamp
how to tune dispatchers? VisualVM - thread timeline shows, if thread polls behind dispatchers are used effectively
don't neglect old good logging [ERROR][05/06/2012 12:55:43.826] [gpdev-bot-system.desk-bot-dispatcher-7] [akka://gpdev/user/poker/lobby/tournament5382577/desk109129/player20121] unprocessed game event: GameEvent(CHAT,None,None)
thanks for listening