Don't fear our new robot overlords – A new way to test on mobile

transcript

Don’t fear our new Robot Overlords!A new way to test on mobile devices

Philip Brechler • europython 2014

Agenda

• Problems when testing on mobile devices

Agenda

• Idea

Agenda

• Idea

• Concept

Agenda

• Idea

• Concept

• Demo

Agenda

• Idea

• Concept

• Demo

• Conclusion and Learnings

Problems in general

CC-BY-SA: Javier Enjuto

Problems in general• You need to check how the app

• You can check coordinates, fonts etc.

• In the end you never know if it looks right

• The human need to double check everything

Problems with simulators

CC-BY: Steve Jurvetson

Problems with simulators• To much computation power

• No memory issues

• No touch interface, mouse pointers are to exact

• Different browser engines

• Some APIs not available

Problems with emulators

CC-BY-SA: Marshall Astor

Problems with emulators• Incredibly slow

• Hard to interact with

• Android: Every manufacturer makes their own interface and browser

• Android: No Google Play Services

Best solution: Testing on real devices with real

inputs and a checking eye

CC-BY: yeowatzup

• Saw a talk about Sikuli for automated testing of websites

• Idea: Sikuli for mobile device testing but with a CI server

Sikuli

Sikuli• Sikuli is a tool for automating

computer tasks with screenshots

• A very python like script language

• Invented at MIT, now developed at Sikuli Labs (University of Colorado)

• Can be used for CI testing with Robot Framework or similar

• In short: Make a screenshot, use CV for detecting icon, tap icon

• With a robot

Making a screenshot

• Android: Use adb command and some bash magic

Making a screenshot

• iOS: Making screenshots of devices is possible using Xcode but not with the terminal

Making a screenshot

• iOS: Making screenshots of devices is possible using Xcode but not with the terminal

• But there is an open source project that reverse engineered some of iTunes’s and Xcode’s magic called libimobiledevice

Detecting an icon

• Use OpenCV to do a template recognition (finding an image in an image)

Detecting an icon

• OpenCV is written in C/C++ but has great python wrappers

Detecting an icon

• Quite simple API for such a complex task

Detecting an icon

• Quite simple API for such a complex task

• Get the coordinate in the screenshot of your searched icon

The robot

The robot• Tapsterbot

• Open Source delta robot

• Made with Arduino, 3 Servos and 3D printed parts

• Costs: ~250€ (With printing)

• Driver written in node.js

Talking to a robot

• The robot is connected via WebSocket

Talking to a robot

• The robot is driven by a node.js „app“

Talking to a robot

• Theoretically the robot could be exchanged quite easily

Talking to a robot

• Theoretically the robot could be exchanged quite easily

• More robots?

Tightening it all together

Tightening it all together• Project Goldeneye

• To test an app or a mobile website just write a standard python unittest test

• All the heavy lifting is abstracted away from you

• Just tell the test to tap and to check

• Writing such tests is quite easy, even your QA can do this

Conclusion

• Goldeneye could be useful for testing mobile apps and especially websites in a CI system

Conclusion

• Testing of existing projects, especially large web projects could be easier

Conclusion

• Robots never get tired

Conclusion

• Robots never get tired

• The robot’s hardware is fine, the software isn’t

Learnings

• A function in a function in a function isn’t normal, but in node.js it is

Learnings

• Never trust a github project with one maintainer

Learnings

• Never trust a github project with one maintainer

• OpenCV is fun and not that hard to learn

Thanks! Questions?

@: philip.brechler@motor-talk-gmbh.de T: @plaetzchen

Don't fear our new robot overlords – A new way to test on mobile

Mobile