Date post: | 22-Jan-2015 |
Category: |
Technology |
Upload: | satish-gorripotu |
View: | 9,800 times |
Download: | 9 times |
Sikuli Satish Gorripotu
oommee
SSIIKKUULLII
SSAATTIISSHH GGOORRRRIIPPOOTTUU
GGssaattiisshh__lleeeellaa@@yyaahhoooo..ccoomm
9885121381
Sikuli Satish Gorripotu
TTOOPPIICCSS AANNDD SSTTAATTUUSS
Topic Comments Page Number
Sikuli Documented
Sikuli and Other Supporting Softwares Required Documented
Installing/Configuring JDK or JRE / Setting Class Path Documented
Installing Sikuli Documented
Starting Sikuli IDE Documented
Stop the script during run time: Documented
Sikuli Working Architecture: Documented
Check Box: Documented
dragDrop: Documented
Sliders Documented
Testing on Skype Application: Documented
Meeting Request Arrival Notifier: Documented
Images Dictionary/Repository: Documented
Open any application in run time. Documented
Empty Recycle Bin: Documented
Run Sikuli from Command Line: Documented
Create Unit Testing Scripts for GUI: Documented
Sikuli Script in JAVA programs: Documented
Sikuli together with other IDE‘s - Eclipse: Documented
Import Packages/ Files Documented
Controlling Sikuli Scripts and their Behavior: Documented
Setting Documented
Controlling Applications and their Windows: Documented
App Documented
Interacting with the User Documented
Listening to Global Hotkeys Documented
General Settings and Access to Environment Information Documented
Region Documented
Extending a Region Documented
Finding inside a Region and Waiting for a Visual Event Documented
Observing Visual Events in a Region Documented
SikuliEvent Documented
Low-level Mouse and Keyboard Actions Documented
Exception FindFailed Documented
Screen Documented
Match Documented
Pattern Documented
Finder Documented
Key Documented
Constants related to Mouse and Keyboard Actions Documented
Sikuli Satish Gorripotu
Extensions Documented
Summary Documented
Java Docs Documented
Jython Basics Documented
References Documented
Excel Handling in Sikuli Yet to Complete
Object Repository Yet to Complete
Handling Synchronization Problems Yet to Complete
Methods that are mostly used Yet to Complete
Maintaining the Test Results Yet to Complete
Frameworks Yet to Complete
Database Testing using Sikuli Yet to Complete
Sending the test results to email by using Sikuli Yet to Complete
Challenges faced Yet to Complete
Sikuli Satish Gorripotu
SSiikkuullii
Sikuli is an open-source research project. This is a visual technology to automate and test graphical
user interfaces (GUI) using images (screenshots).
It includes
1. Sikuli Script,
2. Visual scripting API for Jython
3. Sikuli IDE.
Sikuli Script automates anything you see on the screen without internal API's support.
We can programmatically control a web page, a Windows/Linux/Mac OS X desktop application, or
even an iphone or android application running in a simulator or via VNC.
Download and unzip the Sikuli Using following link
http://sikuli.org/download.shtml
SSiikkuullii aanndd OOtthheerr SSuuppppoorrttiinngg SSooffttwwaarreess RReeqquuiirreedd
1. Sikuli-X-1.0rc2-win32.exe
2. Java Runtime Environment (JRE) 6+
Make sure to have only one instance of each software is available on machine.
You are provided with Admin access to configure the paths in the Environmental variables.
IInnssttaalllliinngg//CCoonnffiigguurriinngg JJDDKK oorr JJRREE // SSeettttiinngg CCllaassss PPaatthh
Install the JDK package and update the JAVA_HOME environment variable (User) in the Advanced
tab. Of System Properties. In system variable edit 'PATH' by adding C:\Program
Files\Java\jdk1.6.0_14\bin;
I . Right click on my computer->Properties->Advanced->Environment variables
II .
1. Click on "New" in system variables
2. Create a new variable as shown below
Variable name = JAVA_HOME
Variable value = C:\Program Files\Java\jdk1.6.0_14 // [path of jdk is installed]
III. Select the existing path variable and edit it as shown below
PATH = % JAVA_HOME%\bin; % JAVA_HOME%\lib;
IV. Click on OK
Sikuli Satish Gorripotu
V . START-> RUN ->type CMD
VI. Now enter "javac" to check whether the java correctly installed or not.
Sikuli Satish Gorripotu
IInnssttaalllliinngg SSiikkuullii
Sikuli-X-1.0rc2-win32.exe is self extracting file. Just double click and click next … next … until
finish whilst follow the instructions.
1. It will automatically install the software.
2. It will configure the environment variables.
3. In desk top or program files we can see the short cut for Sikuli IDE as
SSttaarrttiinngg SSiikkuullii IIDDEE
Double click on the Sikuli IDE will give the below IDE
There are two ways to capture the screen images
1. Click on in the IDE
2. Press a hot-key (Ctrl + Shift + 2)
Sikuli Satish Gorripotu
Screen will look darker and freeze momentarily. The entire desktop becomes like a canvas where we
can draw a rectangle around the target to capture an image.
Now that image will be captured and inserted in IDE.
We can use Auto List command capture to enable the basic commands in the IDE
instead of typing the commands manually.
1. Find
a. find
b. find all
c. wait
d. waitvanish
e. exists
2. Mouse Actions
a. Click
b. doubleClick
c. rightClick
d. hover
e. dragDrop
3. Keyboard Actions
a. type
b. paste
4. Event Observation
a. onAppear
b. OnVanish
c. onChange
d. observe
Below script will open the Keane web site ―www.keane.com‖ and search with the keyword ―Testing‖.
Sikuli Satish Gorripotu
Type:
type ("http://sikuli.org" + Key.ENTER)
miscellanous keys
ENTER - TAB - ESC - BACKSPACE - DELETE - INSERT
function keys
F1 - F2 - F3 - F4 - F5 - F6 - F7 - F8 - F9 - F10 - F11 - F12 - F13 - F14 - F15
navigation keys
HOME - END - LEFT - RIGHT - DOWN - UP - PAGE_DOWN - PAGE_UP
key modifiers
ALT - CMD - CTRL - META - SHIFT - WIN
Sikuli Satish Gorripotu
Stop the script during run time:
Press alt-shift-c
Sikuli Satish Gorripotu
Sikuli Working Architecture:
SSiikkuullii SSccrriipptt
Sikuli Script is a Jython and Java library that automates GUI interaction using image patterns to direct
keyboard/mouse events. The core of Sikuli Script is a Java library that consists of two parts:
java.awt.Robot, which delivers keyboard and mouse events to appropriate locations, and a C++ engine
based on OpenCV, which searches given image patterns on the screen. The C++ engine is connected to
Java via JNI and needs to be compiled for each platform.
On top of the Java library, a thin Jython layer is provided for end-users as a set of simple and clear
commands. Therefore, it should be easy to add more thin layers for other languages running on JVM, e.g.
JRuby, Scala, Javascript, etc.
TThhee SSttrruuccttuurree ooff aa SSiikkuullii ssoouurrccee//eexxeeccuuttaabbllee ssccrriipptt ((..ssiikkuullii,, ..sskkll))
A Sikuli script (.sikuli) is a directory that consists of a Python source file (.py), and all the image files
(.png) used by the source file.
All images used in a Sikuli script are simply a path to the .png file in the .sikuli bundle. Therefore, the
Python source file can also be edited by any text editor.
While saving a script using Sikuli IDE, an extra HTML file is also created in the .sikuli directory so that
users can share the scripts on the web easily.
A Sikuli executable script (.skl) is simply a zipped file of all files in the .sikuli directory. When a script is
passed to Sikuli IDE as a command line argument, Sikuli IDE recognizes its type by check its filename
extension. If a .skl is seen, Sikuli IDE runs it without showing the IDE window. If a .Sikuli is seen, Sikuli
IDE opens it in a source code editor.
Sikuli Satish Gorripotu
SSiikkuullii IIDDEE
Sikuli IDE edits and runs Sikuli source scripts. Sikuli IDE integrates screen capturing and a custom text
editor (SikuliPane) to optimize the usability of writing a Sikuli script.
To show embedded images in the SikuliPane, all string literals that ends with ".png" are replaced by a
custom JButton object, ImageButton. If a user adjusts the image pattern's similarity, a Pattern() is
automatically constructed on top of the image.
To execute a Sikuli script, Sikuli IDE creates a org.python.util.PythonInterpreter and automatically passes
a few lines of headers (e.g. to import Sikuli's Jython modules, and to set the path to .sikuli directory) to
the interpreter.
Once these headers are set, the .py script is simply executed by
PythonInterpreter.execfile().
Working with Check Box:
1. Check the all check boxes:
findAll() : searches the screen for all the matching visual patterns and returns a list of locations.
Then we can use click() function to check the all check box.
2. Check the particular check box:
Sikuli Satish Gorripotu
Step 1: we have to select the region
Step 2: Then identify check box
Below is the code for that.
Sikuli Satish Gorripotu
dragDrop:
The above script will drag and drop all the word files into specified folder.
Sikuli Satish Gorripotu
Working with Sliders
Same procedure as check boxes. But we can use dragDrop() function to move slide.
Way 1:
Way 2:
Need to work with Coordinates. Moving to Null. Need to resolve by Ram Raju.
Find Below Screen shot. There x, Y are the coordinates.
Main Scenario is: Zoom in and Zoom Out of the the word document.
Sikuli Satish Gorripotu
hover(find( ).nearby(200).right().find( ).below().find( ).getCenter().offset(3,18))
Sikuli Satish Gorripotu
Testing on Skype Application:
Now status of the ―Family – Bujji‖ is . We have to wait up to status as .
We can use any of the below way to
Way 1:
while not exists( ):
Sikuli Satish Gorripotu
sleep(5)
Way 2:
wait( ,FOREVER)
Way 3:
wait( ,3600)#wait for an hour
Way 4:
while exists( ):
sleep(5)
Way 5:
waitVanish( , FOREVER)
popup(―Oyee Satti !!!!!! Your Wife Online‖)
Sikuli Satish Gorripotu
Meeting Request Arrival Notifier:
Define a region and focus only on that region to search for a particular visual pattern. So tool don‘t have to
scan the whole screen and we can resize the browser to show just that portion of the map, while leaving a
lot of screen space to do something else.
To do so, click on the ―Create region‖ button.
The entire screen will freeze and turn darker, similar to what happen in the screen capture mode. Simply
draw a rectangle to cover the entire neighborhood map. The region covered by this rectangle is visually
represented as a thumbnail image of the entire desktop where the region is shaded in red.
Sikuli Satish Gorripotu
Next, we capture the screenshot image of the meeting symbol so that we can tell Sikuli Script to watch for
its appearance.
We can write the following script to check the mails and watch for the appearance of a meeting Request
symbol:
Way 1:
Way 2:
Images Dictionary/Repository:
1. Store Images as Predefined.
Sikuli Satish Gorripotu
Filename = VDict()
Filename [ ]= 'Recycle'
Filename [ ] = 'Skype'
2. Store Images in runtime
img = capture()
Filename [img] = input("Please enter Name to Captured image")
3. Retrieve name of the Image on popup.
Way 1:
query = capture()
gotit = Filename[query][0]
popup(gotit)
Way 2:
popup("select a icon in the ")
query = capture()
Result = Filename [query]
if result:
popup(Result[0])
else:
popup("unknown Image")
Open any application in run time.
setThrowException(True)
setAutoWaitTimeout(10000)
switchApp("C:\WINDOWS\system32\calc.exe")
Sikuli Satish Gorripotu
Empty Recycle Bin:
Run Sikuli from Command Line:
PATH-TO-SIKULI/sikuli-ide.exe [options] or
PATH-TO-SIKULI/sikuli-ide.bat [options]
Command Line Options
Sikuli-IDE
--args <arguments>
specify the arguments passed to Jython‘s sys.argv
-h,--help
print this help message
Sikuli Satish Gorripotu
usage:
Sikuli-IDE [--args <arguments>] [-h] [-r <sikuli-file>] [-s] [-t <sikuli-file>]
--args <arguments> specify the arguments passed to Jython's sys.argv
-h,--help print this help message
-r,--run <sikuli-file> run .sikuli or .skl file
-s,--stderr print runtime errors to stderr instead of popping up a message box
-t,--test <sikuli-file> run .sikuli as a unit test case with junit's text UI runner
-r,--run <sikuli-file>
run .sikuli or .skl file
-s,--stderr
print runtime errors to stderr instead of popping up a message box
-t,--test <sikuli-file>
run a .sikuli (.skl not possible) as a unit test with junit‘s text UI runner (A script that can also be
run in IDE‘s Unit Test View).
Example - Run xxxx.sikuli with three arguments:
PATH-TO-SIKULIsikuli-ide.exe -r xxxx.sikuli —args a1 a2 a3
Running sikuli-script.jar from Command line
Run a script without IDE
java -jar %SIKULI_HOME%\sikuli-script.jar path-to-your-script\yourScript.sikuli
Run a script from Command line using the Sikuli contained Jython
This option might be helpful in some cases, where you want to have access to the Jython layer,
before any Sikuli feature is touched. Be aware, that a from sikuli.Sikuli import * is needed to have
access to Sikuli. You might have to take care somehow, that images are found, since bundle path is
not set.
set SCRIPT=path-to-your-script\yourScript.sikuli\yourScript.py
java -cp %SIKULI_HOME%\sikuli-script.jar org.python.util.jython %SCRIPT%
Interactive Sikuli Jython Session from Command Line (Option -i)
sikuli-script.jar
-i
Start an interactive Jython session with the Sikuli environment already in place.
This might be helpful to do some testing on commandline, without having to start the Sikuli IDE. A
specified script and other options will be ignored.
java -jar %SIKULI_HOME%\sikuli-script.jar -i
Create Unit Testing Scripts for GUI:
Sikuli Satish Gorripotu
Sikuli integrates with jUnit and supports unit testing for Graphical User Interfaces (GUI). The unit
testing panel can be opened by clicking the menu View ‣ Unit Test or by the hot key ⌘ -U on Mac
or Ctrl-U on Windows/Linux.
Sikuli IDE aims to minimize the effort of writing code. With Sikuli IDE, a Python class inherited
from junit.framework.TestCase is automatically generated to wrap your unit testing script.
A typical unit testing script consists of two constructing and destructing methods, setUp() and
tearDown(), and a bunch of methods named with a prefix test.
The basic structure of a script is given as following:
TTeessttJJEEddiitt..ssiikkuullii
def setUp(self):
openApp("/Applications/jEdit.app")
wait( , 20000)
close = ( )
# if find(close):
# click(close)
def tearDown(self):
closeApp("jEdit.app")
untilNotExist( , 20000)
def test_textarea_add_del_by_key(self):
type("hello world")
assertExist( )
type("a",KEY_CTRL)
type("\n")
assertNotExist( )
def test_textarea_add_del_by_menu(self):
type("hello world")
assertExist( )
click( )
click( )
type("\n")
assertNotExist( )
def test_new_tab_by_key(self):
Sikuli Satish Gorripotu
type("n",KEY_CTRL)
assertExist(Pattern( ).similar(0.95))
def test_new_tab_by_menu(self):
click( )
click( )
assertExist(Pattern( ).similar(0.95))
def test_new_tab_by_toolbar(self):
click( )
assertExist(Pattern( ).similar(0.95))
def _verify_find(self):
sleep(0.5)
type("hello")
type("\n")
click(Pattern( ).similar(0.90))
assertExist(Pattern( ).similar(0.94))
type("\b")
assertExist( )
def test_find_by_key(self):
type("a long hello world\n")
type("f",KEY_CTRL)
self._verify_find()
def test_find_by_menu(self):
type("a long hello world\n")
click( )
click( )
self._verify_find()
Sikuli Satish Gorripotu
def test_find_by_toolbar(self):
type("a long hello world\n")
click( )
self._verify_find()
def test_textfield_on_toolbar(self):
type("a long hello world\n")
assertExist( )
type("f",KEY_CTRL)
assertExist(Pattern( ).similar(0.85))
type("hello")
assertExist( )
def test_toolbar_print_dialog(self):
click( )
assertExist( )
click( )
def test_menu_submenu(self):
click( )
click( )
assertExist( )
type("test")
click( )
click( )
Sikuli Satish Gorripotu
click( )
click( )
click( )
click( )
assertExist( )
def test_scrollbar(self):
for i in range(1,50):
type("line %d\n" % i)
tail = Pattern( ).similar(0.92)
wait(tail, 10000)
assertExist(tail)
thumb_at_top = Pattern( ).similar(0.98)
assertNotExist(thumb_at_top)
dragDrop( , [0, -1000])
assertExist(thumb_at_top)
assertNotExist(tail)
assertExist(Pattern( ).similar(0.92))
dragDrop(Pattern( ).similar(0.90), [0, 1000])
assertExist(tail)
def test_quit(self):
type("test")
type("e",KEY_CTRL)
type("q",KEY_CTRL)
click( )
Sikuli Satish Gorripotu
assertExist(Pattern( ).similar(0.90))
To run a unit testing script, you need to click on the Run button in the unit testing panel instead of
the ordinary button.
IMPORTANT: Before you try to run your script in this test mode the first time, it has to be saved.
Everytime you change something, you have to save it again, before the next test run.
Sikuli Script in JAVA programs:
The core of Sikuli Script is written in Java, which means you can use Sikuli Script as a standard
JAVA library in your program. This document lets you know how to do that.
After having installed Sikuli on your system, as recommended on the download page, you have to
do the following:
1. Get sikuli-script.jar from your Sikuli IDE installation path.
Sikuli-IDE/sikuli-script.jar
2. Make the native libraries available
3. Include sikuli-script.jar in the CLASSPATH of your Java project.
4. Import the Sikuli classes you need
You can simply ―import org.sikuli.script.*‖ or
import the classes you need. In most cases, you would need at least Region or Screen.
the package name was edu.mit.csail.uid.
Sikuli together with other IDE’s - Eclipse: Install Python plugin (PyDev) and Jython
Since the top level language used by Sikuli script is Python, you need the Python plugin PyDev.
Different from NetBeans, though PyDev is prepared to use Jython as interpreter, you have to install
Jython on your own seperately.
So first install Jython, by downloading the version you want to use from the
http://www.jython.org/downloads.html
Sikuli currently is based on Jython 2.5.1, but you may choose Jython 2.5.2 as well. Install it
according to Jythons installation HowTo.
After installation, make a test from a commandline by typing jython.bat (Windows) or jython
(Mac/Linux) to open an interactive Jython session.
You might type the following (<enter> means pressing the enter/return key)
import os <enter>
import sys <enter>
for e in sys.path: print e <enter> <enter>
type exit() <enter> (to leave interactive Jython)
Sikuli Satish Gorripotu
This shows the current Jython configuration of the Python path. Remember the place where Jython is
installed.
Now we install the PyDev plugin from inside Eclipse: Menu Help -> Eclipse Marketplace. Either
search it or find it on the Popular tab‘s lower part. Simply click the Install button and do what you are
asked to do ;-)
The next step is to tell Eclipse PyDev, where it can find the Jython interpreter. Navigate to the Eclipse
Preferences pane and open the category PyDev and inside the subcategory Interpreter-Jython. First
try to Auto Config by clicking the appropriate button. If this does not work, click the button New,
name the entry and navigate to the folder, where Jython was installed and select jython.jar. After
clicking ok, a window Selection needed might come up: click Select all and ok to finalize this step.
Other options with PyDev are available, but not relevant for our actual matter
http://pydev.org/manual_101_root.html
Configure for using Sikuli script features at runtime
Again we have a difference to Netbeans: The PyDev plugin does not allow library references to
folders inside of jar-files in the respective configuration dialog (it does not insert them to the Python
path), though Jython itself accepts them, when specified on the Python path. So if you want to run
and debug your script in Eclipse, you have to extract the folder Lib/sikuli from sikuli-script.jar
Menu File -> New -> Project -> PyDev -> (open sublist) -> PyDev Project -> click Next button.
On the configuration pane name your project, select Jython as Project type, grammar version 2.5
(higher Python language versions are not supported by Jython 2.5.x) and click the Finish button.
Your project is created. Add at least one you-name-it.py file to the source folder and put from
sikuli.Sikuli import * as the first line.
In the last step, we tell PyDev, where to find the Sikuli libraries.
Goto Menu Project -> Properties -> select category PyDev - PYTHONPATH and go to the tab
External Libraries. We need a reference to path-to/sikuli-script.jar and another one to the
extracted folder Lib containing the folder sikuli.
reference to path-to/sikuli-script.jar
o click button Add zip/jar/egg and select sikuli-script.jar from the Sikuli installation.
reference to the extracted folder Lib containing the folder sikuli
o click Add source folder and select the folder Lib in the place you had it extracted.
o this is not needed, if you have moved the extracted folder sikuli to a folder, that is
already on the Python path (e.g. jython-intallation/Lib/site-packages).
Now you are able to run your first script. Remember, that in every script including the main script,
that you are editing now, as the first line you need from sikuli.Sikuli import *, to have access to the
Sikuli features at runtime.
Sikuli Satish Gorripotu
Everytime later on you might come back to the project‘s preferences with Project -> Properties.
Code Completion works from the start without any further configuration and even steps into the
Java classes where appropriate.
Configuration Eclipse:
1) Open the Eclipse by double clicking on eclipse.exe, Click on OK for workspace launcher. Click on
workbench, it will allow you to move inside the eclipse.
2) Click on File->New->Select Java Project.
Sikuli Satish Gorripotu
3) Enter the Project name and then click on ―Configue JRE‘s‖
Sikuli Satish Gorripotu
Improve the performance of Sikuli
from org.sikuli.script.natives import Vision
Vision.setParameter("MinTargetSize", 6)
Import Packages/ Files
All files must be in Sys path. So Sikuli will find the files automatically, if we give it as
from sikuli import *
Example:
myScriptPath = "c:\\SikuliFiles\\myLibrary"
if not myScriptPath in sys.path: sys.path.append(myScriptPath)
# supposing there is a myLib.sikuli
import myLib
# supposing myLib.sikuli contains a function "def myFunction():"
myLib.myFunction() # makes the call
As Python load the file at once and will continue the library to entire script. If we modify the code in library
file, then Python wouldn‘t reflect the changes. So,
# instead of: import module
Sikuli Satish Gorripotu
import module
reload(module)
# instead of: from module import *
import module
reload(module)
from module import *
myLib.myFunction()
or
myLib import * # which integrates all names from myLib into your current namespace.
myFunction() # We can use directly
Importing from the directory:
import os
# get the directory containing your running .sikuli
myPath = os.path.dirname(getBundlePath())
if not myPath in sys.path: sys.path.append(myPath)
# now you can import every .sikuli in the same directory
import myLib
Scripts in same directory are found automatically
# nothing else needed
# now you can import every .sikuli in the same directory
import myLib
load(jar-file) : with path or no path
if no path is specified, Sikuli first looks into the bundle (the Sikuli folder of the running script) and
then into the extensions folder.
Controlling Sikuli Scripts and their Behavior:
setShowActions(False | True)
True, when a script is run, Sikuli shows a visual effect (a blinking double lined red
circle) on the spot.
exit([value])
Stops the script gracefully at this point. The value is returned to the calling environment.
Setting is a class available in Sikuli Package.
Sikuli Satish Gorripotu
This class has several methods
Class Setting {
1. ActionLogs (True/False) will display the log messages in console
2. InfoLogs (True/False) will display the Info log messages in console
3. DebugLogs (True/False) will display the Debug log messages in console
4. MinSimilarity
Sikuli will search for the minimal similarity of the object in the screen. Like smart Identification
5. MoveMouseDelay Control the mouse move animation.
mmd = Settings.MoveMouseDelay # save default/actual value
click(image1) # implicitly wait 0.5 seconds before click
Settings.MoveMouseDelay = 3
click(image2) # give app 3 seconds time before clicking again
Settings.MoveMouseDelay = mmd # reset to original value
6. DelayAfterDrag
Specifies the waiting time after mouse down at the source location as a decimal value (seconds).
7. DelayBeforeDrop
Specifies the waiting time before mouse up at the target location as a decimal value (seconds).
Above DelayAfterDrag, DelayBeforeDrop can be used as alternative of Wait Time.
# you may wish to save the actual settings before
Settings.DelayAfterDrag = 1
Settings.DelayBeforeDrop = 1
Settings.MoveMouseDelay = 3
dragDrop(source_image, target_image)
# time for complete dragDrop: about 5 seconds + search times
8. SlowMotionDelay
Control the duration of the visual effect (seconds).
9. WaitScanRate
10. ObserveScanRate
Specify the number of times actual search operations are performed per second while waiting for a pattern to appear or vanish.
Example: Region.wait(), Region.exists(), Region.waitVanish(), Region.observe()
Sikuli Satish Gorripotu
def myHandler(e):
print "it happened"
# you may wish to save the actual settings before
Settings.ObserveScanRate = 0.2
onAppear(some_image, myHandler)
observe(FOREVER, background = True)
# the observer will look every 5 seconds
# since your script does not wait here, you
# might want to stop the observing later on ;-)
11. ObserveMinChangedPixels
The minimum size in pixels of a change to trigger a change event.
Ex: Region.onChange() without specifying this value. The default value is 50.
}
Controlling Applications and their Windows:
Note: backslashes \ and double apostrophes ―
In a Sikuli script in normal strings enclosed in ‖ (double apostrophes), these special characters \ and ‖
have to be escaped using a backslash, when you have them inside the string. So for one backslash you
need \\ and for one ‖ you need \‖. In a string enclosed in ‗ (single apostrophes), a ‗ has to be \‘ and a ‖ is taken as such.
To avoid any problems, it is recommended to use the raw string r'some text with \\ and " ...', since there
is no need for escaping. This is especially useful, when you have to specify Windows path‘s or want to
setup command lines for use with App.open(), openApp(), os.popen or Jythons Subprocess module.
a fictive command line example:
cmd = r'c:\Program Files\myapp.exe -x "c:\Some Place\some.txt" >..\log.txt'
openApp(cmd)
Open an application : os.open --> openApp() –> App.open()
Switch to an application : switchApp() –> App.focus()
Close an application : closeApp() –> App.close()
Sikuli Satish Gorripotu
openApp(Application) Deprecated. Now it is App.open()
This function may switch to an already opened application or may open a new instance of the application.
# opens command prompt (found through PATH)
openApp("cmd.exe")
# with parameters (no sense, only to show ;-)
openApp(r'cmd.exe /c start c:\Program Files\myapp.bat')
# opens Firefox (full path specified)
openApp("c:\\Program Files\\Mozilla Firefox\\firefox.exe")
switchApp(application) Deprecated. Now it is App.focus()
This function switches the focus to the specified application and brings its windows to the front.
The window is identified by the application string.
This string (needs not be an application‘s name) is used to search the title text of all the opened
windows for any part of the title matching the string.
Example:
# switches to an existing command prompt or starts a new one
switchApp("cmd.exe")
# opens a new browser window
switchApp("c:\\Program Files\\Mozilla Firefox\\firefox.exe")
# switches to the frontmost opened browser window (or does nothing if no browser window is
currently opened)
switchApp("mozilla firefox")
closeApp(application) Deprecated. Now it is App.close().
Close the specified application.
Parameters: application – the name of an application or part of a window title
This function closes the application indicated by the windows whose titles contain the string
application.
Example:
# closes an existing command prompt
closeApp("cmd.exe")
# does nothing, since text cannot be found in the window title
closeApp("c:\\Program Files\\Mozilla Firefox\\firefox.exe")
# stops firefox including all its windows
Sikuli Satish Gorripotu
closeApp("mozilla firefox")
run(command)
Run command in the command line
Parameters: command – a command that can be run from the command line.
This function executes the command and the script waits for its completion.
Note: Windows: Sikuli‘s strategy on these systems in the moment is to rely on implicit or explicit path
specifications to find an application that has to be started.
Running ―applications‖ can either be identified using their PID (process ID) or by using the window titles.
So using a path specification will only switch to an open application, if the application internally handles the ―more than one instance‖ situation‖.
We usually use App.open("c:\\Program Files\\Mozilla Firefox\\Firefox.exe") to start Firefox. This might open an additional window.
And you can use App.focus("Firefox") to switch to the front most Firefox window
Note: When specifying a path in a string, we have to use \ (double backslash) for each (backslash)
e.g. myPath = "c:\\Program Files\\Sikuli-IDE\\Lib\\" )
App is a class available in Sikuli Package.
This class has several methods.
We can use App class methods in two ways
1. class methods
2. instance methods
Class App {
1. Open
Open the specified application. It opens the specified application and brings its
window the front. Whether this operation switches to an already opened application or opens a new instance of the application depends on the system and application
1. Class Method : open(application)
Usage : App.open("application-identifier")
2. Instance method : open() # App instance someApp was created before
Usage : myApp = App("application-identifier")
myApp.open()
2. Focus
Switch the focus to an application. Parameters are the name of an application or part of a window title
1. Class Method : focus (application)
Usage : App.focus("application-identifier")
Sikuli Satish Gorripotu
2. Instance method : open() # App instance someApp was created before
Usage : myApp = App("application-identifier")
myApp. focus ()
3. Close
It closes the given application or the matching windows.
It does nothing if no opened window or running application can be found.
Whether the application itself is closed depends on whether all open windows are closed
Or a main window of the application is closed, that in turn closes all other opened windows.
1. Class Method : close (application)
Usage : App. close ("application-identifier")
2. Instance method : close () # App instance someApp was created before
Usage : myApp = App("application-identifier")
myApp. close()
4. focusedWindow
Identify the currently focused or the front most window and switch to it.
Sikuli does not tell you, to which application this window belongs.
Region object representing the window or None if there is no such window.
This method always returns a region.
When there is no window opened on the desktop, the region may refer to a special window
such as the task bar or an icon in the system tray.
1. Class Method : focusedWindow (application)
Usage : App. focusedWindow ("application-identifier")
2. Instance method : close () # App instance someApp was created before
Usage : myApp = App("application-identifier")
myApp. focusedWindow ()
Example:
# highlight the currently fontmost window for 2 seconds
App.focusedWindow().highlight(2)
# save the windows region before
firstWindow = App.focusedWindow()
firstWindow.highlight(2)
5. window([n])
Sikuli Satish Gorripotu
Get the series of windows with the matching title.
1. Class Method : App(application).window([n])#an App instance is created on the fly.
Usage : App(application).window([n])
2. Instance method : close () # App instance someApp was created before
Usage : myApp = App("application-identifier")
myApp.window([n])
Parameters: n – 0 or a positive integer number.
If ommitted, 0 is taken as default.
Returns : the region on the screen occupied by the window,
if such window exists and None if otherwise.
Example :
#opens a Firefox browser window and switches to the address field.
# using an existing window if possible
myApp = App("Firefox")
if not myApp.window(): # no window(0) - Firefox not open
App.open("c:\\Program Files\\Mozilla Firefox\\Firefox.exe")
wait(2)
myApp.focus()
wait(1)
type("l", KEY_CTRL) # switch to address field
Afterwards, it focuses on the Firefox application, uses the window () method to obtain the region of
the front most window,
Applies some operations within the region,
And finally closes the window:
# using a new window
firefox = App.open("c:\\Program Files\\Mozilla Firefox\\Firefox.exe");
wait(2)
firefox.focus()
wait(1)
# now your just opened new window should be the frontmost
with firefox.window(): # see the general notes below
# some actions inside the window(0)'s region
click("somebutton.png")
firefox.close() # close the window - stop the process
Below is another example that highlights all the windows of an application by looping through them
# not more than 100 windows should be open ;-)
myApp = App("Safari")
for n in range(100):
w = myApp.window(n)
if not w: break # no more windows
w.highlight(2) # window highlighted for 2 second
Sikuli Satish Gorripotu
}
Interacting with the User
popup(text[, title])
Display a dialog box with an OK button and text as the message.
The script then waits for the user to click the OK button.
Parameters: text – text to be displayed as message
title – optional title for the messagebox
Example:
popup("Hi satish….\n How are you")
input([text][, default])
Display a dialog box with an input field, a Cancel button, and an OK button.
The optional text can be displayed as a caption.
The script then waits for the user to click either the Cancel or the OK button.
Parameters: text – optional text to be displayed as message
default – optional preset text for the input field
Returns:
the text, contained in the input field, when the user clicked OK
None, if the user pressed the Cancel button
Example:
name = input("Please enter your name:")
name = input("Please enter your name to log in:", "anonymous") # a preset input text
When using the parameter default, the text input field will be pre-populated with the given text,
So we click OK/Cancel or edit the content of the input field
Sikuli Satish Gorripotu
Listening to Global Hotkeys
Env.addHotkey(key, modifiers, handler)
Register the specified key + modifiers as a global hotkey.
When the hotkey is pressed, the specified function handler will be called.
Parameters:
key – a character or a constant value defined in Key.
modifiers – Key modifiers, which can be one or multiple constants defined in KeyModifier.
def openAppleMenu(event):
click(Any Image)
# When we pressed Ctrl+Alt+F1, click the top-left apple icon.
Env.addHotkey(Key.F1, KeyModifier.ALT+KeyModifier.CTRL, openAppleMenu)
Env.removeHotkey(key, modifiers)
Unregister the registered global hotkey key + modifiers.
Parameters:
key – a character or a constant value defined in Key.
modifiers – Key modifiers, which can be one or multiple constants defined in KeyModifier.
General Settings and Access to Environment Information
Sikuli Level
Sikuli internally uses the class Settings to store globally used settings.
Publicly available attributes accessed by using Settings.[name-of-an-attribute]
To get it‘s value and Settings.attribute = value to set it.
Jython/Python Level
We can use all settings defined in standard Python/Jython which are available in our system
environment.
The modules sys and time are already imported, so you can use their methods without the need for
an import statement.
sys.path may be one of the most valuable settings, since it is used by Python/Jython to locate
modules, that are referenced using import module.
myPath = "some-absolute-path"
if not myPath in sys.path:
sys.path.append(myPath)
Java Level
import java
Sikuli Satish Gorripotu
# get a value
val = java.lang.System.getProperty("key-of-property")
# set a property's value
java.lang.System.getProperty("key-of-property", value)
Image Search Path
Sikuli maintains a list of locations to search for images when they are not found in the current
.sikuli folder (a.k.a. bundle path).
This list is maintained internally but can be inspected and modified.
getImagePath()
Get a list of paths where Sikuli will search for images.
# getImagePath() returns a Java array of unicode strings
imgPath = list(getImagePath()) # makes it a Python list
# to loop through
for p in imgPath:
print p
addImagePath(a-new-path)
Add a new path to the list of image search paths
removeImagePath(a-path-already-in-the-list)
Remove a path from the list of image search paths
Note: paths must be specified using the correct path separators as double blackslashes .
Default bundle path can accessed and modified.
setBundlePath(path-to-a-folder)
Set the path for searching images in all Sikuli Script methods.
Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli).
getBundlePath()
Get a string containing a path to a folder containing our images used for finding patterns.
Sikuli IDE sets this automatically to the path of the folder where it saves the script (.sikuli).
We can use this function to package our private files together with the script or to access the
picture files in the .sikuli bundles for other purposes.
Sikuli only gives access to the path name, We need other python modules for I/O or other
purposes.
Other Environment Information
Env.getOS()
Env.getOSVersion()
Env.getSikuliVersion()
Example:
if not Env.getSikuliVersion() == "X-1.0rc2":
Sikuli Satish Gorripotu
print "This script needs Sikuli X-1.0rc2"
exit(1)
Env.getClipboard()
Get the content of the clipboard if it is text, otherwise an empty string.
Env.getClipboard().paste()
clipboard will contain what we just pasted.
Env.getClipboard().strip()
To get rid of surrounding white spaces.
Env.isLockOn(key-constant)
Get the current status ( on / off ) off the respective key. Only one key can be specified.
key-constant – one of the key constants Key.CAPS_LOCK, Key.NUM_LOCK, Key.SCROLL_LOCK
Env.getMouseLocation()
Get the current location of the mouse cursor.
Advanced Settings for Tuning Vision Algorithm
Vision.setParameter(param, value)
Vision.getParameter(param)
Get/Set the parameter param of the vision algorithm.
Parameters:
param – a string that indicates the parameter to get.
Returns:
The float value of the specified parameter.
MinTargetSize
MinTargetSize is the minimum image size to which Sikuli can resize.
Sikuli resizes the screen images to a smaller scale for faster matching. This scaling process speeds
up the matching process, but also increases the possibility of false matching.
The default value of MinTargetSize is 12,
< 12. May increase speed but may returns a match that is not what we expect.
Example Jython code.
from org.sikuli.script.natives import Vision
Vision.setParameter("MinTargetSize", 6) # the default is 12. Setting the size to a smaller
value would make the matching algorithm be faster.
Sikuli Satish Gorripotu
Region
Region is a class available in Sikuli Package.
This class has several methods
Class Region
{
class Region Region is a rectangular area on a screen.
It knows nothing about its visual content (windows, pictures, graphics, text, ...).
It only knows the position on the screen and its dimension.
Creating a Region, Setting and Getting Attributes
Region(x, y, w, h)
Region(region)
Region(Rectangle)
Sikuli Satish Gorripotu
Parameters:
x – x position of a rectangle. #upper left corner of the screen (0, 0)
y – y position of a rectangle. #upper left corner of the screen (0, 0)
w – height of a rectangle.
h – width of a rectangle.
region – an existing region object.
rectangle – an existing object of Java class Rectangle
Returns: a new region object.
region can also be created in run-time using Region.selectRegion().
You can create a region based on another region. This just duplicates the region (new object).
Another way to create a region is to specify a rectangle object or to extend an existing region.
Note:
topLeft = Location(reg.x, reg.y) # equivalent to
topLeft = reg.getTopLeft()
theWidth = reg.w # equivalent to
theWidth = reg.getW()
reg.h = theWidth # equivalent to
reg.setH(theWidth)
We can use selectRegion() to interactively create a new region.
setX(number)
setY(number)
setW(number)
setH(number)
Set the respective attribute of the region to the new value.
This effectively moves the region around and/or changes its dimension.
moveTo(location)
Set the position of this region regarding it‘s top left corner to the given location (the x and y values
are modified).
Parameters : location – location object becomes the new top left corner
Returns : the modified region object
Example:
reg.moveTo(anotherLocation) # equivalent to
reg.setX(anotherLocation.x); reg.setY(anotherLocation.y)
setROI(x, y, w, h)
setROI(rectangle)
setRect(x, y, w, h)
Sikuli Satish Gorripotu
setRect(rectangle)
Both methods do the same: setting position and dimension to new values.
setROI() is intended to restrict the search to a smaller area to speed up processing
setRect() should be used to redefine a region (which could be enlarging it).
Parameters: x, y, w, h (all) – the attributes of a rectangle
rectangle – a rectangle object
.
morphTo(region)
Set the position and dimension of this region to the corresponding values of the region given as
parameter.
Parameters:region – a region object
Returns: the modified region object
Note:
reg.morphTo(anotherRegion) # equivalent to
r = anotherRegion; reg.setX(r.x); reg.setY(r.y); reg.setW(r.w); reg.setH(r.h)
getX()
getY()
getW()
getH()
Get the respective attribute integer value of the region.
getCenter()
Get the center of the region or Location as an object.
getTopLeft()
getTopRight()
getBottomLeft()
getBottomRight()
Get the location object of the region‘s respective corner
getScreen()
Returns the screen object that contains this region.
This method makes sense in Multi Monitor Environments
getLastMatch()
getLastMatches()
Returns the best match as a Match object or one or more match objects as an Iterator
object respectively
setAutoWaitTimeout(seconds)
Set the maximum waiting time for all subsequent find operations.
Sikuli Satish Gorripotu
This method enables all find operations to wait for the given pattern to appear until the specified
amount of time has elapsed. The default is 3.0 seconds.
This method is intended for users to override this default setting. Region.find() work like
Region.wait(), without being able to set an individual timeout value for a specific find operation.
getAutoWaitTimeout()
Get the current value of the maximum waiting time for find operations.
Extending a Region
Note: In all cases the current region remains unchanged.
In all cases the new region does not extend beyond any boundary of the screen that contains
the given region.
offset(location)
Returns a new Region object, whose upper left corner is relocated adding the location‘s x and y
value to the respective values of the given region. Width and height are the same. So this clones a
region at a different place.
Parameters : location – a Location object
Returns :a new Region object
Example:
new_reg = reg.offset(Location(xoff, yoff)) # same as
new_reg = Region(reg.x + xoff, reg.y + yoff, reg.w, reg.h)
inside()
Returns the same Region object. Retained for upward compatibility.
region.inside().find() is totally equivalent to region.find().
nearby([range])
Returns a new Region object that includes the nearby neighbourhood of the the current region.
The new region is defined by extending the current region‘s dimensions in all directions by range
number of pixels.
The center of the new region remains the same.
Sikuli Satish Gorripotu
Parameters: range – a positive integer indicating the number of pixels (default = 50).
above([range])
Returns a new Region object that is defined above the current region‘s top border with a height of
range number of pixels.
It does not include the current region.
If range is omitted, it reaches to the top of the screen. The new region has the same width and x-
position as the current region.
Parameters:range – a positive integer defining the new height
below([range])
Returns a new Region object that is defined below the current region‘s bottom border with a height
of range number of pixels.
It does not include the current region.
If range is omitted, it reaches to the bottom of the screen. The new region has the same width and
x-position as the current region.
Parameters:range – a positive integer defining the new height
left([range])
Returns a new Region object that is defined left of the current region‘s left border with a width of
range number of pixels.
It does not include the current region.
If range is omitted, it reaches to the left border of the screen. The new region has the same height
and y-position as the current region.
Parameters:range – a positive integer defining the new width
right([range])
Returns a new Region object that is defined right of the current region‘s right border with a width
of range number of pixels. So it does not include the current region. If range is omitted, it reaches
to the right border of the screen. The new region has the same height and y-position as the current
region.
Parameters:range – a positive integer defining the new width
Finding inside a Region and Waiting for a Visual Event
Besides acting on visual objects, finding them is one of the core functions of Sikuli.
PS: means, that either a Pattern or a string (path to an image file or just plain text) can be used as
parameter.
find(PS)
Find a particular GUI element, which is seen as the given image or just plain text. The given
file name of an image specifies the element‘s appearance. It searches within the region and returns
the best match, which shows a similarity greater than the minimum similarity given by the pattern.
Parameters:PS – a Pattern object or a string (path to an image file or just plain text)
Returns: a Match object that contains the best match or fails if not found
findAll(PS)
Sikuli Satish Gorripotu
Repeatedly find ALL instances of a pattern, until no match can be found anymore, that meets the
requirements for a single Region.find() with the specified pattern.
Parameters:PS – a Pattern object or a string (path to an image file or just plain text)
Returns :one or more Match objects as an iterator object or fails if not found
By default, the returned matches are sorted by the similiarty.
If you need them ordered by their positions, we have to use Python‘s sorted function.
Example:
def by_y(match):
return match.y
icons = findAll(―Image screen shot‖)
# sort the icons by their y coordinates and put them into a new variable sorted_icons
sorted_icons = sorted(icons, key=by_y)
# another shorter version is using lambda.
sorted_icons = sorted(icons, key=lambda m:m.y)
for icon in sorted_icons:
pass # do whatever you want to do with the sorted icons
wait([PS][, seconds])
a Match object that contains the best match or fails if not found
Parameters:
PS – a Pattern object or a string (path to an image file or just plain text)
seconds – a number, which can have a fraction, as maximum waiting time in seconds.
If not specified, the auto wait timeout value set by Region.setAutoWaitTimeout() is used.
Use the constant FOREVER to wait for an infinite time.
If PS is not specified, the script just pauses for the specified amount of time. It is still possible to
use sleep(seconds) instead, but this is deprecated.
If PS is specified, it keeps searching the given pattern in the region until the image appears ( would
have been found with Region.find()) or the specified amount of time has elapsed. At least one find
operation is performed, even if 0 seconds is specified.)
waitVanish(PS[, seconds])
Wait until the give pattern PS in the region vanishes. Returns True if the pattern vanishes within
the specified waiting time, or False if the pattern stays visible after the waiting time has elapsed.
This method keeps searching the given pattern in the region until the image vanishes (can not be
found with Region.find() any longer) or the specified amount of time has elapsed.
At least one find operation is performed, even if 0 seconds is specified
Parameters:
PS – a Pattern object or a string (path to an image file or just plain text)
seconds – a number, which can have a fraction, as maximum waiting time in seconds.
The internal granularity is milliseconds.
If not specified, the auto wait timeout value set by Region.
setAutoWaitTimeout() is used. Use the constant FOREVER to wait for an infinite time.
Sikuli Satish Gorripotu
.
exists(PS[, seconds])
Check whether the give pattern is visible on the screen. And returns a Match object that contains
the best match. None is returned, if nothing is found within the specified waiting time
Does exactly the same as Region.wait(), but no exception is raised in case of FindFailed.
It can be used to symplify scripting in case that we want to know wether something is there or not
to decide how to proceed in your workflow.
It is typically used with an if statement. At least one find operation is performed, even if 0 seconds
is specified.
Parameters:
PS – a Pattern object or a string (path to an image file or just plain text)
seconds –
a number, which can have a fraction, as maximum waiting time in seconds. The
internal granularity is milliseconds.
If not specified, the auto wait timeout value set by Region.setAutoWaitTimeout() is
used. Use the constant FOREVER to wait for an infinite time.
Observing Visual Events in a Region
handler: as a parameter in the following methods, you have to specify the name of a function,
which will be called by the observer, in case the observed event happens.
The function itself has to be defined in your script before using the method that references the
function.
The existance of the function will be checked before starting the script. So to get your script
running, you have to have at least the following statements in your script:
def myHandler(event): # you can choose any valid function name
# event: can be any variable name, it references the SikuliEvent object
pass # add your statements here
onAppear("path-to-an-image-file", myHandler) # or any other onEvent()
observe(10) # observes for 10 seconds
onAppear(PS, handler)
With the given region you register an observer, that should wait for the pattern to be there or to
appear and is activated with the next call of observe().
In the moment the internal find operation on the given pattern is successful during observation,
your handler is called and the observation is paused until you return from your handler.
Parameters: PS – a Pattern object or a string (path to an image file or just plain text.
handler – the name of a handler function in the script
Sikuli Satish Gorripotu
onVanish(PS, handler)
With the given region you register an observer, that should wait for the pattern to be not there or
to vanish and is activated with the next call of observe().
In the moment the internal find operation on the given pattern fails during observation, your
handler is called and the observation is paused until you return from your handler.
Parameters: PS – a Pattern object or a string (path to an image file or just plain text.
handler – the name of a handler function in the script
onChange([minChangedSize], handler)
Parameters: minChangedSize – the minimum size in pixels of a change to trigger a change event.
If omitted: 50 is used
handler – the name of a handler function in the script
minChangedSize
With the given region we register an observer, that should wait for the visual content of the given
region to change and is activated with the next call of observe().
In the moment the visual content changes during observation, our handler is called and the
observation is paused until you return from your handler.
Below example highlights all changes in an observed region.
def changed(event):
print "something changed in ", event.region
for ch in event.changes:
ch.highlight() # highlight all changes
sleep(1)
for ch in event.changes:
ch.highlight() # turn off the highlights
with selectRegion("select a region to observe") as r:
# any change in r larger than 50 pixels would trigger the changed function
onChange(50, changed)
observe(background=True)
wait(30) # another way to observe for 30 seconds
r.stopObserver()
observe([seconds][, background = False | True])
Begin observation within the region.
For each region object, only one observation can be running at a given time.
Sikuli Satish Gorripotu
Parameters:
seconds – a number, which can have a fraction, as maximum observation time in seconds.
Omit it or use the constant FOREVER to tell the observation to run for an infinite time (or until
stopped by a call of stopObserve()).
background – a flag indicating whether observation is run in the background. when set to True, the
observation will be run in the background and processing of your script is continued immediately.
Otherwise the script is paused until the completion of the observation.
stopObserver()
Stop observation within the region.
This must be called on a valid region object. The source region of an observed visual event is
available as one of the attributes of the event parameter that is passed to the handler function
when the function is invoked.
Example: to stop observation within a handler function, call event.region.stopObserver() inside the
handler function.
def myHandler(event):
event.region.stopObserver() # stops the observation
onAppear("path-to-an-image-file", myHandler)
observe(FOREVER) # observes until stopped in handler
}
SikuliEvent
SikuliEvent is a class available in Sikuli Package.
This class has several methods
Class SikuliEvent
{
Sikuli Satish Gorripotu
When processing an observation in a region, a handler function is called, when one of the
registered events
Region.onAppear(),
Region.onVanish() or
Region.onChange()
One parameter, we have access to in handler function is an instance of Class SikuliEvent.
Following attributes of the event, might help to identify the cause of the event and act on the
resulting matches.
Usage: event.attribute
where event is the parameter from the definition of handler function: def myHandler(event):
type
SikuliEvent.Type.APPEAR,
SikuliEvent.Type.VANISH, or
SikuliEvent.Type.CHANGE.
pattern
The source pattern that triggered this event. This is only valid in APPEAR and VANISH events.
region
The source region of this event.
match
In an APPEAR event, match saves the top Match object that appeared in the observed region.
In a VANISH event, match saves the last Match object that was in the observed region but
vanished.
This attribute is not valid in a CHANGE event.
changes
This attribute is valid only in a CHANGE event.
This changes attribute is a list of Match objects that changed and their sizes are at least
minChangedSize pixels.
}
Acting on a Region
PSMRL: which means, that either a Pattern object or a string (path to an image file or just plain
text) or a Match or a Region or a Location can be used as parameter.
P: pattern: a Pattern object. An implicit find operation is processed first. If successful, the center
of the resulting matches rectangle is the click point. If the pattern object has a target offset
specified, this is used as click point instead.
S: string: a path to an image file or just plain text. An implicit find operation with the default
minimum similarity 0.7 is processed first. If successful, the center of the resulting match object is
the click point.
Sikuli Satish Gorripotu
M: match: a match object from a previous find operation. If the match has a target specified it is
used as the click point, otherwise the center of the matches rectangle.
R: region: a region object whose center is used as click point.
L: location: a location object which by definition represents a point on the screen that is used as
click point.
Region
Region is a class available in Sikuli Package.
This class has several methods continuation to above methods.
Class Region
{
click(PSMRL[, modifiers])
Perform a mouse click on the click point using the left button.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
modifiers – one or more key modifiers
Returns:
Number of performed clicks (actually 1). A 0 (integer null) means that because of some
reason, no click could be performed (in case of PS may be not Found).
doubleClick(PSMRL[, modifiers])
Perform a mouse double-click on the click point using the left button.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
modifiers – one or more key modifiers
Returns:
Number of performed double-clicks (actually 1). A 0 (integer null) means that because of
some reason, no click could be performed (in case of PS may be not Found).
rightClick(PSMRL[, modifiers])
Perform a mouse click on the click point using the right button.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
modifiers – one or more key modifiers
Returns:
Number of performed right cilicks (actually 1). A 0 (integer null) means that because of
some reason, no click could be performed (in case of PS may be not Found).
highlight([seconds])
Highlight the region for some period of time.
Parameters:seconds – a decimal number taken as duration in seconds
The region is highlighted showing a red colored frame around it.
If the parameter seconds is given, the script is suspended for the specified time.
If no time is given, the highlighting is started and the script continues.
Sikuli Satish Gorripotu
When later on the same highlight call without a parameter is made, the highlighting is stopped
(behaves like a toggling switch).
Example:
m = find(some_image)
# the red frame will blink for about 7 - 8 seconds
for i in range(5):
m.highlight(1)
wait(0.5)
# a second red frame will blink as an overlay to the first one
m.highlight()
for i in range(5):
m.highlight(1)
wait(0.5)
m.highlight()
# the red frame will grow 5 times
for i in range(5):
m.highlight(1)
m = m.nearby(20)
hover(PSMRL)
Move the mouse cursor to hover above a click point.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
modifiers – one or more key modifiers
Returns:
The number 1 if the mousepointer could be moved to the click point.
A 0 (integer null) returned if no move could be performed (in case of PS may be not Found).
dragDrop(PSMRL, PSMRL[, modifiers])
Perform a drag-and-drop operation from a starting click point to the target click point
indicated by the two PSMRLs respectively.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
modifiers – one or more key modifiers
If one of the parameters is PS, the operation might fail due to not Found.
drag(PSMRL)
Start a drag-and-drop operation by dragging at the given click point.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
Sikuli Satish Gorripotu
Returns:
Number 1 if the operation could be performed.
A 0 (integer null) returned if no move could be performed (in case of PS may be not Found).
dropAt(PSMRL[, delay])
Complete a drag-and-drop operation by dropping a previously dragged item at the given target
click point.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
delay – time to wait after in seconds as decimal value
Returns:
Number 1 if the operation could be performed.
A 0 (integer null) returned if no move could be performed (in case of PS may be not Found).
type([PSMRL], text[, modifiers])
Type the text at the current focused input field or at a click point specified by PSMRL.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
modifiers – one or more key modifiers
Returns:
Number 1 if the operation could be performed,
0 (integer null) if it was not possible or the click couldn‘t be performed (in case of PS may
be not Found).
paste([PSMRL], text)
Paste the text at a click point.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
Modifiers – one or more key modifiers
Returns:
Number 1 if the operation could be performed,
0 (integer null) if not possible or the click couldn‘t be performed (in case of PS may be not
Found).
Extracting Text from a Region
text()
Extract the text contained in the region using OCR.
Sikuli Satish Gorripotu
Low-level Mouse and Keyboard Actions
mouseDown(button)
Press the mouse button down.
Parameters:
button – one or a combination of the button constants
Button.LEFT,
Button.MIDDLE,
Button.RIGHT.
Returns:
Number 1 if the operation is performed successfully, and zero if otherwise.
mouseUp([button])
Release the mouse button previously pressed.
Parameters:
button – one or a combination of the button constants
Button.LEFT,
Button.MIDDLE,
Button.RIGHT.
Returns:
Number 1 if the operation is performed successfully, and zero if otherwise.
mouseMove(PSRML)
Move the mouse pointer to a location indicated by PSRML.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
Sikuli Satish Gorripotu
Returns:
Number 1 if the operation could be performed. If using PS (which invokes an implicity find
operation), find fails and you have switched off FindFailed exception, a 0 (integer null) is returned.
Otherwise, the script is stopped with a FindFailed exception.
wheel(PSRML, WHEEL_DOWN | WHEEL_UP, steps)
Move the mouse pointer to a location indicated by PSRML and turn the mouse wheel in the
specified direction by the specified number of steps.
Parameters:
PSMRL – a pattern, a string, a match, a region or a location that evaluates to a click point.
WHEEL_DOWN|WHEEL_UP – one of the two constants denoting the wheeling direction.
steps – an integer indicating the amoung of wheeling.
keyDown(key | list-of-keys)
Press and hold the specified key(s) until released by a later call to Region.keyUp().
Parameters:
key|list-of-keys – one or more keys (use the consts of class Key).
A list of keys is a concatenation of several key constants using ―+‖.
Returns:
Number 1 if the operation could be performed and 0 if otherwise.
keyUp([key | list-of-keys])
Release given keys. If no key is given, all currently pressed keys are released.
Parameters:
key|list-of-keys – one or more keys (use the consts of class Key).
A list of keys is a concatenation of several key constants using ―+‖.
Returns:
Number 1 if the operation could be performed and 0 if otherwise.
Exception FindFailed
These are the possibilities to handle ―not found‖ situations:
•generally abort a script, if not handled with try: ... except: ...
(the default setting or using setThrowException(True) or setFindFailedResponse(ABORT))
•generally ignore all ―not found‖ situations
(using setThrowException(False) or setFindFailedResponse(SKIP)),
•want to be prompted in such a case
(using setFindFailedResponse(PROMPT))
•advise Sikuli to wait forever (be careful with that!)
(using setFindFailedResponse(RETRY))
Using PROMPT:
Sikuli Satish Gorripotu
Example:
# --- nice and easy
if exists("path-to-image"): # no exception, returns None when not found
pass # it is there
else:
pass # we miss it
# --- using exception handling
# every not found in the try block will switch to the except block
try:
find("path-to-image")
pass # it is there
except FindFailed:
pass # we miss it
# --- using setFindFailedResponse
setFindFailedResponse(SKIP) # no exception raised, not found returns None
if find("path-to-image"):
setFindFailedResponse(ABORT) # reset to default
pass # it is there
else:
setFindFailedResponse(ABORT) # reset to default
pass # we miss it
# --- using setThrowException
setThrowException(False) # no exception raised, not found returns None
if find("path-to-image"):
setThrowException(True) # reset to default
pass # it is there
else:
setThrowException(True) # reset to default
pass # we miss it
setFindFailedResponse(ABORT | SKIP | PROMPT | RETRY)
For the specified region set the option, how Sikuli should handle ―not found‖ situations. The option
stays in effect until changed by another setFindFailedResponse().
Sikuli Satish Gorripotu
Parameters:
ABORT – all subsequent find failed operations (explicit or implicit) will raise exception FindFailed
(which is the default when a script is started).
SKIP – all subsequent find operations will not raise exception FindFailed. Instead, explicit find
operations such as Region.find() will return None. Implicit find operations (action functions) such
as Region.click() will do nothing and return 0.
PROMPT – all subsequent find operations will not raise exception FindFailed. Instead you will be
prompted for the way to handle the situation (see using PROMPT). Only the current find operation
is affected by your response to the prompt.
RETRY – all subsequent find operations will not raise exception FindFailed. Instead, Sikuli will try
to find the target until it gets visible. This is equivalent to using wait( ... , FOREVER) instead of
find() or using setAutoWaitTimeout(FOREVER).
getFindFailedResponse()
Get the current setting in this region.
Returns:
ABORT or SKIP or PROMPT or RETRY
Usage:
val = getFindFailedResponse()
if val == ABORT:
print "not found will stop script with Exception FindFailed"
elif val == SKIP:
print "not found will be ignored"
elif val == PROMPT:
print "when not found you will be prompted"
elif val == RETRY:
print "we will always wait forever"
else:
print val, ": this is a bug :-("
Note: It is recommended to use set/getFindFailedResponse() instead of set/getThrowException()
since the latter ones might be deprecated in the future.
setThrowException(False | True)
By using this method you control, how Sikuli should handle not found situations in this region.
Parameters:
True – all subsequent find operations (explicit or implicit) will raise exception FindFailed (which is
the default when a script is started) in case of not found.
False – all subsequent find operations will not raise exception FindFailed. Instead, explicit find
operations such as Region.find() will return None. Implicit find operations (action functions) such
as Region.click() will do nothing and return 0.
getThrowException()
Returns:
True or False
Sikuli Satish Gorripotu
Grouping Method Calls ( with Region: )
Instead of:
# reg is a region object
if not reg.exists(image1):
reg.click(image2)
reg.wait(image3, 10)
reg.doubleClick(image4)
you can group methods applied to the same region using Python‘s with syntax:
# reg is a region object
with reg:
if not exists(image1):
click(image2)
wait(image3, 10)
doubleClick(image4)
All methods inside the with block that have the region omitted are redirected to the region object
specified at the with statement.
Note for Sikuli X versions up to rc3
This should work for Regions and Region-like objects (Region, Screen, App.window()). Match
objects have to be made a Region explicitly using Region(some_match).
This is a general workaround when facing any problems using with some_region:.
Special Methods for Developers
getRegionFromPSRM(PSRM)
Returns a new Region object derived from the given parameter. In case of PS, internally a find() is
done inside this region. If found, the match is returned. In case RM, just a copy of the given region
is returned.
Params PSRM:
Pattern, String, Region or Match object
getLocationFromPSRML(PSRML)
Returns a new Location object derived from the given parameter. In case of PS, internally a find()
is done inside this region. If found, the match‘s target offset position is returned. In case RM, just a
copy of the given region‘s respective location (center or target offset) is returned.
Params PSRML:
Pattern, String, Region, Match or Location object
}
Screen
Sikuli Satish Gorripotu
Screen is a class available in Sikuli Package.
This class has several methods continuation to above methods.
Since Screen extends class Region, all methods of class Region can be used with a screen object.
Class Screen
{
Screen: Setting, Getting Attributes and Information
Screen([id])
Create a new Screen object
Parameters: id – an integer number indicating which monitor in a multi-monitor environment.
Returns: a new screen object.
It creates a new screen object, that represents the default/primary monitor (whose id is 0), if id is
omitted. Numbers 1 and higher represent additional monitors that are available at the time, the
script is running.
Using numbers that do not represent an existing monitor, will stop the script with an error.
So we may either use getNumberScreens() or exception handling, to avoid this.
Note: If we want to access the default/primary monitor ( Screen(0) ) without creating a new
screen object, use the constant reference SCREEN, that is initiated when your script starts:
SCREEN=Screen(0).
getNumberScreens()
Get the number of screens in a multi-monitor environment at the time the script is running
getBounds()
Get the dimensions of monitor (a rectangle object) represented by the screen object.
Capturing
Capturing is the feature, that allows to grab a rectangle from a screenshot, to save it for later use.
At each time, a capturing is initiated, a new screenshot is taken.
There are two different versions:
Screen.capture() saves the content of the selected rectangle in a file and returns its file name,
Screen.selectRegion() returns the position and dimension of the selected rectangle.
capture([region | rectangle | text])
capture(x, y, w, h)
Returns the path to the file, where the captured image was saved.
In interactive mode, the user may cancel the capturing, in which case None is returned.
Parameters:
region – an existing region object.
rectangle – an existing rectangle object (e.g., as a return value of another region method).
text – text to display in the middle of the screen in the interactive mode.
x – x position of the rectangle to capture
y – y position of the rectangle to capture
w – width of the rectangle to capture
h – height of the rectangle to capture
Sikuli Satish Gorripotu
Interactive Mode: The script enters the screen-capture mode like when clicking the button in the
IDE, enabling the user to capture a rectangle on the screen. If no text is given, the default ―Select
a region on the screen‖ is displayed.
selectRegion([text])
Select a region on the screen interactively.
Returns a new Region object or None, if the user cancels the capturing process
Text is displayed for about 2 seconds in the middle of the screen. If text is omitted, the default
―Select a region on the screen‖ is displayed.
The interactive capture mode is entered and allows the user to select a region the same way as
using the selection tool in the IDE.
Parameters: text – Text to display in the middle of the screen.
}
Multi-Monitor Environments
If more than one monitor is available, Sikuli is able to manage regions and click points on these
monitors.
getNumberScreens() returns the number of available screens.
getBounds() returns the rectangle covered by the default/primary monitor.
Screen.getBounds() returns the rectangle covered by a screen object created using Screen(id).
The monitor, that is the first one based on hardware mapping (e.g. the laptop monitor), will always
be Screen(0). In the Windows settings it is possible to place the taskbar on one of the secondary
monitors, which makes it the primary monitor getting the base coordinates (0,0). The other
available monitors are mapped around based on your settings. But the Sikuli internal mapping is
not changed, so the primary monitor might be any of your Screen() objects. Sikuli takes care for
that and maps SCREEN always to the primary monitor (the one with the (0,0) coordinates). So for
example you have a laptop with an external monitor, that shows the taskbar (is primary monitor):
SCREEN maps to Screen(1)
Screen(0) is your laptop monitor
With its rectangle, a screen object is always identical with the monitor it was created using
Screen(id). Using Region.setROI() to restrict the region of interest for find operations has no effect
on the base rectangle of the screen object.
On the other hand region objects and location objects can be positioned anywhere in the
coordinate system. Only when a find operation or a click action has to be performed, the objects
rectangle or point has to be inside the rectangle of an existing monitor (basically repersented by
Sikuli Satish Gorripotu
Screen(0), Screen(1), ...). When this condition is met, everything works as expected and known
from a single monitor system.
With finding and acting there are the following exceptions:
Point Outside: a click point is outside any monitor rectangle. The clickpoint will be mapped to the
edges or corners of the primary monitor according to the relative position:
to the edges if its x or y value is in the range of the respective edge (right, left, above, below)i
to the corners, if x and y are outside any range of any edge (left/above -> upper left corner, ...)
Region Outside: a region is completely outside any monitor
a click action is handled in the same way as Point Outside
a find operation will always fail
Region Partially Outside: a region is partially outside a monitor but not overlapping another
monitor
a click action is handled in the same way as Point Outside
a find operation will be carried out only in the part of region within the bounds of the monitor,
excluding the area outside the monitor.
Region Across Monitors: a region lies across multiple monitors:
a click action is handled in the same way as Point Outside
a find operation will be restricted to the region within the bounds of the monitor that has a smaller
id.
An interactive capture (the user is asked to select an image or a rectangle via Screen.capture() or
Screen.selectRegion()) will automatically be restricted to the monitor, where it was started.
A scripted capture using a rectangle or a region (i.e. Screen.capture( region | rectangle )), will be
handled accordingly:
Region Outside: no image is captured, None is returned
Region Partially Outside: the returned image will only cover the part inside the monitor
Region Across Monitors: the returned image will only cover the part inside the monitor with the
smallest id.
Location
Location is a class available in Sikuli Package.
Handle single points on the screen directly by its position (x, y).
This class has several methods continuation to above methods.
Class Location
{
Location(x, y)
Returns a new location object representing the position (x,y) on the screen
Parameters: x – x position
y – y position
getX()
getY()
Get the x or y value of a location object
Sikuli Satish Gorripotu
It is possible to get the values directly by location.x or location.y.
It is also possible to set these values directly by location.x = value or location.y = value.
setLocation(x, y)
Set the location of this object to the specified coordinates.
offset(dx, dy)
Get a new location which is dx and dy pixels away horizontally and vertically from the
current location.
above(dy)
Get a new location which is dy pixels vertically above the current location.
below(dy)
Get a new location which is dy pixels vertically below the current location.
left(dx)
Get a new location which is dx pixels horizontally to the left of the current location.
right(dx)
Get a new location which is dx pixels horizontally to the right of the current location.
}
Match
Match is a class available in Sikuli Package.
class Match extends class Region, all methods of class Region can be used with a match object.
An object of class Match represents the result of a successful find operation.
It has the rectangle dimension of the image, that was used to search.
It knows the point of its upper left corner on an existing monitor, where it was found.
This class has several methods continuation to above methods.
Class Match
{
Creating a Match, Getting Attributes
A match object is created as the result of an explicit find operation. It can be saved in a variable
for later use with actions like click().
# m is a reference to a match object, if found
m = find(Some Image)
print m # message area: Match[10,0 30x22] score=1.00, target=center
Sikuli Satish Gorripotu
# m is a reference to a match object, if found
m = find(Pattern(Some Image).similar(0.5).targetOffset(100,0))
print m # message area: Match[10,0 30x22] score=1.00, target=(105,11)
getScore()
Get the similarity score the image or pattern was found. The value is between 0 and 1.
getTarget()
Get the location object that will be used as the click point.
If no offset was specified by Pattern.targetOffset(),click point is the center of the matched region.
If an offset was given, the click point is the offset relative to the center.
Iterating over Matches after findAll()
find operation Region.findAll() returns an iterator object that can be used to fetch all found
matches as match objects one by one.
A reference to the iterator is stored in the respective region and can be accessed using
Region.getLastMatches().
Note: per definition, an iterator can be stepped through only once - it is empty afterwards
To save contained matches for later use, you can convert them to list.
findAll() # find all matches
mm = list(getLastMatches())
Example: using while: with default screen
findAll() # find all matches
mm = SCREEN.getLastMatches()
while mm.hasNext(): # loop as long there is a first and more matches
print "found: ", mm.next() # access the next match in the row
print mm.hasNext() # is False, because mm is empty now
print mm.next() # is None, because mm is empty now
print SCREEN.getLastMatches().hasNext() # is False also ;-)
Example: using with: with default screen
with findAll() as mm:
while mm.hasNext(): # loop as long there is a first and more matches
print "found: ", mm.next() # access the next match
# mm will be None afterwards (destroyed automatically)
Sikuli Satish Gorripotu
Pattern
Pattern is a class available in Sikuli Package.
A pattern is used, to associate an image file with additional attributes used in find operations and
when acting on a match object.
Normally when clicking on a match, the center pixel of the associated rectangle is used.
With a pattern object, we can define a different click point relative to the center using
targetOffset().
Class Pattern
{
Pattern(string)
This will initialize a new pattern object without any additional attributes.
As long as no pattern methods are used additionally, it is the same as just using the image file
name itself in the find operation
Parameters: string – a path to an image file
similar(similarity)
Return a new Pattern object containing the same attributes (image, click point) with the minimum
similarity set to the specified value.
Parameters: similarity – the minimum similarity to use in a find operation. The value should be
between 0 and 1.
exact()
Return a new Pattern object containing the same attributes (image, click point) with the minimum
similarity set to 1.0, which means exact match is required.
targetOffset(dx, dy)
Sikuli Satish Gorripotu
Return a new Pattern object containing the same attributes (image, similarity), but a different
definition for the click.
By default, the click point is the center of the found match.
By setting the target offset, it is possible to specify a click point other than the center. dx and dy
will be used to calculate the position relative to the center.
Parameters: dx – x offset from the center
dy – y offset from the center
getFilename()
Get the filename of the image as a string contained in the Pattern object.
getTargetOffset()
Get a Location object as the target offset of the Pattern object.
}
Finder
Finder is a class available in Sikuli Package.
A Finder object implements an iterator of matches and allows to search for a visual object in an
image file that you provide (e.g. a screenshot taken and saved in a file before). After setting up the
finder object and doing a find operation, you can iterate through the found matches if any.
Note:
1. per definition, an iterator can be stepped through only once - it is empty afterwards
2. it has to be destroyed using finder.destroy(), especially when used with for: or while:
3. When used in a with: construct, it is destroyed automatically
Compared with the region based find operation, no exception FindFailed is raised in case nothing is
found at all (use hasNext() to check). The finder object can be compared to what you get with
region.getLastMatches() when using findAll().
Note: With this version, there is no chance, to get the number of matches in advance. If you would
iterate through to count, afterwards your finder would be empty.
So in this case, you have to save your matches somehow
The workflow always is, that you first do a find operation and afterwards go through the matches
found.
After a complete iteration, the finder object is empty.
So we could start a new find operation again.
Class Finder
{
Finder(path-to-imagefile)
Create a new finder object.
Parameters: path-to-imagefile – filename to a source image to search within
find(path-to-imagefile[, similarity])
Sikuli Satish Gorripotu
Find a given image within a source image previously specified in the constructor of the finder
object.
Parameters:
path-to-imagefile – the target image to search for
similarity – the minimum similarity a match should have. If omitted, the default is used.
hasNext()
Check whether there are more matches available that satisfy the minimum similarity requirement.
next()
Get the next match object.
The returnd reference to a match object is no longer available in the finder object afterwards.
So if it is needed later on, it has to be saved to another variable.
Example 1: basic operations using a Finder
# create a Finder with your saved screenshot
f = Finder()
img= # the image you are searching
f.find(img) # find all matches
while f.hasNext(): # loop as long there is a first and more matches
print "found: ", f.next() # access the next match in the row
print f.hasNext() # is False, because f is empty now
f.destroy() # release the memory used by finder
Example 2: we want to know how many matches (based on the previous example).
# create a Finder with your saved screenshot
f = Finder()
img= # the image you are searching
f.find(img) # find all matches
mm = [] # an empty list
while f.hasNext(): # loop as long there is a first and more matches
mm.append(f.next()) # access next match and add to mm
print f.hasNext() # is False, because f is empty now
f.destroy() # release the memory used by finder
# now we have our matches saved in the list mm
print len(mm) # the number of matches
# we want to use our matches
for m in mm:
print m
}
Sikuli Satish Gorripotu
Key Constants
class Key
{
Applicable usage situations for these predefined constants of special keys and key modifiers can be
found in Acting on a Region and Low Level Mouse and Keyboard Actions.
Special Keys
The methods supporting the use of special keys are
type(),
keyDown(), and
keyUp().
Key.CONSTANT (where CONSTANT is one of the following key names).
String concatenation with other text or other key constants is possible using ―+‖.
type("some text" + Key.TAB + "more text" + Key.TAB + Key.ENTER)
# or eqivalent
type("some text\tmore text\n")
miscellanous keys
ENTER, TAB, ESC, BACKSPACE, DELETE, INSERT,SPACE
function keys
F1, F2, F3, F4, F5, F6, F7, F8, F9, F10, F11, F12, F13, F14, F15
navigation keys
HOME, END, LEFT, RIGHT, DOWN, UP, PAGE_DOWN, PAGE_UP
special keys
PRINTSCREEN, PAUSE, CAPS_LOCK, SCROLL_LOCK, NUM_LOCK
Note: The status ( on / off ) of the keys Key.CAPS_LOCK, Key.NUM_LOCK and Key.SCROLL_LOCK
can be evaluated with the method Env.isLockOn().
numpad keys
NUM0, NUM1, NUM2, NUM3, NUM4, NUM5, NUM6, NUM7, NUM8, NUM9
SEPARATOR, ADD, MINUS, MULTIPLY, DIVIDE
modifier keys
ALT, CMD, CTRL, META, SHIFT, WIN
Sikuli Satish Gorripotu
These modifier keys cannot be used as a key modifier with functions like type(), rightClick(), etc.
They can only be used with keyDown() and keyUp(). If you need key modifiers, use KeyModifier
instead.
Key Modifiers
Methods where key modifiers can be used include:
click(),
dragDrop() ,
doubleClick() ,
rightClick(),
type().
system specific Win
KEY_WIN, KEY_CMD
KEY_META (a synonym for KEY_WIN or KEY_CMD on Windows).
The old modifiers with a KEY_ prefix are deprecated.
Use KeyModifier.CTRL, KeyModifier.ALT, KeyModifier.SHIFT, KeyModifier.META instead.
KeyModifier
class KeyModifier
KeyModifier.CONSTANT (where CONSTANT is one of the following key names).
CTRL
equivalent to the old KEY_CTRL
SHIFT
equivalent to the old KEY_SHIFT
ALT
equivalent to the old KEY_ALT
META
equivalent to the old KEY_META
CMD
equivalent to the old KEY_CMD (and KEY_META)
WIN
equivalent to the old KEY_WIN (and KEY_META)
The modifier constants can be combined to the modifier parameter by either using ―+‖ or ―|‖, if
more than one key modifier is needed.
type(Key.ESC, KeyModifier.CTRL + KeyModifier.ALT)
# or equivalent
type(Key.ESC, KeyModifier.CTRL | KeyModifier.ALT)
They should only be used in the modifiers parameter with functions like type(), rightClick(), etc.
They should never be used with keyDown() or keyUp().
Constants related to Mouse and Keyboard Actions
Sikuli Satish Gorripotu
Mouse related
Class Button : Button.LEFT - Button.MIDDLE - Button.RIGHT
related to mouse wheel actions: WHEEL_DOWN - WHEEL_UP
Keyboard related
Key Modifiers
KEY_ALT - KEY_CMD - KEY_CTRL - KEY_META - KEY_SHIFT - KEY_WIN
A combination of these keys has to be set up using "+" or "|",
e.g. KEY_SHIFT + KEY_ALT or KEY_SHIFT | KEY_ALT | KEY_CMD (used with action methods)
Special Keys
To be used with type() or keyDown()/keyUp():
ADD - ALT - BACKSPACE - CAPS_LOCK - DIVIDE - DOWN - END - ENTER - ESC - DELETE - F1 - F2
- F3 - F4 - F5 - F6 - F7 - F8 - F9 - F10 - F11 - F12 - F13 - F14 - F15 - HOME - INSERT- LEFT -
MINUS - MULTIPLY - NUM_LOCK - NUM0 - NUM1 - NUM2 - NUM3 - NUM4 - NUM5 - NUM6 - NUM7 -
NUM8 - NUM9 - PAGE_DOWN - PAGE_UP - PAUSE - PRINTSCREEN - RIGHT - SCROLL_LOCK -
SEPARATOR - TAB – UP
To be used with keyDown()/keyUp() only:
CMD - CTRL - META - SHIFT - TAB- WIN
To use these constants, write e.g. Key.ENTER. String concatenation with "+" is possible,
e.g. "some text" + Key.ENTER
Constants
FOREVER: can be used with exists - observe - wait - waitVanish
SCREEN : a constant reference to a screen object created using Screen(0) at startup,
The default/primary monitor used as the default region for methods not qualified by
an instance/object/class.
Extensions
Download an Extension
The download of an extension is supported by the IDE through the menu Tools -> Extensions.
Sikuli Satish Gorripotu
We get a popup, that lists the available and already installed extensions and allows to download
new packages or updates for installed ones.
This popup shows a new package not yet installed:
Use an Extension:
To use the features of an installed extension in one of your scripts, just say from
extension-name import *.
Technical Details
Extensions are Java JAR files containing some Java classes (usually the core functions) and/or
Python modules, which define the API to be used in a script.
Sikuli maintains a local extensions directory, where downloaded extensions are stored together
with a hidden list of the installed extensions %APPDATA%\Sikuli\extensions
Once an extension is imported using import extension-name, Sikuli automatically searches and
loads the JAR file of that extension into the current context with load(path-to-jar-file).
Develop an extension
The source structure of an extension named extension-name looks like this:
Java
- org/com
-- your-organization-or-company
--- extension-name
---- yourClass1.java
---- yourClass2.java
---- .... more classes
python
- extension-name
-- __init__.py
-- extension-name.py
The final structure of a JAR (filename extension-name-X.Y where X.Y is the version string) looks
like this:
org/com
- your-organization-or-company
-- extension-name
--- yourClass1.class
--- yourClass2.class
--- .... more classes
extension-name
- __init__.py
Sikuli Satish Gorripotu
- extension-name.py
META-INF
- MANIFEST.MFThe file __init__.py contains at least from extension-name import * to avoid one
qualification level. So in a script you might either use:
import extension-name
extension-name.functionXYZ()
or:
from extension-name import *
functionXYZ()
The second case requires more investement in a naming convention, that avoids naming conflicts.
The file extension-name.py contains the classes and methods, that represent the API,
that one might use in a Sikuli script.
Name the extensions
Sikuli extensions can be Python/Jython modules or Java classes.
For Java classes, following the reverse URL convention of Java is a good idea (for example,
org.foo.your-extension).
However, DO NOT use Java‘s convention for Python/Jython modules.
You need to come up with a unique extension name that does not conflict with existing Python
modules and other Sikuli extensions.
Test the extension
While developing your extensions, you can put the JAR file in Sikuli‘s extension directory or in the
same .sikuli folder as your test script.
The JAR file should not have a version number in its file name, e.g. extension-name.jar.
Because Sikuli starts to search extensions in the .sikuli folder of the running script and then in the
Sikuli extensions folder, it is usually a good idea to put your developing extensions in the .sikuli
folder of your test script.
Another option is to use the load() function with an absolute path to your extension-name.jar.
If this fails, Sikuli goes on searching in the current .sikuli folder and then in the Sikuli extensions
folder.
If load() succeeds, it returns True and puts absolute-path-to-your-extension-name.jar into
sys.path, so you can use import extension-name afterwards.
Guide:
Sikuli Satish Gorripotu
Example:
from guide import *
text(<Some Image on the screen>, "This is Sikuli's logo")
show(5)
Adding Multiple Annotations
The script uses the function tooltip(text) to add tooltips to three links in addition to the text
annotation
from guide import *
text(, "This is Sikuli's logo")
tooltip(,"Previous")
tooltip(,"Next")
tooltip(,"Index")
show(5)
from guide import *
Sikuli Satish Gorripotu
while True:
tooltip(,"Previous")
show(3)
tooltip(,"Next")
show(3)
tooltip(,"Index")
show(3)
Adding Interaction
from guide import *
text(,"Use these to jump to other parts")
dialog("Part 1: Navigation Links")
show()
text(,"Use this to go back to Home")
dialog("Part 2: Logo")
show()
Making a Region clickable
from guide import *
logo = find()
text(logo, "To proceed click this red ...")
clickable(logo)
index = logo.above().right().find()
text(index, "... or click this red")
clickable(index)
show()
Static Annotations
guide.rectangle(PSRM)
Add a rectangular overlay in red on the specified target‘s region.
Parameters: PSRM – a pattern, string, region or match
guide.circle(PSRM)
Add a red circle around the specified target‘s region.
Parameters: PSRM – a pattern, string, region or match
guide.text(PSRM, text)
Add some text (white large letters on dark grey background) left justified below the specified
target‘s region, which is additionally highlighted.
Parameters: PSRM – a pattern, string, region or match
text – a string as text to display
Sikuli Satish Gorripotu
guide.tooltip(PSRM, text)
Add a tooltip (small text in a light yellow box) left justified below the specified target‘s region.
Parameters: PSRM – a pattern, string, region or match
text – a string as text to display
Interactive Elements
guide.dialog(text)
Add a dialog box displaying the given text in the middle of the screen above all othe windows.
Parameters: text – a string as text to display
guide.clickable(PSRM)
Add a clickable element corresponding to the specified target‘s region.
Parameters: PSRM – a pattern, string, region or match
Control
guide.show([seconds])
Show static and interactive components added so far for the specified amount of time.
Parameters: seconds – a decimal number as display duration in seconds
The default duration is 10 seconds. If interactive elements (either one or more clickable elements
or a dialog box) were previously added, it waits until the user interacts with one of these elements.
At this time all elements vanish and are discarded.
Note: If a dialog() element is present, other interactive elements are visible, but not clickable. If
the dialog element is clicked, all elements vanish and are discarded.
Summary of Class and Methods
Region
Methods of Region
above - below - click - drag - dragDrop - dropAt - doubleClick - exists - find - findAll - getCenter -
getH - getLastMatch - getLastMatches - getRect - getROI - getScreen - getW - getX - getY -
highlight - hover - inside - keyDown - keyUp - left - mouseDown - mouseMove - mouseUp - nearby
- observe - onAppear - onVanish - onChange - paste - right - rightClick - selectRegion -
setAutoWaitTimeout - setH - setRect - setROI - setThrowException - setW - setX - setY -
stopObserver - text - type - wait – waitVanish
Creating a Region, Setting and Getting Attributes
getCenter - getH - getLastMatch - getLastMatches - getRect - getROI - getScreen - getW - getX -
getY - selectRegion - setAutoWaitTimeout - setH - setRect - setROI - setW - setX - setY
Extending a Region
above - below - inside - left - nearby - right
Finding inside a Region and Waiting for a Visual Event
exists - find - findAll - wait - waitVanish
Observing Visual Events in a Region
Sikuli Satish Gorripotu
observe - onAppear - onVanish - onChange - stopObserver
Acting on a Region
click - drag - dragDrop - dropAt - doubleClick - highlight - hover - paste - rightClick - type
Extracting Text from a Region
text
Low Level Mouse and Keyboard Actions
getMouseLocation - keyDown - keyUpmouseDown - mouseMove - mouseUp - wheel
Screen
click - drag - dragDrop - dropAt - doubleClick - exists - find - findAll - getCenter - getH -
getLastMatch - getLastMatches - getROI - getW - getX - getY - hover - keyDown - keyUp -
mouseDown - mouseMove - mouseUp - observe - onAppear - onVanish - onChange - paste -
rightClick - setAutoWaitTimeout - setH - setROI - setThrowException - setW - setX - setY -
stopObserver - text - type - wait – waitVanish
capture - getBounds - getNumberScreens - selectRegion
Setting, Getting Attributes and Information
getBounds - getNumberScreens
Screen as (Default) Region
Capturing
capture - selectRegion
Location
above - below - getsetL - getsetL - left - offset - right - getsetL - getsetL
Match
above - below - click - drag - dragDrop - dropAt - doubleClick - exists - find - findAll - getCenter -
getH - getLastMatch - getLastMatches - getScreen - getW - getX - getY - hover - inside - left -
mouseMove - nearby - observe - onAppear - onVanish - onChange - paste - right - rightClick -
selectRegion - setAutoWaitTimeout - setThrowException - stopObserver - type - wait – waitVanish
Sikuli Satish Gorripotu
getScore - getTarget
Finder
find - hasNext - next
Pattern
exact - getFilename - getTargetOffset - similar - targetOffset
VDict
vdict[ ] - vdict[ ] = - [ not in vdict] - del vdict[ ] - get -get1 - get_exact - len( ) - keys
Env
getOS() returns information about the system, the script is running on ( -> General Information
and Settings )
getOSVersion() returns the system's version number, the script is running on ( -> General
Information and Settings )
getMouseLocation() returns the current location of the mouse pointer ( -> Low Level Mouse and
Keyboard Actions )
getClipboard() returns the content of the Clipboard if it is text, otherwise an empty string. ( ->
General Information and Settings )
Sikuli Satish Gorripotu
Sikuli – Class and methods
Class Method Parameter Return value Usage 1 Usage 2
Region
Region(x, y, w, h) Create a region object
Region(region) Create a region object
Region(Rectangle) Create a region object
setX(number)
setY(number)
setW(number)
setH(number)
moveTo(location)
setROI(x, y, w, h)
setROI(rectangle)
setRect(x, y, w, h)
setRect(rectangle)
morphTo(region)
getX()
getY()
getW()
getH()
getCenter()
getTopLeft()
getTopRight()
getBottomLeft()
getBottomRight()
getScreen()
getLastMatch()
getLastMatches()
setAutoWaitTimeout(seconds)
setAutoWaitTimeout(seconds)
getAutoWaitTimeout()
offset(location)
inside()
Sikuli Satish Gorripotu
nearby([range])
above([range])
below([range])
left([range])
right([range])
find(PS)
findAll(PS)
wait([PS][, seconds])
waitVanish(PS[, seconds])
exists(PS[, seconds])
onAppear(PS, handler)
onVanish(PS, handler)
onChange([minChangedSize], handler)
observe([seconds][, background = False | True])
stopObserver()
click(PSMRL[, modifiers])
doubleClick(PSMRL[, modifiers])
rightClick(PSMRL[, modifiers])
highlight([seconds])
hover(PSMRL)
dragDrop(PSMRL, PSMRL[, modifiers])
drag(PSMRL)
dropAt(PSMRL[, delay])
type([PSMRL], text[, modifiers])
paste([PSMRL], text)
text()
mouseDown(button)
mouseUp([button])
mouseMove(PSRML)
wheel(PSRML, WHEEL_DOWN | WHEEL_UP, steps)
keyDown(key | list-of-keys)
keyUp([key | list-of-keys])
setFindFailedResponse(ABORT | SKIP | PROMPT | RETRY)
getFindFailedResponse()
setThrowException(False | True)
getThrowException()
getRegionFromPSRM(PSRM)
getLocationFromPSRML(PSRML)
SikuliEvent
type
pattern
region
match
changes
Sikuli Satish Gorripotu
Screen
Screen([id])
getNumberScreens()
getBounds()
capture([region | rectangle | text])
capture(x, y, w, h)
selectRegion([text])
Location
Location(x, y)
getX()
getY()
setLocation(x, y)
offset(dx, dy)
above(dy)
below(dy)
left(dx)
right(dx)
Match
getScore()
getTarget()
Pattern
Pattern(string)
similar(similarity)
exact()
targetOffset(dx, dy)
getFilename()
getTargetOffset()
Finder
Finder(path-to-imagefile)
find(path-to-imagefile[, similarity])
hasNext()
next()
Button
left
middle
right
App (Application)
Env (Environment)
Key (Key Constants)
VDict (Visual Dictionary)
Sikuli consolidated Methods:
Sikuli Satish Gorripotu
S.No Methods Examples Page Numbers
command line option
--args <arguments>
-h,--help
"-I"
-r,--run <sikuli-file>
-s,--stderr
-t,--test <sikuli-file>
A
1 above() (Location method)
2 Above (Region method)
3 ActionLogs (Settings attribute)
4 addHotkey() (Env method)
5 addImagePath() (built-in function)
6 ALT (built-in variable)
7 App (built-in class), [1]
B
8 below() (Location method)
9 below (Region method)
10 Bundle Path
C
11 capture() (Screen method)
12 changes (SikuliEvent attribute)
13 circle() (in module guide)
14 click() (Region method) Done
15 clickable() (in module guide)
16 close() (App class method)
17 Close (App method)
18 closeApp() (built-in function)
19 CMD (built-in variable)
20 --args <arguments>
21 -h,--help
22 "-I"
23 -r,--run <sikuli-file>
24 -s,--stderr
25 -t,--test <sikuli-file>
26 CTRL (built-in variable)
D
27 DebugLogs (Settings attribute)
28 DelayAfterDrag (Settings attribute)
Sikuli Satish Gorripotu
29 DelayBeforeDrop (Settings attribute)
30 dialog() (in module guide)
31 doubleClick() (Region method) Done
32 drag() (Region method)
33 dragDrop() (Region method) Done
34 dropAt() (Region method)
destroy
E
35 exact() (Pattern method)
36 exists() (Region method)
37 exit() (built-in function)
F
38 find() (Finder method) Done
39 find (Region method)
40 findAll() (Region method) Done
41 Finder (built-in class), [1]
42 Finder() (Finder method)
43 focus() (App class method)
44 focus (App method)
45 focusedWindow() (App class method)
G
46 getAutoWaitTimeout() (Region method)
47 getBottomLeft() (Region method)
48 getBottomRight() (Region method)
49 getBounds() (Screen method)
50 getBundlePath() (built-in function)
51 getCenter() (Region method)
52 getClipboard() (Env method)
53 getFilename() (Pattern method)
54 getFindFailedResponse() (Region method)
55 getH() (Region method)
56 getImagePath() (built-in function)
57 getLastMatch() (Region method)
58 getLastMatches() (Region method)
59 getLocationFromPSRML() (Region method)
60 getMouseLocation() (Env method)
61 getNumberScreens() (Screen method)
62 getOS() (Env method)
63 getOSVersion() (Env method)
64 getParameter() (Vision method)
65 getRegionFromPSRM() (Region method)
66 getScore() (Match method)
67 getScreen() (Region method)
Sikuli Satish Gorripotu
68 getSikuliVersion() (Env method)
69 getTarget() (Match method)
70 getTargetOffset() (Pattern method)
71 getThrowException() (Region method)
72 getTopLeft() (Region method)
73 getTopRight() (Region method)
74 getW() (Region method)
75 getX() (Location method)
76 getX(Region method)
77 getY() (Location method)
78 getY(Region method)
79 guide (module)
getRect
getROI
H
80 hasNext() (Finder method)
81 highlight() (Region method)
82 hover() (Region method)
I
83 Image Search Path
84 SIKULI_IMAGE_PATH
85 import .sikuli
86 InfoLogs (Settings attribute)
87 input() (built-in function) Done
88 inside() (Region method)
89 isLockOn() (Env method)
K
90 Key (built-in class)
91 keyDown() (Region method)
92 KeyModifier (built-in class)
93 keyUp() (Region method)
L
94 left() (Location method)
95 Left (Region method)
96 load() (built-in function)
97 Location (built-in class)
98 Location() (Location method)
M
99 Match (built-in class), [1]
100 match (SikuliEvent attribute)
101 META (built-in variable)
102 MinSimilarity (Settings attribute)
Sikuli Satish Gorripotu
103 morphTo() (Region method)
104 mouseDown() (Region method)
105 mouseMove() (Region method)
106 mouseUp() (Region method)
107 MoveMouseDelay (Settings attribute)
108 moveTo() (Region method)
N
109 nearby() (Region method)
110 next() (Finder method)
O
111 observe() (Region method)
112 ObserveMinChangedPixels (Settings attribute)
113 ObserveScanRate (Settings attribute)
114 offset() (Location method)
115 offset(Region method)
116 onAppear() (Region method)
117 onChange() (Region method)
118 onVanish() (Region method)
119 open() (App class method)
120 open(App method)
121 openApp() (built-in function)
P
122 paste() (Region method)
123 Pattern (built-in class), [1]
124 pattern (SikuliEvent attribute)
125 Pattern() (Pattern method)
126 popup() (built-in function) Done
R
127 rectangle() (in module guide)
128
Region (built-in class), [1], [2], [3], [4], [5], [6], [7],
[8], [9]
129 region (SikuliEvent attribute)
130 Region() (Region method)
131 removeHotkey() (Env method)
132 removeImagePath() (built-in function)
133 right() (Location method)
134 right(Region method)
135 rightClick() (Region method) Done
136 run() (built-in function)
S
137 Screen (built-in class), [1], [2]
138 Screen() (Screen method)
139 selectRegion() (Screen method)
Sikuli Satish Gorripotu
140 setAutoWaitTimeout() (Region method)
141 setBundlePath() (built-in function)
142 setFindFailedResponse() (Region method)
143 setH() (Region method)
144 setLocation() (Location method)
145 setParameter() (Vision method)
146 setRect() (Region method)
147 setROI() (Region method)
148 setShowActions() (built-in function)
149 setThrowException() (Region method)
150 Settings (built-in class)
151 setW() (Region method)
152 setX() (Region method)
153 setY() (Region method)
154 SHIFT (built-in variable)
155 show() (in module guide)
156 SIKULI_IMAGE_PATH
157 Image Search Path
158 SikuliEvent (built-in class)
159 similar() (Pattern method)
160 SlowMotionDelay (Settings attribute)
161 stopObserver() (Region method)
162 switchApp() (built-in function)
setX (Location)
setY (Location)
T
163 targetOffset() (Pattern method)
164 text() (in module guide)
165 text(Region method)
166 tooltip() (in module guide)
167 type (SikuliEvent attribute)
168 type() (Region method) Done
W
169 wait() (Region method) Done
170 WaitScanRate (Settings attribute)
171 waitVanish() (Region method)
172 wheel() (Region method)
173 WIN (built-in variable)
174 window() (App method)
Sikuli Satish Gorripotu
Java Docs
http://sikuli.org/doc/java-x/
S.No All Classes
1 Animator
2 App
3 AppearEvent
4 AppNotFound
5 Button
6 CapturePrompt
7 ChangeEvent
8 Clipboard
9 Clipboard.Charset
10 Clipboard.TextType
11 Clipboard.TransferType
12 Constants
13 Debug
14 DesktopRobot
15 DummyOSUtil
16 Env
17 EventManager
18 ExtensionManager
19 Finder
20 FindFailed
21 FindFailedResponse
22 HotkeyEvent
23 HotkeyListener
24 ImageLocator
25 IRobot
Sikuli Satish Gorripotu
26 IRobot.KeyMode
27 IScreen
28 Key
29 KeyModifier
30 LinuxUtil
31 Location
32 MacUtil
33 Match
34 Observer
35 OpenCV
36 OS
37 OSUtil
38 Pattern
39 Region
40 ResourceExtractor
41 Screen
42 ScreenHighlighter
43 ScreenImage
44 ScriptRunner
45 Settings
46 SikuliAction
47 SikuliActionListener
48 SikuliActionManager
49 SikuliEvent
50 SikuliEvent.Type
51 SikuliEventAdapter
52 SikuliEventObserver
53 SikuliException
54 SikuliScript
55 Subject
56 TextRecognizer
57 TextRecognizer.ListTextMode
58 TransparentWindow
59 UnionScreen
60 Util
61 VanishEvent
62 VDictProxy
63 Win32Util
Sikuli Satish Gorripotu
Basics of Jython
– Python for java Platform
Sikuli Using IDE as Eclipse then, we have two choices.
1. Using Java as a scripting language
2. Using Python as scripting language by downloading Python or Jython plugIn to Eclipse.
If we use default Sikuli Ide then we have to use Python or Jython as scripting language.
http://jythonpodcast.hostjava.net/jythonbook/en/1.0/
Declare Variable
Java –
int x = 0;
Python
x = 0
Reserved Words
and assert break class continue def del elif else except exec finally for from
global or pass print raise return try while with yield
Coding Structure
Java if-statement
x = 100;
if(x > 0)
{
System.out.println("Wow, this is Java");
}
else
{
System.out.println("Java likes curly braces");
}
Python if-statement
x = 100
if x > 0:
print 'Wow, this is elegant'
else:
print 'Organization is the key'
Sikuli Satish Gorripotu
Functions
Example 1:
def my_function_name(parameter_list):
implementation
Example 2:
def my_simple_function():
... print 'This is a really basic function'
my_simple_function()
Example 3:
def multiply_nums(x, y):
... return x * y
multiply_nums(25, 7)
175
Example 4:
def perform_math(oper):
... return oper(5, 6)
perform_math(multiply_nums)
30
Classes
Java - this
Python - self
>>> class my_object:
... def __init__(self, x, y):
... self.x = x
... self.y = y
...
... def mult(self):
... print self.x * self.y
...
... def add(self):
... print self.x + self.y
...
Sikuli Satish Gorripotu
>>> obj1 = my_object(7, 8)
>>> obj1.mult()
56
>>> obj1.add()
15
Statements
Statement Keywords
if-elif-else for while continue break try-except-finally assert def print del raise import
if-elif-else Statement
if <an expression to test>:
perform an action
else:
perform a different action
>>> x = 3
>>> y = 2
>>> if x == y:
... print 'x is equal to y'
... elif x > y:
... print 'x is greater than y'
... else:
... print 'x is less than y'
...
x is greater than y
print Statement
Java Print Output Example
System.out.println("This text will be printed to the command line");
Python Print Output Example
print 'This text will be printed to the command line'
>>> my_value = 'Satish Gorripotu'
>>> print my_value
Satish Gorripotu
>>> print 'My Name is Satish, but people called me as ' + my_value
My Name is Satish, but people called me as Satish Gorripotu
>>> z = 10
>>> print 'I am a fan of the number: ' + z
Traceback (most recent call last): File "<stdin>", line 1, in <module>
TypeError: cannot concatenate 'str' and 'int' objects
Sikuli Satish Gorripotu
To resolve, we have to use below formatting operators
* %s - String
* %d - Decimal
* %f - Float
print 'String of text goes here %d %s %f' % (decimalValue, stringValue, floatValue)
>>> string_value = 'hello world'
>>> float_value = 3.998
>>> decimal_value = 5
>>> print 'Here is a test of the print statement using the values: %d, %s, and %f' %
(decimal_value, string_value, float_value)
Here is a test of the print statement using the values: 5, hello world, and 3.998000
>>> x = 1
>>> y = 2
>>> print 'The value of x + y is: %d' % (x + y)
The value of x + y is: 3
>>> x = 2.3456
>>> print '%s' % x
2.3456
>>> print '%d' % x
2
>>> print '%f' % x
2.345600
try-except-finally
>>> # Suppose we've calculated a value and assigned it to x
>>> x
8.97
>>> y = 0
>>> try:
... print 'Just Print: %f' % (x/y)
... except:
... print ‗Satti, we have a problem....
Satti, we have a problem.
raise Statement
the raise statement is used to throw or ―raise‖ an exception in Python
>>> raise NameError
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError
Sikuli Satish Gorripotu
>>> raise Exception('Custom Exception')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception: Custom Exception
import Statement
# Import a module named TipCalculator
import TipCalculator
# Import a function tipCalculator from within a module called ExternalModule.py
from ExternalModule import tipCalculator
To avode over riding
import tipCalculator as tip
Iteration
Index start from zero
>>> my_numbers = [1, 2, 3, 4, 5]
>>> my_numbers
[1, 2, 3, 4, 5]
>>> my_numbers[1]
2
>>> for value in my_numbers:
... print value
...
1
2
3
4
5
>>> x = 0
>>> while x < len(my_numbers):
... print my_numbers[x]
... x = x + 1
...
1
2
3
4
5
While Loop
Sikuli Satish Gorripotu
while True
perform operation
Example of a Java While Loop
int x = 9;
int y = 2;
int z = x – y;
while (y < x)
{
System.out.println("y is " + z + " less than x");
y = y++;
}
Example of a Python While Loop
>>> x = 9
>>> y = 2
>>> while y < x:
... print 'y is %d less than x' % (x-y)
... y += 1
...
y is 7 less than x
y is 6 less than x
y is 5 less than x
y is 4 less than x
y is 3 less than x
y is 2 less than x
y is 1 less than x
For Loop
for each value in this defined set:
perform suite of operations
Example of Java For Loop
for (x = 0; x <= 6; x++){
System.out.println(x);
}
Example of Python For Loop
>>> for x in range(10):
... print x
...
0
1
2
3
4
5
Sikuli Satish Gorripotu
Basic Keyboard Input
1. raw_input()
2. input()
# The text within the function is optional, and it is used as a prompt to the user
>>> name = raw_input("Enter Your Name:")
Enter Your Name:satish
>>> print name
satish
# Use the input function to evaluate an expression entered in by the user
>>> val = input ('Please provide an expression: ')
Please provide an expression: 9 * 3
>>> val
27
# The input function raises an error if an expression is not provided
>>> val = input ('Please provide an expression: ')
Please provide an expression: My Name is Satti
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1
My Name is Satti
^
SyntaxError: invalid syntax
String Methods
Method Description of Functionality
capitalize() Returns a capitalized copy of string
center (width[,fill])
Returns a repositioned string with specified width and provide
optional padding filler character
count(sub[,start[,end]])
Count the number of distinct times the substring occurs within the
string
decode([encoding[,errors]]) Decodes and returns Unicode string
encode([encoding[,errors]]) Returns an encoded version of a string
endswith(suffix[,start[,end]]) Returns a boolean to state whether the string ends in a given pattern
expandtabs([tabsize]) Converts tabs within a string into spaces
find(sub[,start[,end]])
Returns the index of the position where the first occurrence of the
given substring begins
index(sub[,start[,end])
Returns the index of the position where the first occurrence of the
given substring begins. Raises a ValueError with the substring is not
found.
isalnum()
Returns a boolean to state whether the string contain only alphabetic
and numeric characters
isalpha()
Returns a boolean to state whether the string contains all alphabetic
characters
isdigit()
Returns a boolean to state whether the string contains all numeric
characters
islower()
Returns a boolean to state whether a string contains all lowercase
characters
Sikuli Satish Gorripotu
isspace()
Returns a boolean to state whether the string consists of all
whitespace
istitle()
Returns a boolean to state whether the first character of each word in
the string is capitalized
isupper()
Returns a boolean to state whether all characters within the string are
uppercase
join(sequence)
Returns a copy of sequence joined together with the original string
placed between each element
ljust(width[,fillchar])
Returns a string of the specified width along with a copy of the
original string at the leftmost bit. (Optionally padding empty space
with fillchar)
lower()
Returns a copy of the original string with all characters in the string
converted to lowercase
lstrip([chars])
Removes the first found characters in the string from the left that
match the given characters. Also removes whitespace from the left.
Whitespace removal is default when specified with no arguments.
partition(separator)
Returns a partitioned string starting from the left using the provided
separator
replace(old,new[,count])
Returns a copy of the original string replacing the portion of string
given in old with the portion given in new
rfind(sub[,start[,end]])
Searches string from right to left and finds the first occurrence of the
given string and returns highest index where sub is found
rindex(sub[,start[,end]])
Searches string from right to left and finds the first occurrence of the
given string and either returns highest index where sub is found or
raises an exception
rjust(width[,fillchar]) Returns copy of string Aligned to the right by width
rpartition(separator)
Returns a copy of stringPartitioned starting from the right using the
provided separator object
rsplit([separator[,maxsplit]])
Returns list of words in string and splits the string from the right side
and uses the given separator as a delimiter. If maxsplit is specified
then at most maxsplit splits are done (from the right).
rstrip([chars])
Returns copy of string removing the first found characters in the
string from the right that match those given. Also removes
whitespace from the right when no argument is specified.
split([separator[,maxsplit]])
Returns a list of words in string and splits the string from the left side
and uses the given separator as a delimiter.
splitlines([keepends])
Splits the string into a list of lines. Keepends denotes if newline
delimiters are removed. Returns the list of lines in the string.
startswith(prefix[,start[,end]])
Returns a boolean to state whether the string starts with the given
prefix
strip([chars])
Returns a copy of string with the given characters removed from the
string. If no argument is specified then whitespace is removed.
swapcase()
Returns a copy of the string the case of each character in the string
converted.
title()
Returns a copy of the string with the first character in each word
uppercase.
translate(table[,deletechars])
Returns a copy of the string using the given character translation
table to translate the string. All characters occurring in optional
deletechars argument are removed.
upper()
Returns a copy of string with all of the characters in the string
converted to uppercase
zfill(width)
Returns a numeric string padded from the left with zeros for the
specified width.
Sikuli Satish Gorripotu
Examples:
our_string='python is the best language ever'
# Capitalize first character of a String
>>> our_string.capitalize()
'Python is the best language ever'
# Center string
>>> our_string.center(50)
' python is the best language ever '
>>> our_string.center(50,'-')
'---------python is the best language ever---------'
# Count substring within a string
>>> our_string.count('a')
2
# Count occurrences of substrings
>>> state = 'Mississippi'
>>> state.count('ss')
2
# Partition a string returning a 3-tuple including the portion of string
# prior to separator, the separator
# and the portion of string after the separator
>>> x = "Hello, my name is Josh"
>>> x.partition('n')
('Hello, my ', 'n', 'ame is Josh')
# Assuming the same x as above, split the string using 'l' as the separator
>>> x.split('l')
['He', '', 'o, my name is Josh']
# As you can see, the tuple returned does not contain the separator value
# Now if we add maxsplits value of 1, you can see that the right-most split is
# taken. If we specify maxsplits value of 2, the two right-most splits are taken
Sikuli Satish Gorripotu
>>> x.split('l',1)
['He', 'lo, my name is Josh']
>>> x.split('l',2)
['He', '', 'o, my name is Josh']
Lists
Defining Lists
# Define an empty list
my_list = []
my_list = list() # rarely used
# Single Item List
>>> my_list = [1]
>>> my_list # note that there is no need to use print to display a variable
in the
>>> # interpreter
[1]
# Define a list of string values
my_string_list = ['Hello', 'Jython' ,'Lists']
# Define a list containing mulitple data types
multi_list = [1, 2, 'three', 4, 'five', 'six']
# Define a list containing a list
combo_list = [1, my_string_list, multi_list]
# Define a list containing a list inline
>>> my_new_list = ['new_item1', 'new_item2', [1, 2, 3, 4], 'new_item3']
>>> print my_new_list
['new_item1', 'new_item2', [1, 2, 3, 4], 'new_item3']
Sikuli Satish Gorripotu
Accessing a List
# Obtain elements in the list
>>> my_string_list[0]
'Hello'
>>> my_string_list[2]
'Lists'
# Negative indexes start with the last element in the list and work back towards the
first
# item
>>> my_string_list[-1]
'Lists'
>>> my_string_list[-2]
'Jython'
# Using slicing (Note that slice includes element at starting index and excludes the
end)
>>> my_string_list[0:2]
['Hello', 'Jython']
# Create a shallow copy of a list using slice
>>> my_string_list_copy = my_string_list[:]
>>> my_string_list_copy
['Hello', 'Jython', 'Lists']
# Return every other element in a list
>>> new_list=[2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
# Using a third parameter in the slice will cause a stepping action to take place
# In this example we step by one
>>> new_list[0:10:1]
Sikuli Satish Gorripotu
[2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
# And here we step by two
>>> new_list[0:10:2]
[2, 6, 10, 14, 18]
# Leaving a positional index blank will also work as the default is 0 for the start,
and the length of the string for the end.
>>> new_list[::2]
[2, 6, 10, 14, 18]
Modifying a List
# Modify an element in a list. In this case we'll modify the element in the 9th
position
>>> new_list[9] = 25
>>> new_list
[2, 4, 6, 8, 10, 12, 14, 16, 18, 25]
Boolean Expressions
Example: Testing a String
>>> mystr = ''
>>> if mystr:
... 'Now I contain the following: %s' % (mystr)
... else:
... 'I do not contain anything'
...
'I do not contain anything'
>>> mystr = 'Now I have a value'
>>> if mystr:
... 'Now I contain the following: %s' % (mystr)
... else:
... 'I do not contain anything'
...
'Now I contain the following: Now I have a value'
Using Expressions to Control Program Flow
if-elif-else Statement
Sikuli Satish Gorripotu
Example: Standard if-elif-else
# terminal symbols are left out of this example so that you can see the precise indentation
pi =3.14
x = 2.7 * 1.45
if x == pi:
print 'The number is pi'
elif x > pi:
print 'The number is greater than pi'
else:
print 'The number is less than pi'
Example: Evaluate Empty List
# Use an if-statement to determine whether a list is empty
# Suppose mylist is going to be a list of names
>>> mylist = []
>>> if mylist:
... for person in mylist:
... print person
... else:
... print 'The list is empty'
...
The list is empty
while Loop
Python while Statement
>>> x = 0
>>> y = 10
>>> while x <= y:
... print 'The current value of x is: %d' % (x)
... x += 1
... else:
... print 'Processing Complete...'
...
The current value of x is: 0
The current value of x is: 1
The current value of x is: 2
The current value of x is: 3
The current value of x is: 4
The current value of x is: 5
The current value of x is: 6
The current value of x is: 7
The current value of x is: 8
The current value of x is: 9
The current value of x is: 10
Processing Complete...
Sikuli Satish Gorripotu
esetting Counter Using with-else
>>> total = 0
>>> x = 0
>>> y = 20
>>> while x <= y:
... total += x
... x += 1
... else:
... print total
... total = 0
...
210
continue Statement
# Iterate over range and print out only the positive numbers
>>> x = 0
>>> while x < 10:
... x += 1
... if x % 2 != 0:
... continue
... print x
...
2
4
6
8
10
break Statement
>>> x = 10
>>> while True:
... if x == 0:
... print 'x is now equal to zero!'
... break
... if x % 2 == 0:
... print x
... x -= 1
...
10
8
6
4
2
x is now equal to zero!
for Loop
Sikuli Satish Gorripotu
Comparing Java and Python for-loop
Example of Java for-loop
for(x = 0; x <= myList.size(); x++){
// processing statements iterating through myList
System.out.println("The current index is: " + x);
}
Example of Python for-loop
my_list = [1,2,3,4,5]
>>> for value in my_list:
# processing statements using value as the current item in my_list
... print 'The current value is %s' % (value)
...
The current value is 1
The current value is 2
The current value is 3
The current value is 4
The current value is 5
Enumerate() Functionality
>>> myList = ['jython','java','python','jruby','groovy']
>>> for index, value in enumerate(myList):
... print index, value
...
0 jython
1 java
2 python
3 jruby
4 groovy
If we do not require the use of an index, it can be removed and the syntax can be cleaned up a bit.
>>> myList = ['jython', 'java', 'python', 'jruby', 'groovy']
>>> for item in myList:
... print item
...
jython
java
python
jruby
groovy
Exception Handling
- Need to work on it
JDBC
- Need to work on Data Base conncetions
Sikuli Satish Gorripotu
from __future__ import with_statement
from com.ziclix.python.sql import zxJDBC
# for example
jdbc_url = "jdbc:postgresql:test"
username = "postgres"
password = "jython25"
driver = "org.postgresql.Driver"
# obtain a connection using the with-statment
with zxJDBC.connect(jdbc_url, username, password, driver) as conn:
with conn:
with conn.cursor() as c:
c.execute("select name from country")
c.fetchone()
Excel Handling
- Need to work on Test tada and Test results
Referenced and consolidated from below sites:
http://sikuli.org/trac/wiki
http://sikuli.org/trac/wiki/TitleIndex
http://sikuli.org/trac/wiki/IPAddressMac-0.10?action=history
http://sikuli.org/trac/wiki/WikiStart
http://sikuli.org/trac/wiki/reference-0.10
http://sikuli.org/trac/wiki/RaiMansExamples
http://sikuli.org/trac/wiki/The%20Complete%20Guide%20To%20Sikuli%20X
http://sikuli.org/trac/wiki/A%20Quick%20Tour%20of%20Sikuli%20Script%20IDE?action=diff&ver
sion=7
http://sikuli.org/trac/prefs
http://sikuli.org/docx/
http://sikuli.org/documentation.shtml
http://script.sikuli.org/documentation.shtml#doc/pythondoc-
python.edu.mit.csail.uid.Sikuli.html#python.edu.mit.csail.uid.Sikuli.type-function
http://sikuli.csail.mit.edu/trac/wiki/TitleIndex
http://www.csail.mit.edu/search/node/sikuli
Sikuli Satish Gorripotu
http://www.makeuseof.com/tech-
fun/search/?cx=009717636731598800244%3Aqhe4rh7wuxs&cof=FORID%3A11&q=sikuli#773
https://launchpad.net/sikuli/trunk/0.10
http://jythonpodcast.hostjava.net/jythonbook/en/1.0/
http://www.jython.org/jythonbook/en/1.0/LangSyntax.html#while-loop
http://www.jython.org/jythonbook/en/1.0/LangSyntax.html#for-loop
http://www.jython.org/jythonbook/en/1.0/InputOutput.html#file-i-o
AutoHotkey : same like Sikuli, But takes Keyboard actions.
http://www.autohotkey.com/docs/Tutorial.htm
AutoHotkey supports image recognition and macro recording. I don't see much here that is new.
The bus proximity and lasso search examples are both great ideas, but could each be easily
implemented using either AutoHotkey or AutoIt.