SikuliX - general aspects of scripting ====================================== What is a SikuliX script ------------------------ A SikuliX script is a workflow, that tries to perform actions against the GUI seen on a real screen, that otherwise must be performed manually. Usually it is a series of waiting for images to get visible on the screen and actions (mouse and/or keyboard) on visible GUI elements. The workflow statements usually are complemented with other elements of the used scripting language to implement more complex solutions with decisions, repetitions and states. Example in Python language :: while True: # repeat the body forever wait("image1.png", 10) # wait max 10 seconds for image1 to get visible, otherwise die click("button1.png") # click the button given by image button1 if exists("image2.png"): # wait max 3 seconds for image2 to get visible, return True or False doSomething() # in case True execute function doSomething (defined somewhere else) else: break # in case False get out of the loop, which ends the workflow # start all over with the line after the while # the end of the workflow In the IDE as standard the "foobar.png" are shown as thumbnails of the images (other options available). To write, execute and debug SikuliX scripts you should know about the features of SikuliX and have at least a basic knowledge about the used scripting language. A good starting point is to use the SikuliX IDE, to get used to the SikuliX features. If things become more complex it might make sense, to switch to another IDE that supports your selected scripting language better. Where and how can a SikuliX script be stored -------------------------------------------- A SikuliX script consists of one file containing the statements of your scripted workflow and zero to many image files, that you have captured, to support your workflow. This bunch of related files is meant when we talk about a SikuliX script. How to run such a script is shown in the next chapter. The standard storage for a SikuliX script is a folder/directory having the name that you have given to the script, when saving it from the IDE. It is mandatory, that the contained scriptfile has the same name as the folder, which is guaranteed, when working with the SikuliX IDE. To avoid any problems, you should never directly access the folder content outside the SikuliX IDE, except you really know, what you are doing. On Mac the folder is seen as a package/bundle, which at the top level in Finder hides the content. The folder and its content for the above example (in IDE we named it myscript) :: myscript.sikuli button1.png image1.png image2.png myscript.py There are two other storage variants for SikuliX scripts: zip-file and jar-file, which are intended for distribution of your scripts. Both variants pack the SikuliX script content in one file, which makes distribution easier. Both variants can be run with the means of SikuliX, as long as the target system has a valid SikuliX installation/setup. The jar-file variant has an option to be packed together with the scripting and SikuliX feature support (self contained), so it might be run on systems only having a valid Java installation. - variant zip-file It can be created in the SikuliX IDE with the file menu entry "Export packed source" and gets the ending .skl instead of .sikuli. As such it cannot be reopened for editing nor used for import, it can only be run using SikuliX features on systems having a valid SikuliX setup. In fact it is simply a zip-file, which can be brought back to a .sikuli folder using a zip utility. So be aware, that this variant does not really hide your script content. - variant jar-file (only available in 1.1.2+) It can be created in the SikuliX IDE (file menu entry "Export as jar"). It can be run using SikuliX features on systems having a valid SikuliX setup. The script source is contained in compiled form (Java byte code) and hence not visible, even if one accesses the jar content. So this variant gives some level of secrecy for your scripted workflow. There is a sub-variant, containing the scripting and SikuliX feature support additionally to the script content, which makes it runnable even on systems not having a valid SikuliX setup, but only a valid Java installation. Be aware, that such a jar has at least 60 MB though. `How to make and use it look here `_.         How to run a SikuliX script --------------------------- In the SikuliX IDE a currently open script in the frontmost editor tab can be run by clicking the run button. Saved scripts (.sikuli) and exported scripts (.skl and .jar) can be run from commandline (:ref:`see `) or from within other scripts (:ref:`see `). Using JavaScript ================ One can use **JavaScript** for scripting with SikuliX features. The support for JavaScript is based on the **Java integrated engine Nashorn** (available since Java 7). **BE AWARE** Since Java 11 Nashorn is marked deprecated and might be removed in a future Java version. With Java 13 (available September 2019) it is still there and hence will be there till September 2020. A candidate to be used with SikuliX and JavaScript scripting in the future will be the GraalVM package. People who try already now with SikuliX 2.0.4+ are welcome to give feedback about their experiences on GitHub. The easiest approach for JavaScript scripting: - switch a new editor tab in the IDE to type JavaScript (rigt-click on an empty Tab, select ``Set Type`` in the context menü and then select ``javascript`` from the drop down list). - The bottom status line at the right side now will show ``(javascript)`` - now you can write JavaScript code according to the `specifications of the Java bundled JavaScript interpreter (Nashorn since Java 1.7) `_. - before running the script the first time, you have to save it Example: :: Debug.user("hello from JavaScript"); var img = "img.png"; hover(img); // uses Screen(0) like in Python var scr = new Screen(); print(scr); scr.hover(img); // uses the Region scr The SikuliX features in general have to be used like in Java programming. - All classes from ``org.sikuli.script`` are known (pre-imported). - ``Debug`` and ``Settings`` are also known. the following undotted methods work on Screen(0): :: wait, waitVanish, exists, click, doubleClick, rightClick, hover and you have undotted: :: type, write, paste, run **Be aware:** - FindFailed is not thrown! internally returns null! - If you want to catch exceptions, you have to use the dotted methods. .. _UsingPython: Using Jython and Python ======================= Setup your own Jython environment --------------------------------- Out of the box, SikuliX can be setup with a Jython standalone package, which then is only available to SikuliX to run scripts in SikuliX IDE or from commandline using SikuliX. If you want to use the Jython REPL (interactive commandline) or another IDE like PyCharm, you have to install Jython seperately. In this case no need to have the Jython standalone package, but you can tell SikuliX to use your installed Jython environment: **This only applies to SikuliX 2.0.x+** Previous versions 1.1.x might make problems and are no longer supported **MANDATORY: use Jython 2.7.1** **Make sure you have a valid Java installation (version 8+)** **Note for Mac OSX** If you ever encounter an error like ``ValueError: unknown locale: UTF-8``, then take care, that your environment at runtime of Jython contains these 2 entries: - LC_ALL=en_US.UTF-8 - LANG=en_US.UTF-8 You might use ``export`` or any other appropriate method. **Apply the following steps, to get a Jython environment, that can be used with SikuliX:** - download the installer package from `Jython 2.7.1 Installer `_ - install (usually by double-clicking the package) using the standard setup into an empty folder - test by running ``/bin/jython`` from a commandline, which should open an interactive Jython session, that allows, to run Python statements line by line - make sure, that pip and easy_install are available: - ``/bin/pip`` exists - ``/bin/easy_install`` exists - if this is not the case run ``/bin/jython -m ensurepip`` on a commandline and check again - if you do not get pip ready, don't bother: Jython is useable without it. The caveat: any additional packages have to be installed/setup manually. Additional stuff, if you succeeded with pip: - run ``/bin/pip install jip`` to install the package ``jip``, which allows to add Java libraries easily to your Jython environment - add any needed Python package (must not depend on C-based stuff) using ``pip``, ``easy-install`` or manual methods into ``/Lib/site-packages`` and/or use ``jip`` for adding Java libraries preferably from Maven Central **Usage with SikuliX --- Case 1: Using Jython from commandline or in a Python IDE** You can run SikuliX scripts using ``/bin/jython /youNameIt.py``. In a Python IDE you have to setup your project according to the rules. Prereqisites: - ``sikulixapi.jar`` (NOT sikulix.jar!) must be on the Java classpath at runtime. This can be achieved using one of the methods that come with Jython or the IDE. - to give access to the images use the appropriate features of ``ImagePath``: ``setBundlePath()`` ``and add()`` If this case is planned to be your main usage, you should decide to use plain .py files according to the Python script/module rules and the ImagePath features to acces your images. You can use the SikuliX IDE to capture your images even in plain .py files (:ref:`see plain .py in IDE`). If you want to run scripts from within the SikuliX IDE or from commandline using SikuliX, just open and run the main script unchanged (:ref:`see plain .py in IDE`). **Usage with SikuliX --- Case 2: Using the installed Jython with SikuliX** You have to tell SikuliX about your installed Jython by using the Extensions feature: - start the IDE once and ignore all error messages and close it again - in the ``SikuliX-APP-DATa-folder`` go to the folder ``Extensions`` - `edit the file extensions.txt to point to your Jython install `_ - start the IDE again - you should get to the normal state (a tab ``Untitled`` with type ``jython``) Access Python packages from SikuliX scripts run by SikuliX (GUI or commandline) ------------------------------------------------------------------------------- The following approaches apply to situations, where you want to use Python modules installed somewhere on your system, without the need to manipulate ``sys.path``, meaning, that when using ``ìmport moduleXYZ`` this package is found automatically. SikuliX uses a central repository (``SikulixRepo`` in the following) for internal stuff (native libraries, downloaded artifacts, resources needed at runtime and simailar things). This is a folder in the user's private space (home folder) :ref:`look here `: - Windows: ``%APPDATA%\Sikulix`` - Mac: ``~/Library/Application Support/Sikulix`` - Linux: ``~/.Sikulix`` **Basic preparation** To ``SikulixRepo`` add a folder ``Lib`` and inside add ``site-packages`` (usually already there) **Approach 1** Since an existing folder ``SikulixRepo/Lib/site-packages`` will be recognized and added automatically as the ``1st entry to sys.path``, modules/packages contained in here will be found when imported without any further preperations. This approach can be used, to "overwrite" modules/packages, that otherwise would be found elsewhere on ``sys.path`` (e.g. for testing) **Approach 2** In the folder ``SikulixRepo/Lib/site-packages`` have a file ``sites.txt``, that contains absolute paths one per line, that point to other places, where modules packages can be found. These paths will be added automatically at startup to the ``end of sys.path`` in the given sequence. .. _LoadableJars: Prepare and use your own jar files in the Jython environment ------------------------------------------------------------ You might prepare jar files containing Python scripts/modules/packages, Java classes and other stuff like images, that are intended to be used in the scripting context. **possible use cases** - you want to pack scripted stuff together with other resources into a container ready to be used by yourself or others via import (which is not supported by the .skl packaging method). - you want to secure your script code against modifications by others, that use your distributed jar. Later (possibly only with version 2) there will be a feature available, to run such script containers directly from commandline (``java -jar mystuff.jar parameters``) or by double clicking. **typical jar file structure**:: -- jar rootlevel module1.py # Python module module2.py - folder1 # Python package __init__.py stuff1.py stuff2.py - images # image folder img1.png img2.png - org # Java package - mystuff class1.class class1.class **how to pack such a jar** You might use the Java ``jar utility`` (contained in the JDK). Or use the **SikuliX provided** feature ``Sikulix.buildJarFromFolder(jarpath, folder)``, where jarpath is the absolute path to the jar (the parent folder must exist, the jar is overwritten), that should be created and folder is the absolute path to a folder, containing the stuff to be packed. The content of the folder is copied to the root of the created jar. Just run ``Sikulix.buildJarFromFolder(jarpath, folder)`` in an empty tab in the IDE or in a script, that might do some pre- and/or postprocessing. If the folder contains an ``__init__.py`` on the first level, the given folder is taken as a Python package and as such copied to the root level of the jar, to preserve the package context:: -- packagefolder __init__.py stuff.py becomes a jar -- jar rootlevel - packagefolder __init__.py stuff.py **how to secure your script code using the jar packaging** - Step 1: prepare a folder as in the previous chapter - Step 2: compile the folder into a new folder (see below) - Step 3: pack the new folder into a jar for distribution Run in an empty IDE tab or as part of a script: ``Sikulix.compileJythonFolder(sourcefolder, targetfolder)`` copies the complete content from sourcefolder to targetfolder (the parent folder must exist, the folder is emptied if exists) and then traverses the targetfolder replacing each ``foobar.py`` with it's compiled version ``foobar$py.class``, that contains JVM-byte-code, so your script code cannot be edited anymore in this targetfolder, but still be used with ``import foobar``. **Be aware:** Be sure, your code compiles without errors, because the compile feature either succeeds or fails (compile errors), but you will not get any information about the cause or even the place of the compile problem. Using SikuliX with real C-based Python (version 2.7 up to 3.7 latest) --------------------------------------------------------------------- .. _RealPython: This feature is under development (see on GitHub: `sikulix4python`_ for details and status) Comments and ideas are welcome and can be posted there as issues. Scripting with IntelliJ PyCharm and IDEA with Python plugin ----------------------------------------------------------- **BE AWARE** - you will not see any image thumbnails as in the SikuliX IDE - capturing of your images has to be done with the SikuliX IDE separately into special image scripts/folders, that have to be applied to your scripts using the ImagePath features - code completion will not work, since most features are at the Java level and the undotted Region/Screen funtions (find, click, ...) are only created dynamically at runtime of the script and hence not accessible in the static editor environment - so in the end the only advantages over the SikuliX IDE are:: - a full featured editor and at least syntax evaluation - a full featured debugger, where you can even inspect down into the Java level at a breakpoint Among the free IDE's, that allow to work with Python projects, I made the best experiences over the years with JetBrains's PyCharm (using the free Community Edition) or the Python plugin in JetBrain's IntelliJ IDEA. PyCharm has a weakness, in that code completion while editing does not look into jar-files nor Java classes, while in the built-in Python console after having imported a Java class, code completion about the class attributes/methods works, so having a console open while editing might help as a workaround for inspecting Java classes. IntelliJ IDEA with Python plugin supports complete code completion (while editing and in console) including Java classes, but is a bit more complex to setup and use with just Python. If you are more used to Eclipse, the Eclipse PyDev might be your choice for Python development. The steps to get it running for use with SikuliX are similar to the following steps for PyCharm. If someone wants to document the details for Eclipse PyDev, he/she is always welcome - I will not do it on my own. **Step 1: get Jython up and running** Just follow the steps mentioned in the chapter ``Setup a Jython environment`` above. **Step 2: get PyCharm up and running** Download and install the `PyCharm Community Edition `_. **Step 3a: tell PyCharm to use Jython as interpreter** Start PyCharm and make sure to have closed all projects and be in the start-up window titled ``Welcome to PyCharm``. At bottom right in the menu ``Configure`` select ``Preferences``. You should get a dialog window titled ``Default Preferences``. At the left side select ``Project Interpreter`` and choose your installed Jython by pointing to the contained ``.../bin/jython`` script/command-file. Click ``Apply`` and watch how the Jython setup is analyzed and implemented in PyCharm. If it worked click ``Ok``. .. image:: pycharmDefaultInterpreter.png **Step 3b: tell PyCharm to use Python as interpreter** With respect to SikuliX features used in Python scripts :ref:`look here for details`. **Step 4: setup the environment for a SikuliX-aware Python console** As with the step before get the ``Default Preferences`` dialog open. At the left side select ``Build, Execution, Deployment``... ``Console``... ``Python Console``. To the ``Environment variables`` add antry ``CLASSPATH``, that points to a valid ``sikulixapi.jar``. In the ``Starting script`` you might add the line ``` from sikuli import * ``` so each time you start a console, the prompt is already prepared to know about SikuliX features. .. image:: pycharmDefaultConsole.png Using Ruby ========== Using SikuliX in Java programming ================================= To be written - but until then `read here `_ Using SikuliX in non-Java programming scenarios =============================================== To be written Using RobotFramework ==================== .. _UsingRobot **BE AWARE: Switched off until further notice --- to be implemented as extension** .. versionadded:: 2.1.0 .. versionadded:: X1.1.1 **Valid up to version 2.0.x** You can run ready Robot scripts out of the box in the Sikulix context (IDE or from commandline). The needed Python module robot ( from `robotframework 3.0 `_ ) is bundled with the ``sikulixapi.jar``. At runtime and already with setup, the module is exported to the folder ``/Lib``, which is on ``sys.path`` automatically. So there is no need to have anything else available than a suitable setup of SikuliX. The easiest way is to use the SikuliX IDE with this principal setup :: runScript(""" robot *** Variables *** ${USERNAME} demo ${PASSWORD} mode ${TESTSITE} http://test.sikuli.de *** Settings *** Library ./inline/LoginLibrary Test Setup start firefox and goto testsite ${TESTSITE} Test Teardown stop firefox *** Test Cases *** User can log in with correct user and password Attempt to Login with Credentials ${USERNAME} ${PASSWORD} Status Should Be Accepted User cannot log in with invalid user or bad password Attempt to Login with Credentials betty wrong Status Should Be Denied """) class LoginLibrary(object): def start_firefox_and_goto_testsite(self, page): popup("start_firefox_and_goto_testsite") def stop_firefox(self): popup("stop_firefox") def attempt_to_login_with_credentials(self, username, password): popup("attempt_to_login_with_credentials") def status_should_be(self, expected): popup("status_should_be") the first 2 lines :: runScript(""" robot signal, that you want to run an inline Robot script, that follows on the next lines terminated by ``""")``. This construct is a multiline Python comment, that can be used as a string. **BE AWARE** In the standard, a report HTML will be automatically opened in the standard browser at termination. If you do not wnat that (e.g. in unattended situations) use this instaed as the first 2 lines. Take care: after robot 1 space, a linefeed directly after silent and nothing else. :: runScript(""" robot silent Normally when working with SikuliX features, you have to do some Robot Keyword implementation at the Python level. To Robot you tell where to find these implementation using the ``Library setting``. In this case we have the implementations inline in the same scriptfile according to the Robot rules packed into a Python class having the Keyword methods according to the Robot naming conventions. At runtime this class will be exported to a Python file, whose absolute path is then replacing the Library setting. If you have the Keyword implementations somewhere outside, then you have to put the correct path specification into the Library setting. Another option is to reference a jar file as a Library again according to the Robot specifications. If you now run the script in the IDE, internally a ``robot.run`` will be fired after having setup the script content and the environment. Currently no extra options can be provided for the robot run. As a result you get a folder with the ending ``.robot`` named as your script in the same folder as your script folder containing inputs to and the results from the robot run :: # supposing the script is named testrobot.sikuli # then you get a folder testrobot.sikuli.robot with the content testrobot.robot # the robot script LoginLibrary.py # the Python Keyword implementations # the standard Robot outcome output.xml log.html report.html Still being in the IDE another possible setup would be this way: :: robotScript = """ robot *** Variables *** ${USERNAME} demo ${PASSWORD} mode ${TESTSITE} http://test.sikuli.de *** Settings *** Library /some/path/to/LoginLibrary.py Test Setup start firefox and goto testsite ${TESTSITE} Test Teardown stop firefox *** Test Cases *** User can log in with correct user and password Attempt to Login with Credentials ${USERNAME} ${PASSWORD} Status Should Be Accepted User cannot log in with invalid user or bad password Attempt to Login with Credentials betty wrong Status Should Be Denied """ # here you could do some preprocessing and even modify the above robotscript runScript(robotscript) # eventually do something with the result **BE AWARE** for the keyword library, the ``Name`` in the file name ``Name.py`` and the statement ``class Name()`` **must** be the same and start with an uppercase letter. **BE AWARE ON WINDOWS** the file path must be escaped with 4 backslashes for each backslash like so ``C:\\\\Robot\\\\Libraries\\\\Name.py`` (which leads to the needed 2 backslashes for each backslash as escape in the final robot file) Of course you can use any other method, to fill a string representing a valid Robot script, provided the first line contains the string ``robot`` and only that (denoting the script type for runScript). If in such a case you want to provide an inline Keyword implementation: this does the trick: :: # prepare your script content runScript("robot\n" + scriptContent) # eventually do something with the result # """) # the rest is taken as inline Keyword implementation If you have the need to specify extra parameters to the ``robot.run()``, then you still have the option to stay within the SikuliX context (IDE or from commandline): :: prepareRobot() # takes care for the correct environment workdir = getParentFolder() script = "arobottest/arobottest.robot" robotscript = os.path.join(workdir, script) print "*** trying to run:", robotscript robot.run(robotscript, outputdir=workdir) A library .py file being either in the script folder itself or in the folder containing the script folder is found automatically. So simply the library name is enough in this case. In all other cases you either have to specify the absolute path off the .py script (take care with windows - see above) or use ``addImportPath()`` to add the folder containing the library .py file to ``sys.path``, in which case again only the name is sufficient in the Robot script. It is strongly recommended, to always specify the `outputdir= parameter` since otherwise the reportfiles will be written to the working folder (from where you are running), which might not always be what you want. If you want to use any of these variants outside the SikuliX context (some external Jython or in an IDE like PyCharm) you have to add these 2 lines at the beginning of your main script (as always in such cases): :: import org.sikuli.script.SikulixForJython from sikuli import * to get the SikuliX context ready. Experimental: SikulixServer - run scripts from anywhere with zero-delay =================================================================== .. _UsingRunServer .. versionadded:: X2.1.0 **This feature is completely revised** The design/implementation goals `are here. `_. ... and have to be used as usage guide for now.