Sikuli Trick: Capturing images with the Sikuli Java library

In my last post, I introduced Sikuli, talked about its capabilities and power, and posted a very small code sample that showed the basics of using the Sikuli Java library.

In that code sample, I showed how you could use Sikuli to load an image, search for it on the screen, and click it.  However, there is a big gap in that workflow -- if everything is based on images in Sikuli, where do all of those images come from?

Luckily, Sikuli provides a very nice capability to support the selection, capture, and storage of images from the screen.  If you use their UI, you get it for free.  It's also available in the Java library, but it's not well documented, so I'll post it here for future Googlers to find.

The interesting difference between the code below and the code from my last post happens when the image file does not exist (!imageFile.exists()).  The CapturePrompt.prompt() call freezes the screen and gives you a crosshair to make a selection.  This selection is then passed as a ScreenImage object to the update() callback.  I retrieve the Image object and save it as a file.  This happens the first time the program is run.  Each subsequent time the program is run, it will find the file and then click it.

In my next post, I'll tie this back to the Scala DSL that I posted about on Jan 30 and Feb 4 to make an English-like language for describing procedures that the computer can execute.

----


import java.io.File;
import java.io.IOException;

import javax.imageio.ImageIO;

import org.sikuli.script.CapturePrompt;
import org.sikuli.script.FindFailed;
import org.sikuli.script.Observer;
import org.sikuli.script.Screen;
import org.sikuli.script.ScreenImage;
import org.sikuli.script.Subject;

public class SikuliExample2 {

  public static void main(String[] args) {

    // Create a Sikuli Screen object.
    Screen screen = new Screen();

    // Specify an image that we'd like to click
    String USER_HOME = System.getProperty("user.home");
    String buttonImage = USER_HOME + "/Sikuli/OK_Button.png";
    final File imageFile = new File(buttonImage);

    // If the image file doesn't exist, screen grab 
    // Note, this only needs to be done the first time
    if (!imageFile.exists()) {
      final CapturePrompt cp = new CapturePrompt(screen);
      cp.prompt("'OK_Button.png' not found. Please select it.");
      cp.addObserver(new Observer() {
        public void update(Subject arg0) {
          ScreenImage img = cp.getSelection();
          try {
            ImageIO.write(img.getImage(), "png", imageFile);
          } catch (IOException e) {
            e.printStackTrace();
          }
          cp.close();
        }
      });
    } else {
      // Find the button and click it.
      try {
        screen.click(buttonImage);
      } catch (FindFailed e) {
        System.out.println("Couldn't find: "+buttonImage);
      }
    }
  }
}