Using Scala and Sikuli to create a DSL for analyzing and testing UI Methods

Ever since I first read about GOMS, I've been intrigued by the idea that you could write a description of a process that was both human readable and parseable by a computer.  A language like this could have multiple benefits:

• The instructions to users could double as form a type of automated UI testing -- potentially giving you two benefits for the cost of one.
• You could guarantee that the instructions that you provide to users were correct.  Incorrect instructions would fail to execute during your automated testing (which could be run after every build).  This would catch cases where the developers change the UI without informing the documentation, testing and/or specification teams.
• The instructions could be analyzed using the GOMS methodology to give you free usability metrics (time to complete, learnability, some types of error prediction).
• Other types of analysis related to the structure of the user interface (e.g., which screens and/or controls are most documented, how similar are the procedures used on different screens, etc.)

My interest in this area lead me to pursue my Masters in HCI at Carnegie Mellon with Bonnie John and later work with David Kieras while I was at Soar Technology.  This work was documented in the following papers:

• Hudson, S. E., John, B. E., Knudsen, K., & Byrne, M. D. (1999). “A Tool for Creating Predictive Performance Models from User Interface Demonstrations.” Proceedings of the ACM Symposium on User Interface Software and Technology, pp. 99-103. New York: ACM Press.
• Kieras, D., and Knudsen, K. (2006). Comprehensive Computational GOMS Modeling with GLEAN. In Proceedings of BRIMS 2006, Baltimore, May 16-18.

Unfortunately, funding for that work was not continued, but as I started my User Experience Design work at ProModel -- including elements of UI analysis, specification, documentation and testing -- I was reminded again of the benefits of having a human and machine readable language for describing procedures.

For various reasons, I decided to take a fresh approach.

• I designed my own Domain Specific Language (DSL) based on the guidelines found in the Microsoft Manual of Style for Technical Publications.
• I tried various methods to encode my DSL.  I've been most happy with Scala so far and will have some posts about my experiments with both it's internal and external DSL capabilities.
• I've also incorporated Sikuli (a computer vision-based testing library) to provide a mechanism for automatically executing my procedures.

Well, I guess this is long enough for an introductory post.  Stay tuned for a series of posts that document my prototypes and results thus far...

AutoHotkey Script for a Left Hand Menu

Recently, I've started posting AutoHotkey scripts.  AutoHotkey provides a pretty easy way to automate just about anything in Microsoft Windows without having to become a Windows programmer.  If you'd like to learn more, LifeHacker has a pretty nice introduction to AutoHotkey.

In my last couple of posts, I've been talking about tools and techniques to make it easier to keep your right hand on on your pen when you're drawing using a Wacom tablet, for example.  This script also works pretty well for keeping your right hand on a mouse.

Basically, when you run this script the Win+A key will bring up a window and while the window is open the ASDF keys can be mapped to execute ANY command.  Currently, I'm just popping up test messages, but as a next step I'll be adapting this basic framework to to cover my most common actions while working in Adobe Illustrator, for example.

---


; Left Hand Menu

#a::
Hotkey, a, Test, On
Hotkey, s, Test, On
Hotkey, d, Test, On
Hotkey, f, Test, On

Gui, Font, s10, Verdana  ; Set 10-point Verdana.
Gui, Add, Text, ym, (A) Test 
Gui, Add, Text, ym, (S) Test 
Gui, Add, Text, ym, (D) Test 
Gui, Add, Text, ym, (F) Test 
Gui, Show, w500 h100, Left Hand Menu
Return

GuiClose:
Hotkey, a, Off
Hotkey, s, Off
Hotkey, d, Off
Hotkey, f, Off
Gui Destroy
Return

Test:
MsgBox You pressed %A_ThisHotkey%.
return

Ergonomics of Working with a Graphics Tablet

David Revoy has one of the best articles I've seen about the Ergonomics of Graphics Tablets.  He has experimented with a variety of tablets and desk layouts and, best of all, created these very clear diagrams (such as the one below) to illustrate his article.

As Revoy shows and explains, "Most of CG artists use shortcuts on the keyboard to speed up their work-flow."  In this setup, the right hand is for drawing and the left hand is for shortcut keys (critical for switching between drawing tools, copy & paste, etc.).  The right hand moves back and forth for typing.

I'm writing this post as background for some UI research that I've started doing to improve the usability of software for artists and other pen-heavy users.  The first example is the script I wrote in AutoHotkey to set the location of the Tablet PC Input Panel.

Up next, I'll be looking at the Quill gesture design tool for pen-based user interfaces.  I think some interesting things could be done with simple gestures (like Graffiti unistrokes) in a vector drawing tool that haven't really been explored yet.  I'd like to explore rough drawing text, shapes, etc. and having the computer recognize it and convert it to a desired format in place.  Done right, I think it could lead to a more natural drawing experience while still maintaining the precision of vector drawing tools.

Tablet PC Input Panel and AutoHotkey

As someone who does a lot of drawing, I like to use pen input.  I've used Wacom tablets and, recently, a Samsung Slate Tablet PC.

Like a mouse, a pen input device usually takes up one hand with the other hand on the keyboard for triggering shortcut keys.  This works really well until you have to enter text.  As Allen Newell showed in The Psychology of Human Computer Interaction, moving your hand back and forth from the keyboard to mouse (or pen) can be really disruptive.

The interesting thing about a pen as an input device is that, unlike the mouse, it is very natural to enter text with a pen and Microsoft has very good handwriting recognition built into the Windows operating system starting with Windows XP.  There is a little panel, called the Tablet PC Input Panel (TIP) that can be configured to pop up, accept handwriting, and then output text as if you had typed it.

In general, I've found it to be very useful when I'm drawing some UI design and I want to input little bits of text (labels, buttons, menus, etc.) in the drawing.  One main drawback is that I could never get the TIP to show up exactly when and where I wanted it.

Luckily, with a little tinkering in AutoHotkey, I was able to get exactly what I want -- to have the TIP appear right below my cursor.  The TIP window is a bit odd and can't me moved with the normal AutoHotkey WinMove function so I wrote a script that uses the mouse to drag it into place.  I'm posting the script below to save the next guy the trouble.

----

; Triggered by Win+Z (windows key and Z)
#z::
IfWinNotExist, ahk_class IPTip_Main_Window
   Run, TabTip.exe
Else
   WinShow, ahk_class IPTip_Main_Window

WinGetPos, tipX, tipY, , , ahk_class IPTip_Main_Window
CoordMode, Mouse, Screen
MouseGetPos, mouseX, mouseY
MouseClickDrag, Left, tipX+200, tipY+20, mouseX+100, mouseY-50
MouseMove, mouseX, mouseY

Return

UX Design of a Self-Organizing Volleyball Tournament

One of my hobbies is volleyball.  I used to play about once a week or less, but lately I just can't get enough of it.  I play mostly through the Ann Arbor Rec & Ed and a Meetup.com group called the Ann Arbor Adventure Club.  I play enough with the Adventure Club that I was asked to help organize a doubles volleyball tournament.

Organizing one of these things is a bit trickier than you might expect, especially if you want everyone to have a good user experience!  Also, this group has some unique constraints that hindered a previous attempt to organize doubles volleyball.  In particular, people want the events to be low cost and casual (i.e., drop in when I can).  There are other leagues that cost more and require more of a commitment.

Anyway, by the time I was done organizing this thing, I realized that I had essentially done a small user experience design project -- complete with a visual interface (a printable form - see below).  It gets it's first test on Sunday, but I thought I'd capture what I had done while it's still fresh in my mind.  I'll come back and do another post to let you know how well it worked.

Goals (i.e., Requirements / Constraints)

1. High-quality doubles play: If you can't bump, set & spike somewhat consistently, please take yourself off of the list -- the Thursday night games are oriented towards all levels.
2. Resiliant to "no shows": The first four people to show up will just start playing. Then the next four. After that, we will rotate people in order. Because of this, there won't be fixed teams -- you'll be randomly (sort of) paired with teammates. Still, it would be nice if you would update your RSVP if you know you can't make it.
3. Mostly playing, with some rest: With 2 courts, 8 will be playing at one time. You'll be playing at least 2/3 of the time if all 12 show up. To maximize playing time, there should be no breaks between games (or just very short ones). If you need a rest, let someone play your slot and then you can catch the next one.

Process Design

1. When you arrive, sign in on the spreadsheet (see below).
2. We will use Ann Arbor sand league rules (no set overs, cannot set receive serves, no tipping, carries, etc.). Call your own and your partner's fouls.
3. Once 4 arrive, players 1-4 start playing on the court of your choice. Play 2 games to 21 (first to 21 wins, do not have to win by 2). Players 5-8 start playing on the other court. Players 9-12 wait.
4. When your game is done, each player tallies their wins and losses on the spreadsheet. Each game counts. For example, if players 1 & 2 split the first match against players 3 & 4, they would each mark down 1 win and 1 loss.
5. After the first match is over, players 9-12 would play against players 1-4. If we don't have 12, or if somebody from 1-4 needs a rest, just keep going in order. So if we have only 11 and player 4 needs a rest, it would be players 9, 10, 11 & 1 against players 2, 3, 5 & 6 (players 7 & 8 would wait).
6. One important rule is that you must have a different teammate for each match.
7. Keep rotating for a couple of hours. Always the players with the fewest number of games played get to go next. Ties go to the player who signed in first (e.g., player 4 goes before player 5 if they've both played the same number of games).
8. At noon, we'll set up for the championship match. The player with the best record plays with the second best player against the 3rd and 4th best players. The second court will have the 5th & 6th ranked players against the 7th and 8th for in a consolation match.

Visual Artifact Design (printable sign in & tracking sheet)

The goal of the sheet below was to embed the process described above.  Some specific design details include:

• You are automatically assigned a number by signing in.
• The first 8 players have their teammates assigned automatically by the sheet.  This also serves to teach the players how to fill out the form.
• The "Games Won" columns indicate what you should fill in (a value between 0 and 2) and embed some of the format information (2 games per match).
• As each player fills in their teammate and games won information, it will become visually apparent who plays in the next game -- it will be the left-most, top-most empty cells.  I think this feature will significantly add to the user experience since I think the most likely point of confusion will be figuring out who plays next. (Remember that we're playing with flexible teammates and schedule).
• The match columns headers are rotated 90 degrees to allow room for descriptive labels within the constraints of the page width.
• A double grid and thin lines clearly indicates where to enter match data.  I might consider alternating light grey rows to make this even more clear in the next version.

Design Quotes

I've been doing a bunch of reading over the past six months as I've made the transition from SoarTech where I was a bit of a jack-of-all-trades (project manager, proposal writer, programmer, designer, etc.) to ProModel where I am back to being primarily a visual and UX designer.  As I've been reading, I've been collecting a list of good quotes about design.  Thought I'd take the opportunity to share the best ones.

No design works unless it embodies ideas that  are held common
by the people for whom the object is intended.
--Adrian Forty

Notation is a tool of thought. 
A problem properly represented is largely solved.
-- Philosophy 101

Essentially, all models are wrong,
but some are useful.
-- George Box

The minimal requirement in design is
the organization of all the parts into a unified whole.
-- Eugene Larkin

Simple designs present the minimum amount of information
to achieve maximum effect ... by reducing and regularizing.
-- Michael Terry

Two golden rules of information visualization:
1) show the data, 2) show comparisons.
-- Bret Victor

A proper visualization is a kind of narrative,
providing a clear answer to a question without extraneous details.
-- Ben Fry

Make frequent tasks easy and infrequent tasks possible.
-- Usability Maxim

God is in the details.
-- Ludwig Mies van der Rohe

Script to Extend the Artboard in Adobe Illustrator

It's pretty easy to change the size of the artboard (i.e., canvas) in Adobe Illustrator, but as I've said in some previous posts I've started to experiment with large format wireframes / navigation diagrams.  When you zoom way out (or print small) it looks like a navigation diagram, but you can zoom way in (or print poster size) to see the wireframe details.  I'll put up an example in a later post.

Anyway, I tend to start with the artboard at tabloid (17x11) or 2x tabloid (34x22) and then grow the artboard incrementally as I go.  I change the artboard regularly enough and in such a predictable way that I went ahead and made the script below:

----

if (documents.length == 0) {
    alert("Must have a document open!");
}

TABLOID_LANDSCAPE_W = new UnitValue (17, "in");
TABLOID_LANDSCAPE_H = new UnitValue (11, "in");

curWidth = activeDocument.artboards[0].artboardRect[2];
curHeight = activeDocument.artboards[0].artboardRect[3];

isWidth = confirm("Extend width?");

if(isWidth) {
    newWidth = curWidth + TABLOID_LANDSCAPE_W.as("px");
    activeDocument.artboards[0].artboardRect = [0,0,newWidth,curHeight];
} else {
    newHeight = curHeight -TABLOID_LANDSCAPE_H.as("px");
    activeDocument.artboards[0].artboardRect = [0,0,curWidth,newHeight];
}

----

It's pretty simple, but I thought I might as well share the idea and the script for whomever might find it useful.  Here's how it works:

1. First check to make sure a document is open.
2. Define the width and height that I'm going to use to extend the artboard.
3. Get the current width and height of the artboard.
4. I use a confirm dialog to ask if you want to extend the width, otherwise the script will extend the height.
5. Finally, calculate the new width or height and extend the artboard.