Yesterday, I had the chance to participate in a formal usability study at the University of Ottawa. Marconi Lanna’s master’s project is an improved code difference visualization tool. I had already played with the tool in the past and found it interesting but only saw its true value during the formal usability test where I was given specific tasks to perform. I will keep the details about this tool for a later post when it becomes publicly available, but participating in the study helped me realize a few things:

  1. Creating a proper usability study is hard.
    • There are so many ways you can introduce bias in a test, it isn't funny. For this particular test, I feel that it was very close to being unbiased and most of my initial concerns were addressed.
    • One concern that does remain is the fact that the test cases were crafted by the researcher and could have been manipulated to generate a specific set of results. However, crafted test cases are required for any kind of comparison between the users. I personally feel the test was unbiased as it presented potential flaws in both the default tool and the proposed tool.
    • This particular test was for a tool used by software developers. As you probably know, some developers demonstrate productivity levels an order of magnitude higher than others.  You can imagine that this makes it hard to analyze productivity results collected during a study.
  2. Don't forget to make the test subjects communicate
    • If a subject does a task and says nothing, you don't learn 10% of what you could have learned about their thought process. In usability studies, the thought process is the most important thing to understand.
    • Everyone is different, and having them express their concerns/procedures helps you analyze the results.
    • This is not a discussion. Because of the desire for unbiased results during the questions, you cannot ask specific questions (which may lead to bias) or comment on the answers (which will definitely lead to bias). However, after everything is done and recorded, having discussions with the subject can enhance your overall understanding of what they said while thinking aloud.
  3. Working on usability is a self-improvement task that you must work on every day.
    • It's easy to continuously postpone enhancements to the the non-functional aspects (performance, security, reliability, usability, etc.) of a software system when you're a small startup.
    • You should log even seemingly superficial non-functional concerns in your bug database just as with any other issue. Customers that complain are rare and you must take their comments seriously.

Following my own advice, I want to look at one of the usability issues that I filed in our bug system a few months ago, our bug #1377. I like revealing some internal details of the issues we face and appreciate your input on our strategies. Note: if you're not interested in a real-world example at a complex usability issue, skip to the conclusion of this post as this could be a snooze-fest!

A closer look at one of our usability issues

Drink too much eggnog and you too will have usability issues Usability is one of our core values when building software. However, we sometimes slip up, as with anything! Looking back at the comments received over the last year by our users, I can clearly identify the single most important usability problem we have with our wide range of software solutions for the franchise industry. The issue is related to our party/event/appointment booking module which is a part of our franchise point of sale (POS) software. I'd like to drill down into this feature and present the usability issue and how it came to surface.

Background information

A POS system basically lets you sell items in a retail store. Our is integrated with a frequent buyer program, and we register information concerning each member in the database. Depending on the franchise brand, different information is tracked. (We can do cool things when customers are willing to provide information such as their postal or email address, but as a general rule, all fields are optional). We also have a party booking module which allows parents to book their child's birthday party at a store. The feature accomplishes two things: it acts as a calendar system (to avoid booking overlaps) and is integrated with the sale module to support deposits. Obviously, when someone books an event at a store and pays a deposit, the system needs to remember who paid what, so that when the big day comes, the deposit can be applied to the transaction.

To make a long story short, we offer a number of configuration options in the background to support different scenarios used by different franchises. Here are some examples:

  • Book a child birthday party in one of multiple rooms.  You reserve the room for a period of time for your exclusive use.
    • This is akin to booking a meeting room.
  • Book a space in one of multiple play areas. You reserve your spot in the play room, but the room is shared up to its capacity.
  • Book a particular employee for your haircut or spa services.
    • Rooms are not the only resource that can be booked. This ties in to the employee management system.
  • Book a table, a pitcher of beer, and that hot new employee to serve at your table.
  • You're booking a particular resource and pay a: 
    • Flat fee (party package)
    • Flat fee per guest
    • The fee will depend on what is bought at a later date.
  • Stores have the possibility to ask for a deposit, and if they do, it can be of a particular % of the transaction or a fixed dollar amount.
    • The store decides if deposits are refundable. (in the event of a cancellation).

As you can see, there are numerous elements at play here that make it non-trivial in our context, where we design the booking system to meet each franchise's particular booking needs. What is important at a high-level, however, is we defined the following use cases when researching the requirements for this module.

  1. (Deposit is required) A member is in the store and wants to book an event.
  2. (Deposit is required) A member is on the phone, wants to book and event and will come by later this week to pay for the deposit.
  3. (Deposit is not required) A member wants to create a booking.
  4. Employees want to peruse the list of bookings for a particular date range.
  5. Employees want to add an additional deposit amount to a transaction (regardless of the fact that deposits are required or not)
  6. Employees want to finalize an event, and apply the total deposit amounts to the transaction.
  7. An employee books a "fairy princess" birthday party for another employee, without their knowledge or consent, and invites their friends and family. The birthday boy frantically tries to cancel the event as soon as possible, to avoid long-term trauma.
  8. (and a few more, less important scenarios).

We chose to divide our scenarios contextually

We decided that the best way to solve this problem was to create two ways to access the booking information, one which is contextually associated to a store member and one for higher-level tasks.

  1. From the existing sales page in the POS (for all scenarios that work with money)
    • The cashier first selects a member from the frequent member program (or create a new account)
    • The system displays a new button called bookings
    • The cashier selects bookings and progresses through the following menus (create a new booking, additional deposit, cancellation, finalize event).
    • The system then returns back to the sale page, where the cashier can process the sale/refund.
  2. From the events page in the POS (for all scenarios that don't deal with money)
    • The system displays an Outlook-like calendar of all the events, allowing employees to peruse the events
    • Employees can view availabilities, change rooms, change employees, change date&time, enter additional event information from this page.
    • Employees can create an event without a deposit, which the system presents as being in the "Deposit Required" state.
      • This is to support scenario #2 listed above. You'd create a tentative booking, which you'd be willing to double book if the person never comes by to pay for the deposit.
      • Also, this is used by franchises that don't require deposits instead of going to the sale page, and becomes similar to a paper calendar in a hair salon.

The usability problem

After being in use for about 18 months, I can state that we need to step back and re-evaluate this feature, to improve its usability. The system works fine when used as intended, but I have observed the following issues:

  • If deposits are not required, untrained cashiers book events in the event page and never finalize them.
    • They do open up the sale page, possibly select the member, and sell whatever was bought without associating the sale with the booking.
    • This is bad because we're left with open events and can't track cancellation rates, sales per member, etc.  (bad for the store)
  • If deposits are required, untrained cashiers assume they need to go to the events module to do anything related to events.
    • They book an event, but the system marks it as "Deposit Required" and doesn't let them change the deposit amount.
    • This is bad as they are under-using the system, not following store policies, and sending in support requests to have the system explained to them. (bad for the store and bad for LavaBlast)

As a side note, I am honestly always surprised when a cashier expects to book an event, accept money but not track any information to associate the deposit with the person making the payment. That's like their store manager expecting them to come in to work and not bothering to write their name on their timesheet.

Root cause

  • We've divided the scenarios in two groups: the ones that deal with money and the ones that do not.
  • Combine this with the following facts:
    • Some stores don't use deposits. Therefore, money is only involved when finalizing the event at which point the main task is selling products/services, not dealing with events.
    • Even if we use deposits, we allow the creation of temporary bookings (in the deposit required state).

To recap, to improve usability, we've defined processes that contextually lead the cashier to performing their tasks. However, since the core starting point (deposit amounts) varies completely from one franchise store to another, our contextual division doesn't make sense.

Possible Solutions

  • Possible solutionsSolution #1: In the store events page, after selecting an event, give the option to the cashier to add a deposit, cancel or finalize the event. This would bring the user from the events page to the sale page and allow them to follow our defined processes.
    • Possible issue: We're making them switch contexts implicitly. When they're done, they would wonder why there were in the sale module instead of the events module. (do we force them back to the events module when they are done?)
    • Possible issue: they want to finalize an event that has already occurred (to get rid of "open" events, caused by the described usability issue).
  • Solution #2: In the sale page, make the booking button available even when no member has been selected. Once clicked, prompt to select a member or offer to peruse the list of bookings. Selecting an event would auto-select the member, and bring us to the usual process.
    • Can be combined with the previous solution.
    • Possible issue: This allows cashiers to do anything at any time while in the sale module and loses the contextual value of the process we've defined. One main problem in the POS industry is the constant desire to put all options on the page, flooding the user with too many options and making the POS hard to use and increases training costs.
  • Solution #3: Do the previous solutions and get rid of the top level events page.
    • If everything can be done from the sale module (and in most cases the sale module is where the cashier spends most of their day), why bother having a separate mode, that adds confusion.
    • Possible issue: cashiers won't instinctively feel that the sale page is where they want to go for event bookings, they might not even discover that events exist!
  • Solution #4: Cut the feature set to attack the root causes directly
    • If deposits are required, get rid of the feature that allows us to create temporary bookings.
      • Possible issue: people will complain that they can't do this anymore. As deposits are required, this is a case of tough love.
    • If deposits are never required, don't integrate with the sale module at all.
      • Possible issue: decrease the perceived value of the event module and lose a few potential reports (which are not in use at this point).

At this point, we'll probably end up implementing #1, #2, and #4. One of our core technological themes is usability, therefore we're willing to live with cutting a few features for an improved process. However, the issue is still up for discussion and I'd love to hear your opinions on this specific matter or on similar problems you've experience in your software startups.

Conclusion

We've built a flexible core to support the varied needs of franchisors. Everything works, but are we offering too many features to our customers? Should we cut back to make it simpler on everyone? In our case, I think so. If I had to sum up the usability problem discussed today into a high level lesson learned, I'd state the following: Don't separate use case scenarios into different groups if a configuration option invalidates the grouping premise.

During this holiday season, I think I'll take some time to re-read a few of my favorite books on usability:

  1. Steve Krug - Don't Make Me Think
  2. Joel Spolsky - User Interface Design for Programmers

Although these are great reads, they deal with the low-level user interface design problems, not with higher-level software engineering feature interaction problems in software product lines. I'd be curious to know if there any good resources on this subject (other than overly academic conference papers/workshops). I'd love to hear more first-hand, real-world experiences from other bloggers.

Easy to use software is like eggnog.

  • Reaching the desired results takes time. (The best eggnog takes at least three weeks to make).
  • Users will get a kick out of it. (With as much bourbon as milk, eggnog will definitely get people talking.)
  • At a high level, anyone can claim to be a pro. However, the devil is in the details.
  • You need to listen to your users to make the recipe better in the future.
  • What would the holiday season be without it? (okay, that doesn't make sense!)
kick it on DotNetKicks.com