Introducing Infomaki: Bite-sized Usability Testing
Written on February 16, 2009 by Michael Lascarides
As we roar into a 2009 that is filled with major Labs projects, we have a constant need to get as much input from our users as possible. This need is fulfilled in many ways, from statistical analysis to focus groups to in-depth personal interviews to online surveys. Over the last month, however, it has become apparent that we needed a tool in our kit that would allow us to get simple usability questions in front of users with a minimum of fuss.
To be sure, there are strategic questions that require a lot of setup and deep knowledge about the respondents. We have a lot of those questions, and we are asking them in all their properly-sampled, audience-segmented glory, often with the assistance of consultants and our Strategy department. But often, what is needed is just a sanity check– a reassurance to our team that we are on the right track.
Our design sessions frequently result in debate about which of two words is more compelling or accurate for our users, or whether a particular button is noticeable in a particular location. When we can, we test designs on real people using paper or digital prototypes, but it is impractical to test every day; sitting down with real people is not always as simple as you’d expect, what with the schedules of busy New Yorkers.
Given that we have a large amount of traffic on our web sites, and an audience willing to help us where they can, we’ve naturally been drawn to web-based usability tests to fill this gap in our testing regimen. After all, once a web-based test is set up, we can go about our jobs while data points quietly accumulate in the background. We’ve used SurveyMonkey in the past for full-fledged traditional surveys, and we’ve also evaluated OptimalSort for online card sorting, and been inspired by the Five Second Test. But none of them were a perfect fit, for reasons including complicated interfaces, inflexible setup, privacy policy hassles, and/or lack of a way to embed a link back to our site when the test is completed.
So, we set out to create our own rapid-testing usability laboratory from scratch, and last Tuesday we launched it, in rough beta form. Nicknamed Infomaki*, it’s showing a lot of potential even in its first 48 hours.
It’s not groundbreaking technology. Built on a Rails back-end (my rapid-prototyping framework of choice), it currently supports two kinds of tests: standard multiple choice (with optional “Other:” box) and a “Where would you click…?” screenshot (see image above) that records click locations. But it’s a bit different from the other tools mentioned above in that:
1. Each question is free-standing
2. The user can answer as many or as few questions as they want
3. It’s incredibly easy for the team to insert questions immediately
The main idea is to have a pool of dozens of independent questions available at any given time, from which a respondent will see a random selection.
The pitch we’re using (in a thin 12px-high banner at the top of our site header) is “Answer a single question and help us improve our web site!” It’s simple, it’s a breeze. Then immediately after the easy question is answered, we ask politely if they’d like to answer another. As such, we’re finding that even with the “one question” pitch, the average number of questions answered is around 8, and an astonishing 90% of users answered more than one question. It’s the potato chip of surveys: can’t eat just one.
So far, this “no commitment” approach seems to be outpacing our traditional surveys. As a point of comparison, the last time we ran a survey off of the NYPL site (using the same banner at the top of the site), it was a regular, please-answer-these-questions pitch leading to a 8-page, 19 question SurveyMonkey-powered survey. Over 14 days, that survey received 7,341 individual answers to questions from 520 respondents, 60% of whom completed the whole survey. Infomaki, on the other hand, garnered over 6,900 answers from 840 respondents in its initial 48-hour maiden voyage.
This makes a difference in the way these survey questions can be approached: rapid feedback leads to rapid turnover. We’re mining the vast middle ground between putting a full survey in the field with full protocols and methodologies, and asking people in the office “Does this look right to you?” Designers can post a question in the morning and have several dozen responses by the afternoon. Ideally, we’d like to work out a way that this tool can be “baked in” to the new NYPL.org so that user feedback becomes an ongoing, always-on process.
Granted, there are issues with this approach. For starters, we don’t get a rounded profile of all Library users. It’s safe to assume that Infomaki respondents are among our more web-savvy patrons. But as long as we’re aware of that limitation, it might not be such a bad thing when posting questions directly related to the web site. And future releases will include the ability to add demographic questions into the mix, to build a richer profile of the users who are answering multiple questions.
We’ve already received a lot of feedback, despite not even blogging the launch (one little post on Twitter was the only announcement made). Most seriously, there appears to be a database-related bug where certain users are seeing repeated questions (no visitor is supposed to see the same question more than once), and we’ve taken down the link to the app while we work this out (though you can still feel free to click around). Some have also complained about the lack of a “I have no idea where to click” button in the screenshot questions, but I like forcing the respondent to make their best guess in these circumstances. We’ll probably address the issue with language to this effect rather than a change in the logic.
Internally, the application is optimized to store all results from varied types of questions in a single common database table, which makes it extremely easy to analyze response statistics. Our plan is to release the findings to any interested parties (any grad students out there want some raw statistical data?).
This is a brand-new, “zero-point” release of this application, and we’ll be making LOTS of changes in the upcoming weeks, including support for a wider variety of questions, cleanup to interfaces, and possibly some interstitial entertainment (Library trivia questions, anyone?) to keep people engaged with the application as long as possible.
More significantly, we plan a full Open Source release of the application very soon, so you can download and tinker with your own copy. Stay tuned to this blog for release details.
* Oh, and in case you’re wondering, the name Infomaki is a pun on Japanese food (a not-insignificant amount of which fuels the Digital Experience Group daily). “Maki” are those hand rolls of rice, seaweed and tasty treats, rolled up together and chopped into bite-sized cross-sections. Seemed somehow appropriate to what this tool does to user feedback.
UPDATE: We’ve put a fix in place for the repeating-questions bug. If you get the same question multiple times in the same session, please note it in the comments. Watch out: many of the questions are VERY SIMILAR, which is by design.
Filed in: Open source, Ruby on Rails, Usability, User testing, infomaki.
[...] more about Infomaki here. SHARETHIS.addEntry({ title: “Design in the Wild: Infomaki’s Bite-sized Usability Testing”, [...]
[...] Usability The Digital Experience Group of the New York Public Libraries (NYPL) decided to try a bold experiment. Knowing that they wanted as much user feedback as possible and that they had willing [...]
[...] end to gather survey data, or if they working towards something like we at NYPL Labs are doing with Infomaki, our new usability tool developed by Michael Lascarides, our user [...]
[...] Labs Infomaki: A lightweight usability testing service The New York Public Library Labs has begun testing a new light usability testing system to supplement their formal tests, inspired by the Five Second Test method which was devised by [...]
What a great idea! Your tool could be used not only in libraries, but also in businesses and other ventures! When is your goal to have the open source version ready for release?
[...] Infomaki from NYPL Labs can be found here [...]
“I have no idea where to click” – would be an interesting option if there was a way to tell if someone had stalled – and can’t decide where that information might be.
Hi,
I’m a member of Optimal Workshop. We are the team behind OptimalSort. Thanks for your interest in our card sorting application. Sorry it didn’t quite work out for you though.
Having said that we also saw a need for a quick tool for performing web-based usability sanity checks. As a result we built Chalkmark, a tool which does much the same thing as Infomaki. We’ve been in beta since late last year and at the moment the tool is free for anyone who needs to use it. We have also just launched another simple tool called Treejack for validating information architectures.
Anyway before I sound like I’m spamming your comments thread, we’d like to congratulate you on the success of the tool, it looks great. It is also cool to see people independently coming to the same conclusions, it hopefully shows we are both on the right track.
Steve
@Todd We need to go about setting up license & repository, then make sure our unit tests are pretty robust, but there’s a lot of enthusiasm here for an OSS release. I’d say “weeks” rather than “months”.
@Julie For now we’ve added a caption that says “No Idea where to click? Make your best guess”, but we’re still requiring a click. We’d like to see what people try first when they’re baffled rather than just opting for “I’m baffled”.
@Steve No worries about the “spam” :) We love Optimal Sort, but we have very strict rules here about privacy on third party sites, and the requirement of email entry for participants was ultimately a deal-killer for us. Also, once we decided to launch this site, we built a roadmap for extending it (with user stories, segmenting, etc etc) which we couldn’t do with a hosted product. But even though it didn’t work out for us, I highly recommend your tools to anyone else who might be reading this!
[...] Introducing Infomaki: Bite-sized Usability Testing | NYPL Labs (tags: tools testing usability webben nypl testverktyg) [...]
1. The bite sized usability test is kick-ass, superb. I found it by accident and could see that it took me ten seconds to find out where i would donate money. Obviously the NYPL home page is not especially user-focused.
One comment – there was no ability to create an unstructured comment, so it took another 90 seconds to find this blog.
How do I provide specific input to this group?
Thanks for being so persistent, Peter! The intention wasn’t to capture unstructured feedback, so we’re not asking for it (it takes much longer to process… the structure is part of the fast turnaround time). But feedback is always appreciated, so feel feel to post on this blog or email me directly at michael_lascarides at nypl dot org.
[...] Way more detail here: http://labs.nypl.org/2009/02/16/introducing-infomaki-bite-sized-usability-testing/ [...]
[...] 0 Comments Categories: Uncategorized The New York Public Library Labs just introduced a nifty new tool built on Ruby for quick usability testing. The tool, infomaki, is being touted as a [...]
I just run 20 or so questions on Infomaki, and I believe I received one particular question three times. I believe the question was “Where would you click to find blogs by NYPL staff” or something along those lines. I didn’t notice any differences between the images presented, but I wasn’t concentrating on that. Also, I stopped doing the tests because I was asked a survey question (”How old were you when you first went to a library?”) and when I responded it took me to an error page.
Thanks for the heads-up, Ian! An update with some new functionality earlier in the week seems to have broken something. We’ve taken down the link to the site while we roll it back into the garage for repair. We’ll be back shortly!
[...] Introducing Infomaki: Bite-sized Usability Testing 


share: [...]
I just found this blog from a post on the Google Analytics blog and am soaking up all the great info here. This post really interested me because it looks like a great way to get a lot of valuable information.
One question: are you looking at how long people take to click on their answer? While it’s definitely useful to see where people click, how long it takes them to find their answer is also revealing. I see that you have Google Analytics running on each page so the average time spent should be easily retrievable through their reports.
It would be great if you could share any insights you learn in a later blog post.
[...] even more lightweight usability testing goodness, take a look at the NPYL Labs’ infomaki. « a short history of internet scare [...]
[...] eventually summed up and displayed as heat maps. Infomaki has been developed by the people over at NYPL Labs, and just a few days ago finally went open source (requires Ruby on Rails 2.2+). One interesting [...]
[...] in tests of other interfaces and you will come away with useful ideas to apply to your own design. NYPL’s Infomaki does the same thing. They’ve created their own usability test site, and I spent a fair amount [...]
[...] then discovered NYPL Labs, a section of the NYPL website dedicated to the NYPL digital experience. There I was able to read [...]
[...] worthwhile announcement from the Labs blog is their development of an open source web usability survey (which they are calling Infomaki). They survey only asks users to answer one question at a time, [...]
The age ranges the survey asks for have an overlap, [19-24] [24-29]. Being 24, I fit both, but could select only one.