Introducing Infomaki: Bite-sized Usability Testing
Written on February 16, 2009 by Michael Lascarides
As we roar into a 2009 that is filled with major Labs projects, we have a constant need to get as much input from our users as possible. This need is fulfilled in many ways, from statistical analysis to focus groups to in-depth personal interviews to online surveys. Over the last month, however, it has become apparent that we needed a tool in our kit that would allow us to get simple usability questions in front of users with a minimum of fuss.
To be sure, there are strategic questions that require a lot of setup and deep knowledge about the respondents. We have a lot of those questions, and we are asking them in all their properly-sampled, audience-segmented glory, often with the assistance of consultants and our Strategy department. But often, what is needed is just a sanity check– a reassurance to our team that we are on the right track.
Our design sessions frequently result in debate about which of two words is more compelling or accurate for our users, or whether a particular button is noticeable in a particular location. When we can, we test designs on real people using paper or digital prototypes, but it is impractical to test every day; sitting down with real people is not always as simple as you’d expect, what with the schedules of busy New Yorkers.
Given that we have a large amount of traffic on our web sites, and an audience willing to help us where they can, we’ve naturally been drawn to web-based usability tests to fill this gap in our testing regimen. After all, once a web-based test is set up, we can go about our jobs while data points quietly accumulate in the background. We’ve used SurveyMonkey in the past for full-fledged traditional surveys, and we’ve also evaluated OptimalSort for online card sorting, and been inspired by the Five Second Test. But none of them were a perfect fit, for reasons including complicated interfaces, inflexible setup, privacy policy hassles, and/or lack of a way to embed a link back to our site when the test is completed.
So, we set out to create our own rapid-testing usability laboratory from scratch, and last Tuesday we launched it, in rough beta form. Nicknamed Infomaki*, it’s showing a lot of potential even in its first 48 hours.
It’s not groundbreaking technology. Built on a Rails back-end (my rapid-prototyping framework of choice), it currently supports two kinds of tests: standard multiple choice (with optional “Other:” box) and a “Where would you click…?” screenshot (see image above) that records click locations. But it’s a bit different from the other tools mentioned above in that:
1. Each question is free-standing
2. The user can answer as many or as few questions as they want
3. It’s incredibly easy for the team to insert questions immediately
The main idea is to have a pool of dozens of independent questions available at any given time, from which a respondent will see a random selection.
The pitch we’re using (in a thin 12px-high banner at the top of our site header) is “Answer a single question and help us improve our web site!” It’s simple, it’s a breeze. Then immediately after the easy question is answered, we ask politely if they’d like to answer another. As such, we’re finding that even with the “one question” pitch, the average number of questions answered is around 8, and an astonishing 90% of users answered more than one question. It’s the potato chip of surveys: can’t eat just one.
So far, this “no commitment” approach seems to be outpacing our traditional surveys. As a point of comparison, the last time we ran a survey off of the NYPL site (using the same banner at the top of the site), it was a regular, please-answer-these-questions pitch leading to a 8-page, 19 question SurveyMonkey-powered survey. Over 14 days, that survey received 7,341 individual answers to questions from 520 respondents, 60% of whom completed the whole survey. Infomaki, on the other hand, garnered over 6,900 answers from 840 respondents in its initial 48-hour maiden voyage.
This makes a difference in the way these survey questions can be approached: rapid feedback leads to rapid turnover. We’re mining the vast middle ground between putting a full survey in the field with full protocols and methodologies, and asking people in the office “Does this look right to you?” Designers can post a question in the morning and have several dozen responses by the afternoon. Ideally, we’d like to work out a way that this tool can be “baked in” to the new NYPL.org so that user feedback becomes an ongoing, always-on process.
Granted, there are issues with this approach. For starters, we don’t get a rounded profile of all Library users. It’s safe to assume that Infomaki respondents are among our more web-savvy patrons. But as long as we’re aware of that limitation, it might not be such a bad thing when posting questions directly related to the web site. And future releases will include the ability to add demographic questions into the mix, to build a richer profile of the users who are answering multiple questions.
We’ve already received a lot of feedback, despite not even blogging the launch (one little post on Twitter was the only announcement made). Most seriously, there appears to be a database-related bug where certain users are seeing repeated questions (no visitor is supposed to see the same question more than once), and we’ve taken down the link to the app while we work this out (though you can still feel free to click around). Some have also complained about the lack of a “I have no idea where to click” button in the screenshot questions, but I like forcing the respondent to make their best guess in these circumstances. We’ll probably address the issue with language to this effect rather than a change in the logic.
Internally, the application is optimized to store all results from varied types of questions in a single common database table, which makes it extremely easy to analyze response statistics. Our plan is to release the findings to any interested parties (any grad students out there want some raw statistical data?).
This is a brand-new, “zero-point” release of this application, and we’ll be making LOTS of changes in the upcoming weeks, including support for a wider variety of questions, cleanup to interfaces, and possibly some interstitial entertainment (Library trivia questions, anyone?) to keep people engaged with the application as long as possible.
More significantly, we plan a full Open Source release of the application very soon, so you can download and tinker with your own copy. Stay tuned to this blog for release details.
* Oh, and in case you’re wondering, the name Infomaki is a pun on Japanese food (a not-insignificant amount of which fuels the Digital Experience Group daily). “Maki” are those hand rolls of rice, seaweed and tasty treats, rolled up together and chopped into bite-sized cross-sections. Seemed somehow appropriate to what this tool does to user feedback.
UPDATE: We’ve put a fix in place for the repeating-questions bug. If you get the same question multiple times in the same session, please note it in the comments. Watch out: many of the questions are VERY SIMILAR, which is by design.
Filed in: Open source, Ruby on Rails, Usability, User testing, infomaki.
[...] more about Infomaki here. SHARETHIS.addEntry({ title: “Design in the Wild: Infomaki’s Bite-sized Usability Testing”, [...]
[...] Usability The Digital Experience Group of the New York Public Libraries (NYPL) decided to try a bold experiment. Knowing that they wanted as much user feedback as possible and that they had willing [...]
[...] end to gather survey data, or if they working towards something like we at NYPL Labs are doing with Infomaki, our new usability tool developed by Michael Lascarides, our user [...]
[...] Labs Infomaki: A lightweight usability testing service The New York Public Library Labs has begun testing a new light usability testing system to supplement their formal tests, inspired by the Five Second Test method which was devised by [...]
What a great idea! Your tool could be used not only in libraries, but also in businesses and other ventures! When is your goal to have the open source version ready for release?
[...] Infomaki from NYPL Labs can be found here [...]
“I have no idea where to click” – would be an interesting option if there was a way to tell if someone had stalled – and can’t decide where that information might be.
Hi,
I’m a member of Optimal Workshop. We are the team behind OptimalSort. Thanks for your interest in our card sorting application. Sorry it didn’t quite work out for you though.
Having said that we also saw a need for a quick tool for performing web-based usability sanity checks. As a result we built Chalkmark, a tool which does much the same thing as Infomaki. We’ve been in beta since late last year and at the moment the tool is free for anyone who needs to use it. We have also just launched another simple tool called Treejack for validating information architectures.
Anyway before I sound like I’m spamming your comments thread, we’d like to congratulate you on the success of the tool, it looks great. It is also cool to see people independently coming to the same conclusions, it hopefully shows we are both on the right track.
Steve
@Todd We need to go about setting up license & repository, then make sure our unit tests are pretty robust, but there’s a lot of enthusiasm here for an OSS release. I’d say “weeks” rather than “months”.
@Julie For now we’ve added a caption that says “No Idea where to click? Make your best guess”, but we’re still requiring a click. We’d like to see what people try first when they’re baffled rather than just opting for “I’m baffled”.
@Steve No worries about the “spam” :) We love Optimal Sort, but we have very strict rules here about privacy on third party sites, and the requirement of email entry for participants was ultimately a deal-killer for us. Also, once we decided to launch this site, we built a roadmap for extending it (with user stories, segmenting, etc etc) which we couldn’t do with a hosted product. But even though it didn’t work out for us, I highly recommend your tools to anyone else who might be reading this!
[...] Introducing Infomaki: Bite-sized Usability Testing | NYPL Labs (tags: tools testing usability webben nypl testverktyg) [...]
1. The bite sized usability test is kick-ass, superb. I found it by accident and could see that it took me ten seconds to find out where i would donate money. Obviously the NYPL home page is not especially user-focused.
One comment – there was no ability to create an unstructured comment, so it took another 90 seconds to find this blog.
How do I provide specific input to this group?
Thanks for being so persistent, Peter! The intention wasn’t to capture unstructured feedback, so we’re not asking for it (it takes much longer to process… the structure is part of the fast turnaround time). But feedback is always appreciated, so feel feel to post on this blog or email me directly at michael_lascarides at nypl dot org.
[...] Way more detail here: http://labs.nypl.org/2009/02/16/introducing-infomaki-bite-sized-usability-testing/ [...]
[...] 0 Comments Categories: Uncategorized The New York Public Library Labs just introduced a nifty new tool built on Ruby for quick usability testing. The tool, infomaki, is being touted as a [...]
I just run 20 or so questions on Infomaki, and I believe I received one particular question three times. I believe the question was “Where would you click to find blogs by NYPL staff” or something along those lines. I didn’t notice any differences between the images presented, but I wasn’t concentrating on that. Also, I stopped doing the tests because I was asked a survey question (”How old were you when you first went to a library?”) and when I responded it took me to an error page.
Thanks for the heads-up, Ian! An update with some new functionality earlier in the week seems to have broken something. We’ve taken down the link to the site while we roll it back into the garage for repair. We’ll be back shortly!
[...] Introducing Infomaki: Bite-sized Usability Testing 


share: [...]
I just found this blog from a post on the Google Analytics blog and am soaking up all the great info here. This post really interested me because it looks like a great way to get a lot of valuable information.
One question: are you looking at how long people take to click on their answer? While it’s definitely useful to see where people click, how long it takes them to find their answer is also revealing. I see that you have Google Analytics running on each page so the average time spent should be easily retrievable through their reports.
It would be great if you could share any insights you learn in a later blog post.
[...] even more lightweight usability testing goodness, take a look at the NPYL Labs’ infomaki. « a short history of internet scare [...]
[...] eventually summed up and displayed as heat maps. Infomaki has been developed by the people over at NYPL Labs, and just a few days ago finally went open source (requires Ruby on Rails 2.2+). One interesting [...]
[...] in tests of other interfaces and you will come away with useful ideas to apply to your own design. NYPL’s Infomaki does the same thing. They’ve created their own usability test site, and I spent a fair amount [...]
[...] then discovered NYPL Labs, a section of the NYPL website dedicated to the NYPL digital experience. There I was able to read [...]
[...] worthwhile announcement from the Labs blog is their development of an open source web usability survey (which they are calling Infomaki). They survey only asks users to answer one question at a time, [...]
The age ranges the survey asks for have an overlap, [19-24] [24-29]. Being 24, I fit both, but could select only one.
[...] 8 – Infomaki [...]
That is an awfully astounding column you’ve posted.Thanks a lot for that a fantastically amazing post!
wholesale new era hats
Sorensen Pontiac Medicine Hat RAVINGSERAPH.COM
http://ravingseraph.com/ – buy xanax online no prescriptionhttp://ravingseraph.com/ – buy xanax without prescription I used this last year and still have some of it left over for panic attacks.
What Are Drug Factories buy cheap meridia Diet pill wonder http://www.texasstatecycling.com/ – meridia sibutramine
Regenerative Knee Medicine meridia drug Moreover, take a note that you just don?t prefer this drug if you want to become perfectly healthy an overnight. http://www.texasstatecycling.com/ – meridia sibutramine
Why Practice Medicine In Private generic phentermine online Buy Phentermine online and save your money! Worldwide delivery, no prescription, no hidden fees! http://www.gorgamonkeys.com/ – phentermine diet pills
Pepcid Ac Maximum Strength Tablets http://freemyps3.com/ – order meridia online buy sibutramine without prescription These drugs include Phentermine and also include prescription drugs, and all over the counter drugs.
Medication For Laryngitis http://www.redhotchiliocean.com/ – provigil no prescription medication – Provigil should never be taken by patients that are under the age of 16. provigil drug
Facts About Drug Screening http://www.hatsformen.org/ – lunesta eszopiclone For those who are struggling with sleep onset a dose of one mg is usually sufficient. purchase lunesta
Scabies Medicine Shampoo zoloft sale The dosages are different and only a qualified medical practitioner can establish beyond doubt what strength will befit your condition and yield the most practical results. http://www.originalobamawristband.com/ – buy zoloft
Picture Of Morphine Drug topamax drug Buy Topamax (Topiramate) at reputable online pharmacy and save extra money! No prescription! No hidden fees! buy generic topamax Topamax treats many different conditions in both adults as well as children, as indicated. http://www.ourdailybreadcafe.com/ – topamax cost
Zanax Drug Side Effects http://www.livingfrontroyal.com/ – meridia diet If your doctor failed to advise an exercise routine or failed to inform you about how you can achieve a reduced calorie diet, please ask your doctor about it. order meridia online
Wilbert Brown Drug Conviction acomplia prescription online This drug is a first new class therapeutic agents called Cannabinoid-1 Receptor blocker that treats obesity and related conditions. http://www.jomtienbeachhotel.com/ – discount generic acomplia
Root Canal Medication http://www.cheaplipitorforsale.com/ – atorvastatin online This is because negligence on the part of the patient in communicating changes in physical conditions to the doctor might result in sudden kidney failure too. lipitor 40 mg
Thanks, http://longcreeksd.com/ – xanax no prescription Xanax has also been used for anxiety that is associated with moderate to severe depression. xanax online pharmacy
[...] Introducing Infomaki: Bite-sized Usability Testing | NYPL Labs [...]
The main Benefit of Exercise is your health is good and your body will remain fit and you looks good,..
Its also good to reduce fats of the body,.