Usability testing – the value in getting live reactions

Usability Evangelists

Usability testing has been an integral part of the team’s output for well over a year now. It is safe to say it has changed the way we see how users look at our software. Typically, whenever we undertake a usability test, it usually consists of a usability ‘show and tell’ where the recorded tests are relayed to the rest of the project team. Everyone makes notes as the tests are played and then we discuss our thoughts by debriefing afterwards. It worked really well, but the project team were missing a sense of connection with the person using the software. We knew we could do better.

Seize the moment

We’ve recently been working on a new piece of software which is an asset register for all of our systems. The main challenge for the team was to make a solution which gets some very specific and technical information and try to make this as user friendly as possible. It wasn’t what information was being captured that we were redesigning, but how we were capturing it.

Because this software was much more about the user’s experience, we wanted to have that connection with the users and so opted to seize the moment and undertake live usability tests.

…and now for the technical bit

We’re going to get a little geeky for a moment – if you glaze over when you hear computing acronyms, or think a Raspberry Pi is something you have after your meal, skip to the next section.

************** START OF TECHNICAL DESCRIPTION **************

Streaming live tests would obviously have some technical hurdles that needed to be overcome. Despite our offices offering free Wi-Fi, understandably it is restricts machines from accessing hosted services on the same network. We needed to come up with a portable and quick means to stream the sessions that was self-contained.

Using a Raspberry Pi and a spare Wi-Fi router, we created a network that a laptop could connect to wirelessly and either send or receive streams. The Raspberry Pi has a non-GUI version of Raspian installed with a RTMP server which loaded on boot so that it was truly headless. After a bit of testing we found that the wireless could be a bit of a bottleneck so we connected the Pi to the router via an Ethernet cable and ensured devices were connected with the wireless-N standard. What was even better was that we noticed we could power the Raspberry Pi with the USB port on the router meaning it only ran off one electrical outlet – niiice!

We used two laptops to actually conduct the streaming. One with OBS installed which recorded and encoded the feed then streamed it to the RTMP Raspberry Pi server. The other connected to the stream so it could be connected to a bigger display for the audience.

If anyone wants more detail on how this was set up, feel free to contact me.


*************** END OF TECHNICAL SECTION *********************

The Ceremony

We booked six appointments for the day. The tests were conducted in one meeting room and the project team were hosted in an adjacent room where they observed. The tests ran pretty much the same as a standard usability test except that we gave the project team an opportunity to ask questions at the end. This was something we wanted to utilise because of the nature of the software and it was definitely what brought most value to conducting the tests live.

It is always interesting to see the reactions of people who have never experienced a usability test before. Everyone always gets value out of observing – whether it is to understand how people interact to the software, the content that is in it or just the desirability of the product. We also found that by having attendees know that the tests were live also resolved one of our biggest problems – people talking over the tests. Attendees seemed to be much more engaged, concentrating to the tests more than compared to recorded sessions – perhaps because they thought if they missed something, they knew they couldn’t pause the video or go back and hear it again.

Running live tests also had one huge benefit that we weren’t anticipating. Because we had the project team to hand, if a problem appeared in a number of tests, it was possible to quickly change the software and test the tweaks immediately. Having a much more proactive approach to the way we responded to usability issues meant we could test, ideate, apply and test again in a single sitting – guerrilla iteration.


Running live usability tests definitely have a place in our UX toolkit and I would go as far as to say it is the preferred means of running usability tests (….here comes the but…) BUT live tests do take so much more time to prepare. The flexible nature of a recorded test means that we can get feedback very quickly about something we are not too sure on – doing this live adds so much more overhead. Running live tests does have its place, making more sense as a series of tests throughout a day so that it becomes much more time effective.

Because of this, recorded tests aren’t going anywhere soon, but it has a new competitor when it comes to getting valuable feedback from users.

The (slightly technical) next steps

One of the side affects to having a stand alone wireless network to facilitate the stream is that there is no internet connection. This meant that because there was no means to a remotely hosted service, we had to host all of the software in a development environment on the streaming computer. This added a lot of load to the streaming computer, because not only was it encoding the video stream, but also running a development environment. Also, because of the geographical limitation of the wireless network, it meant we had to conduct the tests in close proximity to where people were observing.

I have a couple of ideas to how we can improve the set-up for the next set of testing, so watch this space!

Tom Thornton


    • Thanks Neil! We try not to be in the same room as where the tests are (even for recorded ones) so that we don’t add bias to the subject. We’re really trying to understand how a person would use the software without any help, so to have 10 or so people watching behind them could mean it changes the person’s habits – we want to make the whole thing as informal as possible as we’re not testing them, we’re testing the software. Another benefit is that we can talk freely in the other room if we want to point any observations out, however that’s a bit of a double edged sword as I touched on in the blog.

      If you haven’t already heard of it, I suggest a great book called Rocket Surgery Made Easy – it is great if you’re thinking about introducing usability tests to your organisation.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s