Negotium App: Evaluating User Engagement and Content Quality Study
CoEx Lab at CMU Human-Computer Interaction Institute
Skills
Evaluative research, agile team
Timeline
2024 Spring, 3 months
Team
2 Designers, 2 Engineers, 1 Researcher, 1 PM
I worked as the UX researcher on leading a diary study for Negotium App, an interactive mobile application focusing on negotiation skills. I conducted the entire study myself, from protocol drafting to execution to result analysis. This project helped the team to gain insights into users’ usage behaviors, proposing solutions to enhance consistent user engagement and app content consistency.
As the first official pilot test of the MVP version of the app, this project provided the team with a clear direction for redesign and set the priorities of features. More importantly, this study validated the team’s effort, and we are now looking forward to conduct a long-term version testing.
9
Usability pain points
5
Common themes
4
Design flow
1
Change in strategy
What is Negotium?
Negotium is an app designed to empower women with the knowledge and strategies needed for effective negotiation in the workplace, classroom, or everyday life. Research shows that women negotiate less often than men, leading to inequalities like leadership and wage gaps. By providing scalable access to training tools, Negotium helps women advocate for themselves more effectively.
Project Goal
We just launched the app in TestFlight and incorporated new content/features since the last testing, but we were unsure if and how users will consistently engage with the app in their daily lives. The general goals are to:
Collect users’ feedback on learning content,
Identify motivations and issues for consistent engagement,
Evaluate the overall ease of use of the app in context.
How (consistently) will users engage with the app feature and learning content in their daily lives, and what factors influence this engagement?
METHODOLOGY
Choosing the Right Method
No one else in the team had experience with legit user studies before, so I had full control over the project and designed it myself.
Since our goal is to see if users will consistently use our app, I decided to use Diary Study because we wanted to gain contextual knowledge and learn users’ natural behaviors in consistent way. It was important to go beyond a one-time testing session to learn users/leaners’ feedback on the app feature and learning modules over time with our high-fidelity prototype.
The team was onboard with me -- diary study allowed us to observe trends and frequency over time and gain deeper understanding into contextual needs, while minimizing the bias caused by our own intervention.
Version 1 & Internal Testing
Well, people forgot...
Due to budget and time constraints, we decided to recruit 5 participants, offering each $20 compensation.
I initiated internal testing where team members completed one module daily and fill out After-Session feedback every time they finish a learning session in the app and an End-Of-Day (EOD) feedback form.
However, almost all the members forgot to finish the After-Session form and only filled out the EOD form when I sent out an email remind at 5pm everyday. How might we keep minimum interaction while reminding users to log in data everyday?
Version 2
Conducting research
Based on internal testing results, I refined the feedback process and made couple changes:
- Combined after-session and EOD feedback to prevent members from forgetting one and to reduce overload.
- Revised questions to exclude those inappropriate for the daily feedback form.
- Created a built-in button for the form link and removed the email reminder.
User Testing Flow V1
User Testing Flow V2
Now, the Diary Study spanned a week, with an initial 30-minute onboarding session and $5 compensation for participants. Throughout the week, whether the participants finished something or did not do anything, they need to submit a daily feedback form. At the study's end, participants underwent a 45-minute debrief interview for $15 compensation.
After confirming with the rest of the team, I created a screener survey as well as a protocol draft. We thought that this is effortless for the users while making sure that we capture at-the-moment thoughts. Due to budget and time constraints, we recruited 5 participants through social media based on our targeted users.
SYNTHESIS & RESULT
Analyzing the Key Insights
After collecting all the necessary results, we conducted a synthesis workshop and used affinity clustering to analyze the data. Based on the daily forms data, I created a persona that encapsulates common user habits (how they typically use the app), users’ preferences and fears (their interests and primary concerns), and their motivations and needs related to the app.
Overall, we had an 80% engagement rate and relatively high interests in recommending it to others (7.5/10)
While most of the users stick to the end, there was a significant drop on the third and forth day in terms of both response rate and “did nothing” responses in the form.
We also had 5 main insights based on the clustering
Based on our affinity diagramming result, I concluded 5 main insights regarding learning Negotium content, app consistent engagements, general user pattern, app visual, and app usage & features feedback.
Scroll to see more
Insight 1: Content should be connected with real life scenarios.
While learning terminologies is a good starting point, content should connect with real life. Users were overwhelmed by lots of terminologies in the app and might forget due to lack of review or missing connection with real life scenarios.
Insight 2: Users appreciated the hand-on learning experiences.
All users thought that, unlike text and short responses, quizzes & chat were much better in helping them to engage in the app because of their interactiveness, how they provide different negotiation strategies, and provide practice opportunities.
Insight 3: Improving the visibility and clarity of app overall features.
Overall, there was a bad visibility and clarity for other app features. Users were confused about the Flight Log and Daily Review and wanted clarification or prompting for these. Half of the participants did not know what “my profile” features were for.
Insight 4: Connect gamification features with app content.
A lack of connection between the current gamification system and the main content has resulted in less motivated users. Participants said that the gamification did not motivate them because they are not well integrated with the main content.
Insight 5: Building a collaborative learning experience
Users enjoyed using the app, and they wanted to find partners or a community (rather than trying to compete with someone else) to learn negotiation together.
"I'm working in a project team for one of my mechanics classes, and I think just like seeing how the simulations communicate with different people has helped me figure out what the best way to communicate with my teammates is. " -- P5
Deliverable
I first prepared a slides-show presentation with the team and received positive feedback towards the richness of the data. To further push for actions, I organized all the results according to different insight sections in a list and held an ideation session. All the team members gathered together to discuss if and how we want to proceed with each finding!
NEXT STEP
Project Outcome & Impact
As the first official pilot test of the MVP version of the app, this project provided the team with a clear direction for redesign and set the priorities of features. More importantly, this study validated the team’s effort, further proving the effectiveness of the app.
Key Change of Strategy: collaboration rather than competition
One major strategic change was the shift away from creating a leaderboard to encourage competition and user engagement. From this research, participants indicated that they were more interested in negotiating with others and collaborating rather than competing. While further research might be needed to validate this finding, the team decided to focus on improving the gamification and its integration to resolve one of the bigger usability issues in the app.
What should we do next?
The next step after this research would be resolving small usability bugs, conducting content review to ensure consistency, and re-designing the gamification systems & profile pages.The team is also interested in conducting a longer version of the diary study to gain more insights into long term learning.
My takeaway
Importance of checking testings details
There was a miscommunication between me and the engineer, and I didn't know we only had an iOS version, so I didn't include that in the screener. This led to serious issues with recruiting participants as not all of them had phones with iOS system. Although we found ways to help each participant obtain a device, this was a lesson for me to make sure that I needed to check every single details of the testing process.
Being a support for the team as well as be supported by the team
I joined the team as the sole researcher, while the rest of the team members had been collaborating with each other for a long time. Because it was such a small team, I decided to start by learning contextual information and identifying what people were frustrated or concerned about before making any research plans. This not only helped me to adjust research focus and synthesize results that were more useful for each member but also helped in gaining support and trust from them.
