Exploring the Impact of Culture on UX Research Methods

Introduction

About This Project

I completed this research as my senior thesis for the University Honors Program at CSUSB under the supervision of Associate Professor Rob Ray. Over three semesters, I developed a research topic, conducted a literature review, designed the study, recruited and screened participants, carried out testing sessions, and analyzed data. I also presented my research at CSUSB's 12th annual Meeting of the Minds.

This research was approved by the CSUSB Institutional Review Board.

Summary

UX is a traditionally Western field. As it has grown and become practiced internationally, many researchers have questioned whether UX research methods are as effective in non-Western cultures. Existing research suggests that culture does have an impact on UX research methods [2, 3, 4, 5, 6]. However, many of these studies compare methods across broad, international groups [2, 3, 4]. As a result, their findings may be limited to the dominant cultural groups of each country [1]. Additionally, while there are studies that compare UX research methods across cultural groups within an individual country, these studies tend to occur outside of the West [3, 5]. In general, I found a Western vs non-Western framing and an assumption that traditional UX research methods are effective across all of the West.

I tested three UX research methods with Black, Asian, and Latinx women. In a mock usability study, I asked participants to think aloud, complete a survey, and then interviewed them. Then, in a follow-up interview, we discussed their experiences with each of the three methods. Based on these discussions, the Black, Asian, and Latinx women in this study seem to have experienced UX research methods differently— Black and Latinx women experienced tensions during the think aloud, while Asian women did not.

Literature Review

What do we know about culture and UX research methods?

In my research, I found two types of studies that investigate the role of culture in UX research. The first type of study is what I call the “international study.” International studies test and compare UX research methods across multiple countries. One example of this is a 2002 study by Evers. In this study, Evers tested the think aloud, survey, and interview with English, North American, Dutch, and Japanese participants.

The second type of study is what I call the “national study.” National studies also test UX research methods across cultural groups. However, these studies have a narrower scope— they compare cultural groups within one country. For example, a 2011 study by Hsieh tested the survey and interview with younger and older Taiwanese participants. There have been similar studies in Malaysia [3], India [3], and Namibia [5].

Both national and international studies have found that culture does have an impact on the efficacy of UX research methods. Evers found that surveys posed problems for English, Dutch, and Japanese participants. Hsieh found that younger Taiwanese participants were more likely to give feedback and criticism than older Taiwanese participants.

What don't we know?

There are some issues to consider with the research so far.

First, international studies lack specificity. Because they compare such broad groups, it's important to question who exactly is being represented. Who were the North American participants in Evers' study? What does all of North America have in common that the entire continent was being compared to individual countries? As a Black and Mexican-American person, these were my first questions. Later, I found that Evers was referring to US Americans. Although I was initially mistaken, there was still a question of representation. And, while I first asked this question of Evers' study, it's a valid question for any of the international studies. How well were marginalized communities represented in these country-level groups?

National studies don't necessarily have this problem because of their narrow scope. However, it's worth noting that these studies occur in non-Western countries. That means there is a level of specificity that we aren't engaging with in the West. In the case of the US, we have the findings from Evers' international study. How might our findings differ if we specifically centered the experiences of marginalized US Americans? These questions highlight the need for a national study in the US.

Methods

We tested UX research methods with...

In this study, we tested UX research methods with Black, Asian, and Latinx women. All of our participants were students at California State University, San Bernardino. Their ages ranged from 18 to 24 years old. It's important to note that none of them were Art or Design majors. This was to ensure that they weren't overly familiar with web design concepts or the critiquing process.

We did this by...

Mock Usability Study

First, we conducted a mock usability study for ConnectIE.org's mobile website. In the usability study, I asked participants to complete think aloud tasks, fill out a survey, and then interviewed them. At the end of the usability study, I asked them to briefly rank each of the methods from most to least favorite. Ranking the methods wasn't a metric— it was a strategy to open up discussion in the follow-up session. It was also a way to avoid framing any of the methods as problematic.

Follow-Up Interview

In the follow-up, I asked participants to explain their rankings. We discussed their attitudes toward each of the methods in detail. Then, we talked about any behaviors I observed during the study. After discussing the methods, we talked about the role of culture in the study.

Findings

Black, Asian, and Latinx women believe culture can have an impact

Responses to the question, "Do you think a person's culture can play a role in the way they participate in these methods?"

After experiencing the three methods, everyone in our study said they believed culture could have an impact. They described ways they could imagine a person's culture influencing their behavior and comfort during the study. For example, multiple participants mentioned politeness in East Asian culture and suggested that East Asians with strong cultural ties might not feel as comfortable giving criticism in a usability study.

Latinx women believed their culture had an impact

Responses to the question, "Can you see ways in which your culture played a role in the study?"

Although everyone believed culture could have an impact, only Latinx women believed that their culture actually played a role in the study. They expressed concerns over whether their thoughts were important enough to say aloud. For example, they were unsure if their opinion on the color palette was relevant to the study. They attributed this to their upbringing not just as Latinx people, but specifically as Latinx women.

In contrast to the think aloud, the interview eased this tension since they were directly asked for their thoughts and opinions. Being prompted by questions and asked to elaborate reassured them that their thoughts were important— even when it came to seemingly minor things like color.

Black and Latinx women appeared to fear judgment

Participants who expressed concerns about their tech-savviness being tested

Throughout our discussions in the follow-up interview, Black and Latinx women expressed concerns that their tech-savviness was being tested. During the think aloud, they worried that making a mistake on the website would reflect poorly on them. This tension seemed to be more associated with being observed than the act of thinking aloud.

Conclusion

Are there cultural differences within the US?

In some cases, there may be. However, this study can't answer that question as it applies to the entire US. Based on the data, we can only say that the Black, Asian, and Latinx women in this study had these particular experiences. And, we must acknowledge that, although we've viewed these experiences through the lens of culture, we can't say that culture was the cause.

Limitations and Further Research

This study was limited in multiple ways. First, the sample size of Black women was incomplete, while the overall sample size of the study is not significant. Additionally, my own identity may have influenced participants and their level of comfort with me, especially on the topic of culture.

If these findings were to be validated in future research that addresses these limitations, then we might ask: Do these tensions impact the participation of Black and Latinx women in the UX research process? And, how might we ease these tensions and increase the comfort of Black and Latinx women?

Acknowledgments

Thank you!

This project would not have been possible without resources from the University Honors Program, the Research Assistance Desk at John M. Pfau Library, my faculty mentor Rob Ray, and all of the women who participated in this study.

Land Acknowledgment

We recognize that California State University, San Bernardino sits on the territory and ancestral land of the San Manuel Band of Mission Indians (Yuhaaviatam). We recognize that every member of the California State University, San Bernardino community has benefitted and continues to benefit from the use and occupation of this land since the institution's founding in 1965.

Consistent with our values of community and diversity, we have a responsibility to acknowledge and make visible the university's relationship to Native peoples. By offering this Land Acknowledgment, we affirm Indigenous sovereignty and will work to hold California State University, San Bernardino more accountable to the needs of American Indian and Indigenous peoples.

References

1. Costanza-Chock, S. (2020). Design Justice: Community-led practices to build the worlds we need. The MIT Press.
2. Evers, V. (2002). Cross-Cultural Applicability of User Evaluation Methods: A Case Study amongst Japanese, North-American, English and Dutch Users. CHI '02 Extended Abstracts on Human Factors in Computing Systems.
3. Hsieh, H. C. L. (2011). Exploring the Impact of Cultures on Web Usability Test. In International Conference on Human Centered Design, 47-54.
4. Oyugi, C., Dunckley, L., Smith, A. (2008). Evaluation Methods and Cultural Differences: Studies Across Three Continents. NordiCHI '08: Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges, 318-325.
5. Paterson B., Winschiers-Theophilus, H., Dunne, T. T, Schinzel, B., Underhill, L. G. (2011). Interpretation of a cross-cultural usability evaluation: A case study based on a hypermedia system for rare species management in Namibia. Interacting with Computers, 23(3), 239-246.
6. Vatrapu, R. & Pérez-Quinones, M. (2006). Culture and Usability Evaluation: The Effects of Culture in Structured Interviews. Journal of Usability Studies, 10(4), 156-170.