In the new era of distributed online spaces it is no longer sufficient to look at human behavior just on local servers and storage sites. From exploring on-line videos to searching vast information spaces with keyword search interfaces, the information scope and societal impact have changed. HCI researchers therefore need new strategies to study, respond, and adapt to changing circumstances as the context for research changes. Quantitative methods such as social network analysis (SNA) and surveys are appropriate techniques for addressing some research questions in some types of online social spaces, but they have limitations. For some studies, qualitative methods, specifically applied ethnography, may offer the only way to deeply understand users' behavior, intentions and feelings within the social-digital ecology of the Web.
The goal of this research is to examine the issues that researchers encounter when attempting to understand diverse types of behavior in large online spaces using ethnographic approaches. To accomplish we will conduct a two-stage study. Stage 1 will produce a meta-analysis of current research literature on YouTube, Facebook, Twitter, Wikipedia and other large-scale social systems, and on keyword search interfaces, to identify the methods that were used by researchers, under what assumptions, and which outcomes these studies reported. Stage 2 will produce a grounded theory analysis of case studies of how a sample of HCI researchers do qualitative research, particularly ethnography, in these vast and changing information spaces.
Large scale online environments, ranging from social networks (e.g., Facebook), to peer production networks (e.g., Wikipedia), information dissemination tools (e.g., Twitter), outlets for creativity (e.g., Flickr, YouTube) and collective curation communities (e.g., Encyclopedia of Life, Pinterest), have become a mainstay of online activity. These environments enable users to exchange information, engage in intellectual production of textual and audio-visual content, collaborate on a massive scale, and engage in various discretionary activities. For many researchers these activities and the traces they leave are a goldmine of research opportunities, enabling an intimate glimpse into the social dynamics of technology use, and the processes of engagement, collaboration, production and attachment on a scale larger than ever before. Where previously online studies focused on relationships formed in small groups such as dyads, families, communities, and groups within organizations, large scale online environments expand the field to include networked relationships encompassing hundreds of thousands and even millions of people. Perhaps not surprising, these rich and often relatively easy to collect data drive much of the research done on large scale social environments to be quantitative assessments of the strength and divergence of relations created among users, and the structure of the network created by them. Quantitative studies can enable researchers to identify changes and trends, areas of particular activity, information-flow directions, gate-keepers, those at the center of networks or those at the outside, those with many links to others or singletons. Yet quantitative studies are not well suited to answer questions related to human values, motivations and meanings, since they offer a rich but incomplete picture of behavior and intention. This research exploratored the opportunities and challenges that qualitative researchers face when studying large scale online environments. To do that we conducted survey research and interviews with researchers known for doing qualitative research, in which we discussed the ways in which researchers reconciled the opportunities offered by large scale online environments and the practical and theoretical problems they face. In addition we examined papers from the ACM Digital Library in order to gauge the popularity of qualitative methods for studying large scale online environments. Our research has surfaced many challenges that come from the negotiation between the size and scope of the field, on one hand, and the focused nature of qualitative work, on the other. Determining the unit of analysis in large constrantly changing online environments was highly problemmatic, especially when the interface was often layered and it and the platform and tools frequesntly changed. There were also ethical issues about which data could be used and even though it was in the public domain, whether it should be used. The practical issues revealed by interviews and questionnaires included the product of pairing various combinations of tried and tested methods that have been used in studying smaller scale settings, with large scale environments. These challenges were set against a broader framework of theoretical challenges, speaking to methodological choices and power structures within the broader research community: for example, qualitative research of large scale online environments is not "pure" qualitative work. It stems from the research question and not from the methodological proficiency of the researcher; it is relatively methodologically agnostic and incorporates other, often quantitative, to allow for multiple perspectives and to complement the qualitative work; and, it suffers from power issues due to its standing between purely qualitative research and the positivistic tradition. While these challenges are by no means foreign to qualitative work, they are exacerbated by the scope and attributes specific to large scale online environments. One way is to address these issues by focusing on qualitative researchers of large scale online environments as a community of practice, and craft mechanisms to support them. Communities of practice are defined as "an informal aggregation of individuals engaged in common enterprise… characterized by the shared manner in which its members act and how they interpret events." Indeed, from the themes that came up from the data, emerged a clear image of a community of practice that is important to the broader information science research community. Focusing on the challenges as having an impact not just on the immediate qualitative research community, but on the broader research community may also lead to a better understanding the relatively low number of papers discussing large scale online environments that we found in the meta-analysis. The challenges outlined by our study participants also illustrate the need for a community-wide discussion through which we acknowledge that qualitative methods, along with quantitative methods needed to gain a deeper understanding of how people interact in large scale environments and how these environments evolve and change. Through our studies many other researchers are thinking more about these challenges, including many graduate students.
