A framework for assessing the impact of new media

A framework for assessing the impact of new media

In Digital Activism 101, Mary Joyce writes that activists can use digital technology to do only five things: shape public opinion, plan an action, protect activists, share a call to action, or take action. Then rinse and repeat. (Heh I love that phrase; stole it from Gunner’s awesome Another Cloud is Possible presentation.) Five seems to be real popular. The guys who wrote Blogs and Bullets: New Media in Contentious Politics lay out a five-level framework to assess the impact of new media. They caution that when it comes to democratization and social change, new media can have both positive and negative effects. (That reminds me of what I refer to as Kentaro Toyama’s mantra: technology magnifies human potential and intent. Good and bad, my friends.) What follows are my notes — copying and summarizing work by Sean Aday, Henry Farrell, Marc Lynch, John Sides, John Kelly, and Ethan Zuckerman. I encourage you to read their report.

Five pathways to manifest, five pathways to measure

I really tried to make that sound like Lord of the Rings (#fail). Anyway, these five levels are pathways by which change can be manifest and measured. We can ask whether or not new media is transforming:

1. Individuals — Changing how citizens think or act?
2. Intergroup relations — Fostering/undermining connections? Propaganda, memes & political communications surrounding elections?
3. Collective action — Facilitating collective action? Changing perceptions? Opportunities? Network structures?
4. Government policies — How is government learning about and using new media? What are they doing with it?
5. External attention — Increasing international attention? Consequences?

Aday and his crew point out that these five levels may be mutually reinforcing (changes in individual attitudes align with changes in the possibilities for collective action and the nature of external attention) and nonlinear (for example, change at the individual level may go in a different direction than the dynamics of external attention). Now on to each level…

Individual transformation

Individuals who participate in or exposed to new media may…

• Develop new political competencies (participate more readily or effectively in real-world politics, process information differently)
• Change political attitudes, orientations, or identities
• Intensify political attitudes, orientations, or identities (i.e., radicalization)
• Produce a more active or passive approach to political action (i.e., reducing or increasing likelihood to engage in off-line political activity)

Competencies example: Women may more readily engage in a mediated public sphere, providing an experience with public political engagement that in the past would have been denied. Citizens can become more passive, by (1) leading them to confuse online rhetoric with substantial political action, or (2) diverting attention away from productive activities. Methods: Measure individual transformation with instruments that assess individual attitudes, expectations, or beliefs — experiments, survey research, focus groups, structured interviews.

Intergroup relations

New media may reshape discussions and debates within and across groups, changing relationships and attitudes. Is new media:

• Fostering or undermining connections among like-minded people or groups, and between different groups?
• Playing a distinctive role in spreading propaganda or hateful images?
• Helping reinforce in-group identity in pre-conflict and conflict situations?
• Furthering cross-community communication in post-conflict situations?

Methods: Link analysis to measure contact within and across political or societal boundaries. Survey research to explore whether consumers of new media tend to consume material produced by people in their in-groups, people in their out-groups, or third parties. Experimental research to measure causal impact of exposure to different kinds of media.

Collective action

New media may affect the potential for individuals and groups to organize, protest, or take other forms of collective action. Has new media:

• Reduced transaction costs for contentious political action?
• Created flat vs. hierarchical networks of collective action?
• More effectively created & disseminated focal points (iconic images)
• Changed political opportunity structures? (attract international attention, expose divisions within elite, enable challengers to engage elites in new ways)
• Changed perceptions about others’ opinions and beliefs — so that others feel safer coming forward in support of a previously taboo position once they see how many online peers share their views

Method: This level of analysis would most likely require careful case study research in order to assess the changes produced by the various proposed mechanisms.

Government policies

This was “regime policies” in the report. I prefer government. I also switched up the wording a bit. Can/do governments:

• Learn about how to deal with new media from other countries? (And is the learning curve getting easier?)
• Actively use new media to react to others’ use of new media, or both?
• Use new media just as effectively as citizens?
• Divert attention from domestic challenges by using new media to whip up patriotism and xenophobia?

New media may help activists — but they may also help repressive regimes. Some governments have been caught off guard by new media activism,

but others have responded by co-opting, shutting down, or overwhelming activists. Governments may learn from the experience of other countries and increasingly act preemptively against particular new media forms when conflict might be brewing (for example, China blocked Twitter during a tense period shortly after the Iranian “Twitter revolution”).

External attention

New media may garner attention from outside actors, mobilizing political sympathy or hostility and creating new opportunities to generate power internally. Questions:

• What attracts external attention to one country (say Sudan) and not another (Congo)?
• How does information bias (say, perceptions generated by liberal, educated “bridgebloggers”) distort external attention?
• Does external attention lead to “cheap talk” among external supporters (like green Twitter profiles) or more costly actions (money, protest)?
• Can international attention push protesters to take risks in the mistaken belief that they will get substantial support from abroad?
• When, if ever, can external attention on its own have internal political consequences?
• Do new media create linkages with diasporas, and can these linkages lead to radicalization?
• Do new media affect the ability of states to engage in counterinsurgency strategies?

The Iranian Twitterers, for instance, framed the confrontations around images and actions that attracted Western attention. But their success in doing so likely depended on the rebroadcast of images and footage in traditional (mostly English-language) media. The millions of Twitterers who colored their profiles green in support of the Iranian protesters could not prevent the Iranian regime from attacking its opposition. As one “tweet” cruelly put it: “Note to would-be revolutionaries: you can remove the green tint from your pictures now; it didn’t work.” There are limits to internet solidarity: The Save Darfur movement mobilized attention and sympathy, but failed to save Darfur.

Methods: How to assess?

• Measure linkage patterns and content
• Track memes across media outlets

There has been progress in developing techniques to collect and analyze vast amounts of data from the Internet. We can capture the flow of information and communications in real time, while also reaching back far enough to establish baseline conditions from which significant deviations stand out. And we can identify and trace a sufficiently representative selection of new media. Remember it’s important to work in multiple languages — not only English.

So what’s changed? Show me the progress!

• Widespread adoption of syndication formats, including RSS and Atom — Syndication allows a software program to “subscribe” to a URL and be alerted when new content is posted to that URL. Aggregators designed for research purposes can sub- scribe to thousands of feeds, retrieving every story posted to a weblog or newspaper Web site, a process that is far more efficient and accurate than previous spidering-based approaches to data collection.
• Automated content analysis — Tools becoming more widespread, powerful, and affordable. Reuters’ OpenCalais provides a free web service that allows researchers to classify fifty thousand documents per day.

Link analysis

• Analyzing link patterns between Web-based media that habitually inter-link surfaces networks
• Networks are important motivators of social phenomena and political behaviors (example: terrorist recruitment often draws on preexisting networks of friends and acquaintances)

New media network maps reveal:

• Who is talking to whom
• Key nodes or hubs that link many other media sources
• Patterns of affinity or antagonism

By combining network analysis and content analysis, researchers can reveal the central identity of different media sources and of the networks in which they exist. SIDE NOTE: For years I lurked on SOCNET, a social networks discussion forum. You can search their archives. I like Valdis Krebs in particular, especially when he blogs about how SNA experts can work with a social justice organizations to kick some serious butt (related: How to Research a Slumlord).

Content analysis

• Measure content that is published or discussed in new media
• Measure how content spreads across networks

Example: Berkman Center’s Media Cloud project — open-source software that monitors an array of digital media, processes texts, identifies and displays patterns, and allows for more detailed analysis:

• Subscribes to tens of thousands of RSS or Atom feeds, collects newly published stories shortly after publication, and indexes content for analysis and retrieval
• Tracks the appearance of a word or phrase across media sources and across time
• Tracks words that commonly appear in conjunction with the search term
• Compares patterns of appearance and conjunction across sources
• Retrieve the text or Web page of any story
• Simple graphic visualization of results

Meme tracking

Term coined by Richard Dawkins. Jure Leskovec, Lars Backstrom, and Jon Kleinberg (memetracker.org) have shown that prominent political memes typically spread from mainstream media to blogs, rather than vice versa, with characteristic patterns of spikes in the intensity of discussion.

• Meme — an ideas or “discrete cultural forms” that spread from individual to individual, perhaps changing as they do, in processes that are loosely analogous to genetic evolution
• Researchers track how individual phrases are transmitted across electronic networks, as well as how they change and mutate

Use memes to:

• Begin to establish causality — by identifying the flow of specific concepts, terms, or issues through new media
• Identify the origin and flow of ideas — Do ideas move from the blogosphere to the mainstream media or vice versa? In which sector of political society did an idea originate?
• Understand how memes change as they move across communities (within limits)

Linking the framework to good strategy

In my recent rant about making good strategy I went on about listening and SMART objectives. I’d like to try going through this framework in the early on in that process, and especially when figuring out how to gage success. Also I wonder if it would help to map out a better theory of change, and make better decisions about audiences, messages, and tactics. Certainly understanding constraints and have a sense of what’s possible helps. On that note, I leave you with Civil Resistance 2.0: A New Database of Methods and a reminder: Don’t choose your tactics until the end. But, when you do, get really wild and consider the entire range. Then execute them well. That’s where the rubber hits the road. Become a Project Manager Ninja!