Impact Stories: Designing and Measuring Engagement

In the first panel HI Deputy Director Debika Shome participated in live case studies of engagement and impact strategies for two films: Tales from the Organ Trade and The Secret Trial 5. Since Tales from the Organ Trade has just been released and The Secret Trial 5 is still in progress, the filmmakers had very different strategy needs.

Impact Stories was a lively interactive session with real-time strategy development and audience participation. The session gave media creators a sense for process, timing and tools for planning an engagement campaign and impact assessment. Jen Gilomen from BAVC and Tricia Finneran from Impact Story Strategy focused on goal setting, audience segmentation and creating user profiles. Debika Shome from HI focused on impact evaluation, data and metrics for The Secret Trial 5, which examines the human impact of Canada’s war on terror. Tim Horsburgh from Kartemquin Films moderated the panel and led audience participation.

The filmmakers moved through each stage of the process from goals to audience to impact with insights and takeaways that had the potential to deepen their upcoming campaign and their potential for success.

Campaigns by Design

Interactivity was a big part of the second session as well. Campaigns by Design sold out early and was limited to a smaller group of filmmakers who were interested in workshopping a project or idea and collaborating with other makers to design their project’s campaign. Jen Gilomen from BAVC and Debika Shome from HI led the workshop and talked about the latest trends and tools for doing this work. BAVC focused on story, goals and audience. HI focused on how to move from story to measurable impact and how to translate broad goals to more concrete metrics drawing from the recently released Impact Playbook.

Topics included challenges and opportunities with impact measurement and metrics to inspire impact measurement. Audience members workshopped, frameworks and specific examples drawn from past HI projects. Participants were eager to apply our findings to their own work and found the audience engagement and impact measurement strategies particularly helpful.

With both sessions, HI’s goal was to demystify the evolving field of impact measurement for entertainment and provide audiences with manageable, step-by-step approach to start implementing impact measurement today. Although it’s difficult to condense more than three years of case studies, we hope these best practices will benefit the wider documentary filmmaking community.

We look forward to more hands-on interactions with media creators as we work to develop products and tools that help media makers. We’re particularly looking forward to seeing how impact measurement for The Secret Trial 5 and other projects workshopped at Hot Docs evolve over the next year and beyond based on input from the Hot Docs sessions.

Top Image: Participants workshop their ideas during Campaigns by Design

What can documentaries teach narratives?

Steven Soderbergh’s widely discussed keynote on the state of the film industry set the tone for conversations on distribution and creative freedom throughout the festival. In anticipation of these shifts, festival organizers created Artist to Entrepreneur (A2E) a new programming series that connects narrative filmmakers to technologists and distribution experts.

At A2E, HI participated on a panel focused on adapting engagement and measurement strategies pioneered by documentaries for narrative filmmakers. Shaady Salehi of Active Voice emphasized the importance of strategic planning for engagement, and Jennifer Gilomen of BAVC provided case studies of documentaries that had successfully engaged audiences. HI discussed the importance of audience measurement and real-time data during the narrative distribution process. Many of these strategies were drawn from the recently released Impact Playbook and re-imagined for the optimization of distribution of narrative films.

As narrative filmmakers increasingly use crowdfunding and community screening platforms, we believe that data can help successfully engage backers and viewers. In complex and competitive viewing environments, these tools will help filmmakers of all stripes find audiences.

Image: Still from A River Changes Course | Migrant Films

Interactive Impact

The San Francisco International Film Festival also presented an opportunity to talk with audiences and filmmakers about impact. HI took part in a salon titled Interactive Impact: Making change Through Audience Engagement to share our perspective and learn from other participants in this emerging field.

The salon introduced a filmmaker perspective to the discussion of social change. Kalyanee Mam, director of A River Changes Course and Jeremy Teicher director of Tall as the Baobab Tree (we highly recommend both films) spoke about how filmmakers can navigate creative and advocacy positions related to their films. The panel and  audience also discussed how to best manage partnerships with nonprofits and the challenge of finding the time and energy to both make films and engage their audiences to make change.

Mam plans to screen her film, which chronicles issues of urbanization, labor, and the environment in modern Cambodia, throughout the country, often to citizens who lack access to media. Teicher is working with Girls Not Brides to address child and adolescent marriage, an issue that is dealt with in his film. Both films are complex and aesthetically rich—they remind us of the power of film to explore the nuances of important global issues.

Top Image: Interactive Impact Panel photographed by Pat Mazzera | courtesy of the San Francisco Film Society

NFB: Interactive documentary

The National Film Board of Canada has a storied history of documentary production, as evidenced by its long list of Academy Awards. Today the NFB is a is a leader in interactive documentary and an example of innovation in public media.

At TFI Interactive, NFB head of digital content Loc Dao discussed recent (Bear 71) and upcoming projects, as well as the organization’s philosophy for creating compelling interactive content. Dao and his colleagues have followed recent trends in technology adoption that show that audiences are increasingly using mobile and tablet devices. The NFB uses this information to match their stories to a platform, an idea that was echoed throughout the conference.

In a post-broadcast world where the audiences can access many types of content on a variety of devices, storytellers and designers at the NFB think critically about the context in which users encounter their work. For example, their recent project The Loxleys and the War of 1812 is an interactive graphic model that takes advantage of the high resolution and touch-screen interface of tablets. By matching content with form, the NFB is able to create compelling interactive experiences.

 Image: Still from The Loxleys and the War of 1812 | NFB/interactive

Localore: Community-driven media

Like the NFB, Localore is a public media initiative launched by the Association of Independents in Radio (AIR). Its participants produce media in a variety of formats, with an emphasis on audio. The ten lead Localore producers gathered from across the country to present their work at TFI Interactive.

These projects demonstrated how interactive public media can reach populations who aren’t fully represented or served by other media outlets. Curious City lets its Chicago listeners set the agenda for an urban sleuthing show. Reinvention Stories chronicles the response of Dayton residents to their city’s changing economic conditions. Planet Takeout uses the ubiquitous urban Chinese carryout as a site to explore the lives of Boston residents. Black Gold Boom investigates the massive but underreported transformations around North Dakota’s ongoing oil rush. Many of these media-rich projects use Zeega as a platform for interactive web publishing.

AIR also represents a model for public media collaboration in the digital age. Their support of innovative public media projects like Localore brings interactive storytelling to wider audiences. Overall, publicly funded media played an integral role at TFI Interactive.

Image: Lucky Cat, Wok N’ Talk, Jamaica Plain, Boston | Planet Takeout, CC BY-NC-SA 2.0

Caspar Sonnen: understanding an emerging field

One difficulty in evaluating interactive media is the field’s accelerating pace. HTML5, tablets, user-generated content, cloud filmmaking, crowdfunding—it seems that every year there are new languages for media makers to learn. Dealing with “newness” is the challenge of working in new media.

Caspar Sonnen took a holistic, historical approach to interactive documentary in his talk at TFI Interactive. Sonnen is the New Media Coordinator of the International Documentary Film Festival Amsterdam, a leading global forum for documentary media. In 2007, he launched the IDFA doclab, which runs a competition program in digital documentary storytelling. From the perspective of a curator, he maintains a broad view of the field and thinks systematically about effective interactive work. Sonnen suggested a four part framework for understanding interactive work, to which we’ve added our own interpretation:

•  Story: Compelling narratives are the foundation for all great documentary.
• Interface: Interactive documentaries are unique because they aren’t limited to single medium or distribution channel. The chosen interface should enhance the story and engage the audience.
• Capturing reality: Interactive documentaries should capture the complex nature of our interactive world.
• Digital data: Data is now a narrative unit. Digital interactions leave traces which can be used to tell a story or analyzed to better understand issues and audiences.

This system for evaluating these projects inspired HI to think further about developing adaptable frameworks for understanding this wide-ranging field. Variety is one of the great strengths of interactive documentary—narratives can be realized on their own terms. Capturing and celebrating this variety is a challenge as we continue to develop our own methods for comparison and analysis of these stories.

These highlights are only a small sample of the talks from TFI Interactive 2013. For a full recap of the day, see their Storify.

Top Image: Blabdroids from the Robots in Residence at TFI Interactive | Alexander Reben

HI researchers and the MSiF tech team, including SVA Interaction Design student Minsun Mini Kim and Columbia Graduate Film student Paul Chen, worked throughout the day to record, analyze, and visualize engagement data from two separate groups of participants. This engagement data was collected using Q Sensors provided by the Affective Computing Group at the MIT Media Lab. The sensors measures movement, skin temperature, and electrodermal activity. This last measure serves as an indicator of arousal, such as one’s engagement with the environment, anxiety, or excitement.

The tech team translated the Q Sensor electrodermal activity data into an engagement heat map of each participant as they encounter the story’s content. This heat map (created by Minsun Mini Kim and shown below) illustrates the individual beats of the immersive story along a timeline, according to Joseph Campbell’s Hero’s Journey. The map also shows the physical locations of those story beats,  which path each participant took, and how engaged they were at each point in the story with blue areas being the lowest points of engagement and red areas being the highest points.

The heat map was annotated with photos and participants’ qualitative feedback for each beat of the story. In this instance, a participant with particularly high levels of engagement data also reported feeling “liberated” during an emotional beat of the story meant to elicit feelings of transformation and epiphany.

At the end of the experience, participants are shown that the science fiction immersive experience is a metaphor for Lydia’s life and foster care. HI researchers found that, despite the low engagement readings, much of the feedback from the participants cited this aspect of the experience as one of the most interesting and impactful.

This “debrief” with Lydia was also important from a design perspective. During the final presentation Lance spoke about the origins of the idea for MSiF and his goals of designing with the stakeholders in a social cause (such as foster care advocacy group, Orange Duffel Bag), as opposed to designing solutions for them.

The presentation ended with three main takeaways:

• Immersive stories can deeply connect people to social issues.
• Audience members enjoy engaging with interactive stories.
• Data-driven storytelling can innovate social science research and optimize stories for greater impact.

The feedback from the audience was generally positive with commentators emphasizing the ways in which the experience made them feel. One woman said, “If I had watched a documentary, of course I would’ve cried but I would not have felt [that].”

My Sky is Falling will be featured again at the upcoming conference, DIY Days, and expand to include real-time data analysis of participants’ narrative choices and engagement. Be sure to check back for a follow-up post on how we further analyzed participant feedback. Additionally, the Envision presentation can be viewed in its entirety here.

Image: IFP Video Recording | YouTube.com

Following the on-site event, we have continued to work with BAVC and the Producers Institute participants to build their understanding the social impact of their work. Now, we are proud to introduce the “Impact Playbook” a guide that shares these lessons with the wider field. Specially designed for today’s media makers, the Impact Playbook features applied strategies and best practices to make sense of new data sources and social impact.

The changes in the media landscape in this period have been profound. As Director of Independent Media Jen Gilomen writes in the forward, “When I started at BAVC twelve years ago, “social media” meant watching TV with friends.  Google was just starting out, and Facebook, YouTube, and Twitter didn’t exist.” Today, VOD data, Facebook Likes, and user-generated content can overwhelm media makers if they don’t think logically and holistically about what each metric means. The Impact Playbook builds on existing frameworks like The Fledgling Fund’s “Assessing Creative Media’s Social Impact” and the Center for Social Media’s “Social Justice Documentary: Designing for Impact” to speak directly to media makers at the center of this changing field.

The Impact Playbook situates these emerging data sources in a larger practice of understanding impact. By integrating data collection and analysis into the creative and outreach process, our aim is to empower media makers to understand the impact of their work; to use data to better understand their audiences and their own stories. We want to build this field collaboratively and hope that the Impact Playbook will open conversations and lead to new strategies, tools, and frameworks as social issue media evolves.

The full version of the Impact Playbook is now available for download. We welcome any comments or questions.

For HI, these emerging platforms present new opportunities to study how participants interact with social issue narratives. In the immersive narrative, My Sky is Falling (MSiF), participants explore a theatrical, narrative world. The story centers on the issue of the foster care system in the United States using science fiction metaphors. To understand the impact of this compelling and informative story, HI researches will analyze real-time user engagement data, which is collected as participants move through the experience wearing an Affectiva Q Sensor. This physiological data will be mapped against an evaluation framework that assesses changes in audience comprehension, attitude, and response. Our goal is to merge creative design with data science to measure the influence of innovative narratives on contemporary social issues.

The MSiF experience was initially created as a final project for the Columbia University graduate film course, Building Storyworlds for the 21st Century, in the Fall 2012 semester. Along with the Reboot team and other creative technologists, HI researchers helped the students to create the narrative by analyzing data from classroom exercises as well as designing frameworks for data collection and measurement of the final project.

MSiF will also run at the upcoming conference, DIY Days on April 27. Data and analysis from both events will be posted here.

One of the most remarkable achievements of the night came in the Best Documentary Short category where Inocente, the story of a young, homeless Latina artist took home the Oscar. The fact that Inocente won the award is not a surprise; the beautifully made film combines issue advocacy and storytelling in a compelling way. The surprise is the way the film was funded; with its Academy Awards triumph, Inocente became the first Kickstarter-backed film to win an Oscar.

A few weeks ago, HI Researcher Doris Domoszlai  had the opportunity to watch Inocente at Barnard College’s third annual Athena Film Festival. The screening was followed by a Q&A with producer Yael Melamede. Melamede provided some insight into the evolution of the filmmaking process as well as the film’s impact, not only in its subject’s life, but also in the lives of others.

The Fledgling Fund, one of the film’s main backers, was instrumental in the creation of Inocente‘s impact strategy, working with Melamede and six other social justice organizations to gauge the impact that they hoped to see the film make. Oscar visibility will surely help in this process. HI researchers used social media monitoring service Crimson Hexagon to informally track Inocente’s social media buzz. The results were clear: on February 25, a day after the Oscars, there were 6,126 posts about the film, jumping from a mere 216 just a day before.

The visibility from film’s Oscar victory will help amplify the impact work that the filmmakers were already engaged in. According to Melamede, art workshops are often held in conjunction with screenings of the film, where the artist (also named Inocente) often makes appearances, in addition to her “doing more and more advocacy work.” Additionally, the filmmakers are working on free educational companion curricula that will be distributed with the film, which would address many of the issues the film addresses, such as teen homelessness, violence, and immigration.

The creation of companion curricula was just one of the goals the filmmakers incorporated into their highly successful Kickstarter campaign. It took the filmmakers a month to raise the $50,000 they needed in order to complete post-production of the film, and they raised $2,257 more than their total goal, with nearly 300 backers.

Inocente’s Kickstarter success highlights the burgeoning field of new media funding platforms. Although Kickstarter is one of many crowdfunding websites, its Oscar recognition makes it the ideal candidate for the media funding study currently underway at the Harmony Institute. The study will take an in-depth look at documentary films and their Kickstarter campaigns in order to better understand the role of crowdfunding. The study will be released on The Ripple Effect shortly.

This theme resonates strongly with HI’s current research, from our work at the 2012 BAVC Producers Institute to our latest research tools for media makers. Media that Matters 2013 opened this conversation to a diverse group of thinkers, nonprofit leaders, and storytellers. Jessica Clark, author of the Center for Social Media’s excellent guide, “Designing for Impact,” lead a pre-conference workshop that encouraged filmmakers to think strategically through both production and rollout to track impact and iterate, amplifying what works and subtracting what doesn’t, at each stage.

Wendy Levy’s opening keynote, “Measurement, Meaning, and Momentum: Storytelling in the Age of Data” set the tone by situating the unique power and value of storytelling in our increasingly data-informed world. Levy emphasized that data and impact measurement complement effective narratives. In her words, ”data can facilitate creative discovery and originality.”

Later panels expanded on this theme. Nonprofit measurement expert Katie Delahaye Paine stressed the need for setting measurable, observable goals and drawing meaningful inferences from data instead of getting caught up on tools and software. Sheila Leddy discussed the Fledgling Fund’s new Impact Grants, which represent a strong commitment from funders for measuring and communicating impact. Leddy focused on a broader understanding of impact that isn’t overly reliant on any single numerical indicator, but rather tells the narrative of impact by adding context and qualitative information.

Following Media that Matters 2013, the groundwork is in place to build a rigorous understanding of the impact of social issue media. As filmmakers embrace data as part of their iterative design process, they can use insights to hone their work and effectively communicate results to the wider community. Although we are still at the early stages of this shift, we are looking forward to developing creative, innovative, and rigorous tools to better understand impact.

HI Executive Director John Johnson summed up the situation in a recent article in GOOD: “This industry-shifting research provides valuable tools to help filmmakers hone their craft of creating impactful, moving stories.” The next challenge is the hard work of research and analysis to ensure that “impact” and “data” aren’t just buzzwords, but central to this new generation of stories.

If the Federal Reserve account balance falls in today’s treasury statement, does it make a sound?

Past and present HI data analysts Brian Abelson (now an Open News Fellow at the New York Times) and Burton DeWilde recently competed in a Bicoastal Datafest analyzing money’s influence in politics — and won!

Funded by the MacArthur Foundation, organized by the Sunlight Foundation, and hosted by Columbia and Stanford Universities, the Datafest brought together journalists, data analysts, academics, and web developers for two frenetic days of data-driven story-telling focused on revealing the influence of money in politics and producing reusable tools for others to do the same. The event’s overarching goal, “to contribute to making data analysis and computational thinking a common skill in journalism, social sciences, and the nonprofit sector,” fell very much in line with Harmony Institute’s ethos.

Participants formed cross-disciplinary teams to tackle an array of interesting challenges. We (Abelson and DeWilde) teamed up with journalists from Reuters, the Daily Beast, and Huffington Post to represent CSV Soundsystem, our informal, weekly data/journalism workgroup.

Our project, dubbed FMS Symphony, began with the liberation of a dataset. For eight years, the U.S. Federal Government has saved its daily account statements itemizing cash deposits and withdrawals, public debt, and other financial measures as human-readable text files; however, these files have never been accessible programmatically, which would allow for systematic analysis of the government’s finances. To solve this problem, we scraped thousands of these files from the Financial Measurement Service’s (FMS) website, painstakingly parsed them in Python, then converted them into “the first-ever electronically searchable database of the Federal government’s daily cash spending and borrowing.”

From this, we produced a handful of interactive data visualizations, including a day-by-day comparison of the legal debt ceiling, actual public debt, and the amount of discussion on the topic in the Congressional record. (The Summer 2011 debt ceiling standoff is immediately recognizable.) Given the complexity of our 61-dimensional dataset, we opted to produce a multi-sensory graphic through sonification: We literally made music of our data. Chords are daily movements — money in, high/major chord; money out, low/minor chord — with flourishes representing movements up and down that are more than 2.5 standard deviations from the mean of a rolling two-month window. We also included two contrapuntal riffs reflecting the distance between accumulated federal debt and the legal debt ceiling (in the left channel) and the federal interest rate (in the right channel). Lastly, we used principal component analysis to rotate the 60 account line-items and plotted the 15 highest-loaded components as Chernoff faces that mutate day-by-day according to the particularities of each account statement.

For our efforts, we were awarded Best in Innovation by a distinguished panel of judges and voted Best in Show by the participants at both Columbia and Stanford. The Datafest’s other winning projects addressed a wide range of issues:

• One person asked, “Does Congress Really Suck?” and presented the answer as an interactive web experience.
• The influence of money, power, and people in Silicon Valley was mapped with an eye toward the upcoming debate on immigration reform.
• Another team used a common statistical law to search for potential cases of fraud in local government spending patterns and built a tool to empower others to do the same.

All this goes to show what can be accomplished when smart, talented, and engaged people get together to work toward a common goal — whether advocating for transparency in politics or understanding the influence of narrative media.

Games are a huge component of the entertainment marketplace—Industry analysts project that gaming revenue, including sales from mobile devices would reach $78.5 billion in 2012. However, detailed social research on gamers is still an emerging field. In order to better understand some of the issues at play, we spoke with Cornelia Brunner of the Center for Children and Technology.

Brunner has worked in education for over 40 years, starting at Sesame Street and moving on to experimental and progressive classrooms that incorporate new media and technology. Pivoting off our work in game evaluation, she explained her ongoing research on games in educational settings. We asked Brunner, what would an inclusive, meaningful game look like?

One of Brunner’s insights, culled from ongoing psychological research with students, is that different people conceive of technology in different ways. More specifically, she has observed that people access technologies through gendered perspectives. Brunner terms the perspectives “butch” and “femme” to emphasize the social construction of gender roles and avoid confusion with biological sex. Importantly, members of either sex can interact with technology from either perspective, in varying degrees.

Generalizing through extended content analysis, Brunner has formalized a set of dichotomies that can be applied to technology while noting that these categories are fluid and subject to change over time. Butch perspectives prefer centralized technology solutions that transcend limitations, while femme perspectives want diffuse artifacts that help share and communicate. For example, when asked to draw a picture of the World Wide Web, the butch tendency was to draw a representation of a browser window, while the representative femme illustration resembled a network map.

This framework is particularly useful for analyzing video games. Brunner identified two gendered dimensions that apply to many games: collision-arrangement and shooter-puzzle.

The red letters (F,B) refer to femme and butch perspectives.

An interesting consequence of video game development is that it has been easier to develop video games in the butch-dominant quadrant of collision and shooter. Tetris, perhaps the best pure example of a game based upon the principles of arrangement, stands as a historical outlier to this theme. Only relatively recently in the history of gaming have simulations been powerful enough to begin to realistically model complex social interactions as seen in The Sims (2000) or more recently in games like Prom Week (2012). Brunner also noted that mobile technology, such as smartphones and tablets incorporate more femme traits like  expressive, exploratory, focused on communication than butch characteristics.

This understanding of gendered perspectives provides researchers a tool for designing and evaluating games that appeal across a broader range of perspectives.

Image: Player interface from Prom Week | University of California, Santa Cruz