Can an AI-Powered Tech Tool Help Ford Evaluate its Impact on Complex Social Issues?

In September, the data analytics firm UpMetrics issued a somewhat jargony press release announcing a new partnership with the Ford Foundation “to support the expansion of Ford’s data-driven impact storytelling capabilities.”

Or, to put it more clearly, Ford is using UpMetrics’ platform to attempt to aggregate both quantitative (facts and numbers) and qualitative (fuzzier, but nonetheless important, evidence of impact) data, to more thoroughly evaluate Ford’s own progress in advancing its many missions.

Shiny new nonprofit solutions from tech companies are frequently presented in glowing terms, and the philanthrosphere is known for chasing after the next innovation in grantmaking and grant reporting. But how it plays out on the ground can be far less exciting, particularly for nonprofits if they have to jump through a new set of hoops. At the same time, more funders, at least in rhetoric, seem to be embracing the messier work of social movements and trending away from a historic obsession with metrics. All that got us curious about how this new partnership will work.

What we learned about the Ford/UpMetrics partnership should reassure Ford’s partners on at least a few fronts—the foundation and vendor are taking several security measures, and grantees won’t actually do anything with the platform themselves. That said, foundations' embrace of new tech platforms to measure the success of their strategies does raise a lot of questions—some old and some new—about who decides what data is important and who controls the process of gathering and evaluating it.

What is UpMetrics, and how is Ford using it?

UpMetrics is a web-based application that allows users to create “stories” by aggregating both quantitative and qualitative (think a news article or social media post) results from a large variety of sources in a single place. A program associate at Ford can transfer data about its grantees from the funder’s grants management system, Fluxx, into the UpMetrics app. Fluxx founder and former CEO Jason Ricci worked at UpMetrics from 2019 to 2020. An UpMetrics spokesperson told me via email that while Ricci is one of several individuals at UpMetrics who had inroads at Ford based on past relationships, “there isn’t a relationship between Fluxx and UpMetrics and no correlation between the two contracts.”

That same program associate can then capture news stories, anecdotes, photos, policy changes, outside polls and other stories and view and analyze all of that information in one place to understand the effect that Ford’s overall efforts are having in a given focus area.

In an effort to make the task of quantifying the unquantifiable easier, UpMetrics also uses Watson AI to assign “sentiments,” or values, to qualitative information. In effect, a given Ford program associate using UpMetrics will be able to report, for example, how many artists received community outreach training from Ford grantees, how often the artists who received that training show up in later news articles about arts outreach efforts in Ford’s service areas, and whether those news articles portray positive effects to the greater community from those outreach efforts. Add in a heaping helping of government reports on potentially related data (like public school test outcomes), posts on community blogs, and even tweets, and Ford can then use that collected data to track progress.

UpMetrics allows a user (whether a funder or a nonprofit) to “layer” these bits of information to “look at the trajectory of that story [or effort] over time, to see if there’s any patterns in there,” said Annie Rhodes, chief strategy officer for UpMetrics.

“Ford is hoping that the UpMetrics platform will help give it a more complex, nuanced data tool to help the funder learn more about how social change happens and continuously improve Ford’s support to partners addressing inequality,” said Ford Strategy and Evaluation Officer Subarna Mathes. “How do we first put the onus on ourselves to say, how are we actually learning about and making the case for some of the claims that we’re making? Not because we’re passing those judgments on to our grantees… but really because we’re frankly all just trying to understand how complex social change happens.”

To that end, Ford’s program associates have been tasked with aggregating the wide range of data required to create that nuanced, “layered” picture. The results will be used to evaluate both the impact of Ford’s grantmaking and the success of its own efforts to provide “thought partnership” and otherwise move the needle on its issues.

Grantees left out

One concern about the new application is that Ford’s grantees weren’t involved in making the decision to use UpMetrics, and they aren’t part of the process of deciding, for example, whether or not to attach positive or negative “sentiments” to the qualitative data that Ford program associates are uploading to UpMetrics. And the foundation does not plan on sharing what raw data from outside sources was compiled in the platform.

“I think when you leave grantees out of the process, whether it’s at the beginning when you’re even thinking about a company like this, or in the middle, during defining the metrics, or at the end, when the analyzed results start coming in, you kind of start to see that the funders get to own the process more and more,” said Stephanie Peng, National Committee for Responsive Philanthropy senior associate for movement research. “The funder gets to decide what impact is. The funder gets to decide how they want to share that data.”

To be fair, Mathes said Ford does not require grantees to take on the burden of adopting and using UpMetrics themselves, and has no plans to. Ford emphasizes that the program will not be used to track progress on a grantee-by-grantee basis. Rather, UpMetrics data will be included alongside other input, including grantee input, as Ford evaluates its strategies and refines its 10-year plans.

Ford “has a long way to go” to make the grantee input process during strategy development “more robust,” but “we are ensuring that we both ask our existing grantees, as well as key stakeholders, key actors who are engaged but not part of our sphere,” Mathes said.

Who owns the data, and how secure is it?

Both Ford and UpMetrics stressed that Ford owns the data it shares with the platform. Rhodes said that UpMetrics neither uses the data for its own purposes, like demonstrating its platform to potential customers, nor shares it with any third parties. Further, Rhodes said, “the data that it captures is not specific to an organization, but more about an overall impact goal that everyone is driving toward.” Mathes at Ford said that the program associates who are using UpMetrics are trained internally about what information should, and should not, be shared with the program.

Ford plans to discuss the results it generates from working with UpMetrics with its grantees, but does not plan to share the information it has used to compile those results, saying that the data it’s using comes both from the grantees themselves but also from third-party sources that are widely available online. “Grantees have access to all the data we do,” a Ford spokesperson said via email.

Of course, individual grantees may well not have Ford’s capacity to collect that third party data, online or not. For UpMetrics’ part, Rhodes said that the company “look[s] to our funder partners to drive the collaboration around the data with their grant partners.”

Both Ford and UpMetrics do seem to be taking the security of the data that winds up on the platform seriously. Rhodes said that UpMetrics advises its customers to “de-identify” data before uploading it to the platform, requires secure user authentication to access the data, and said “data visualizations shared outside of the UpMetrics platform only show aggregated metrics and are restricted from filtering to prevent a viewer’s ability to drill down into a subset of data,” to find, for example, the contact details for a particular grantee. For further security, Ford has designated a single program associate in each program team to upload information to the platform.

Ford isn’t alone in turning to UpMetrics to get a handle on the fuzzier information that can point to the success or failure of particular strategies. According to Rhodes, nonprofits—not foundations—are currently the company’s largest customer base, with 50% of those nonprofits contracting with and paying UpMetrics directly. Most of the other 50%, she said, are able to use UpMetrics courtesy of sponsorships from anonymous donors. In other cases, funders make UpMetrics available to their grantees voluntarily, Rhodes said.

Clarification: This story has been updated to clarify information about how Ford plans to use UpMetrics, including that Ford will not be downloading data from the platform, and that UpMetrics does not produce reports.

Featured