The CHIRON Trustworthiness Calculator: Origin & Rationale

What is the CHIRON trustworthiness calculator? The idea for a trustworthiness calculator began with member-to-member conversation during a community engagement studio (CES). CES members started kicking around the idea of creating some kind of a “social credit” score for researchers. Often attributed to—although frequently mischaracterized—a decentralized system in China, the essence of a social credit score is about quantifying a largely amorphous quality. Other CES members and, subsequently, academic workgroup members continued to iterate on the idea of a “trust score,” at times referencing review sites like Yelp or Trustpilot. 

How is the trustworthiness calculator different from review sites?

• The trustworthiness calculator does not aggregate data from external sources. If we were to devise something akin to a true social credit score for researchers, we would need access to data about “standard” topics like the researcher’s publications, grant sources, institutional affiliations, and professional memberships, and information that is not a standard part of big health data researcher portfolios, like the researchers’ track record with community engagement in their work. Needless to say, some of these data streams don’t (yet) exist. Moreover, we feared that using the data that we do have might reproduce institutional inequalities. 

• The trustworthiness calculator does not include a space for reviews from the public. As with external data, we were concerned about reproducing bias with the trustworthiness calculator. The entire CHIRON project team has been focused on making sure the whole CHIRON toolkit is a space that supports and encourages growth, rather than a space of censure. 

What is trust in the context of biorepository-enabled research? There is broad recognition of the importance of trust in research. Past (and not-so-distant past) failures of the research enterprise have understandably rendered researchers suspect in many communities, and, frankly with the rise of big health data research many more communities may have reason to be skeptical of the good intent of researchers. Developing and increasing biorepository-enabled researcher’s capacity for trustworthy research practice is the driving force behind the CHIRON toolkit.

Across the toolkit have assumed a pragmatic stance to the question of trust by focusing on trustworthy practices:

• Maximizing transparency in the conduct of research

• Recognizing (and hearing!) the communities represented within biorepository data

• Nudging researchers from a solely individually-focused to group-oriented conceptual framework of research benefit and harm

• And, aspirationally, expanding researchers’ conception of justice in research from a transactional redistribution of research benefits to a relational drive for participatory parity and bi-directional collaboration

How does the CHIRON trustworthiness calculator fit into all of this? Given the enthusiasm of the CES members and our academic workgroup and that the goal of the CHIRON toolkit is to help center communities in biorepository-enabled research, we figured it would be worth taking a stab at creating a tool to help researchers map their community-oriented research practice. The resulting self-assessment trustworthiness calculator—which, unsurprisingly, incorporates several topics addressed elsewhere in the toolkit—provides researchers a score which corresponds to bands of community orientation and highlights next steps in the journey for integrating community interests in repository research. One additional challenge we faced when designing the calculator is the nature of computational research itself: many communities may be involved at once. We ask users to consider a single community (or several communities in turn) as they complete the calculator—which may provide a range of scores depending on the community under consideration. Results are scored in ranges that scaffold learning and point to opportunities for learning and further engagement. We are piloting the trustworthiness calculator within the CHIRON toolkit pilot (pilot-within-a-pilot), soliciting feedback from those who use it. 

The Limits of Self-Assessment. Although the trustworthiness calculator is a useful tool for researchers to assess some of the qualitative characteristics of their proposed research, it is built on self assessment. The benefits of self-assessment are that it can be useful for prompting researchers to consider factors about which they may not have spent much time thinking. Drawbacks of self-assessment include that, by its very definition, it is subject to bias and distortion. For communities, a self-evaluation tool like the CHIRON trustworthiness calculator does not require researchers who may have done harm to engage in remediation. Instead, self-scorers can “game” the calculator, making the choices that best fit with their own desired outcome.

Redemption and reparation. The CHIRON trustworthiness calculator brings forward and makes legible the need for researchers to engage in the work of redemption and reparation. Low scores might indicate to researchers the necessity of critically (re)considering their positionality, methods, transparency, and prior history with the communities with which they desire to interact. Willingness to engage in critical, personal assessment requires that researchers be open to critical feedback. More than this, it requires the researchers to be willing to respond to criticism in ways that are both intended to remedy deficiencies and are actually effective. Importantly, investigators must recognize that their desire to “do good” does not trump individual and community notions of privacy, safety, and security.

Beyond the CHIRON trustworthiness calculator. Researchers have long recognized the harmful impacts of well-intentioned public health programs around the globe. Some in health and medicine are taking concrete steps toward restorative practice. The REParations and Anti-Institutional Racism (REPAIR) Project at the University of California, San Francisco, for example, focused on strategically addressing anti-Black racism in science, medicine, and health care. To do this, over the course of three years, REPAIR designed curriculum, created spaces for critical dialogue with previously harmed communities, and cross-disciplinary research. A three-year program may not provide the sustained change. It does, however, begin to demonstrate institutional movement to repair the distrust that communities have toward researchers.