Drafting Guidelines for Emerging Scholarship

Over the past few months, a few of us in ExperimentalHumanities@Iliff have been participating in an ongoing conversation about resources and guidelines for the support and evaluation of scholarship that doesn't fit easily into traditional print publishing models. In concert with Ted Vial, chair of the AAR Publications Committee, we have collaboratively worked up a draft statement on resources for the support and evaluation of digital scholarship in religious studies. The draft is below and we share it to encourage input from others. Please use the comment area at the bottom of the page or the embedded hypothes.is tools at the top right of the page to share your comments, concerns, questions, and insights. We look forward to the conversations. 

DRAFT - Resources for the Support and Evaluation of Digital Scholarship in Religious Studies

0.0 Preamble

0.1 Contributors (alphabetically): Justin Barber, Timothy Beal, Elizabeth Coody, Pamela Eisenbaum, Mark George, Michael Hemenway, and Micah Saxton.

0.2 We have drawn upon the work of many to compose this set of resources. Several guild associations have produced guidelines for evaluating digital scholarship including:

0.3 There are two other resources we recommend as starting places for exploring the questions and issues that arise regarding the support of digital scholarship:

0.4 What do we mean by digital scholarship? This begs the question, what do we mean by scholarship? We imagine all scholarship as digital in some fashion, given the use of digital technologies in all forms of scholarship and publication. More specifically here, by “digital” we mean methods of scholarship that are not governed primarily by the mechanics and ideologies of print publication. Here is one example of this kind of definition from a digital project trying to be self reflective about definitions such as this:

'Digital' can mean a lot of things, but in this case it has a simple function to identify academic work that is governed more by internet sensibilities than those of print. Relatedly, internet sensibilities could encompass many different values, but here I use internet sensibilities to signal a focus on process more than product, collaboration more than independent scholarly activity, and interface design as a critical part of content production.
(Michael Hemenway, “Space as Thesis” at http://aproximatebible.postach.io/post/space-as-thesis)

0.5 Our investment in these resources is not merely a product of necessity given the shifts in scholarly practice in the academy. As a guild, we affirm the value of digital scholarship for the advancement and enrichment of the field of religious studies. This is not an attempt to eradicate or overthrow traditional forms of scholarship. Rather, with Sarah Barrow, we see these emerging approaches as opportunities for alterity.

it is time to normalise alternative ways to publish and circulate ideas. This statement is not an attempt to undermine the enormous value of the physical ‘book’ or the rigour and review that goes with its publication; rather it is to do with seeking acknowledgement for and trust in alternative ways of doing and presenting research, valorising interdisciplinary and collaborative effort, and accepting that high-quality academic endeavour might result in something ‘other.'
(Sarah Barrow, "The Impossible Constellation," in The Academic Book of the Future, Kindle location 586.)

1.0 Medium / Design / Collaboration / Process: New Key Terms for Evaluating Digital Scholarship

1.1 As a department or institution, it is important to articulate clearly the core values we have around scholarship. What is “scholarship” in religious studies? What is “research”? How are they related to publication? How are they different from or related to other work we do as religionists (e.g., teaching and service)? What criteria do we use to decide whether something is scholarship or research? For example, AHA defines scholarship as “a documented and disciplined conversation about matters of enduring consequence.” Does this definition provide enough guidance for values based evaluation, or does your department or institution need something different?

1.2 The need for critical reflection on questions such as these become especially acute and urgent in contexts of media revolution such as ours, when we find ourselves increasingly self-aware of untheorized and poorly articulated assumptions and values about the media in and through which we carry out and evaluate scholarship. On our present media horizon, new terms of engagement are emerging. For example:

1.2.a Medium can no longer be taken for granted (e.g., in the humanities, as article-length or book-length continuous print text or simply a digital display of it). Medium must be taken seriously as an integral part of any project. To transpose McLuhan's famous dictum, the medium is the project. This often means evaluators must participate in a medium that is unfamiliar to them. Such participation could demand different timelines and modes of review (see below), addressing questions emerge as to the interrelation of media, research, and publication. Is the medium appropriate to the project? How does medium affect the project and its dissemination? Is the medium theorized in the project? How might the medium contribute to less passive engagement with the project by other scholars?

1.2.b Design of interface, apparatus, and/or data structure should be considered a scholarly activity and evaluated as such. From the back end (e.g., coding) to the front end (e.g., interface), does the design demonstrate sustained and careful attention to the particular capacities and affordances offered by the technologies used?

1.2.c Collaboration is natural to most digital scholarship. Common assumptions of individual authorship may not apply. How do we measure the scholar's most essential role(s) in the project? What responsibilities does the scholar have in curating and negotiating the relationships in a project team? That is, what are the scholar's roles in the processes of research and design and also in the social processes of the collaboration itself?

1.2.d Process is implicated in all of the above questions. Indeed, in digital scholarship as well as in scholarly work in other emerging media technologies, the collaborative process often overshadows the “final” product (if there is such a thing). How do we evaluate and account for scholarly activities beyond writing, editing, and publishing, such as design, making, curating, facilitating, and programming?

2.0 Recommended Guidelines for Evaluating Digital Scholarship

2.1 Many institutions are already well underway in revising their procedures and standards for the appointment, tenure, and promotion of candidates in light of the changing media environment of religious studies specifically and academic culture more generally. Likewise, many graduate programs have begun revising related standards of evaluation for dissertations and theses. Many other programs, however, have not begun either process, and urgently need to do so. Such revisions are important not only for fairness to faculty and graduate school candidates working in new media but also for recruiting and retention, since many of the most important and innovative graduate students and junior scholars are doing this kind of work, and will not apply to or stay at an institution that does not adequately recognize the value of that work.

2.2 Recognizing that the specific procedures for revising such standards and evaluation processes vary widely across departments and institutions, we recommend that the AAR make additional resources available to them, including: model documents of procedures and standards for evaluation and standards from other institutions (i.e., similar to the AAR's syllabus repository); fora at the AAR for discussing issues related to the revision and implementation of such standard and procedures; and perhaps consultants who might meet with departments to help educate faculty in the issues and facilitate substantive engagement in the revision process, including reflecting on questions such as those in the previous section.

2.3 The following are recommendations of specific procedures and criteria that should be considered in the process of revising departmental standards. They are not ranked, merely enumerated. While these recommendations focus on evaluating faculty for tenure and promotion, many of them are easily adaptable to the context of evaluating the digital scholarship of graduate students.

  • 2.3.a Evaluation in the work's native environment. Above all, it is crucial to emphasize that scholarship must be evaluated in its native, medium-specific environment. As the CAA/SAH Guidelines put it succinctly, "it is crucial that digital work be seen in the environment for which it was designed. Scholars deserve to have their work taken seriously, including the digital contribution. Hence, all work of digital scholarship must be evaluated in its appropriate environment." This need drives most of the procedures, measures, and criteria that follow.
  • 2.3.b Internal committee. When at all possible, the number of members of the internal committee who have expertise/experience in relevant media technologies should be proportional to the use of these media technologies within the chosen, media specific environment of the project, even if that committee member does not have content expertise in the project's subject field. The committee might also consider contracting an external evaluator with media-specific expertise in order to help the committee engage and evaluate the work on its own terms.
  • 2.3.c External peer evaluators. The list of external peer evaluators should include scholars who have expertise/experience in the candidate's media technology as well as in her/his academic field in order to be able to offer media-specific peer assessment of the scholarship (e.g., appropriateness of the medium to the project, quality of back-end and front-end design, etc.).
  • 2.3.d Collaborator evaluations. Collaborators (internal or external) on the candidate's project should provide evaluations of the candidate's contributions to the larger process, including responsibilities in curating and negotiating the relationships in a project team.
  • 2.3.e Project narrative. The candidate should provide a project narrative that describes and documents the project and her/his roles in it as another important means of enabling the committee to "evaluate scholarship on its own terms as process" (CAA/ SAH Guidelines). This narrative might include specific benchmarks for evaluation and progress throughout the process. The project narrative could also include specific descriptions of the hardware and software requirements for viewing and participating in the project. In addition, the project narrative should credit project collaborators by mentioning their names and the contributions they made to the project.
  • 2.3.f Project demonstrations. The candidate might present annual demonstrations to department faculty throughout the project's development process. Reports on and/or recordings of these demonstrations would then become part of the candidate's file. Such demonstrations could be especially important when evaluating projects that have no clear endpoint or launch date.
  • 2.3.g Metrics of access and impact. There should be a clear standard for measuring evidence of the accessibility and impact of the project, that is, quality and quantity of access and use by other scholars and students (incl. links to it in other projects), and the development of new projects based on its technologies and/or design. For example, is the code of a project made available through a public collaborative repository such as GitHub? If so, are there forks (copies) and contributors pushing the project into new areas or applications? What kinds of conversations is the project stimulating, both technically and theoretically?
  • 2.3.h Grants. Is the project involved in conversations about funding for humanities research and teaching in this emerging media age? What grant funding has the project procured and what grant applications have been submitted or are planned to support the project?
  • 2.3.i Awards and other recognitions Has the project won any awards or received any recognition from other scholars or organizations in the study of religion (e.g. peer reviews or peer participant)?