Summer School - Opening the Black Box of Quality: Reflecting on Scholarly Practice in the Social Sciences and Humanities

Reflecting on and assessing the quality of scientific work is part and parcel of scholarly practice – be it in various work situations or the different stages of an academic career. As scholars, we judge the quality of a book that has recently been published; as teachers, we give advice to students on how to increase the quality of their projects and grade term papers; as doctoral students, we assess the quality of related work to position ourselves in a particular field of expertise; and, not least importantly, we constantly reflect our own work in terms of implicit and often informal standards of quality that govern our working practices and the formation of our academic selves.

It thus does not come as a surprise that scientific quality has become an important research topic in itself. A large fraction of the literature addresses the procedures and practices of peer-review. Over the past decades, scholars have presented rich analyses of peer review processes. Studies in this area of research show how such procedures of quality assessment are organized and direct attention to their biases and structural flaws in different ways
(Lamont 2009, 2012; Wenneras 1997). Other studies emphasize the multiple negotiations that take place in quality assessments and indicate that quality is a fluid category co-constitutive with disciplinary boundaries, ideas about center and periphery, and conventions concerning the design of a ‘good’ paper or study (Hirschauer 2010; Lack/Markschies 2008).

One of the main insights of this literature is that peer review does not merely measure scientific quality or render academic performance transparent. Rather, peer review processes constitute academic performance and achievement as ‘objects’ that can be measured (Strathern 2000a). This is an important finding given the rise of what is called ‘audit society’ (Power 1997) or ‘audit cultures’ (Strathern 2000b). The processes in which academic
performance is assessed render visible some practices while obscuring others. In this way, they attach importance differently, privilege certain practices at the expense of others, and, as a consequence, also structure the self-evaluation and identity formation of researchers and scientific institutions. E.g. the choice of publication strategies or between writing one’s PhD-thesis as a monograph or as a set of papers is no longer made according to epistemic criteria alone. Together with methodological and scientific standards of quality, questions of career trajectories or access to prestigious institutions and positions on the basis of a publication record play an increasing role. Additionally, in the case of research proposals, such assessments have led to the emergence of an economy of promise in which the ability to articulate intriguing, while still plausible, promises has become an important asset. Thus today, the question of how academic peers assess the quality of scientific work is more than
ever a pressing and disputed problem for scholars in all disciplines.

The proposed Summer School will first of all introduce its participants to the aforementioned debates of leading scholars (e.g. Hirschauer, Lamont). Second, and against this background, it will direct attention to the multiple scholarly practices of attributing and negotiating quality that structure our academic lives beyond the institutional settings of formal peer review. In the literature, these mundane practices have not received similar attention until now; instead, they tend to remain black-boxed in much the same way as they are in both academic and policy debates. To open up this black box implies that attention is directed also to those instances in which scientific work is being valued in often implicit ways. The Summer School will use a comparative optic as a heuristic device to render visible the invisible. In particular, we will draw on the disunity and heterogeneity of scientific cultures as concerns their quality standards and practices. In this sense, the students’ and lecturers’ diverse disciplinary backgrounds within the social sciences and humanities will become an essential resource for all participants to engage in a debate on what counts as good, appropriate, innovative, original, etc. scholarly work in specific fields and situations – and according to which criteria of worth. In the context of the Summer School, our attempt to open up this black box will be guided by focusing particularly on how quality is attributed, contested, and negotiated in writing and reading (Hyland 2004, 2012). In critically reflecting these practices, we will thus also engage with the question of what constitutes a ‘good’ PhD-thesis.