بهبود فرایند بررسی با فن آوری اطلاعات

Peer review is the engine of scholarship where new knowledge is legitimized. Despite technological advances in publishing and communication, the process of review has not changed since it became prevalent over 100 years ago. This paper describes how information technology can be used to improve the peer review process. Taking a combined design science and natural science approach, we design and test a prototype system based on the principles of structured communication. Through an exploratory study, we find that our proposed system is viewed more favorably by both authors and reviewers across several dimensions, including fairness, convenience, and value.

Information technology has fundamentally changed the production and dissemination of new knowledge. For example, in academia knowledge producers (authors) create manuscripts on sophisticated word processing software, share drafts and discuss ideas using a variety of communication tools, and produce print-ready journal copy using desktop publishing software. Teams of scientists can share and work on complex projects in ways that were simply not possible a few years ago, while authors of textbooks are now experimenting with electronic delivery mechanisms that will radically change the distribution model of the publishing industry. Despite these technological advances, the process for evaluating the content of publications remains largely unchanged. The focus of this paper is peer review, the process by which new knowledge is legitimized by its acceptance and dissemination to the wider community. Peer review is often described as an instance of decision making (e.g., [33]), and as an example of knowledge production and dissemination. Peer review is also time consuming and expensive. Editors believe that the largest “cost” of producing a journal is reviewing and editing [22]. Weber [34] estimated a reviewing “opportunity cost” of $24,500 for each published paper in Management Information Systems Quarterly. IT has already impacted the peer review process, most notably in the use of web-based document management tools that can manage the process of submission, review, and arriving at an editorial decision. Yet the peer review process itself is essentially the same as it was since it became prevalent over 100 years ago [28]. Most information technologies simply automate the review process (e.g., [9]). The current peer review process of (often) slow back and forth deliberation among authors, reviewers, and editors continues the legacy of an earlier era defined by infrequent, high-cost communication. Watson [32] argues that though the Internet has improved some aspects of publishing, most of the changes are simple and focus on the tasks of publication (“alpha level”), there has been little to no change to people's roles (“beta change”) or a restructuring of the underlying system (“gamma change”). There is, however, a fundamental dilemma in researching how to use IT to improve the peer review process. There is no current literature that fully elaborates the potential problems. Further, there are no comprehensive behavioral models that can explain the impact of proposed IT enabled improvements. Therefore, we propose that both a design science and natural (behavioral) science research approach is necessary to build new utility into the peer review process and understand the impact of such proposed changes [15], [20] and [31]. Cao et al. [5], who build on Hevner et al. [15], March and Smith [20], and Nunamaker et al.'s [24] design science approaches, suggest that a multi-methodological, cross-paradigm research approach that combines design and behavioral science will yield more powerful and insightful results. Research on the peer review process is at an early stage and very little aggregate knowledge has been accumulated, a combined approach will therefore improve the chances that important behaviors are identified early and only the most useful technical artifacts are built. As Hevner et al. [15] argue, truth and utility are inseparable and “an artifact may have utility because of some as yet undiscovered truth. A theory may yet to be developed to the point where its truth can be incorporated into design” (p. 80). The remainder of this paper is structured using a combined design and natural science approach. First, we provide perspectives on peer review that illustrate the importance of the problem and our approach. Second, we apply the design science perspective to create a process based model to analyze the traditional peer review process and to serve as a baseline for changing the process. We propose a new process based on structured communication and implemented using a prototype system. Third, and in parallel, we apply the natural science perspective to develop a set of behavioral propositions to evaluate changes to the peer review process. The two perspectives of design and natural science come together in an exploratory study to evaluate the impact of the prototype system. In the final sections, the results of the study are discussed.

This paper analyzes the basic processes of peer reviewand proposed changes to the review process using an IT enabled structured communication approach. We investigated the problem from both a design science perspective through the development of a prototype system, and a natural science perspective by integrating previous literature into behavioral propositions that guided the evaluation of the prototype. An exploratory study provided further insights on the prototype system and on the underlying design and behaviors. This paper contributes to the literature at three different levels: • From the design science perspective we prototype and demonstrate how structured communication can improve thepeer reviewprocessbyemployingartifactsthatincrease mutual understanding. An evaluation of the prototype sug- gests that future design research should consider inte- grating the design concepts of “ discussion forums ” and “ work fl ow ” to peer review. Newer design concepts also need to consider reviewer motivation and look for new and easier ways to engage reviewers. We believe our work is one of the fi rst to consider and demonstrate how changing the process of review can lead to improvements. • From the natural science perspective, we show how the theoretical perspectives of procedural justice, the rational actor, and economic valueareimportantto understandpeer review. These perspectives can also be applied to other forms of peer review such as product reviews on the web or audits of accounting practices. Procedural justice speaks to concerns of fairness that will likely always exist in any process that involves evaluation of one group by another. The rational actor perspective provides insights into theadministrative aspects of any kind of review, and the economic perspective provides insights on why people choose certain behaviors. The combination of these per- spectives into propositions is the fi rst step toward devel- oping a comprehensive behavioral theory of peer review. The evaluation of the prototype provided additional in- sights including expanding the conception of value, adding learning as a concept, and suggesting re fi nements to the initial set of concepts. • This paper also contributes to implementing and re fi ning design science methodology. In line with suggestions from the literature [5,15,31] we extend the application of the design science perspective by combining it with the natural science perspective. Rather than settle for traditional benchmark style measures such as ease of use or perfor- mance, we developed behavioral theoretical propositions in conjunction with prototype development. The process of developing the prototype and in parallel, considering eval- uation measures and theoretical approaches allowed us to converge on what we believe are key design artifacts (e.g., structured communication, work fl ow) and behaviors (e.g., fairness, value, learning) that might have been ignored by traditional approaches. The propositions provide a sub- stantive basis for evaluating current and future systems, and a speci fi c direction for future research. To summarize, the combined approach is powerful because it generates new insights and lends more credibility to the results by anchoringdesigntobehavioral issuesof truthandrelevance and anchoring behaviors to design issues of utility and feasibility. The structured communication process proposed in this paper is relatively easy to implement, and we encourage editors of conferences and journals to continue the process of experimentation. Moreover, given that review is, at its core, a process, we believe thatfutureIS researchcan continuetoadd signi fi cant value to this very important aspect of academic scholarship as it is increasingly applied to other domains.