In peer review, reviewers often have formal and informal occasions to interact with other reviewers of the same grant proposal or submission. The former happens, for example, when grant proposals are evaluated by sitting review panels who are tasked with collectively deciding the fate of a particular proposal. Despite the scholarly consensus that interactions (and thus social influence) can alter the outcome of collective decision making, to date social influence has rarely been a theoretical ingredient in simulation models of peer review. In our contribution we use a simple agent-based model of peer review to show that social influence processes can indeed change the dynamics and outcome of peer review. With our simulation experiment we showcase the incorporation of social influence processes in peer review models as a promising direction for future research.

Does social influence matter in peer review?

Sentiment analysis methods have been developed to automatically extract opinions from texts, e.g. online reviews. Here we explore the idea of adopting sentiment analysis to extract grant reviewers' opinions. This paper examines the relationship between the sentiments of grant reviews and reviewed proposals’ funding decisions made by agencies. We define peer reviews’ prediction as to what degree reviews’ positiveness/negativeness, via sentiment extraction, can indicate proposals’ chance of success in receiving funding. Building on a corpus of peer review texts and related documents from an Irish science funding agency, we conduct sentiment extraction by manual and two simple automated coding (TextBlob and VADER). A comparison between the three results has been made. We find the manual and automated coding results to be overall consistent, but manual coding is still the most accurate and reliable for peer review studies. Furthermore, the three coding results show strong prediction of proposals’ funding decisions, which indicates that reviewers' opinions are largely adopted by agencies.

Peer Reviews' Prediction in Proposals' Funding Success: A Sentiment Analysis of Grant Reviews

Legal scholarship has an ambiguous identity, somewhere between the humanities and the social sciences, having features in common with both. It shares many of the characteristics of social sciences since the law is a social phenomenon, but when normativity and legal certainty are concerned, legal scholarship is probably closer to humanities (Peruginelli & Faro, 2018). At the same time law is distinct from both. There is an interesting debate on the role of academic legal research in which consideration is given to 'law as a practical discipline', 'law as humanities' and 'law as social sciences' (Siems & Mac Sithigh, 2012). Because of the peculiarities of legal scholarship, it is not easy to assess legal journals according to the standard quantitative metrics applied in social sciences, and qualitative assessment of the published content is more appropriate. Therefore, we compared the peer review process employed by legal journals in three countries, Croatia, Italy and Spain, to find out their characteristics. 
In this study, 34 Croatian, 36 Spanish and 40 Italian law journals (selected randomly from the 153 top ranking Italian law journals) were analysed. For each journal, we collected basic set of metadata, as well as a document providing the description of the peer-review process. We collected 107 documents in four languages (English, Croatian, Italian and Spanish) which were analysed in order to find out characteristics of the peer review process. For text analysis, we developed an extensive categorization dictionary. Nine main categories in the Categorization dictionary are peer review (process and reviewers), reviewer's characteristics (academic level, autonomy, competence), reviewer's provenance (external, internal), peer review blindness/openness (single-blind, double-blind, open), number of reviewers (one, two), evaluation criteria for submissions (originality, methods, relevance, clarity, accuracy, etc.), peer review outcome (report, acceptance, rejection), ethical issues (editorial standards and codes, conflict of interests, confidentiality, research integrity), manuscript type (original scientific article, professional article, etc.).
The most frequent categories are "peer review", "submissions' evaluation criteria" and "manuscript type", present in more than 90% of the collected documents. "Reviewer's competences" are the most represented subcategory in "“reviewer's characteristics“, as high level of reviewers expertise in the legal sciences is important for law journals. 73% of analyzed journals are declaring anonymous peer review, 25% double-blind peer review, and only one journal employ single-blind peer review. 42% of documents contained information on "two reviewers", 8% "one reviewer" and 7% "three or more reviewers".
Ethical issues are certainly under-represented in documents describing peer review process and editorial policies issued by Italian, Spanish and Croatian law journals. Only 32% of the documents mention conflict of interests, and all other subcategories are present even less than 30%. 
Croatian legal journals are addressing research misconduct and different publication types more than the other two countries. Also, they consider manuscript originality as very important, but some important submission evaluation criteria, reviewer characteristics and peer review outcomes are not present. On the other hand, Spanish legal journals are addressing more reviewer's competences, but some important ethical issues are missing. Italian legal journals take care about the reviewer's competences, academic level and objectivity, as well as anonymity of the peer review process, but expected peer review outcomes and ethical issues should be mentioned in a greater extent.
References
Gutwirth, S. (2009). The evaluation of legal science. The Vl.I.R.-model for integral quality assessment of research in law : what next ? In H. Eisendrath & J. P. Van Bendegem (Eds.), It takes two to do science. The puzzling interactions between science and society (pp. 69–80). VUBPress.
Peruginelli, G., & Faro, S. (2018). Research quality evaluation: The case of legal studies. In The Evaluation of Research in Social Sciences and Humanities (pp. 103–129). Springer Cham.
Siems, M., & Mac Sithigh, D. (2012). Mapping Legal Research. Cambridge Law Journal, 71(3), 651–676.




Peer Review in Legal journals

The time required for a vehicle to travel different routes in the daily traffic of large cities varies and changes constantly, impacting the daily lives of everyone dwelling in those cities. Trying to predict such time is essential to evolve in understanding the behavior of vehicle traffic. On the other hand, due to the vast amount of data generated in this context, it is necessary to use new ways of dealing with this problem. This paper presents an exploratory analysis of the behavior of batch and stream learning algorithms for predicting the trip duration time for vehicles going through different routes. We understand that batch learning algorithms are not necessarily adequate for being used in stream mining situations. However, we would like to have a testbed to analyze the behavior of stream learning algorithms. For our experimental analysis, we used real data from three specific routes. The results show that the use of data stream learning for this domain yields promising results

An Exploratory Study Using Stream Learning Algorithms to Predict Duration Time of Vehicle Routes

Intelligent transport support systems have had a major impact on people’s urban mobility. In large urban centers, transportation services still need ways to optimize vehicle supply in certain areas, according to the demand in each of them. This work proposes a stream learning methodology for taxi demand prediction, that tackles concept drift detection on a real taxi data stream.

Ginevra Peruginelli

5

5

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES