Several countries partly fund universities on the basis of scientific performance. In this context, peer review is often used to assess the quality of scientific products. In large research assessment exercises peer review may become costly, and bibliometric indicators have been suggested as an alternative. A recurrent question is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review at the institutional level. Additionally, we also quantify the internal agreement between two peer reviewers. We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two peer reviewers. This suggests that bibliometric indicators may indeed be considered as an alternative to peer review for large institutional research assessment exercises.

Metrics and peer review agreement at the institutional level

Legal scholarship has an ambiguous identity, somewhere between the humanities and the social sciences, having features in common with both. It shares many of the characteristics of social sciences since the law is a social phenomenon, but when normativity and legal certainty are concerned, legal scholarship is probably closer to humanities (Peruginelli & Faro, 2018). At the same time law is distinct from both. There is an interesting debate on the role of academic legal research in which consideration is given to 'law as a practical discipline', 'law as humanities' and 'law as social sciences' (Siems & Mac Sithigh, 2012). Because of the peculiarities of legal scholarship, it is not easy to assess legal journals according to the standard quantitative metrics applied in social sciences, and qualitative assessment of the published content is more appropriate. Therefore, we compared the peer review process employed by legal journals in three countries, Croatia, Italy and Spain, to find out their characteristics. 
In this study, 34 Croatian, 36 Spanish and 40 Italian law journals (selected randomly from the 153 top ranking Italian law journals) were analysed. For each journal, we collected basic set of metadata, as well as a document providing the description of the peer-review process. We collected 107 documents in four languages (English, Croatian, Italian and Spanish) which were analysed in order to find out characteristics of the peer review process. For text analysis, we developed an extensive categorization dictionary. Nine main categories in the Categorization dictionary are peer review (process and reviewers), reviewer's characteristics (academic level, autonomy, competence), reviewer's provenance (external, internal), peer review blindness/openness (single-blind, double-blind, open), number of reviewers (one, two), evaluation criteria for submissions (originality, methods, relevance, clarity, accuracy, etc.), peer review outcome (report, acceptance, rejection), ethical issues (editorial standards and codes, conflict of interests, confidentiality, research integrity), manuscript type (original scientific article, professional article, etc.).
The most frequent categories are "peer review", "submissions' evaluation criteria" and "manuscript type", present in more than 90% of the collected documents. "Reviewer's competences" are the most represented subcategory in "“reviewer's characteristics“, as high level of reviewers expertise in the legal sciences is important for law journals. 73% of analyzed journals are declaring anonymous peer review, 25% double-blind peer review, and only one journal employ single-blind peer review. 42% of documents contained information on "two reviewers", 8% "one reviewer" and 7% "three or more reviewers".
Ethical issues are certainly under-represented in documents describing peer review process and editorial policies issued by Italian, Spanish and Croatian law journals. Only 32% of the documents mention conflict of interests, and all other subcategories are present even less than 30%. 
Croatian legal journals are addressing research misconduct and different publication types more than the other two countries. Also, they consider manuscript originality as very important, but some important submission evaluation criteria, reviewer characteristics and peer review outcomes are not present. On the other hand, Spanish legal journals are addressing more reviewer's competences, but some important ethical issues are missing. Italian legal journals take care about the reviewer's competences, academic level and objectivity, as well as anonymity of the peer review process, but expected peer review outcomes and ethical issues should be mentioned in a greater extent.
References
Gutwirth, S. (2009). The evaluation of legal science. The Vl.I.R.-model for integral quality assessment of research in law : what next ? In H. Eisendrath & J. P. Van Bendegem (Eds.), It takes two to do science. The puzzling interactions between science and society (pp. 69–80). VUBPress.
Peruginelli, G., & Faro, S. (2018). Research quality evaluation: The case of legal studies. In The Evaluation of Research in Social Sciences and Humanities (pp. 103–129). Springer Cham.
Siems, M., & Mac Sithigh, D. (2012). Mapping Legal Research. Cambridge Law Journal, 71(3), 651–676.




Peer Review in Legal journals

The time required for a vehicle to travel different routes in the daily traffic of large cities varies and changes constantly, impacting the daily lives of everyone dwelling in those cities. Trying to predict such time is essential to evolve in understanding the behavior of vehicle traffic. On the other hand, due to the vast amount of data generated in this context, it is necessary to use new ways of dealing with this problem. This paper presents an exploratory analysis of the behavior of batch and stream learning algorithms for predicting the trip duration time for vehicles going through different routes. We understand that batch learning algorithms are not necessarily adequate for being used in stream mining situations. However, we would like to have a testbed to analyze the behavior of stream learning algorithms. For our experimental analysis, we used real data from three specific routes. The results show that the use of data stream learning for this domain yields promising results

An Exploratory Study Using Stream Learning Algorithms to Predict Duration Time of Vehicle Routes

Intelligent transport support systems have had a major impact on people’s urban mobility. In large urban centers, transportation services still need ways to optimize vehicle supply in certain areas, according to the demand in each of them. This work proposes a stream learning methodology for taxi demand prediction, that tackles concept drift detection on a real taxi data stream.

A Methodology for Taxi Demand Prediction Using Stream Learning

Smart cities should be built on harmonious relationships with the environment, with rational measures to use environmental resources in many ways. Resources essential to human life such as water and energy must have special attention. With the advance of technologies use, human being becomes one of the main promoters of information and solutions for smart cities. Stream machine learning is an approach where data becomes available sequentially, called data stream, which adapts the prediction model with fixed window-sized data stream. The objective of this work is to analyze the effectiveness of buildings energy consumption using stream learning. To achieve this goal, MOA tool was used to evaluate the performance of the selected stream learning algorithms on an an open database. Batch algorithms were also used as testbed, to analyze the performance of stream algorithms related to them

Elias Sanz-Casado

5

1

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES