FSAM Conversation Letter 3 – April 19, 2021
On the Third Webinar, March 26, 2021
of IFSAM Webinar Series on Management Research Evaluation
Co-chairs:
Xavier Castañer, President Elect, International Federation of Scholarly Associations of Management (IFSAM)
Kerry Brown, ANZAM President & member of our Council
This conversation letter contains an extensive summary of the contributions of the panellists and cochairs and a short commentary by two participants, by panellist Amanda Wilson (Wiley) and Andrew Jack (FT). The summary from the co-chairs has been drafted by Elisa Martinelli, IFSAM deputy secretary, and edited by the co-chairs.
Keywords: research evaluation; scholarly journals; ranking system; impact; economic model; open access (OA); open research; citation metrics; quality signalling; standard; DORA
Panellists:
– James Cleaver, F1000 Research Ltd Manager of strategic partnerships with learned societies (former T&F Manager of portfolio of management & economics), UK
– Amanda Wilson, Editor, social sciences Wiley, USA
– Heloise Berkowitz, Co-EiC, m@n@gement, France
– Liyuan Zhang, President, economy & management publishing division, HUASHANG Enterprise,China
– Marrisa Fernando and Sirichai Preudhikulpradab, EiCs ABAC ODI, Thailand
–Martina Bihn and Juno Kawakami, editors in Germany and Japan respectively at Springer Nature Group, business, economics, law & political science
– Felipe Zambaldi, Co-editor of RAE, ANPAD, Brasil
Following the first webinar on Jan 29 with representatives of different stakeholders in the field, the second webinar on Feb 24 with representatives of the stakeholder group of university presidents and management school deans from around the world, the third webinar gathered representatives of scientific journal EiC and editors of publishing houses with management journals and books.
We asked them to choose a couple of the following questions and briefly tell us about their viewpoint:2
1- what is the situation of publishing scientific research in management (from your specific vantage point, i.e. type of publication) in these pandemic times?
2- which types, formats (articles, books, handbooks, textbooks,…) and dimensions of scientific research in management should the focus be in?
3- what is your organization and your personal view on open science and open access? What is your view on the Dora declaration?
4- how could quality (which dimensions) of scientific management publications and their diffusion (reading) improve as evaluated by different audiences (including managerial practitioners across sectors)?
a. How would you seek to demonstrate and measure the societal value of your publications?
b. Would you recommend a switch away from journal impact factors in assessing their scientific quality? If so, what would you recommend as alternative/s?
5- do you see value in moving towards a standardized unique reporting system (e.g. Orcid for authors, Ror for institutions)?
We next summarize some of the key insights we believe each brought, providing later a summary of the chat, the chairs’ conclusions and the Q&A part.
1. Panellists’ contributions
James Cleaver (F1000 Research Ltd manager of strategic partnerships with learned societies, UK) argued that there is a fragmentation or polarization of the research system caused by the fact that research and thus faculty evaluation processes are too centered on journal rankings and metrics, especially in business schools. James believes that the “unit of currency” in publishing should be the article and not the journal, and this is even more the case in an Open Access/Open Research world.
The DORA declaration has existed for more than a decade, advocating for not just using simple journal metrics (IF, journal raking lists) but a wide array of measures to assess research. F1000 is an original signatory of DORA while Taylor and Francis signed only in February 2021 but it immediately displayed on its journals’ web-site a detailed metrics page with an explanation on how these metrics are calculated and used, which are their limits and so on. James thinks that this is the kind of culture and system we need in assessing research. F1000 relies on article-level metrics (downloads, views and citations) in order to value the article on its own merits rather than on its venue of publication. James suggests considering other systems to assess research, including values - like
HumetricsHSS (equity, openness, community, etc.) - and real-world impact - such as Altmetrics (social media interactions, news coverage, etc.). This can bring to a more holistic view of what a researcher is doing and contributing, especially if we are able to enhance the role of open research activity in research assessment: sharing, reviewing, increasing transparency in author contributions (via the system called CRediT), and engaging in sustainable activities. James advocates for a mix of qualitative and quantitative measures to align incentives. Actually, we have to make a choice: continue to rely on imperfect journal ranking lists or change the evaluation culture. Amanda Wilson (Editor, social sciences Wiley, USA) describes the situation of publishing scientific research in management in these pandemic times. According to her, Wiley has had an increase in submissions, evidencing a gender imbalance and acknowledging the key importance of giving reliable online access to contents. Print and distribution were challenged by the pandemics but Wiley managed to cope with it and was involved in many virtual conferences. In terms of open science and open access (OA), Amanda shares that Wiley is completely committed to this transition. It successfully launched more than 200 fully Gold OA journals in 2020 and during the current year it is going to double the number of Gold OA journals thanks to the acquisition of Hindawi. Amanda observes that across the academic community there is an increasing need for greater transparency throughout the research process, increase diversity, favor more inclusive research practices and encourage data to be open. There is much research coming out than ever in the management field, which is increasing at 6% a year, leading to a surplus of output which requires to clearly identify more structured processes.
Regarding OA in the management field Amanda believes there are some barriers to OA progress as sometimes OA is perceived as predatory and low quality. Moreover, few fully OA management journals of high-quality are currently available and the discussion on OA in key management conferences is poor. Dora has a good aim but the management community should be more committed in supporting the change.
Heloise Berkowitz (Co-EiC, m@n@gement, France) depicts her journal as free to submit, free to publish and free to read, fully supported by French academic associations and institutions and strongly based on the voluntary contributions made by editors, authors and reviewers. Heloise shares how their diamond model positioning is deeply nourished by a view of science as a “global public good”, based on the assumption that our society is better able to progress and to solve problems if and when science is freely accessible to everyone. Heloise thinks that the best way to accomplish this and develop diffusion is through open access (OA): visibility generates usage and usage turns into impact. OA maximises audience. This is connected with a broader trend in the EU towards Open science and Mission oriented science: the former concerns a policy priority in the EU about making research results and data available to everyone as early as possible, while the latter consists in dealing with big societal challenges. Heloise finds them in full contradiction with the current publication system and incentives’ structure. Open science and mission science are both involving commitment by institutions in supporting alternative and innovative ways of publishing and evaluation that could differ from the current standard ones. This is what the French National Institute for Research and Science (CNRS) is doing, advocating for open access and the DORA declaration. Heloise reports that the CNRS economics and management session, composed by peers to evaluate other researchers, has just officially renounced to journal rankings. A French initiative of 100% open repository of publications is under the way too. Heloise underscores that these are tangible signals that something is changing.
Liyuan Zhang (President, economy & management publishing division, HUASHANG Enterprise, China) described the publishing activity carried out by her organisation and its current strategic objectives. The first one is concerned with planning a series of books for innovating future education.
The second one refers to training college teachers and students in writing and publishing articles in global journals. The third one is to enhance studies on cultural industries. Liyuan focused her speech on how to evaluate the quality of academic work and textbooks, underscoring the importance of the policy orientation of the Chinese government which encourages research which contributes to the country development and the important role played today by China in science in general as well as by cutting-edge research in AI, new materials, content-technologies, smart medicine, 3nanotechnologies, cloud computing, etc. in particular which may be examples of new discoveries to include in textbooks. According to Liyuan, new subjects and contents in light of multidisciplinary integration are needed together with a particular attention to practical problems.
Martina Bihn and Juno Kawakami (editors in Germany and Japan respectively at Springer Nature
Group, business, economics, law & political science) briefly presented some of their most successful journals/books in business and management edited by Palgrave and Springer. Martina reminded us that Springer Nature is a signatory of DORA, supporting the variety of metrics it requires. Martina thinks that top journals are not problematic and that we do not have to look only at articles: journals reflect the evolution of research agendas. In terms of the effect of the pandemic, Martina also reported an increase in submissions to journals. Then, she underscored Springer Nature’s policy towards open access (OA) and its strong commitment to speed up the transition in a sustainable way. The main issue in the management research field with OA is having the funds to afford it, so they are working on transformative agreements in different regions/countries in order to bring leading journals into OA. They are also working on societal impact visibility, data availability, as well as on more transparent peer reviews. Martina thinks that an open research environment is also a way to foster research on societal relevant areas, citing as example the Springer Nature’s Sustainable Development Programme. Juno added on the OA book side in business and management, especially regarding textbooks and professional books series. Juno manages also a number of monograph series in Japan and the rest of Asia, several of which are indexed in Scopus. Nowadays, topics gaining importance are digital transformations, blockchain, fintech, SCM and logistics (enhanced by the pandemic spread), sustainability, diversity and inclusion, among others. Similar methods of evaluation used for journal articles exist for books: Juno said that figures on downloaded chapters are one of the key metrics they use, broken down by region, together with book sales, citations and recommendations, popular book reviews in magazines and academic journals, awards and prizes. Regarding OA books in business and management, Juno underscored that this kind of books enjoys 2.5 more citations and 10 times more downloads, on average, within the same kind of monograph or textbook category. This kind of books has also a more geographically diverse readership, reaching about 60% more countries than non-OA books. Springer Nature is also trying to ensure access to funding for book authors, even if this is still in authors’ hands for the majority of cases.
Marrisa Fernando and Sirichai Preudhikulpradab (EiCs ABAC ODI, Thailand) described their international journal: an OA journal, published biannually, using OJS platform (as all the university journals they publish). Marrisa considers it beneficial as it is free of charge for downloading and very good for knowledge sharing, as the final aim is to further research, reaching and engaging as many researchers as possible. In terms of quality and sustainability, Marrisa thinks it is the responsibility of the Editor and of the Editorial board to ensure them along with the integrity of the published research. DORA honours the diversity of academic activities and represents a good standard. Marrisa reported that in Thailand the Thai Citations Index (TCI) serves as a database of academic papers published in Thai journals, both in native and English languages, and they also have the Asian Citation Index (ACI) relative to the research undertaken in ten Asian countries. In relation to the pandemic, Marrisa observes the emergence of new management topics in terms of leadership, behaviour, innovation and new methodologies in terms of data gathering, enhanced by technology.
In terms of societal value, Marrisa believes in the importance of understanding the context specificities. Sirichai added on the kind of articles they are looking for. Their journal welcomes works from practitioners and academics, focusing on organisational topics not only from highly experienced researchers, but also from emerging ones (Master and PhD students). They are particularly interested 4on the impact that research can immediately have on organisational systems, people, processes and technologies, especially in articles that leverage different research methodologies.
Felipe Zambaldi (co-editor of RAE, Brazil) presented RAE, a general management journal celebrating 60 years this year. For more than fifty years it was published mainly in Portuguese and it is very relevant locally. In 2009 it was indexed in WOS and Scopus, enlarging the audience although attractiveness of their journal still remains an issue related to its impact factor (IF). Now it is published in three languages, as English and Castilian (‘Spanish’) are accepted too. RAE main readership is in academia and submissions from abroad are rising, especially from Spain and Latin America. Brazil has its own system of evaluation through the Cielo platform - an American platform indexing journals - and a regulatory agency (CAPES) assessing the quality of publications. There is an increasing pressure to get more international. Felipe observes that access to research worldwide is increasing while he thinks that management schools and higher education-related institutions in general are relying too much on journal metrics, leaving small room for researchers to publish in niche outlets.
The submission rate to RAE in 2020 increased, especially for topics concerned with gender imbalance and socio-related issues. He also observes that many marketing scholars are changing their research interests from consumer behaviour to digital platforms. Felipe doesn’t think that relevant research should only be published in top journals; in order to produce a higher societal impact, Felipe thinks we should produce research involving stakeholders like managers rather than only collecting information from them.
2. Summary of the chat
Panellists’ insights raise a lively debate on the webinar chat that we report next as we consider it highly beneficial to the overall discussion.
Nicholas O’Regan argues that the issue is that in CABS and ABDC lists, early career researchers (ECR) use the ranking to get recognition and be promoted. James (Cleaver) agrees but he underlines that ECRs only do that because of institutional demands on them. If institutions focused less on the publication outlet and more on some of the other areas he mentioned in his speech instead, everyone would be a lot happier. Andrew Jack (from the FT) adds that the challenge with article-based metrics is impact (citations), which takes several years to emerge; prestige publications provide some earlier indications of quality but he posits that we should find alternative ways to swiftly benchmark for high quality research before/instead of citations. James does not agree as he reaffirms that the publication outlet tells us nothing about the quality of the individual article. Andrew Jack replies underscoring that it takes on average 8-10 years in many academic fields to get a significant/representative citation count, so that’s not very fresh or indicative of the research quality of a business school today - or presumably much use in faculties seeking to hire or promote based on research assessments.
Nicholas reminds that while many schools have signed up to DORA, many still specify CABS rated journals in their recruitment adverts. Recruitment panels tend to rely on heuristic modes such as rankings. He doubts that it is feasible to ignore rankings and argues that we should instead spend time reading the actual publications. Andrew Jack agrees that acceptance/publication in a high impact journal is still a metric of quality. Self-published individual articles have much less consideration, unless widespread and rapid reactions/critiques can be considered. He wonders what alternative early indicators of quality could be used at scale. James replies that this is why citations are not a great measure: downloads, views, and other ways of assessing societal impact, etc. are much better than citations. For example, a poor piece of work could be widely cited because people want to underscore 5how much they disagree with some earlier findings. Andrew Jack counteracts that there are growing efforts to use data analytics/network analysis to "clean" citations, to remove self-citation, "negative" citations and weight influential other citations. Downloads, views, etc. should definitely be considered - but also need cleaning to weight the nature of those viewing.
Later on, when the discussion shifted more into social impact measures emerged, Andrew Jack launched the discussion on how to measure those consistently: UK REF is interesting, but it is based on a small selection in one country. How could it be replicated consistently for a broader range of academic outputs around the world? James intervened writing that stakeholders must invest more in the infrastructure to do that – to measure diversity of research outputs, digital footprint, metadata and discoverability. This then can enable delivering visibility and credit for all research activities rather than a final article in a so-called "top journal". James thinks that the recognition of research as an iterative process and not as a one-off publishing event is key and can help to overcome the need for journal lists and journal hierarchies.
In relation to OA, many people from the audience commented on the high costs to publish OA, sometimes prohibitive for individuals.
Madeleine Barrows (BAM CEO) asked the panellists about what they think the management learned societies could be doing to influence the problems associated with the current academic (journal rankings-based) culture. Amanda (Wilson, from Wiley) replies that it would be helpful if the different learned societies in management could start by setting up coordinated debates on the role the ranking lists have, discussing the positives and negatives of these lists - and the blockers to change. James adds that in his opinion, societies should be highlighting and advocating for the value of all the types of research, teaching, and knowledge-exchange activity that their members do and that this should be considered by higher ed institutions when assessing performance for promotions, etc., in order to help their members gain the credit and visibility for everything they do. Andrew Jack contributes suggesting that learned societies could agree to open access (possibly switching to an author/b school pay to publish model), explore/share the widest possible range of metrics (downloads, views, etc) and foster curated conversations/reactions to articles. Xavier (Castañer, then IFSAM president elect) shares that this is what IFSAM is trying to do, to foster this dialogue among the member academies, through this webinar series and the upcoming Council meeting on May 10th. Amanda recognises this effort and she commits to support an ongoing program of discussions amongst the key global societies.
3. Chair’s conclusions
Xavier, as co-chair, concluded pointing out five issues raised by the panellists’ speeches.
The first one concerns scholarship, trying to respond to the following question: what is research about? Scientific research is about scholarship, that is: scientifically creating (new) knowledge. Xavier thinks that one of the key elements regarding this concern is the double-blind review system, one of the most important practices to guarantee scholarship. So, a critical issue is how the publishers’ call for transparency might affect the double-blind review system.
The second issue is quality signalling, brought by James (Cleaver) as well as Andrew Jack in the chat.
What is the (early) signal of scientific quality? There is a delay in the articles’ publication and their citations, thus possibly the need to find a combination of signals.
The third issue is probably the most important in the path towards OA: the economic model of management research. Who pays for producing scientific knowledge? How much? OA require authors to pay to be published, so this is an important issue to deal with. And: who pays for accessing those publications? Heloise (Berkowitz) brought the concept of a public good. Is scientific research a pure public good? Public economics defines public goods as a good which is characterized by nonexclusive provision and non-rival consumption. Xavier thinks we can agree that a journal article exhibits non-rival consumption as we can all read it at the same time without impairing each other’s capability to read it, while he is not sure that provision is non-exclusive, because up to now someone had paid for those journals to be accessible at universities. Indeed, some of the new models Heloise was talking about are challenging this economic model in which publishers are central actors.
The fourth issue regards research “relevance” to other actors, outside scholarship (scientific contribution). It is now labelled as “impact”, managerial or towards society. The challenge is to have measures of impact that could be replicable all over the world. But how do we measure relevance?
Relevance to whom?
The last issue concerns standards. Xavier refers to an issue Felipe put forward: with the diffusion of these metrics across the world we are leading the production of scientific knowledge towards a single standard.
Xavier concludes underscoring the presence of a collective action challenge: academics, as individuals, are the most fragmented kind of stakeholders. Academic associations in different countries/regions and their federations (like IFSAM with this webinar series) sometimes try to provide a platform for expressing their voice, but this is rare, again due to the challenge of collective action among so many individual actors. Then we have a second, less numerous stakeholder group, thousands of management schools and universities in the world. And finally, the publishers, which are more concentrated. He thinks that there is an issue on “who can do what” in a most effective way. Then, Xavier opens the discussion.
4. Summary of the Q&A after the conclusions
Kerry Brown (ANZAM president and webinar co-chair) asks Amanda to highlight some of the key insights on transformative agreements. Amanda replies that publishers have signed the new transformative agreements with national library groups to enable authors to publish OA (with the APC fee agreed between institutions and publishers up front) - and to remain compliant with funder mandates around OA. Amanda observes a lack of deep understanding around how business models for scholarly journals work, today very complex. Particularly in the last decade, with the drive to OA, the funding issue has become key not only for authors – who started to get funds only if their research could be made available for everyone - but also for the library community which simply does not have the budget to keep paying for licenses.
Martina adds that this situation came up in other areas than HSS in which there was large funding at risk. She suggests that in an area like management we should be more open to new models. When everything will be OA, Martina thinks that no one will have to pay apart from the service offered by publishers. The latter are developing a wide offer for the society and an improved level of personalisation to authors, helping in disseminating research, making data available and rendering the process more transparent.
Nicholas Oregan asked in the chat whether the current publishers’ business model is not a utopian one. Amanda replies that the current model is more difficult to manage. A hybrid situation is not ideal.
Ranking lists and IF make difficult to start a new journal or a new area of research: people are not willing to publish on it as long it doesn’t have an IF and at the same time it requires an immense investment for publishers and take times.
Xavier comments that Amanda has raised an important point: the asymmetric access to economic resources in different universities. However, he underscores that in the management field academics are doing the reviews for free, editing the article for free, etc. The central, core input of scientific publications is knowledge and academics are actually providing it for free. How is the new system going to affect this? Xavier reports also on the fact that new regulations are pushing scholars to render data available in an OA manner, something that it makes sense only in a pure public good context, in order to accelerate scientific knowledge creation. But, at the same time, the business and management fields are characterised by competition: rendering data OA can soon lead to cancel this comparative advantage.
James replies that this is why we need to think about research, OA and data sharing as a big interconnected web issue that we have to simultaneously address. He also recognises a problem regarding competition and the OA proposal of sharing data, but he also underscores the possibility to find collaborators not only competitors, so as to take research further and faster. For James the issue is what you want the research culture to be in the future, given the marketisation of research culture which has taken place. Amanda agrees with James: research should be concerned with unlocking new knowledge, working collaborative rather than in competition. Martina adds that there are always two perspectives: the individual one and the societal one. From the latter point of view is better if everything is open and available. She welcomes a move towards an open world where there is maximum distribution, maximum cooperation, maximum access and availability; however, she underscores that there is also some payment in this move. She states that “we are going easy from paying to read it, but we are not going away from paying to produce the framework, the setup, the investment in IT solutions, the channels through which all of this will be distributed”.
Martina suggests to see it from this perspective: not as the payment of a product but as the payment of a service. Sirichai suggests to identify a societal IF to be added to the individual one and suggests to consider the pricing structure of OA in order to make it affordable. Martina adds that there are also waivers available for those who do not have funds/grants available to publish their research. It is difficult to fix the right price as the services offered by different publishers vary. But there are pressures on prices and on delivering more contents for the same investment on the side of the libraries and institutions; publishers are negotiating new deals.
Xavier believes that management scholars are very generous with their time and with knowledge creation: again, academics peer review for free, edit for free as the way in which an article is produced is a co-authoring between the authors, the reviewers and the action editor. Xavier evidences a significant decrease in using secondary databases pretty available and instead to create own unique datasets to protect, that no one else can use, because how we can at the end remunerated is within the name that is attached on the top-quality publications (for hiring and tenure/promotion). To be the first in publishing a certain theory, perspective, model, etc. and be cited by others as being contributing in this regard is the way in which academics can get promoted and recognised by the system. Xavier agrees with James that unless there is a change in the incentives’ system that management schools and universities at large use to recognize and reward the contribution of their
Faculty, that pressure is not going to decline. Xavier also suggests to be careful in discriminating within citations, as James and Andrew pointed out, as the scholars’ aim is to identify (research) gaps and, in so doing, criticizing the work of others is the rule (in the literature review that each scholarly endeavour has which ends with gap identification).
Madeleine intervenes to add on the chat exchange with James related to OA. She writes about the need to avoid that the move to open access publishing have unintended consequences on the research ecosystem - e.g. (as Amanda noted) global South academics being unable to afford OA APCs and unable to publish in the most-read titles; learned society publishers losing the funds that currently support their research grants. Then there are problems for some associated with the way OA mandates are being developed by funders - e.g. around the requirement for open licenses, which cannot work for commercially or otherwise confidential datasets, or privately-owned images - so some researchers can be inadvertently locked out of being able to publish.
James shares the way for researchers to think about data is the principle of "as open as possible, as closed as necessary". There are sometimes good reasons for not sharing data - it exposes personal data, GDPR, etc., put someone at risk, or has commercial protections around it. In those cases, sharing can be waived or partial sharing can occur. He completely agrees with Madeleine and Amanda’s points on the global inequalities of transformative agreements. Heloise underscores that public university and research is not concerned by this, but business schools are: financial incentives in favour of article publication are also making things worse. Marrisa agrees with Heloise and underscores that the Asian model is one of collaboration among academics, practitioners and academic institutions for knowledge sharing, rather than financial considerations.
Madeleine says that in her opinion OA tends to funnel on “one size fits all”: the learning society is a complex ecosystem and not a simple circle based on someone who is writing and another one who is reading, but communities are benefited in other ways. She affirms that open science is great in principle but there are some significant areas in which it doesn’t work. Profits for commercial publishing by authors are going back to the community in form of grants. The point is not simplifying the discussion as it is not a simple system. Amanda agrees on the complexity of the issue and she reports that she feels that, contrary to what is happening in other disciplines, in the management community there is no interest on OA.
Andrew, thanking for the brilliant and interesting insights and discussion, identifies the health and medicine sectors as the source of the debate on open science. He concurs with the fact that is it really desirable to have an open approach when it comes to diseases, but there seems to be something special with business schools as the incentives’ structure and the organisation is very different. In his view, OA has been pushed very legitimately in medicine as the key founders are governments and philanthropic organisations requiring sharing and open access. Business schools are much more autonomous, as they derive much of their income from tuition fees or consultancy, not from third parties; so, issues on accountability are less important. This is a serious, difference in terms of the field’s fundamental structure that need to be addressed. He sees the tension in a move towards total open access. From a practical viewpoint, Andrew thinks that rankings and datasets do have an important role to play in accountability and in incentivising and pushing in a more desirable direction academics and institutions; he also believes in the importance of books and chapters and broader intellectual forms of outputs and collaboration and in the need to find ways to incorporate them: the challenge is how to do that, to pragmatically see whether beyond very reductionist metrics there are other metrics, consistent enough so that they can be used to be widely applied to assess research quality in the recruitment process, schools and universities, and so on.
To sum up, indeed, as several participants put it, the ecosystem of academic publishing is becoming increasingly complex, with potentially asymmetrical impact across countries and differently endowed universities even within countries. It was very interesting to hear publishers to express their surprise for the little attention learned societies in the management field have so far given to open access and open science, and how they welcome our voices. We hope we will continue to contribute as an agora and platform for such conversations.
