David Lillis: Preserving Excellence in New Zealand Research

Introduction

Since the 1990s, New Zealand’s research effort has been directed towards the achievement of outcomes and investment decisions carried out primarily on the basis of excellence and potential to achieve those outcomes. Until recently, each of our funding instruments embodied excellence or "science quality" as a condition of funding but, unfortunately, excellence has diminished in importance in funding decision-making, especially within the Performance Based Research Fund (PBRF).

Over decades, various advisory committees have advised successive governments on how to get the most from our innovation system. For example, twenty years ago the Science Enterprises Group stated that an investment framework should provide the most appropriate basis for allocating scarce resources to research and development on the basis that investment should maximize benefit to New Zealand, that measurable benefits should occur in economic, social and environmental areas, and that investment decisions should balance the needs of today with the anticipated needs of tomorrow (SEG, 2004). Of course, to achieve measureable benefits requires excellent research and a credible system for judging excellence and estimating future benefits.

Unfortunately, in New Zealand science is in danger of becoming devalued through the imposition of artificial equality with ‘other ways of knowing’ and through funding processes that are based partly on cultural or ethnic affiliation. Concomitantly, the notion of excellence in research has been diluted too; for example, through proposed increases to weightings for funding decision-making that are based strongly on ethnicity.

Who Decides on Excellence?

What we mean by “excellence” in research differs for different people and institutions, but nevertheless it should be possible for New Zealand to achieve greater coherence on the contestable, imprecise and diffuse nature of this concept, what we can capture though qualitative and quantitative indicators, and what is missing when we define a necessarily limited set of indicators or judgements for either ex-ante or ex-post evaluation.

As in all other domains of work, excellence is obligatory in any kind of research, but the stakes are particularly high when the proposed research is intended to influence decisions that affect peoples’ lives, the environment, governance or other areas of development (Mendez, 2021).

Who decides on what it is that constitutes excellence? Is it policy-makers? Funders? Is it the taxpayer? Is it the researchers themselves or perhaps Government? Is excellence absolute or does it depend on the present conditions or on the purpose of the work? Is it immutable or can it vary over time? Does it depend on the discipline or field? Is excellence different for public research and private research?

Does it depend on the kind of institution concerned? Should it involve the notions of benefits or impact? If so, what are the anticipated benefits and what is impact? Should excellence, benefit and impact be treated together or separately?

What about the creation of knowledge for its own sake? Maybe we can draw distinctions between research that brings about changes in levels of understanding, knowledge and attitude, and the use of research that results in improvements to practice and policy-making. Are such uses central to the notion of excellence or to be considered separate from it?

How should we conceive of the notion of excellence and its function as a proxy for scientific quality, and who should judge excellence? Unfortunately, in New Zealand we do not yet have consensus on what it is that constitutes excellence, nor a sufficient public appreciation of the great importance of excellent research that underpins policy or application to the health and wellbeing of the public, within industry, the environment or other domains. Hence this article.

Excellence as a Contested Concept

Ferretti et al. (2018) refer to excellence as an essentially contested concept and note extensive debate around the meaning of excellence, its capacity to provide quantitative assessments of research activity and its potential to support policy.

Research excellence might pertain to objects (books, publications, software, seminars, new products, processes, Intellectual Property etc.) or to services or to people. It may concern what we could call the standard of the work (e.g. robustness of experimental design, data collection, research methods, analysis and reporting of findings). Perhaps it should embody novelty or originality. Though some research is helpful in confirming prior research, perhaps our definition of excellence should include end use and the potential to deliver benefit. Most certainly, it should not include political or cultural ideology or “research” that is intended to disseminate political perspectives.

Imbuing science with ideology harms the scientific enterprise and leads to a loss of public trust. If we continue to undermine merit, our universities will become institutions of mediocrity rather than places of creativity and accomplishment, leading to the loss of the competitive edge in technology. Thus, we need to restore our commitment to practices grounded in epistemic humility and the meritocratic, liberal tradition (Abbot et al., 2023).

We may agree with excellence as some multi-dimensional qualitative or quantitative criterion of both ex-ante and ex-post evaluation of research. Further, we may agree that excellence as a condition of funding should never be compromised, as otherwise we achieve systemic delivery of lightweight, mediocre research that has little influence, or even negative influence if flawed research is adopted within policy-making. Thus, we must protect excellence at all costs.

Possible Indicators of Excellence

Much criticism of peer review centers on subjectivity (e.g. Yates, 2005), where the results of an evaluation depend on the value-based opinions of reviewers. The judgements of peers (those working in the same or similar research areas) are mandatory for funding decisions, but cannot capture all of the relevant dimensions of excellence. Thus we may design a set of qualitative and quantitative indicators that capture dimensions such as the:

1. Judgement of peers
2. Standard of work and its delivery
3. Importance of the research to policy, the people, the environment or to the economy
4. Educational and human capital benefits to researchers and students of participation in the research
5. Potential impact of the research in stimulating further research
6. Judgements on future benefits and outcomes.

Bibliometric and scientometric (the quantitative study of science, communication in science, and science policy) approaches may be utilised in some cases to evaluate research retrospectively and to judge proposals prospectively, but are both labour- and cost-intensive and may not lend themselves easily to outcome-based decision-making.

Citation counts and impact factors can introduce various biases. The impact factor, calculated each year for scientific journals on the basis of the average number of times that their articles are referenced in other articles, was never intended to evaluate individual scientists, but instead as a measure of journal quality. However, often the impact factor is used in this manner, so that scientists are ranked by weighting each of their publications according to the impact factor of the journal in which it appeared (Alberts, 2013). Alberts and others believe that misuse of the journal impact factor is destructive, inviting gaming that can bias journals against publishing important papers in certain fields (such as social sciences and ecology) that are much less cited than others (such as biomedicine). Among many others, Tregoning believes that the current system is subjective, biased and built on personal networks and that confusion over how to judge scientific productivity is sapping scientific productivity (Tregoning, 2018).

Another problem is that far more people are engaged in certain kinds of research (for example, medical research) than in other kinds (for example, in language evolution or ecology). Consequently, a mediocre paper in the former can be cited more than an outstanding paper in the latter.

Other workers have discussed problems concerning impact factors. Rather than judging the value of a paper on its significance and the quality of the work, or judging peers on the aggregate of their individual merits, we now judge them on the average number of citations received for papers published in journals in which they choose to publish (Marks et al., 2013).

A relatively small percentage of papers published in Nature and Science generate the majority of citations, so that impact factors have considerable skew. Review papers tend to get highly cited, but they may contain little that is novel. It is very hard to reduce quality to a number, as quality is so discipline-specific. However, many people, especially early-career researchers, try to produce papers that will interest high impact journals. This tendency can result in overstated claims and failure to test alternative hypotheses. A further discussion of indicators as measures of quality is given in Appendix 1, while Appendix 2 discusses the impacts, benefits and outcomes that can emerge from excellent research. Appendix 3 gives a short account of the issues involved in evaluating research.

Should Excellence be Stable through Time and across Disciplines?

Research excellence can be about process as well as outcomes, whereby the former need not automatically transform into the latter. Of course, research excellence can be described in diverse ways. Thus, while some definitions consider research excellence with primary reference to the peer review system, others broaden the notion of excellence to include wider societal impact. What counts as excellent research today might not necessarily count as excellent research in the future, and any definition of research excellence might well be subject to revision. Thus, twenty years from now, will excellence in research relating to the nation’s reaction to global pandemics or relating to the response of ecosystems to climate change look the same as they do today?

Perhaps excellence should be inherently a malleable and flexible construct that may vary through time and is different within different jurisdictions and within different research areas. For example, a government-funded institute might have a very different definition from that of a private institution. Thus, excellence may tell us little about the importance of the research and instead tells us about who decides. Clearly, it is important to determine the goals and interests that excellence may serve. Research excellence could be thought of as going beyond a high standard. However, now we must define ‘a high standard in research’ and determine how standards should be set and why and by whom.

We recognize the value of certain research that enhances knowledge without evident immediate material gain, but may believe that, where possible, research should support industry, the health and wellbeing of the people and their education, the protection and sustainability of our environment and developing the economy.

Excellence in New Zealand Research

Administered by the Royal Society Te Apārangi, the Marsden Fund invests in excellent, investigator-led research that is aimed at generating new knowledge and long-term benefit to New Zealand. It supports excellent research that expands the knowledge base and contributes to the development of people with advanced skills. The primary objectives of the Marsden Fund center on excellent investigator-led research and advancement of knowledge. Its Terms of Reference require applications to be assessed primarily on potential for significant scholarly impact (demonstrable contribution to shifting understanding and advancing methods, theory and application across and within disciplines) because of the novelty, originality, insight and ambition of the proposed research; demonstrated ability and capacity to deliver and development of research skills.

Given the diversity of disciplines and research organisations, funding considerations and consideration of researchers for fellowships vary by field and application. For example, in nominations for Fellowship of the Royal Society Te Apārangi, for applied scholars, consideration takes account of the researchers’ history of:

1. Successful promulgation or uptake of new products, processes, Intellectual Property or services based on the innovation or new knowledge
2. Major changes in relevant public policy or government investment or operational strategy for example in conservation, education, emergency management, environmental protection, health, justice, or social policy
3. Major cultural or social change within communities of significant size
4. Major changes to practice in a professional community, at least at a national level
5. Major environmental change.

For less applied research, considerations vary by field, Thus in the sciences, the Royal Society Te Apārangi looks for outstanding publications, perhaps supported by evidence of the impact of the research. In the humanities it looks for outstanding publications such as monographs, articles, and book chapters. In the social sciences it looks for outstanding publications, which may include commissioned investigative reports, but may be supported by peer-recognition and end-user recognition. Finally, in technology, the applied sciences and engineering it looks for a combination of publications, Intellectual Property, evidence of the importance of the research, peer-recognition and end-user recognition. Thus, within the Royal Society Te Apārangi, excellence is a somewhat flexible and variable quality that depends on the specific type and purpose of the research and on the track records of the researchers.

Another of New Zealand’s funds is the Performance-Based Research Fund (PBRF), which aspires to enhance research quality by supporting excellent tertiary sector research. Within the PBRF, formulae used to allocate funding to research organisations involve numeric weightings that have increased recently for Māori researchers, Māori-oriented research and Māori postgraduate degree completions. The benefits are likely to include concomitant increases in Māori, Pacific and Māori-related employment in tertiary organisations and consequent domestic gains relating to equity and sense of worth. However, the funding process must be supervised closely so as not to disadvantage significantly other excellent research of potential benefit to New Zealand, attenuate the worth of non-Māori researchers and, in addition, lead to diminished credibility of New Zealand’s aggregate research effort. We could also experience a decline in international university rankings and in ability to compete in international tertiary education (Lillis, 2023).

Traditional Knowledge in Science?

The PBRF supports research activities that include the advancement of the traditional knowledge of the first people to inhabit New Zealand (mātauranga Māori). However, is traditional knowledge science? Certain elements of most or all traditional knowledge have a scientific basis but certainly not on the level of modern science - so is it possible for traditional knowledge to constitute excellent science? Possibly - under very broad definitions of science excellence or, more precisely, since no single definition can ever suffice, under particular typologies of excellence in science. However, a commonly-held opinion among scientists is that the world views of communities of the past have a valued place today and substantive contributions to make, but in the discovery of empirical, universal truths fall short of science (Corballis et al., 2021). Thus, traditional knowledge should be valued, but must sit outside the ambit of modern science.

However, while well-intended, the implementation of funding mechanisms that are based partly on ethnic affiliation represents a probable diminution of excellence and is potentially damaging unless managed very carefully. New Zealand will not achieve long-term success in tertiary education and research by proportioning resources on the basis of ethnic or cultural affiliation, nor by appointing academic staff on the basis of ethnicity rather than of genuine research and teaching potential.

Several academics known to me assert that excellence should be the only criterion for funding and that preferring one ethnicity over others can only lead to divisiveness and the erosion of quality (Lillis, 2023). We agree partially but not fully with this belief, because we have a responsibility to invest taxpayers’ money in such a way as to deliver benefit and not all excellent research can do so. However, we should be concerned about the future of basic research in New Zealand, where increased emphasis on traditional knowledge mitigates against basic research, especially in “blue skies” research, where applications are not yet apparent.

Unfortunately, much traditional knowledge has little or nothing to do with science, so in the present environment we must prevent public funding of traditional practices such as carving or songs as science. Generally, we assess science on the basis of suitability of methods, but no one has yet offered an epistemology for the traditional knowledge of various nations, including New Zealand.

There are clear dangers in funding that is based on ethnic affiliation, but even worse is to make it a requirement for assessment. This requirement is now a significant problem in New Zealand. Effectively, we have appropriation from the science budget, with little or no accountability for its use. There is no major objection to funding work on traditional knowledge, such as mātauranga Māori research, provided that we have a metric for quality and provided that such research is not confused with science.

Falling Short of Excellence

In what ways may research fall short of excellence, apart from drawing on data of poor quality, methodological flaws and incorrect analysis and poor reporting? Appendix 3 discusses issues involved in evaluating research but for now we offer two examples of New Zealand research that may fall short of excellence in that they appear to involve motivated reasoning. In the very first sentence of the abstract of one publication we read:

Racism is an important determinant of health and driver of racial/ethnic health inequities.

Does this not pre-suppose the existence of a problem? More objective would be:

In this study we investigate whether or not patient-reported racism is associated with differences in racial/ethnic health outcomes and, if so, the extent to which patient-reported racism is associated with the observed differences.

Their conclusion is that experiences of racism potentially lead to poorer healthcare and healthcare inequities through higher unmet need, lower satisfaction and more negative experiences of healthcare. The authors of the study state that the health system has a critical role to play in addressing racism within healthcare and supporting societal efforts to eliminate racism and ethnic inequities.

They may be fully correct here, or partly correct. However, many health professionals believe that in New Zealand racism is almost non-existent in health (e.g. Knight, 2022). In any case, the concern here centers on the possibility that research involving motivated reasoning may be adopted and applied widely within policy. There is a very real danger when “research” advances political agendas rather than constituting robust inquiry, and we must be vigilant in relation to the conflation of science with the promulgation of political perspectives. Even well-intentioned articles on social or political issues, such as racism or poverty, do not constitute research unless certain criteria are met, including elements of investigation, testing, evaluation, critical analysis and debate (Lillis and Schwerdtfeger, 2021).

Another paper asserts that understandings of ethnic health inequities are often situated in victim blaming and cultural deficit explanations, rather than in the root causes. It states that for Indigenous populations, colonization and racism are fundamental determinants of health inequities. Further, it asserts that the Te Kupenga Hauora Māori model takes colonization as a critical determinant of health inequities, underpinning all levels from basic to surface causes. They emphasize that privilege, alongside racism, plays a causative role in Indigenous health inequities. Similar assertions are made very frequently in relation to primary and secondary education in New Zealand.

They may be incorrect, partly correct or wholly correct in identifying privilege and racism as causes of inequality in health but other factors, such as socioeconomics, lifestyle choices and genetics play major roles in differential outcomes across communities.

We must at all costs avoid confusion between research and activism. Most of us agree on the need for justice and equality of opportunity, but research is the pursuit of truth. In some activist-led, social justice-oriented "research", there is the risk of sub-standard methods, motivated reasoning or un-objective reporting of findings that go unchallenged - the "Anything Goes" school of thought, of which Mario Bunge warned us twenty years ago (Bunge, 2006).

Sub-standard research is not helpful as we attempt to progress towards a better world, especially when it is incorporated within policy and decision-making, such as in education and healthcare. The ever-present danger is that we dilute our notions of excellence in research - to the detriment of society at large.

Depoliticizing Science and Strengthening Merit­-based Practices

Abbot et al. offer various suggestions for depoliticizing science and strengthening merit­-based practices. They include the following:

1. Government research funding be distributed exclusively on excellence (their word is “merit”)
2. Academic departments and conferences select speakers on scientific, rather than ideological, considerations
3. Admissions, hiring and promotion be merit-­based and free from ideological tests
4. Scientific papers are published and retracted on the basis of scientific, rather than ideological, grounds or on the basis of public pressure.

We agree with these recommendations. Further, whatever definition of excellence is used, we may agree on the desirability of excellence and that we have duty of care to protect the highest academic standards across all disciplines and areas of research. We contend that there must be no diminution of established definitions of excellence in order to satisfy either political or social agendas, however compelling those agendas may appear to be. Appendix 4 gives a synopsis of Abbot et al., a very important paper - In Defense of Merit in Science - and their thoughts on excellence in research.

A review of the entire funding system in New Zealand is well overdue. It has become unwieldly, with many overlaps and the requirement to address traditional knowledge and world views has become a problem for many science and engineering researchers. Perhaps we should move to a simplified system under a new Ministry of Research, Science and Technology, preferably including universities and CRIs within it. One can envisage a fund, such as Marsden, covering fundamental research; another for mission-led and more applied research, and research translation, commercialization and knowledge transfer as a third. A fourth fund could comprise whatever replaces the PBRF, but focused on the development of capability, recalling that PBRF money was originally research postgraduate student funding from Government. A fifth fund might comprise traditional knowledge research, recognising that in a zero sum game, each fund will subtract from the others. Its criteria for access would undoubtedly be different from those of other funds.

Protecting Excellence for the Future

Perhaps we should opt for tight situation-dependent judgements of excellence. For example, within Marsden, the criteria for excellence in physics-based proposals would differ from those for social research. Such an approach may require segmentation in instruments such as the PBRF. We could accord higher ratings for Māori researchers when they are engaged in, and preserving or promoting, mātauranga Māori, but absolutely not when they are engaged in quantum physics or other areas along with people of all races and ethnicities.

We could impose specific exclusions. Thus, decolonisation and related activism are not to be considered within a priori assessments of research proposals. If, for example, proposals turn up evidence that points to decolonisation as important (along with definitions of what it actually is), then such a conclusion is perfectly reasonable. However, if someone applies for a grant to continue researching this area, we should allow only discussion of evidence and exclude debate that pre-supposes anything about decolonisation as poor quality science and downgrade the score accordingly - in other words, penalise low quality research that is based on presuppositions.

Cabinet has asked the Tertiary Education Commission, working with the Sector Reference Group for the PBRF Quality Evaluation 2025, to consider how the definitions of research and research excellence can be improved. Indeed, there is always room for improvement. However, it is crucial that we exercise due diligence if we engage in broadening our definitions of research excellence because there is great potential for the taxpayer to fund research that would not be considered excellent in other countries (Lillis, 2023) and that could lead to inadequate policy and decision-making.

If we are to make for a better world, then there is much that we can achieve by learning to live together as individuals, within workplaces and across nations. We can also make for a better world through the conceptual institutions of education and research, but those institutions must be protected at all costs. Especially, we must ensure that our research is of excellent quality and can contribute substantively to health and wellbeing, to the health of our environment and to our prosperity and quality of life.

References

Alberts, Bruce (2013). Impact Factor Distortions. SCIENCE VOL. 340.

Abbot, D., A. Bikfalvi, A.L. BleskeRechek, W. Bodmer, P. Boghossian, C.M. Carvalho, J. Ciccolini, J.A. Coyne,. Gauss, P.M.W. Gill, S. Jitomirskaya, L. Jussim, A.I. Krylov, G.C. Loury, L. Maroja, J.H. McWhorter, S. Moosavi, P. Nayna Schwerdtle, J. Pearl, M.A. Quintanilla Tornel, H.F. Schaefer III, P.R. Schreiner, P. Schwerdtfeger, D. Shechtman, M. Shifman, J. Tanzman, B.L. Trout, A. Warshel, and J.D. West, “In Defense of Merit in Science.” Journal of Controversial Ideas 2023, 3(1), 1; 10.35995/jci03010001, pp1-26.

Bunge, Mario (2006). In praise of intolerance to charlatanism in academia.

Corballis, Michael; Clements, Kendall; Cooper, Garth; Elliffe, Doug; Nola, Robert; Rata, Elizabeth and Werry, John. "In Defence of Science". New Zealand Listener, 31 July 2021. p4.

Ferretti, Federico, Angela Guimaraes Pereira, Daniel Vertesy and Sjoerd Hardeman. Research excellence indicators: time to reimagine the ‘making of’? Science and Public Policy, 45(5), 2018, 731–741, doi: 10.1093/scipol/scy007, Advance Access Publication Date: 14 February 2018

Hammersley, M. (2008). Troubling criteria: A critical commentary on Furlong and Oancea’s framework for assessing educational research. British Educational Research Journal , 747-762.

Hemlin, S. (1999), ‘Utility evaluation of academic research: six basic propositions’, Research Evaluation, 7(3), pages 159-166.

Knight, Lawrie (2022). Fact checking Māori health claims that led to the Pae Ora (Healthy Futures) Bill: 
https://breakingviewsnz.blogspot.com/2022/03/dr-lawrie-knight-fact-checking-maori.html

Lillis, David (2000). “Towards a New Science Evaluation System for New Zealand.” Research Evaluation, volume 8, number 2, August 2000, pages 145–150, Beech Tree Publishing.

Lillis, David and Schwerdtfeger, Peter (2021). The Mātauranga Māori - Science Debate:
https://breakingviewsnz.blogspot.com/2021/12/david-lillis-and-peter-schwerdtfeger.html

Lillis, David (2023). Capture of Research Funding in New Zealand?
https://breakingviewsnz.blogspot.com/2023/03/dr-david-lillis-capture-of-research.html

Marks, Michael, Mark Marsh, Trina A. Schroer and Tom H. Stevens (2013). Misuse of Journal Impact Factors in Scientific Assessment. Traffic 2013; 14: 611–612

Mendez, Ethel (2021). Evaluating Research Excellence: Main Debates. The International Development Research Centre:
https://idrc-crdi.ca/sites/default/files/2021-04/Brief-Final-English.pdf

SEG (2004). A Framework for Research & Development Investment in New Zealand. Final Report from the Science Enterprises Group. October 2004.

Tregoning, John (2018). How will you judge me if not by impact factor? Nature, 21 June 2018, Vol 558.

Ware, M. (2011). Peer review: recent experience and future direction. New Review of Information Networking, 23-53.

Yates, L. (2005). Is Impact a measure of Quality? Some Reflections on the Research Quality and Impact Assessment Agendas. European Educational Research Journal, Volume 4 (Number 4), 391- 403.

Appendix 1

Indicators as Measures of Quality

Mendez (2021) reminds us that the debate on the use of indicators or metrics in research evaluation centers on their validity as measures of quality. Indicators such as citation counts, impact factors, and publication rates rely on the assumption that published research is already research of good quality because it has undergone some quality review process. However, indicators are sensitive to the subjectivity of peer review and therefore may replicate biases in the review of colleagues or competitors.

Other factors, such as the reputation of the author or the persistence of a researcher to get published, may play a role in what is published (Ware, 2011). Therefore, quantitative indicators are not always the most effective measures of intrinsic quality but instead are proxies that can be moderated by other factors.

Perhaps it is necessary to take excellence as a multi-dimensional concept, agree that a range of indicators is necessary to describe it, but accept that in practice it is impossible to attain a complete description. Perhaps we should consider both inputs and outputs in funding decision-making. Perhaps excellence is not a notion in the absolute, but exists relative to objectives, recognizing that indicators and metrics correspond to social and political needs and are not simply the results of technical processes.

Appendix 2

Impacts, Benefits and Outcomes

If we are to include the notion of research impact (stimulation of further research, citations, benefits, outcomes) within the definition of excellence, then we must identify the range of possible outcomes of research. We may discriminate between purely academic impacts (where research influences within academia) and external impacts (where research influences parties that are external to the research institutions). Perhaps we can think of outcomes as material benefits contributing to quality of life (Lillis, 2000). These benefits could include increased industrial competitiveness, quality of the environment, effective social regulation and enhancements to skills and learning. ‘Outputs’ signify only those process developments from research that are routine and represent no more than achievement at the programme or project level. Outputs may in practice include publications, patents, equipment and software (Lillis, 2000).

Kinds of innovation and outcome that may accrue from research include the following:

1. Industrial innovations
2. Industrial productivity improvement and quality enhancement
3. Social outcomes
4. Environmental, resource and risk-management outcomes and innovations
5. International issues requiring bilateral or multilateral action; for example, the Law of the Sea, ozone depletion and global pollution
6. Underpinning of standards and regulations
7. Discovering, defining and classifying utilised resources
8. Overcoming key barriers and closing knowledge gaps
9. Human capital enhancement and mobility.

Clearly, any evaluation system must be capable of apprehending, evaluating and, where possible, measuring the scale of these and other kinds of innovations and outcomes by directing attention to appropriate units of analysis (e.g. institutions, businesses, sectors). Of course, we recognise that much research is utilised indirectly (for example, new basic science knowledge applied in engineering or medicine) and that some research and development, especially that conducted within the private sector, may go unreported.

Certain research enhances knowledge without evident immediate material gain, and may have cultural or historic significance (e.g. Andrew Wiles’ solution of Fermat’s Last Theorem) and human capital benefits for those who participate in the research. Therefore, where possible, definitions of research excellence and associated evaluations should include the development of human capital and creation of knowledge. Specifically, knowledge outcome indicators could include tangible outputs, new methods and human capital development and retention.

Appendix 3

Research Evaluation and Excellence

Of course, we have need for excellence and for ex-ante and post-evaluation of significant research but linking funding to research evaluation may bring incentives for researchers to focus on areas where they can generate rapid results, hence compromising academic freedom and the production of research that can have long-term effects (Yates, 2005; Hammersley, 2008; cited in Mendez, 2021).

However, when evaluating outcomes of research and development retrospectively, issues of causality and attribution arise. In practice, often it is impossible to demonstrate that particular outcomes (for example, rise in GDP, increase in sector sales, improved public health or a cleaner environment) arose because of particular research investments. As Hemlin noted many years ago, sound methods for assessment of causality between public-funded research and productivity are lacking (Hemlin, 1999). Indeed, at the national level commonly-used indicators (e.g. inventions, patents, innovations) may not be reliable, and indirect measures (for example, the notion that medical research reduces mortality and shortens illnesses) are not linked clearly to research and development as the only cause.

Appendix 4

In Defense of Merit in Science

Abbott et al. (2023) use the word ‘merit’ to mean largely the same construct as ‘excellence’. They remind us that merit (excellence) must be the key metric to judge and evaluate scientific claims and that the merit of an idea should be evaluated through scrutiny. They say that the ultimate test of the merit of a claim is its ability to predict the functioning of the universe as elucidated through replicable experiment and observation, but not whether it feels right or comports with a particular worldview or group interest. They assert that ideological orthodoxies deserve no place in science and that to ensure that the best scientific ideas are put forward, merit must also be applied in order to evaluate research proposals and prospective students and faculty. Here, merit comprises the scientific claims contained in research plans, the quality of the proposed methods, and the expertise and academic track records of the people involved.

Abbott et al. state that in assessing merit and scientific promise, despite their limitations, quantitative metrics bring benefits. While merit cannot be quantified by simplistic formulas (e.g. number of publications multiplied by some impact factor), using numeric data to quantify scientific output is a useful component of evaluation because it provides a quantitative measure of productivity. They believe that good practice uses a combination of quantitative metrics and qualitative assessments that include letters from reviewers assessing how influential, original and innovative the work is.

They view objective quantifiable metrics (such as publications) as one important dimension of merit, but agree that merit cannot be reduced to counting. So – they pose the rhetorical question as to whether one superb publication is more valuable than four good publications. They concede that such questions involve judgment calls on which different people and institutions may disagree. Of course, what makes a published report superb may differ among fields and institutions. Although subjective judgments should play an important role in evaluations of merit, they recognize that subjective judgments are vulnerable to bias.

Dr David Lillis trained in physics and mathematics at Victoria University and Curtin University in Perth, working as a teacher, researcher, statistician and lecturer for most of his career. He has published many articles and scientific papers, as well as a book on graphing and statistics.