Leon Benade, Nesta Devine & Georgina Stewart
Te Kura Mātauranga School of Education, Auckland University of Technology
The Performance-Based Research Fund, commonly known as PBRF, is a system for evaluating the research work of university academics and consequently of the universities in Aotearoa New Zealand. All academics submit to the PBRF panel a ‘portfolio’ of their work – a summary, emphasising the highlights of the research they have done in the previous 6 years. This is assessed according to a set of criteria (e.g. see here and here for the criteria used in the most recent round). These criteria are believed to define ‘quality’ in the field of academic research. The system has been in operation since 2002. It has been reviewed in 2004, 2008 and 2012/13, and a 2019 Review is now well underway. In this blog post, we use the Terms of Reference for the 2019 review (published by the Ministry of Education in 2018) to consider the problems and potential changes to the PBRF system.
Foundations and aims of the PBRF
The theoretic foundation of the PBRF system lies in New Public Management theory, particularly in the ‘agency problem’, which sees employees as problematic agents of their employers. This theory is based on the concept of ‘homo economicus’, the rational autonomous chooser who is motivated by self-interest. It is difficult for teachers to further their own self-interest by seeking larger profits. So, this theory holds, instead of pursing their employer’s goals they pursue their own self-interest by other means – ‘featherbedding’, that is, devoting more of the employer’s resources to their own comfort than is necessary. In New Zealand this concept was summed up in the term ‘provider capture’ and was used to disempower teachers at all levels of the education system.
When the PBRF was created in 2002, the challenge for its designers was to create a system which would convert intellectual productivity into measurable outputs. Such a system would serve the dual purpose of monitoring each individual’s work, and rewarding the institutions they worked for, presumably for conducting satisfactory techniques of motivation and surveillance. The universities’ share in a defined amount of money varies according to the performance of their academics.
Impacts of the PBRF
Academics themselves get little in return for the substantial work they put into recording and finding evidence for their outputs: the researchers who ‘earned’ it see very little of it, as the university can spend it as they like, for instance on student scholarships.
Apart from the amount of money the government gives to universities depending on their gradings, PBRF has other effects. An ‘A’ or ‘B’ grading, discretely communicated, can be very helpful to a lecturer’s career. The university that gets the highest ranking overall is publicly celebrated, and its ranking enables that institution to attract more research funds, and possibly more significantly, more international students. International students are considerable contributors to university budgets.
Changes over time
Like any device meant to ‘describe’ or ‘measure’ which then becomes a goal to be reached, the PBRF has distorted the field it seeks to record. In the first round, high value was put on the numbers of outputs and international connections. Consequently, the universities sought to employ ageing professors with long publishing lists – at the expense of young academics who were less likely to have a large number of published papers. Given that the academic workforce was already advancing in years, this was a recipe for future disaster in workforce demographics.
Subsequent PBRF rounds have tried to compensate for this by putting more emphasis, or giving more value to, younger ‘emerging’ academics. There has also been an attempt to recognise a wider scope of academic output than just the conventional peer-reviewed academic journal article, although these are still the most highly prized form of output.
The 2019 review
The 2019 review aims to assess the effects of the changes in the PBRF system since the last review. The espoused purpose of the PBRF has not been surveillance, and there are some (weak) inbuilt protections for individuals – for example, the only reference to confidentiality in the information about PBRF on the TEC website is in the section on Researchers’ requests for assessment information, which says: “To ensure confidentiality, if we have any concerns related to your identity, the information will not be released.” Its public purpose has been to recognise and support research excellence at the tertiary level, in the assumption that research excellence leads to high-quality research-led teaching. The 2019 review seeks to improve the payoff of investment in research by improving the performance of PBRF so that future research will have nation-wide impact. It is not clear what the mechanism for that will be.
According to the Terms of Reference, “PBRF has supported the development of a strong research culture across tertiary education organisations over the last 16 years” (Ministry of Education 2018 p. 1). However, it is not clear how or if the PBRF has achieved this, as it is not possible to maintain a kind of control group which does not take part in PBRF. The surveillance culture, which depends on the compliance and competitiveness of the individual academic creates a huge amount of stress and tension amongst university staff, including the considerable numbers who are employed to administer the system and exhort academics to increase, enhance, and curate their production.
The assumption built into the design of the PBRF is the basic premise of neo-liberal thinking, that competition in a market, even an artificial one, improves ‘quality’. There is no evidence for this belief – indeed often the reverse happens as competitors cut prices and hence skimp on materials and workmanship in an effort to retain the competitive edge. In New Zealand we have seen our most prestigious universities sink through international league tables as universities from other countries figure out how to ‘play the game’. It is unlikely that the quality of their research has actually deteriorated over such a short period.
The 2019 Terms of Reference do not question the ideas underlying PBRF, but direct the panel to check if PBRF is consistent with other government Initiatives, like the National Research Charter. However, there may be a problem inherent in the desire to produce ‘consistency’: reducing variation also reduces the likelihood of new ways of thinking.
More interestingly, the review Terms of Reference ask the panel to consider the concept of ‘the individual as the unit of measurement’ perhaps signalling increased interest in collaborative approaches to research.
Reducing the focus on the individual may well be a good thing if it reduces individual workload. Many people feel they could have written several papers with the time and effort they put into preparing their PBRF portfolio. However, the increasing focus on ‘impact’ is worrying. ‘Impact’ can be very difficult to prove, and may take years, even decades to occur, depending on the discipline. Universities have a legislated responsibility as ‘critic and conscience’ of our society. Our fear is that that ideal may be swamped by the need to prove ‘impact’ measured purely by instrumental, fairly unsubtle means.
Leon Benade is an Associate Professor at AUT – Auckland University of Technology. His research interests are teachers’ work, school policy, ethics, philosophy in schools, critical pedagogy, and the New Zealand Curriculum. Leon’s current research work focuses on how ’21st century learning’ impacts the work of teachers and school leaders, particularly in relation to the establishment of Innovative Learning Environments (ILE) and digital pedagogies. Leon is a co-editor of the New Zealand Journal of Educational Studies, NZARE’s official journal.
Nesta Devine is a Professor of Philosophy of Education at AUT – Auckland University of Technology. Her research interest span education policy and theory, prison education, Pasifika teachers, and school exclusion. Nesta’s research challenges the conventional ways of thinking that can lead to inequities for different groups of learners. Nesta is the Book Reviews editor of the New Zealand Journal of Educational Studies, NZARE’s official journal.
Georgina Stewart is an Associate Professor at AUT – Auckland University of Technology. Georgina’s research interests build on her background as a Māori teacher of science, mathematics and Te Reo Māori in English-medium and Māori medium secondary schools in Auckland and Whangarei. Specifically, Georgina’s research centres on the nexus between language, knowledge, culture and education, and aligns with Kaupapa Māori, feminist and poststructuralist philosophical traditions. Georgina is a co-editor of the New Zealand Journal of Educational Studies, NZARE’s official journal.