TEC moves on PBRF rorting

The Tertiary Education Commission appears to be taking action over the attempts by universities to improve their relative PBRF ranking, by hiding staff with low research scores from the census done in June. I blogged extensively on this a couple of weeks ago. It does not gain the university any extra funding from the PBRF, but it does lead to them having a better average rating, and hence a higher rank.

In a consultation paper just released, the TEC states:

36 The incentives for TEOs to maximise the number of PBRF-eligible staff whose EPs meet the standard for a funded Quality Category (ie. “A”, “B”, “C”, “C(NE)”) relate to the funding that those EPs attract, and the positive contribution that funded Quality Categories make to the calculation of the AQS. It is important that the calculation of the AQS for TEOs be equitable and reliable because of the reputational impact of the relative ranking of TEOs based on the AQS.

37 As a high AQS will tend to enhance the reputation of a TEO, there are strong incentives for TEOs to minimise the number of PBRF-eligible staff whose EPs would not meet the requirements for a funded Quality Category (ie. those assigned an “R” or “R(NE)”), either by supporting these staff to develop sufficient evidence of research activity, or by seeking to have them deemed ineligible for the PBRF.

38 There are strong incentives for TEOs to take steps to minimise the number of individuals reported as PBRF-eligible, where their evidence portfolios are not likely to be assigned a funded Quality Category (A, B or C), because unfunded Quality Categories reduce the reported AQS, and by extension the ‘ranking’ of TEOs, subject areas, and nominated academic units.

39 The TEC places considerable reliance on TEOs to apply the staff eligibility criteria in a manner that is consistent with both the wording, and importantly, the intent, of the staff eligibility criteria.

40 As part of the preparations for the 2012 Quality Evaluation, the TEC has contracted KPMG to undertake an audit of TEOs. The first stage of the audit process has indicated that there appears to be differences in Human Resource practices across TEOs. In particular, differences in the way in which TEOs are applying the staff eligibility criteria in relation to staff whose EPs are not likely to meet the standard for a funded Quality Category.

A nice summation of the problem, and nice to see the have sent in auditors who have discovered how some universities are trying to rort the system.

41 The extent of these differences indicates that there is potential for these practices to affect the credibility of the published results of the 2012 Quality Evaluation, including the way in which TEOs are ranked (at both the overall organisational level, and by subject areas), and for the reporting of the relative quality of research by panel, subject area, and nominated academic unit generally.

The translation is no one would believe the results of the rankings, with the current practices employed by universities.

The TEC recommends:

55 This option involves the TEC making changes to the reporting framework for the Quality Evaluation so that any staff whose EPs were assigned the “R” or “R(NE)” Quality Categories would be excluded from the calculation of the AQS at all levels.

56 The main advantage of this approach is that it serves to remove one of the most significant incentives for TEOs to manage the employment relationships of staff for the purposes of maximising their ‘ranking’ through the Quality Evaluation measure. This approach does this by ranking TEOs solely on the basis of staff whose EPs were assigned a funded Quality Category. There would therefore be no rationale other than standard employment practices to make changes to the employment agreements of staff.

57 This approach would also ensure the results of all participating TEOs would be treated equitably and no individual TEOs would be unduly advantaged or disadvantaged by their application of the staff eligibility criteria or HR practices.

This seems a sensible way forward. It is a pity that it is necessary. If the universities were not trying to rort or skew their rankings by manipulating employment contracts, this would not be necessary. But relaying on the universities to act ethically and within the spirit of the policy seem too much to hope for, so the TEC has recommended this change.

It is good to see the Government heeding the concerns over what was happening, as it was going to reduce the rankings to a bit of a joke – the university with the most manipulative HR department would probably end up being the one which got the highest rating.

It will be interesting to see what the Tertiary Education Union says on this proposed change, as they had also expressed concern over the manipulation of employment contracts to improve ratings.