How do researchers in Spain wish to be evaluated?

Results from a survey of 875 research academics in Spain on what criteria should matter most in research evaluation — and why the answers are both encouraging and sobering.

Leer en Español

We recently published the results and full data from a survey asking 875 research academics living and working in Spain for their views on how important a selection of 39 different criteria should be in the evaluation of candidates’ research CVs in a recruitment process. In a second part of the survey we also asked participants about their level of agreement with a series of statements regarding research evaluation practices.

The initial motivation of this survey was to show that academics in Spain are dissatisfied with the way their research output is currently being evaluated and to propose alternatives to improve the assessment of scientific/academic quality. Looking at the results, I don’t think we can say that this objective has been achieved.

In the second part of the survey, 88% of the participants consider that “publish or perish” is an existing problem, 84% think that evaluation based on the number of published articles fosters questionable authorship strategies, and 71% agree that such a way of evaluation promotes bad research practices. While this seems encouraging, the number of published articles was ranked as the second most important evaluation criterion (after being PI or co-PI in research projects). Does this mean that counting the number of articles is bad but it is the best we have? At the same time, 70% agree that it would be preferable to base research evaluation on a pre-determined number of the best contributions. Could this method, which is already being applied by some national evaluation agencies, be pointing in the right direction?

Despite so much being said and written about the irrelevance of the Impact Factor for the evaluation of individual authors, this criterion was rated as the tenth most important, with 54% of the participants considering it “very” or “fairly important”. The Hirsch index was considered less important and ranked 16th.

Another disheartening observation was that publishing pre-registered studies was not considered relevant for research assessment, ranking 36th in the order of importance, with 51% of the respondents rating it as “not very important” or “not important at all”. At the same time, publishing preprints was the least important criterion, rated as not very important or not important at all by 60% of the participants. Although the importance of preprints did not vary by discipline or age, it is promising that preregistration was considered more important among younger researchers (22–40 years old) in the life sciences, with “only” 28% of respondents considering it unimportant.

The inclusion of an interview in the recruitment process, which currently only involves the submission of a CV in a specific format that varies from one institution to another, was supported by 79% of the respondents and ranked ninth in order of importance (only one place above the IF!). Interestingly, when dividing the results by gender, it appears that women tend to consider the inclusion of an interview as less important than men: this criterion ranks seventh for men and seventeenth for women.

Another crucial question for the Spanish assessment system is who should be involved in the evaluation committees: only internal members of the department opening the position, only members external to the department, or both. This question is particularly relevant as the Spanish evaluation system has been repeatedly accused of endogamy, favouring candidates affiliated to the hiring department to the detriment of external applicants who often have more experience and merit for the position. According to our results, the majority of respondents believe that evaluation committees should be composed of both internal and external members, but still 12% believe that candidates should be evaluated only by internal members.

In general, it seems that there is still a long way to go to raise awareness in Spain, where researchers still value journal-level indices for individual assessment and do not recognise the importance of practices that clearly improve research quality, such as the pre-registration of experimental studies. Despite these concerns, our survey also shows that most researchers recognise that the current system leads to bad practices and even suggests broadly-supported improvements, such as using a pre-determined number of best contributions for evaluation, or including an interview by a committee composed of external and internal members.

The observations highlighted here are just a few that called my attention. Anyone interested can access the full data and the report using the links below, and also explore the interactive chart shown further down with various filter options.

Zenodo PDF Report Tableau Version

opinionItems = [
  "Publicar los resultados de la investigación es parte de la responsabilidad y trabajo de un académico",
  "El fenómeno \"publish or perish\" está presente en España",
  "La evaluación basada en número de publicaciones incentiva estrategias de autoría cuestionables",
  "La evaluación basada en número de publicaciones incentiva prácticas de investigación cuestionables",
  "El proceso de selección debería incluir una entrevista personal",
  "Deberían considerarse periodos sin producción justificados",
  "Las comisiones de contratación deberían contar con miembros internos y externos",
  "La evaluación debería basarse en un número predeterminado de las mejores contribuciones",
  "Las comisiones de contratación deberían contar solo con miembros externos",
  "Las comisiones de contratación deberían contar solo con miembros internos"
]

impLevels = ["Muy importante", "Bastante importante",
             "Medianamente importante", "Poco importante",
             "Nada importante", "Sin respuesta"]

agreeLevels = ["Totalmente de acuerdo", "De acuerdo",
               "Ni en desacuerdo ni de acuerdo", "En desacuerdo",
               "Totalmente en desacuerdo", "Sin respuesta"]

impColors = {
  return {
    "Muy importante": "#1a9641",
    "Bastante importante": "#91cf60",
    "Medianamente importante": "#d9ef8b",
    "Poco importante": "#fdae61",
    "Nada importante": "#d73027",
    "Sin respuesta": "#d9d9d9"
  };
}

agreeColors = {
  return {
    "Totalmente de acuerdo": "#1a9641",
    "De acuerdo": "#a6d96a",
    "Ni en desacuerdo ni de acuerdo": "#d9ef8b",
    "En desacuerdo": "#fdae61",
    "Totalmente en desacuerdo": "#d73027",
    "Sin respuesta": "#d9d9d9"
  };
}

impLabels = {
  return {
    "Muy importante": "Very important",
    "Bastante importante": "Fairly important",
    "Medianamente importante": "Moderately important",
    "Poco importante": "Not very important",
    "Nada importante": "Not important at all",
    "Sin respuesta": "No response"
  };
}

agreeLabels = {
  return {
    "Totalmente de acuerdo": "Strongly agree",
    "De acuerdo": "Agree",
    "Ni en desacuerdo ni de acuerdo": "Neither agree nor disagree",
    "En desacuerdo": "Disagree",
    "Totalmente en desacuerdo": "Strongly disagree",
    "Sin respuesta": "No response"
  };
}

factorLabels = {
  return {
    "Publicar los resultados de la investigación es parte de la responsabilidad y trabajo de un académico": "Publishing results is part of an academic's responsibility",
    "El fenómeno \"publish or perish\" está presente en España": "The \"publish or perish\" phenomenon is present in Spain",
    "La evaluación basada en número de publicaciones incentiva estrategias de autoría cuestionables": "Evaluation based on number of publications encourages questionable authorship strategies",
    "La evaluación basada en número de publicaciones incentiva prácticas de investigación cuestionables": "Evaluation based on number of publications encourages questionable research practices",
    "El proceso de selección debería incluir una entrevista personal": "The selection process should include a personal interview",
    "Deberían considerarse periodos sin producción justificados": "Justified periods without output should be considered",
    "Las comisiones de contratación deberían contar con miembros internos y externos": "Hiring committees should include internal and external members",
    "La evaluación debería basarse en un número predeterminado de las mejores contribuciones": "Evaluation should be based on a pre-determined number of best contributions",
    "Las comisiones de contratación deberían contar solo con miembros externos": "Hiring committees should include only external members",
    "Las comisiones de contratación deberían contar solo con miembros internos": "Hiring committees should include only internal members",
    "Actividades de transferencia / patentes": "Transfer activities / patents",
    "Artículos de divulgación": "Popular science articles",
    "Autoría de libros": "Book authorship",
    "Capítulos de libro": "Book chapters",
    "Charlas por invitación": "Invited talks",
    "Contratos de transferencia": "Transfer contracts",
    "Diferenciación de la investigación con respecto al departamento": "Research differentiation from department",
    "Edición de libros": "Book editing",
    "Entrevista ante un tribunal evaluador": "Evaluation committee interview",
    "Estancias en centros internacionales": "International research stays",
    "Estancias en centros nacionales": "National research stays",
    "IP o co-IP de proyectos": "PI or co-PI of projects",
    "Informes para instituciones": "Institutional reports",
    "Índice de impacto": "Impact Factor",
    "Índice h": "h-index",
    "Número de autores": "Number of authors",
    "Número de citas recibidas": "Number of citations received",
    "Número de publicaciones en revistas indexadas": "Publications in indexed journals",
    "Número de publicaciones en revistas no indexadas": "Publications in non-indexed journals",
    "Organización de simposios internacionales": "Organisation of international symposia",
    "Organización de simposios nacionales": "Organisation of national symposia",
    "Participación en comités organizadores de eventos internacionales": "Participation in international event organising committees",
    "Participación en comités organizadores de eventos nacionales": "Participation in national event organising committees",
    "Participación en equipo editorial de revista internacional": "International journal editorial board",
    "Participación en equipo editorial de revista nacional": "National journal editorial board",
    "Participación en proyectos": "Project participation",
    "Posición de la revista en ranking": "Journal ranking position",
    "Posición autoría": "Authorship position",
    "Pósteres en congresos de nacionales": "National conference posters",
    "Pósteres en congresos internacionales": "International conference posters",
    "Preprints": "Preprints",
    "Presentaciones en congresos internacionales": "International conference presentations",
    "Presentaciones en congresos nacionales": "National conference presentations",
    "Proceedings": "Proceedings",
    "Proximidad de la investigación con respecto al departamento": "Research proximity to department",
    "Publicaciones con datos y scripts en abierto": "Open data and scripts",
    "Publicaciones con pre-registro": "Pre-registered publications",
    "Revisión para revistas internacionales": "International journal peer review",
    "Revisión para revistas nacionales": "National journal peer review"
  };
}

data = transpose(rawData)

uniqueVals = {
  const u = (col) => [...new Set(data.map(d => d[col]))].filter(v => v && v !== "Sin respuesta").sort();
  return {
    Genero: u("Genero"),
    Edad: u("Edad"),
    Estatus: u("Estatus"),
    Area: u("Area")
  };
}

genderLabels = {
  return {
    "Todos": "All",
    "Hombre": "Male",
    "Mujer": "Female",
    "Otro / Prefiero no contestar": "Other / Prefer not to say"
  };
}

ageLabels = {
  return {
    "Todos": "All",
    "22-30": "22-30",
    "31-40": "31-40",
    "41-50": "41-50",
    "51-60": "51-60",
    "Más de 60": "Over 60"
  };
}

statusLabels = {
  return {
    "Todos": "All",
    "Catedrático/a": "Full Professor",
    "Contratado/a Doctor/a": "Associate Professor (tenured)",
    "Estudiante de doctorado": "Doctoral student",
    "Investigador/a Postdoctoral": "Postdoctoral researcher",
    "Otro": "Other",
    "Profesor/a Ayudante Doctor/a": "Assistant Professor",
    "Profesor/a Titular": "Associate Professor",
    "Ramón y Cajal / similar": "Ramón y Cajal / similar"
  };
}

areaLabels = {
  return {
    "Todos": "All",
    "Arte y Humanidades": "Arts & Humanities",
    "Ciencias": "Sciences",
    "Ciencias de la Salud": "Health Sciences",
    "Ciencias Sociales y Jurídicas": "Social & Legal Sciences",
    "Ingeniería y Arquitectura": "Engineering & Architecture"
  };
}

viewof selGenero = Inputs.select(
  ["Todos", ...uniqueVals.Genero],
  { label: "Gender", format: x => genderLabels[x] || x }
)
viewof selEdad = Inputs.select(
  ["Todos", ...uniqueVals.Edad],
  { label: "Age", format: x => ageLabels[x] || x }
)
viewof selEstatus = Inputs.select(
  ["Todos", ...uniqueVals.Estatus],
  { label: "Position", format: x => statusLabels[x] || x }
)
viewof selArea = Inputs.select(
  ["Todos", ...uniqueVals.Area],
  { label: "Area", format: x => areaLabels[x] || x }
)

filtered = {
  let d = data;
  if (selGenero !== "Todos") d = d.filter(r => r.Genero === selGenero);
  if (selEdad !== "Todos") d = d.filter(r => r.Edad === selEdad);
  if (selEstatus !== "Todos") d = d.filter(r => r.Estatus === selEstatus);
  if (selArea !== "Todos") d = d.filter(r => r.Area === selArea);
  return d;
}

sampleSize = {
  const ref = filtered.filter(r => r.Factor === opinionItems[0]);
  return ref.length;
}

html`<p style="text-align:center; color: var(--bs-secondary); font-size: 0.9rem; margin-top: 0.5rem;"><em>N = ${sampleSize} respondents</em></p>`

function computePct(rows, levels, items, exclude) {
  const subset = exclude
    ? rows.filter(r => levels.includes(r.Valor) && !items.includes(r.Factor))
    : rows.filter(r => levels.includes(r.Valor) && items.includes(r.Factor));

  const counts = {};
  for (const r of subset) {
    const k = r.Factor + "|||" + r.Valor;
    counts[k] = (counts[k] || 0) + 1;
  }

  const totals = {};
  for (const [k, v] of Object.entries(counts)) {
    const factor = k.split("|||")[0];
    totals[factor] = (totals[factor] || 0) + v;
  }

  const result = [];
  for (const [k, v] of Object.entries(counts)) {
    const [factor, valor] = k.split("|||");
    result.push({ Factor: factor, Valor: valor, Pct: Math.round(v / totals[factor] * 10000) / 100 });
  }
  return result;
}

impData = {
  const pcts = computePct(filtered, impLevels, opinionItems, true);
  const firstPcts = {};
  for (const r of pcts) {
    if (r.Valor === impLevels[0]) firstPcts[r.Factor] = r.Pct;
  }
  const factors = [...new Set(pcts.map(r => r.Factor))];
  factors.sort((a, b) => (firstPcts[a] || 0) - (firstPcts[b] || 0));
  return { pcts, factorOrder: factors };
}

agreeData = {
  const pcts = computePct(filtered, agreeLevels, opinionItems, false);
  const firstPcts = {};
  for (const r of pcts) {
    if (r.Valor === agreeLevels[0]) firstPcts[r.Factor] = r.Pct;
  }
  const factors = [...new Set(pcts.map(r => r.Factor))];
  factors.sort((a, b) => (firstPcts[a] || 0) - (firstPcts[b] || 0));
  return { pcts, factorOrder: factors };
}

Plotly = require("https://cdn.plot.ly/plotly-2.27.0.min.js")

chart1 = {
  const { pcts, factorOrder } = impData;
  const div = document.createElement("div");

  const traces = impLevels.map(nivel => {
    const vals = factorOrder.map(f => {
      const match = pcts.find(r => r.Factor === f && r.Valor === nivel);
      return match ? match.Pct : 0;
    });
    return {
      x: vals,
      y: factorOrder.map(f => factorLabels[f.normalize("NFC")] || f),
      type: "bar",
      orientation: "h",
      name: impLabels[nivel],
      marker: { color: impColors[nivel] },
      text: vals.map(v => v >= 4 ? v.toFixed(2) + "%" : ""),
      textposition: "inside",
      textfont: { size: 9, color: "#333" },
      hovertemplate: "<b>%{y}</b><br>" + (impLabels[nivel]) + ": %{x:.1f}%<extra></extra>"
    };
  });

  const layout = {
    barmode: "stack",
    title: { text: "<b>Importance of different factors for research CV evaluation</b>", font: { size: 14 } },
    xaxis: { title: { text: "% of responses", standoff: 10 }, ticksuffix: "%", range: [0, 100] },
    yaxis: { title: "", tickfont: { size: 9 }, automargin: true },
    legend: { orientation: "h", y: -0.12, x: 0.5, xanchor: "center", font: { size: 10 } },
    margin: { l: 10, t: 50, b: 90, r: 10 },
    height: 1200
  };

  Plotly.newPlot(div, traces, layout, { responsive: true, displayModeBar: false });
  return div;
}

chart2 = {
  const { pcts, factorOrder } = agreeData;
  const div = document.createElement("div");

  const traces = agreeLevels.map(nivel => {
    const vals = factorOrder.map(f => {
      const match = pcts.find(r => r.Factor === f && r.Valor === nivel);
      return match ? match.Pct : 0;
    });
    return {
      x: vals,
      y: factorOrder.map(f => factorLabels[f.normalize("NFC")] || f),
      type: "bar",
      orientation: "h",
      name: agreeLabels[nivel],
      marker: { color: agreeColors[nivel] },
      text: vals.map(v => v >= 4 ? v.toFixed(2) + "%" : ""),
      textposition: "inside",
      textfont: { size: 9, color: "#333" },
      hovertemplate: "<b>%{y}</b><br>" + (agreeLabels[nivel]) + ": %{x:.1f}%<extra></extra>"
    };
  });

  const layout = {
    barmode: "stack",
    title: { text: "<b>Opinions and proposals</b>", font: { size: 14 } },
    xaxis: { title: { text: "% of responses", standoff: 10 }, ticksuffix: "%", range: [0, 100] },
    yaxis: { title: "", tickfont: { size: 9 }, automargin: true },
    legend: { orientation: "h", y: -0.25, x: 0.5, xanchor: "center", font: { size: 10 } },
    margin: { l: 10, t: 50, b: 110, r: 10 },
    height: 550
  };

  Plotly.newPlot(div, traces, layout, { responsive: true, displayModeBar: false });
  return div;
}