Introduction

On 2025-01-20 the NBIS Training Steering Group had a meeting on course evaluations. The first question was 'How are evaluations evaluated?'.

What are the ELIXIR evaluation questions?

You can find the ELIXIR evaluation questions here.

This paper deals only with the mandatory questions 5 to and including 9.

It is common practice that courses are evaluated by surveys [Brazas & Ouellette, 2016][Gurwitz et al., 2020][Jordan et al., 2018].

This does not mean that the questions in such surveys are useful. For example, 2 out of 3 teachers of one NBIS course have the shared verdict that the NBIS questions are useless, where 1 is neutral and reasoning 'it is what we commonly do'.

Source?

You can find one teacher's written-down criticism here. He openly uses the term 'useless'.

You can find the meeting notes where it was decided unanymously against using the survey in unmodified form here. In this meeting, only orally, the verdict on this survey was:

• 2x: useless
• 1x: neutral, as '[such surveys] is what we commonly do'

This disagreement is used as a starting point in evaluating the NBIS course evaluation questions, where the literature is searched for how these questions came to be and by which criteria the best were selected, in the hope of establishing the usefulness of this survey.

Which alternatives are suggested?

One alternative that is suggested is to use learning outcomes:

• An example evaluation using learning outcomes

Another alternative that is suggested is to ask for written feedback only to rate teachers:

• An example reflection that used written feedback for its teachers

Also, there is a clear rule to assess the usefulness of an evaluation questions: 'An evaluation question is useful if it is used in a reflection'. Reflections that apply these rules:

• Example 1, where some questions are assessed to be useless
• Example 2, where all questions are used

Besides discussing the current survey question, this paper is the first to give a fully transparent process on how, with the same goals in mind, a similar set of evaluation questions were developed and how the best questions of this set were selected.

References

• [Brazas & Ouellette, 2016] Brazas, Michelle D., and BF Francis Ouellette. "Continuing education workshops in bioinformatics positively impact research and careers." PLoS computational biology 12.6 (2016): e1004916.
• [Gurwitz et al., 2020] Gurwitz, Kim T., et al. "A framework to assess the quality and impact of bioinformatics training across ELIXIR." PLoS computational biology 16.7 (2020): e1007976. website
• [Jordan et al., 2018] Jordan, Kari, François Michonneau, and Belinda Weaver. "Analysis of Software and Data Carpentry’s pre-and post-workshop surveys." Software Carpentry. Retrieved April 13 (2018): 2023. PDF