Discussion¶
First and foremost, one can argue that a questionnaire is simply a questionnaire. That the development of a questionnaire is 'just' a group effort. That the people in such groups are constrained by time. And that this is a good enough excuse. However, this overlooks the fact that at least sixteen thousand participants have taken the time to fill in this questionnaire. If we care about our participants, maybe we should care about the usefulness of a questionnaire we send to each of them.
RQ1: What is the history of the ELIXIR evaluation questions?¶
Although this is not formally published, maybe the current survey (or the surveys it is based on) have been developed by evidence-based best practices, yet in unpublished and more informal documents. However, it seems unlikely that references to the literature are used in informal communication, yet subsequently omitted when a formal academic paper is written. To us, it seems more likely that only a shallow research search has been performed, hence that there was little awareness of the evidence-based literature at the time of writing.
RQ2: Can the current ELIXIR evaluation questions be criticized?¶
The majority of the ELIXIR mandatory evaluation questions, in the section to assess course quality can be criticized. Three out of five questions (i.e. questions 5, 6 and 9), however, simply seem to be in the wrong session. Why it was chosen to put these questions in the section called 'quality metrics', instead of a (new) section with a better fitting name is unknown.
Of the two questions that do seem in place, they seem to be measuring the same thing: course satisfaction (on its own), and recommending a course (because the learner is satisfied with the course). It is unknown why these two similar questions are both in the evaluation and it would be interesting to see how strong the correlation is between the answers on these two questions.
Regarding these two questions, however, only one is modestly supported
by the literature ('Would you recommend the course?'),
where the other ('How satisfied are you with the course?')
has strong support against it. Note that the former question came
from the Data Carpentries questionnaire (as described
in [Jordan et al., 2018]), where the latter has been added by ELIXIR.
Epilogue¶
We know that a teacher reflecting on his/her work is one of the
best ways to increase his/her teaching quality.
Or: 'student ratings can only become a tool for enhancement when they
feed reflective conversations about improving the learning process and when
these conversations are informed by the scholarship of teaching and
learning [Roxå et al., 2021].
The other best way for teachers to improve is to do peer observations.
Note that neither practice needs an evaluations.
If we really care about teaching quality, shouldn't we encourage doing the things that actually work?
References¶
[Roxå et al., 2021]Roxå, Torgny, et al. "Reconceptualizing student ratings of teaching to support quality discourse on student learning: a systems perspective." Higher Education (2021): 1-21.