Results of RQ2¶
What are the ELIXIR evaluation questions?
You can find the ELIXIR evaluation questions here.
This paper deals only with the mandatory questions 5 to and including 9.
With the goal of the SFT ('to improve the course and its materials')
in mind, here we go through the mandatory
questions that resulted from the process
described in the results of Research Question 1.
The relevant questions are found in Section 3 - Quality Metrics
of the NBIS short-term evaluation.
Here, we go through each of these questions in detail.
Question 5¶
5. Have you used the tools/resource(s) covered in the course before?
- Never - Unaware of them
- Never - Used other service
- Occasionally
- Frequently
Question 5 is an interesting way to evaluate the quality of a course, because it is about something learners have done before the course took place. Searching the literature for 'using previous experience in course evaluations' (and sentences alike) resulted in zero hits.
What are the metrics for this question?
These are the metrics collected at 2025-01-24 7:04 Stockholm time (https://training-metrics-dev.elixir-europe.org/feedback-report):
| Reponse | n | Frequency (%) |
|---|---|---|
| Never - Unaware of them | 4350 | 23.5 |
| Never - aware of them | 3838 | 20.8 |
| Never - Used other service | 1803 | 9.7 |
| Occasionally | 6974 | 37.7 |
| Frequently | 1528 | 8.3 |
Question 6¶
Question 6 is another interesting way to evaluate the quality of a course, because it is about the usefulness of the topic being taught, combined with predicting the future. Searching the literature for 'using self-predicted future use of content in course evaluations' (and sentences alike) resulted in zero hits.
What are the metrics for this question?
These are the metrics collected at 2025-01-24 7:16 Stockholm time (https://training-metrics-dev.elixir-europe.org/feedback-report):
| Reponse | n | Frequency (%) |
|---|---|---|
| Maybe | 2822 | 15.1 |
| No | 105 | 0.6 |
| Yes | 15792 | 84.4 |
Question 7¶
Question 7 attempt to measure course quality by asking the learner
if he/she would recommend the course.
This question originates from one of the two evaluations that
this ELIXIR evaluation is based
on ([Jordan et al., 2018]).
Searching the literature
for 'using course recommendation in evaluation'
(and sentences alike) resulted in one relevant hit.
This paper, [Ang et al., 2018], shows that using
this question may indeed be a valid way to asses course
quality [Ang et al., 2018].
What are the metrics for this question?
These are the metrics collected at 2025-01-24 8:27 Stockholm time (https://training-metrics-dev.elixir-europe.org/feedback-report):
| Reponse | n | Frequency (%) |
|---|---|---|
| Maybe | 19597 | 89.5 |
| No | 1790 | 8.2 |
| Yes | 519 | 2.4 |
Question 8¶
8. What is your overall rating for the course
- Poor (1)
- Satisfactory (2)
- Good (3)
- Very Good (4)
- Excellent (5)
Question 8 too attempts to measure course quality by asking the learner
to rate it.
This question is absent from the two
questionnaires (i.e. those described in [Brazas & Ouellette, 2016]
and [Jordan et al., 2018]) this questionnaire is based one.
Searching the literature
for 'using course satisfaction in evaluations'
(and sentences alike) resulted in many relevant papers.
The most important paper is a meta-analysis,
which concluded that there is no relation
between training quality and participant
satisfaction [Uttl et al., 2017] and this meta-analysis
gives some examples how problematic this metric is.
What are the metrics for this question?
These are the metrics collected at 2025-01-24 8:28 Stockholm time (https://training-metrics-dev.elixir-europe.org/feedback-report):
| Reponse | n | Frequency (%) |
|---|---|---|
| Excellent | 7736 | 37 |
| Very good | 8437 | 40.4 |
| Good | 3543 | 17 |
| Satisfactory | 993 | 4.8 |
| Poor | 192 | 0.9 |
Question 9¶
Question 9 is an interesting way to measure the course quality, based
on the learner being willing to answer questions on the future.
It seems more likely that question should be placed outside
of the section Section 3 - Quality Metrics.
Searching the literature for 'using future contact in course evaluation' (and sentences alike) resulted in zero relevant hits.
What are the metrics for this question?
These are the metrics collected at 2025-01-24 8:32 Stockholm time (https://training-metrics-dev.elixir-europe.org/feedback-report):
| Reponse | n | Frequency (%) |
|---|---|---|
| No | 8756 | 49.7 |
| Yes | 8860 | 50.3 |
References¶
[Ang et al., 2018]Ang, Lawrence, Yvonne Alexandra Breyer, and Joseph Pitt. "Course recommendation as a construct in student evaluations: will students recommend your course?." Studies in Higher Education 43.6 (2018): 944-959.[Brazas & Ouellette, 2016]Brazas, Michelle D., and BF Francis Ouellette. "Continuing education workshops in bioinformatics positively impact research and careers." PLoS computational biology 12.6 (2016): e1004916.[Jordan et al., 2018]Jordan, Kari, François Michonneau, and Belinda Weaver. "Analysis of Software and Data Carpentry’s pre-and post-workshop surveys." Software Carpentry. Retrieved April 13 (2018): 2023. PDF- [Uttl et al., 2017] Uttl, Bob, Carmela A. White, and Daniela Wong Gonzalez. "Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related." Studies in Educational Evaluation 54 (2017): 22-42.