Skip to content

talk.md

Latest commit

 

History

History
148 lines (85 loc) · 4.47 KB

talk.md

File metadata and controls

148 lines (85 loc) · 4.47 KB

Goal: Implement a model of adaptive assessment in Numbas, for:

  • diagnostic tests
  • assessment for learning

DIAGNOSYS

  • A knowledge graph of topics, linked by dependency.
  • One question per node on the graph.
  • Classify each node as "passed" or "failed".
  • After answering a question, mark it and its backward/forward dependencies as passed/failed.
  • Learning objectives are subsets of the nodes, e.g. "Algebra", "Calculus".
  • Use student's qualifications to estimate starting point.
  • Limited "lives" for retrying a question.

A graph of 7 nodes. Each is labelled with a question mark. There are lines linking some of the nodes

The same graph. One of the nodes is now highlighted.

The highlighted node has been marked with a tick.

All nodes linked to the highlighted node on the left are marked with a tick.

Instead, the highlighted node is marked with a cross. All other nodes are still labelled with question marks.

All nodes linked to the highlighted node on the right are marked with a cross.

"Mastery"

Inspired by Duolingo.

  • Roughly linear.
  • Each topic has several questions.
  • All questions must be answered correctly.
  • Failed questions are put back on the end of a queue.

A graph of 6 nodes. The third and fourth node both link to the second and fifth nodes. The fifth node is highlighted and an arrow from it points to a list of items numbered 1 to 4.

Item number 1 is marked with a tick and shaded out.

Item number 2 is marked with a cross.

Item number 2 moves to the end of the list. Now item number 3 is at the top.

Mathspace

Talk by Mo Jebara at EAMS 2018

  • Inner loop: immediate question feedback
  • Middle loop: pick a question within a topic
  • Outer loop: pick a topic

The Mathspace knowledge graph - nodes linked by edges

Item response theory

  • Update P(pass topic) after each answer.
  • Stop asking questions when confidence is high.

Another model (Möbius?)

  • Estimate student's knowledge level on a linear scale.
  • Move up or down based on answers.
  • Ask N questions, chosen based on student's level.

Implementation in Numbas

  • Exam author defines topics and learning objectives.
  • Topics have "depends on" / "leads to" relations.
  • One question group per topic.
  • Controlled by a diagnostic algorithm.
  • Some built-in, can extend or write your own.

(See the documentation)

A flow chart. First node labelled "Initialise state", leads to a node labelled "Pick a question". That leads in a circle to "Student answers", then "Produce options". The path then branches. Following round the circle leads to "Update state", then back to "Pick a question". The other branch leads to a final node labelled "Test ends".

The same flowchart, with annotations describing feedback at each stage. After "pick a question", feedback is "% completion, estimate of level, current topic". After "Student answers", feedback is "Usual part feedback". After "Produce options" and before "Update state", feedback is "Correct/incorrect; suggested next step". After "Test ends", feedback is "Scores for objectives; summary text."

Success

I've reimplemented DIAGNOSYS in Numbas.

numbas.mathcentre.ac.uk/exam/22135/diagnosys

Writing an adaptive test is hard

Need to write lots of questions.

Must think hard about model of knowledge, and relations between topics.

Two tweets. First is by Howie Hua: Your brain throwing out the definition of a derivative once you learn the power rule. Below is a picture of Indiana Jones replacing an idol labelled with the limit definition of a derivative with a sandbag labelled with the standard derivative of a power of x. Second tweet by David Butler: But did you know the definition before you learned the power rule?

Partial success!

To do:

  • Use DIAGNOSYS on students
  • Improve the editing interface: more easily configurable settings
  • Implement some other models

Your input is very welcome!