How to evaluate new word learning

In our project, we used a bespoke Word Knowledge Profile. This was designed to measure the depth of knowledge that teenagers had about the words which we taught them in the sessions and the matched control words.

For each word, we asked them to repeat the word, think of a rhyming word, spell the word (because this took a while, we only asked them to spell some of the words), rate their own knowledge of the word, use the word in a sentence, tell us what the word meant, think of when they would use or hear the word and think of a time that they ‘performed’ the word (e.g. think of a time they evaluated something). This allowed us to measure changes to how the teenagers could understand the word, use the word and relate the word to their own lives.

The table below gives more detail on this.

Examples of how we scored the answers correctly versus incorrectly are given in a document here (in relation to the teenagers telling us what the word meant).

A worksheet for completing the Word Knowledge Profile is provided here.  This features the words that we taught and the matched words from our project. Of course, you can adapt this to include the words that you teach in your word learning sessions!

Item Description
Word repetition Participants were asked to repeat each word after one adult model.
Rhyme production Participants were asked to generate a rhyming word for each word.  If they were unable to think of a real word, they were encouraged to think of a non-word.  Both real word and non-word rhymes were accepted as correct.
Spelling Participants were asked to spell six of the 20 words. Phonetically plausible attempts were accepted as correct.
Knowledge rating Participants were asked to rate their own knowledge of the word according to a red-amber-green system.  They were given the following instructions (Beck, McKeown and Kucan 2002):

Red: I do not know this word.  I have not heard it before and cannot use it.

Amber: I know something about this word.  I have heard it before but am not sure how to use it.

Green: I know this word.  I know what it means and can use the word.

If the participant rated the word as ‘red’, the participant was usually unable to complete the rest of the Profile for this word.

Ability to use the word in a sentence Participants were asked to use the word in a spoken sentence.  This was scored as correct if the participants produced the word as a verb in a sentence which was syntactically and semantically correct.  Sentences without an appropriate semantic component such as I exploited something were not scored as correct.
Word meaning task Participants were asked to describe the meaning of each word. This was scored correctly if the student indicated that they knew something about the meaning of the word.   Detailed definitions or multiple contexts were not required for a mark but were noted on the Profiling sheet.  Very few responses included a detailed definition; therefore this liberal scoring procedure was adopted.
When or where would you use this word? Participants were asked to think of an example of when or where they might use the word.  This was used as an indication of the participants’ engagement with the word and the likelihood of the word being used.  If the participant was able to give an example of when they would use the word, their response was scored correctly (e.g. ‘I would use eliminate during a computer game’).
Personal context Participants were asked to give an example of a personal context in which they had executed the action of the word e.g. for evaluate ‘Can you give me an example of when you have evaluated something?’.  If the participant was able to give an example of when they had completed the action, their response was scored correctly (e.g. ‘I evaluated the word learning lessons in the interview’)