Web-based British Sign Language Vocabulary Test

The purpose of the web-based British Sign Language Vocabulary Test (BSL-VT; Mann, 2009; Mann & Marshall, 2012) is to assess deaf children’s vocabulary knowledge in British Sign Language (BSL) “by specifically measuring the degree of strength of the mappings between form and meaning for items in the core lexicon” (Mann & Marshall, 2012, p. 1031). The authors wanted to address one of the main weaknesses of most standardized vocabulary assessments, which is that they measure only one aspect of a person’s vocabulary knowledge. In the pilot study it was developed for children between the ages of 4-15 years (Mann & Marshall, 2012).

 

Development of the instrument

The web-based BSL Vocabulary Test consists of four tasks each of which taps a different degree of strength of vocabulary knowledge. The tasks are: (1) meaning recognition, (2) form recognition, (3) meaning recall, and (4) form recall. The test draws from a model for second language learning by Laufer and colleagues (Laufer, Elder, Hill, & Congdon, 2004; Laufer & Goldstein, 2004), which demonstrated that (spoken) vocabulary develops incrementally and that there is a hierarchy in the degree of strength of vocabulary knowledge. Based on this model, the same items are used across all tasks. Each of the four tasks of the BSL Vocabulary Test consists of (the same) 120 vocabulary items. The test includes two receptive and two production tasks. The two receptive tasks use a multiple-choice format and can be self-administered. The two production tasks require an administrator, who scores each response, based on four options and also documents the given response in a text box on the computer screen, using English glosses.

(1) Meaning recognition task: test takers see a pre-recorded BSL sign followed by four pictures. They have to select the picture that corresponds to the meaning of the signed prompt (Video 1). In the (2) form recognition task, test takers see a picture, followed by four pre-recorded BSL signs and have to select the sign that matches the meaning of the picture prompt (Video 2). In the (3) form recall task, test takers see a picture and have to produce the corresponding BSL sign and in the (4) meaning recall task children see a pre-recorded BSL sign and have to generate another BSL sign with an associated meaning (Since the test was developed, this task has been changed and the test taker now needs to generate three associated signs).

 

Video 1: Examples of the BSL Vocabulary Test, comprehension task 1 (© Mann, 2009b)

Video 2: Examples of the BSL Vocabulary Test, comprehension task 2 (© Mann, 2009b)

 

The items for the test were selected from several sources: (1) a BSL norming study (Vinson, Cormier, Denmark, Schembri, & Vigliocco, 2008), (2)  receptive vocabulary test for German Sign Language (Perlesko; Bizer & Karl, 2002), (3) commonly used, standardized, English vocabulary tests and (4) based on feedback by deaf and hearing researchers and teachers from the school who collaborated with the authors during the item development (Mann & Marshall, 2012). Excluded from the item pool are fingerspellled signs and depicting/pointing signs. The order of the items in each set is randomized each time someone is taking the test. The items of the test belonged to the grammatical categories of nouns, verbs, and adjectives. Based on research for spoken language acquisition which shows that the first acquired words are nouns, the ratio across the three grammatical categories is 8:1:1 for children < 10 years old and 6:2:2 for older children (> 10 years old). In order to address the issue of regional variations of signs, any signs that were known to be subject for regional variation were excluded (including colors and numbers).

Distractor development: For the two receptive skills tasks, which use a multiple-choice format, it was important to develop sound distractors. These tasks consist of four types of responses: (1) the target, (2) a phonological distractor, (3) a semantic distractor, and (4) a visual or an unrelated distractor. They are presented randomly for each item. Signs that were known to be iconic were excluded as much as possible. These items, many of which are commonly used in spoken English vocabulary tests, include categories like body parts and animals, or numbers (Mann & Marshall, 2012). For the remaining iconic items, visual distractors were used.

Participants: Twenty-four deaf children aged 4-15 participated in the study. They came from five programs (three schools for the Deaf, two units/resource bases). All programs used BSL as language of instruction. Of these 24 children, 12 were male and 12 female. Their average age was 11;2. All of the participants had a hearing loss of >70 dB in their better ear. They were either native signers or strong signers, who all used BSL as their preferred language/means of communication. The testing took place in the schools, in a quiet room. For each participant a background questionnaire (online) was completed which included questions about the child’s age, parental hearing status, type of hearing loss, and communication used at home. The questionnaire was filled out by the teacher or a speech language therapist. The children also completed a Nonverbal IQ Task (NVIQ) using Raven’s Progressive Matrices (Raven, 1994) in order to see if any of the test participants has a cognitive delay. All participants also completed the four tasks of the BSL Vocabulary Test.

In a follow-up study, an additional 43 children were assessed, resulting in a total of 67 deaf children (37 male, 30 female) aged 4-17 years (Mann, Roy, & Marshall, 2013, p. 98). One difference to the pilot study was that the newly added participants had more variable BSL skills and also included children from diverse language learning backgrounds and children with additional needs. The goal of this study was to investigate if some key variables in deaf signing children such as parental hearing loss and additional needs affect deaf children’s vocabulary knowledge in BSL (Mann et al., 2013).  Average scores for three of the four tasks were reported.

Procedure: The test was administered by a hearing researcher who is a fluent signer. The test was presented individually for each child. As for the two receptive tasks, the children could operate the mouse and select the appropriate answer themselves. For the two production tasks, the child had to produce an answer in BSL and the researcher entered the answer with English glossed during test administration. All results were saved automatically in the database on the Web server. All four tasks were completed in two sessions, and each session included two tasks. There was about one week between the first and the second session to minimize learning. Each session lasted about 30 minutes. Before the tasks started, the children saw pre-recorded instructions in BSL and could practice on two items.

Scoring: The responses for the two receptive skills tasks were scored as (1) for correct and (0) for incorrect and saved automatically. For the production tasks, four answer choices were provided for the researcher to code the child’s answer live. The four scoring choices were (1) correct sign, (2) partially correct sign, (3) wrong sign/different sign, and (4) do not know. These scores were presented in codes (e.g., CS=correct sign) in order not to affect the child’s motivation. The criteria for “correct” were when the child could provide the expected BSL sign, “partially correct” when the answer was less expected (e.g., VEGETABLE instead of CARROT), but still suggesting that they knew the meaning of the sign. The gloss was also entered in a textbox below the checked answer.

Psychometric properties: In order to investigate inter-rater reliability on the judgment of the responses, a deaf native signer also coded the responses of three children and compared it with the researcher’s coding. The mean reliability was 94.5% (range: 91-97%) for the form recall task and 95.2% (range: 93-97.5%) for the meaning recall task (Mann & Marshall, 2012).

Content validity is given, even when not mentioned explicitly by the authors (Mann & Marshall, 2012), by the way in which the items were developed.

In addition, statistical significant correlations were found between each of the four tasks and age (bivariate correlations) (Table 1), showing that the test is age sensitive.

 

Table 1: Correlation of the four BSL tasks and age (alpha level .013 to compensate for multiple comparison, k=4)), N=24 (from Mann & Marshall, 2012, p. 1040)

 

Meaning recognition

Form recognition

Form recall

Meaning recall

Age

R(24) = .766, p < . 001

R(24) = .758, p < . 001

R(24) = .845, p < . 001

R(24) = .511, p < . 001

 

There are plans to further develop the BSL vocabulary test to a fully computer adapted version (Mann et al., 2013).

The BSL Vocabulary Test has been adapted to American Sign Language although with a reduced number of items (80 instead of 120) appropriate for use with for children aged 6-11 years. (W. Mann, personal communication, January 3, 2014).

 

Availability

Currently the test can be used for free by schools and researchers through  City University's homepage, but a login is required which can be obtained from  Wolfgang Mann, City University London (Mann et al., 2013).

 

Strengths and Weaknesses

Strengths: (1) test assesses different degrees of vocabulary knowledge in deaf children, (2) reporting of (some) psychometric properties, (3) computer-based format makes use of the BSL test easy to administer.

Weakness: (1) no norms available yet.

 

The BSL-VT has been adapted to  American Sign Language.

AUTHOR

Summarized by Tobias Haug (2013; in cooperation with Wolfgang Mann).

For more information regarding this test, please contact  Wolfgang Mann at the University of Roehampton.