Workshop - computational approaches to analysing student performance: methods and applications

Speaker: Peter Foltz, Pearson and University of Colorado, Institute for Cognitive Science

How do we analyse written and spoken data and turn results into useful information?

• training, feedback, personalisation, tracking, better decision-making

Automated scoring of writing

• writing and feedback
• segue into Latent Semantic Analysis
• what can be scored? not just K-12 ELA essays
• application to formative and large-scale learning analytics

will talk about general methods and approaches

more conceptual than mathematical or procedural

turning open-ended language into performance data

• hand scoring is not feasible for large-scale learning analytics - writing, speaking, chat
• multiple choice and logged interaction data isnt always enough to get at skills and knowledge
• analysis of open-ended spoken and written expression is a key to access higher level cognitive processes

what is the overall motivation?

•  techniques to allow data analytics access a wider range of assessment items
• improve student learning
• improve knowledge/ decision-making of instructors, administrators

automated language analysis

•  complexity of language can be distilled by mathematical methods (see Caroline's session yesterday for details)
• permits automatic assessment and feedback o written and spoken communication

automated scoring approach

• learn to score from human scored student responses
• measure the content and quality of responses by determining: the features that human scorers evaluate when scoring a response; how those features are weighed and combined to produce these scores

scoring approach

• obtain 200 - 1000s of pre-scored essays
• extract features from essays
• use machine learning to combine features to predict scores

Says word count accounts for 60% correlation to high score - smart kids write more (can't agree with that)

Aspects of writing

content-based scoring - Latent Semantic Analysis to capture the 'meaning' of written English

Bell Labs - Yellow Pages problem - eg search for cars - get carpools etc - but not where to buy cars - have to search for automobiles. The synonymy problem.

What does that have to do with automated scoring?

• LSA reads lots of text
• learns what words mean and how they relate to each other
• result is a 'semantic space'
• every word is represented as a vector
• every paragraph represented as a vector

reading comprehension
content scoring - every essay represented as a vector
new essays are placed based on the words they contain

demo interface

tools available

A score is not enough

• detect off-topic or highly unusual essays
• detect if it may not score an essay well
• detect larding of big words, non-standard words, swear words, too long, too short

Reliability and Validity

• has been tested on millions of essays
• generally agrees with a single human reader as often as 2 human readers agree with each other

used in

• situational judgment tasks and memo writing
• scoring physician patient notes
• language testing and translation

how do we know it works?

• comparison against - human scores, external validity criteria
• improvements in process/ visualisations - teacher scoring speed, ability to find students 

No significant difference between this software and 2 human scorers

Scenario-based writing - asking students to answer several questions about a hypothetical, yet realistic, scenario.

CLA - automated scoring.

Situated writing assessment
have worked with military on critical thinking, problem solving, leadership.

Automated assessment of diagnostic skills
actor trained - drs scored on process

Example

what helps in formative writing?
 (Writing Next, 2007; Carnegie Corporation, 2011)

• teaching students strategies for planning, revising and editing their compositions (effect size .82)
• teaching students how to assess their own writing (.46)
• explicitly and systematically teaching students how to summarise texts (.82)
• providing feedback
• monitor students' writing

challenge for implementing formative writing assessment in the classroom

WritetoLearn example

Scorecard

break

showed teacher dashboard - which students are having trouble with the essays?

impressive graphs showing improvement in assessment scores by the number of revisions. biggest gains in 'ideas', smallest in conventions (eg grammar) why? give more feedback on each iteration.

Automated scoring of spoken performance

• assessing understanding and spoken language proficiency in everyday and workplace contexts
• english fluency
• foreign language skills
• expression, use and comprehension of job relevant content

Kinds of tests (mostly English language assessment)

watch a video and describe what is happening

repeat a sentence

listen to a story and do a re-tell (fluency + comprehension skills)

others

describe a picture or graph - looking at vocabulary, language use, pronunciation and fluency

read a passage aloud

how it works

multiple pieces of information - wave form, spectrum, words segmentation

collect a lot of sound samples, human rated, train system to predict human ratings, run validation

What next?

• combined text _ context - need to be able to combine the textual situation with the contextual learning situation
• machine learning is moving fasst
• NLP is moving slower
• getting  a number is not enough