Overview of LASI13 for me

Just a few thoughts as we conclude our week at Stanford.

I realise I have a very different perspective on the proceedings. Many attendees are active teachers/ researchers, working on their individual projects within their own courses. For me, I am interested in how we can take advantage of analytics at scale.  In other words, where are the big gains for my institution as a whole? So my first takeaway is reflection on where we are/ should be focussing our efforts.

1. We are currently trialing a dashboard to give students feedback on their level of engagement with the university and their course. While many at LASI have been negative about dashboards, I am still optimistic about the prospect of getting big wins from giving students access to data that we already collect. From LASI13, I have encountered and become very interested in Ruth Crick's work on Learning Power and Rebecca Ferguson's work dispositions. I will explore the feasibility of incorporating ELLI in the dashboard.

2. Am thinking about whether we are becoming guilty of 'provider capture' - producing data and analytics that we are excited about, but that don't actually change anything. ie students dont make use of it, and teachers don't change anything as a result. Will keep thinking on this, but with this in mind ...

3. I came away even more convinced about the urgent need for the "numeracy" course we are developing - ie providing opportunities for students to develop knowledge of math/statistics/ probability without having to focus on equations and mathematics such that they can be informed citizens & professionals. Had a wonderful discussion with Ian Witten (author of Weka). He recommended his project Computer Science unplugged

"activities introduce students to underlying concepts such as binary numbers, algorithms and data compression, separated from the distractions and technical details we usually see with computers."

Exactly what we are trying to do but in numeric literacies!

4. After participating in the workshop on automated scoring and marking, I began to rethink my current stance. The current work is interesting and focussed on giving students a 'mark' and some feedback. My immediate reaction is that it wouldn't be as good as the feedback students would get from a good academic who is skilled at providing feedback. However, some students don't always have that experience, and I continually get survey results saying that students want more feedback. So my questions, is, would this kind of automated feedback be better than little (or no) feedback?

Also, I like David Boud's ideas about reducing student reliance on 'external' marking and feedback and helping them to develop self-assessment skills (see for example http://www.tandfonline.com/doi/abs/10.1080/713695728#.Udb46VPOkbo. Trying to see how Pearson's work could support this approach to supplement their current approach of giving a mark and automated feedback.

After tonight's 14+ hour flight home, I may well have other reflections :)


Interesting links/ posts (to me anyway)

This site was recommended after an informal discussion about the challenges of leading people - a Stanford academic has written a book called 'The no asshole rule' - this is his blog

Kristen deCerbo from Pearson's

Kristen DiCerbo

Blog of what she is taking away

Martin Hawksey ‏@mhawksey used a tool called Topsy to provide a visual documentation of #LASI13
http://hawksey.info/labs/topsy-media-timeline.html?key=tvBQ-F8CcGi2cOQONBzHg8g

Didn't get to Gooru workshop (search engine for learning resources) but must look at. http://www.youtube.com/watch?v=nq9HTYiv7jM

And, on a personal level, LOVE the Moves App that Erik Duval put me onto

Friday 5th July

Open Platforms for pedagogical innovation
Piotr Mitros
Chief Scientist, edX

Mentioned the Concord Consortium (STEM resource finder)

Lots of open learning analytics - an integrated and modularised platform (2011) SIemens et al

What is edX?

circle below

research                      <--->      quality       <--->          access                 links to research

understanding learning     improve learning      educate 1 billion people


What is edX Learning Sciences?

• MITx platform
• open-ended grading
• insights
• crowdsourcing

Says in the beginning he didn't know what he was doing, but had  two and a half months before students were enrolled. example of Circuits and Electronics

Goal: lightweight pedagogical innovations

Similar goals

• ridiculously easy to use
• you don't need your sysadmin - you cannot take down the LMS
• data access without PII
• build a community - open standard (not just edX)

• normalisation
• self-assessment
• randomised problems
• stop feature
• multistage adaptive testing 

missing piece - standard way to process data.

Offline analysis (lots of graphs)

dashboards - analyse what student has done, respond with a hint

Groupwork

What have people got out of the week?

Great Google doc created here

Workshop - computational approaches to analysing student performance: methods and applications

Speaker: Peter Foltz, Pearson and University of Colorado, Institute for Cognitive Science

How do we analyse written and spoken data and turn results into useful information?

• training, feedback, personalisation, tracking, better decision-making

Automated scoring of writing

• writing and feedback
• segue into Latent Semantic Analysis
• what can be scored? not just K-12 ELA essays
• application to formative and large-scale learning analytics

will talk about general methods and approaches

more conceptual than mathematical or procedural

turning open-ended language into performance data

• hand scoring is not feasible for large-scale learning analytics - writing, speaking, chat
• multiple choice and logged interaction data isnt always enough to get at skills and knowledge
• analysis of open-ended spoken and written expression is a key to access higher level cognitive processes

what is the overall motivation?

•  techniques to allow data analytics access a wider range of assessment items
• improve student learning
• improve knowledge/ decision-making of instructors, administrators

automated language analysis

•  complexity of language can be distilled by mathematical methods (see Caroline's session yesterday for details)
• permits automatic assessment and feedback o written and spoken communication

automated scoring approach

• learn to score from human scored student responses
• measure the content and quality of responses by determining: the features that human scorers evaluate when scoring a response; how those features are weighed and combined to produce these scores

scoring approach

• obtain 200 - 1000s of pre-scored essays
• extract features from essays
• use machine learning to combine features to predict scores

Says word count accounts for 60% correlation to high score - smart kids write more (can't agree with that)

Aspects of writing

content-based scoring - Latent Semantic Analysis to capture the 'meaning' of written English

Bell Labs - Yellow Pages problem - eg search for cars - get carpools etc - but not where to buy cars - have to search for automobiles. The synonymy problem.

What does that have to do with automated scoring?

• LSA reads lots of text
• learns what words mean and how they relate to each other
• result is a 'semantic space'
• every word is represented as a vector
• every paragraph represented as a vector

reading comprehension
content scoring - every essay represented as a vector
new essays are placed based on the words they contain

demo interface

tools available

A score is not enough

• detect off-topic or highly unusual essays
• detect if it may not score an essay well
• detect larding of big words, non-standard words, swear words, too long, too short

Reliability and Validity

• has been tested on millions of essays
• generally agrees with a single human reader as often as 2 human readers agree with each other

used in

• situational judgment tasks and memo writing
• scoring physician patient notes
• language testing and translation

how do we know it works?

• comparison against - human scores, external validity criteria
• improvements in process/ visualisations - teacher scoring speed, ability to find students 

No significant difference between this software and 2 human scorers

Scenario-based writing - asking students to answer several questions about a hypothetical, yet realistic, scenario.

CLA - automated scoring.

Situated writing assessment
have worked with military on critical thinking, problem solving, leadership.

Automated assessment of diagnostic skills
actor trained - drs scored on process

Example

what helps in formative writing?
 (Writing Next, 2007; Carnegie Corporation, 2011)

• teaching students strategies for planning, revising and editing their compositions (effect size .82)
• teaching students how to assess their own writing (.46)
• explicitly and systematically teaching students how to summarise texts (.82)
• providing feedback
• monitor students' writing

challenge for implementing formative writing assessment in the classroom

WritetoLearn example

Scorecard

break

showed teacher dashboard - which students are having trouble with the essays?

impressive graphs showing improvement in assessment scores by the number of revisions. biggest gains in 'ideas', smallest in conventions (eg grammar) why? give more feedback on each iteration.

Automated scoring of spoken performance

• assessing understanding and spoken language proficiency in everyday and workplace contexts
• english fluency
• foreign language skills
• expression, use and comprehension of job relevant content

Kinds of tests (mostly English language assessment)

watch a video and describe what is happening

repeat a sentence

listen to a story and do a re-tell (fluency + comprehension skills)

others

describe a picture or graph - looking at vocabulary, language use, pronunciation and fluency

read a passage aloud

how it works

multiple pieces of information - wave form, spectrum, words segmentation

collect a lot of sound samples, human rated, train system to predict human ratings, run validation

What next?

• combined text _ context - need to be able to combine the textual situation with the contextual learning situation
• machine learning is moving fasst
• NLP is moving slower
• getting  a number is not enough

Thursday 4th July

Apache Hadoop
Doug Cutting, Chief Architect, Cloudera

Big data's time has come

graph of Moore's Law

same has held true for other aspects of IT, size, speed, cost etc exponential improvement

how we process data will change too - once industries can harness data we will see big changes there too.

some people love, others hate, the term 'big data'. Who cares?

what is big data?

• its scalable - distributed, commoditized, reliable computing 

trends -

open source's time has come
open software - most based on Linux

at Apache (NFP, 140 projects), quality is emergent

• is flexible - spend a lot of time upfront with relational databases - what are the columns going to be, because presupposing what the Qs are going to be. system should accept raw data, project schemas onto it. general notion of not pre-conceiving the data (schema on read) is the difference between big data vs traditional relational data. Data hard to move so as much as possible want to operate on data where it lives.

Apache Hadoop started as a batch system - started out trying to do build an index and search engine similar to what Bing does. Google published some papers on what they were doing, he decided that way a better way of doing things. Implemented that.  Hadoop is a massive batch processing system. if you want to do some processing

Has added tools to Hadoop ecosystem eg

• Apache Pig - data flow language
• Hive - SQL engine
• Mahout - library of machine learning programs
• HBase - after a paper from Google - lets you store values under keys in effectively huge table, incredible rates of insert and lookup. no SQL stores
• cloudera impala - SQL engine - executes SQL queries interactively
• Solr - search technology with distributed scalable search.

Hadoop becoming the big data operating system

User
Application
Operating System
Hardware

Big data fuels innovation

"more data beats better algorithms"

"need a platform you can shove your data into and have access to a rich set of tools to enable you to explore your data."

big data - spreadsheets on steroids

recommended video
Peter Norvig - The Unreasonable Effectiveness of Data

+---
Plenary: Playing Nicely in the Sandbox: Ed Tech, Education Researchers, & Business
Speakers: Alan Louie (Imagine K12 - incubator for startups who want to build for K12), Steve Schoettler (startup Junio), Nicole Forsgren Velasquez

Where is the win-win?

Alan
get 20K - have to spend 2 months in Staford area, 10 teams per 6 month period. took knowledge we have about startups and how it can be applied for K12.

Needs to be more collaboration between researchers and startups.

Edtech startups pick fast and cheap - exactly opposite for academics - go for good.

some people keep researchers as far away from startups as possible.

Nicole
Initially looked at as though she was an alien because she came from business

accept that people are going to monetise things. Edtech startups have data - so contact someone from a Faculty of Business.

Steve
sucker for hard problems.

has found that working with others requires a multidisciplinary approach. startup owners manual says start with cheap and scratchy and iterate to get it working. lots of startups are missing key elements.

often see great ideas but they havent thought of things like, how to get a teacher to use this? need for scalable, reliable technology.

have to join and bring together all disciplines to address these end to end problems.


mention of zynga.com - free online, social games

+---

Panel: Analytics for 21st Century Skills
Panelists: Rebecca Ferguson, Ruth Deakin Crick, Peter Foltz

what analytics literacies do people need for 21st century?
how can analytic tools/ technologies help people become literate?

literacy as

• reading and writing
• reading the environment - understanding the joint result of complexity among supporting elements (people, technology, context)
• writing the environment - contributing, changing, augmenting, designing, modding, evaluating

new literacies of eLearning

1. multimodal - communicate across multiple platforms eg storify, pinterest
2. multiactor
3. sociotechnical
4. collaboration
5. emergence - comfortable with continuously evolving environment

Peter
Data analytics for 21st century kills: focus on writing, speaking & communicating

• higher level thinking skills becoming increasingly important in the workplace
• common core PISA, ATC21S, DEAG
• general competencies - cognitive, problem solving etc, leadership

assessment of the free expression of knowledge and skills

to demonstrate, learners must be able to process and generate it independently
think, talk and write using effective skills

MCQ doesn't cut if but hand scoring of written and spoken tests not scalable

how can technology meet this challenge:

• convert written and spoken performance into measures of skills and abilities
• need engageing and realistic items that train and test people with the context and content for the workplace
• reliable, valide, efficient, cost effective
• help personalisation, realtime feedback, decidion-making by teachers, administrators

therefore Pearson doing:

Automated language analysis
complexity of language can be distilled by mathematical methods - use computational linguistics, NLP, machine learning, automated speech recognition

eg writing

• summative K-12 writing
• formative writing practice with feedback
• situation judgment tasks - what would you do?
• tasks and simulations - write an email to your boss etc

speaking assessment

• english/ foreign language proficiency
• reading fluency
• create a sentence with these three words
• describe what you see in a video

team communication

• predict team performance measure and warn instructors
• automated monitors for students and teachers in learning chat rooms

Comments on PISA2015

how do we incorporate new kinds of thinking?

how do we emphasis students working together?

Q
Intelligent essay assessor - any progress beyond word count? Yes

Ruth
21C Skills: EnquiryBlogger

everybody has a little list

European Commission list

Learning to learn: perspective from Theory and Practice, Routledge

competence is more than one thing - includes identity, learning power, knowledge skilss & understanding = competence in the world   <-- used in EnquiryBlogger

See  in journal
Crick & Claxton (2004) Developing and Effective Lifelong Learning Inventory: the ELLI Project, Assessment in Education.

See also assessment article

Rebecca
EnquiryBlogger: reflection and relationships

EnquiryBlogger site

showed example of a real blog - built on standard collaborative Wordpress basis. Couldnt help but notice spelling

ELLI Spider: learning power

structuring knowledge

clicking on a blob will take you to that blog post

Mood view: managing mood

learningemergence.net/tools/enquiryblogger
Wordpress plugin

Workshop - quantified self technologies and analytics

Quantified Self Technologies and Analytics
Erik Duval, Abelardo Pardo

quantifiedself.org

Asked what attendees use

most said fitbit - wear constantly and get feedback on kms walked etc fitbit.com

also rescuetime - checks browser use and social media use - therefore what is and isnt productive in how you use your time. Can set goals eg no more than 1 1/2 hours per day in email.

Moodpanda - tracks mood - get a map of where you feel differently

Lift - android - express a goal, create a 'habit' and start checking in. get little graphs showing how you are doing

Abelardo
Observe only while engaged in the activity, then turn off (magic switch).

sensors are becoming a commodity

Groupwork
1 - what would you like to measure?

brainstorm then discussion - making creates opportunities for innovation ie collecting data just because you can, can pay unexpected dividends eg using Runkeeper - when in a foreign place, ask it to recommend a run (Runkeeper knows how far you typically run etc)

wearable & ubiquitous 'in' and 'out' fitbit gives Erik a little buzz once he's gone a few Km

2 - come up with a QS scenario and how you could use it

Venice unfolding example  

Erik's blog on the workshop

Wednesday 3rd July

Theme of day said to be linking psychology and data

Using Learning Analytics to Illuminate Student Learning pathways in an Online Fraction Game

Taylor Martin, Nicole Velasquez
activelearninglab.org

created games to help students understand fractions. see http://play.centerforgamescience.org/refraction/site/ 

looks much like the game my grandsons love to play - use laser beams to divide up screen to get fractions eg half of a half of a half is an eighth.

used analytics to look for clusters of approaches

cluster 1 - minimal (careful, slow approach) low success rate
2 - Haphazard trying a lot, moving around a lot, success low.
3 - explorers (some iterations but not a lot, thoughtful) medium level success
4 - strategic explorer (exploring a lot of things quickly) success high
5 - careful - take medium time, considered, go straight to solution

post-test score not associated with strategy

Self-regulated learning and trace data,
Phil Winne

Basic idea is that learners lack tactics and strategies. They also don't have data about their learning tactics, or about which tactics have effects.

See
Winne (2011) - Cognitive & Metacognitive Analysis of Self-Regulated Learning
bit.ly/19Y1OAW 

Demonstrated nStudy software that enables students to create bookmarks, highlight and save quotes, tags, notes, documents, concept maps. tool for learners to engage and share information.

Affect and engagement during learning,
Sidney D’Mello

tracking emotions in learning
study learning contexts
note affective states
develop computational models

take home message
confusion can be beneficial to learning

Building the educational data scientist
Ryan Baker, Piotr Mitros, Marcelo Worsley

Ryan
Creating a grad program MS in Learning Analytics but has to be approved by NY state. Interime  Masters in Cog Studies in Ed, Focus in LA.

@ Teachers College, Columbia University

Creating a Masters in Learning Analytics, but for regulatory reasons Master of Cognitive Science with a focus on LA

Everything (curriculum etc) will be open access

(note his forthcoming MOOC on big data in Education)

Marcelo
www.lse.cs.cmu.edu

Piotr
(Chief Scientist at edX)

what employers want
complex problems solving
communication,
critical thinking,
ability to get things done.

also some core skills:
mathematical maturity
informatice
cognitive science and communications


need to give a more broad theoretical background and tie back to learning - but with a focus on data

edX has zero people as 'data scientists' neither does Udacity

John reminds us of the different mix of skills required for different roles as per the analysers book.

(image not so clear so here are the horizontal (roles) and vertical (skills) headings - ie red blocks are statistical knowledge required)

                Data business person   data creative   data developer   data researcher
Business
Big Data
Math/OR
Programming
Statistics

discussion

• importance of data curators - requires skills and knowledge of: ontology, database design, meta-data structures, technical sides of storage, legal requirements, records management etc
• also need to highlight importance of empirical research methods - used in conjunction with data mining
• critique - lots of discussions about 'techniques', not much about ethics etc. 
• Q to Piotr re no data scientists - says they built on Google experience: hired people who are talented, have communications skills etc

other links
University of Michigan UG program  School of Information's UGrad program

+---

Panel: Funding and Institutional Support – CERAS 101
Chair: Taylor Martin
Speakers: Edith Gummer (NSF), Ed Dieterlie (Bill and Melinda Gates Foundation), Suzi Hewlett (Australian Office of Learning & Teaching), Susie Vaks DePianto (Google)

Edith
Data-intensive research to improve learning and teaching

Data Intensive Ideas Lab

Ed from Gates Foundation
Funding & Institutional Support

Central goal in K-12 is for 80% or more to graduate college-ready. Currently 55M students in K12 pipeline, many behind and likely not to graduate high school college ready. Want to accelerate learning and get students back on track. Each year 4.2M children enter K.

Breakthroughs leading us closer to personalisation of learning for all learners. called it a confluence of breakthroughs moving us closer to the personalisation of learning for all learners

One of these is hexagons is Advanced Learning Analytics
This is why they have invested in this event.

frameworks for puzzling through a potential investment
optimism - taking on messy challenges NEED
collaboration - if you want to run fast, run alone, if you want to run far, run together APPROACH
rigor - BENEFIT (what will you compare your work to?)
innovation -  ALTERNATIVES

Map of building blocks

building blocks

Suzi Hewlett

Good overview of OLT's program.

Workshop - Dispositional Learning Analytics

Held in the lounge area of the the building so acoustics not great. Simon Buckingham-Shum chairing the session via Skpe :)

Website, speakers, links here

Played a PPT introduction featuring Simon's voiceover

began with some definitions
what is a learning disposition - enduring tendencies in the way that you behave, goes deep.

as educators we are trying to create intentional learners.

ref to John Dewey - must be the will or desire to employ intentions

Katz talks about habits of mind not as mindless habits. Contrasts attitudes and dispositions

Dweck - need for growth mindset - believe in your talents and abilities and that they can be developed through passion, education and persistence

Ruth Deakin Crick - contrast and compare students and teachers dispositions. common is location of self in relation to knowledge

John Seely Brown - looking at the profiles of what it means to be effective in 21st C - argues that resilience will be the defining concept. dispositions are now at least as important as knowledge and skills. cannot be taught, can only be cultivated



Thanks Simon!
See http://www.c-spanvideo.org/clip/4457327

argument about complex adaptive systems - turbulent times, unprecedented demands on capacity to adapt and relearn

need to develop personal and systemic resilience

Framing the challenge: designing learning ecosystems for white water

do you have the capacity to persist?

worried about

• disengaged learners who know curriculum and have ability
• high test achievers but who go to pieces outside of comfort zone

 ethics of analytics at front of discussion

If students and teachers have better dispositions for learning, can we demonstrate a difference?

can we move from self-report and researcher-report to automatically compute DLA directly from activity traces + other datasets

+--

Ruth Deakin Crick

headlines & key ideas

dispositions are key levers for sustainable learning in 21C conditions of uncertainty, risk and challenge

learning eco-systems are complex and unpredicatable and cannot be sustained by external control: self-directed learning is therefore crucial

images
 - metaphor of jumping over hurdles - pre-set, high stakes, one after the other
- nature scene with road in middle - aim to make it real but still a prescribed path through the curriculum
- in a wilderness with no path

dispositions can be modelled - 'learning power'

learning power
mix of values, attitudes and dispositions which together are necessary for fuelling an individual's engagement with new learning opportunities through identity formation and scaffolding.

7 dimension of learning power

being stuck & static --> changing and learning
data accumulation --> meaning making
passivity --> critical curiosity
being rule bound --> creativity
being robotic --> strategic awareness
fragility and dependence --> resilience

Learning power correlates with other success criteria (but its a complex relationship)

• distinct patterns of learning power profiles for under-achieving students
• lp is positively associate with pro-social behaviour and negatively with violent behaviour

self-reporting questionnaire ELLI Effective Lifelong Learning Inventory

see Learning Futures Report, London, Paul Hamlyn Foundation

showed cohort analytics for educator and organisational leaders - pie charts

LA makes the invisible visible

fried egg model .. learning is an emergent property

modelling dispositions makes connections between ....

See also http://www.slideshare.net/Ruthdeakincrick/learning-dispositions-and-transferable-competences-pedagogy-modelling-and-learning-analytics

Simon Buckingham Shum and Ruth Deakin Crick (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd International Conference on Learning Analytics & Knowledge (Vancouver, 29 Apr-2 May, 2012). ACM. pp.92-101.
Open Access Eprint: http://oro.open.ac.uk/32823 / Slides/Replay

Deakin Crick, R. (2012). Student Engagement: Identity, Learning Power and Enquiry – A Complex Systems Approach. In S. Christenson, A. L. Reschly & C. Wylie (Eds.), Handbook of Research on Student Engagement (pp. 675-694). New York: Springer.

Deakin Crick, R., & Yu, G. (2008). Assessing Learning Dispositions: Is the Effective Lifelong Learning Inventory Valid and Reliable as a Measurement Tool? Educational Research, 50, (4), 387-402.

Dweck, C. S. (2000). Self-Theories: Their Role in Motivation, Personality, and Development. New York: Psychology Press.

Learning Power, ViTaL Partnerships website: http://www.vitalpartnerships.com/learning-power

Shaofu Huang (2013). Prototyping Learning Power Modelling in SocialLearn. Social Learning Analytics Symposium, The Open University, UK (June 20, 2013). Webcast/slides

+---
Dave Paunesku http://stanford.io/1cO7c5T  on the GPA impact of changing student mindset (a la Dweck)

David S. Yeager and Gregory M. Walton (2011). Social-Psychological Interventions in Education: They’re Not Magic. Review of Educational Research, vol. 81, no. 2, 267-301. DOI: 10.3102/0034654311405999. http://rer.sagepub.com/content/81/2/267.

+---

Chris Goldspink
Layers, Loops and Processes: Multi-level Analytics in Learning Systems

PPT slides available

starting with individual agency and engagement in learning

If you want to gauge a student's interest you first have to uncover disposition

Showed complex diagram derived from structural equation modelling

references
Ruth Deakin Crick, Chris Goldspink & Margot Foster (2013). Telling Identities: Learning as Script or Design? Learning Emergence Discussion Paper (June, 2013). PDF

Ruth Deakin Crick, Steven Barr & Howard Green (2013). Evaluating the Wider Outcomes of Schools: Complex Systems Modelling. Learning Emergence Discussion Paper (June, 2013). PDF

Shaofu Huang (2013). Modelling Learning Dynamics in an Authentic Pedagogy Setting. Presented at the Systems Learning and Leadership Seminar, Graduate School of Education, University of Bristol, UK (May 24, 2013). Prezi Slides


Discussion about importance of carefully scaffolding the process of handing over agency/responsibility for learning to students.

What sort of learning architecture will provide rapid feedback at multiple levels to make the invisible visible and enhance responsible agency?

 +---
Learner Dispositions: Big Data Meets Focused Social Science Research
Nelson Gonzáles, Chris Goldspink and Ruth Deakin Crick

PiersonLabs: http://www.piersonlabs.com

Declara -personalisation through persistent data

ELLI - learning dispositions through data mashup

ESA - rapid prototyping through social collaboration

Showed a recommendation Engine

Platform Architecture

looking at social collaboration

Declara's semantic search and predictive analytics:

• intuitively understand the intent of social interactions
• push personalised content and connections to them

 this enables

personalised

adaptive

project-based

learning and professional development

Slide of next generation user interface


OTHER WORKSHOPS

Heard good stories about
Datawrangling workshop with "open" tools http://slidesha.re/13s0NxA

Tuesday 2nd July

Tuesday 2 July

Session 1

Market Dynamics to Accelerate the Educational Data Science Field:

Making a market for learning analytics

Stephen Coller (Bill and Melinda Gates Foundation) @eduforker

His interest in this field began with a conversation with Roy Pea about the systems architecture required for learner ecosystems to provide personalised learning. Gates foundation sees great potential for this field to improve experience of learning.

To improve the instructional core you need to:

• change content
• increase the knowledge and skills of instructors
• alter the relationship of the student to the teacher and the content

Have not focused enough attention on what students are actually doing. A K-12 study ref? showed a bias in focus on what the instructor does (Pie chart showed about 80%) student performance was only about 10%. (we have known this in higher ed for a long time!!!! Hence the Learning2014 project at my university)

How might market forces contribute? Showed a diagram showing four components in a circle students -> product markets -> institutions -> resource markets -> students

Steps
1. A user-experience that gets heads nodding and hearts racing
Students "I am paying a s***-load. What are my gains? Where am I heading?"

2. A (credentialed) product of Service to meet them
Product markets "tasks predict performance" -- mentioned Minerva model (high end model - poses challenges, transdisciplinary teams to work on them). Think this now high-end model may trickle down to community colleges.

3. Demand aggregated at sufficient scale to produce (data and dollar) resources
Institutions "How do I evaluate student outcomes? How do I assess and improve instruction?"

4. Responsive brokerage of data and dollars to meet and fuel follow-on demand
Resource markets "What is good content? What is an effective task?"

What role should foundations play?

(better image via Simon Buckingham-Schrum at pic.twitter.com/Zl0qEUdHWN)

horizontal axis - demand nascent to emerging to established
vertical axis - economic incentive for suppliers low to high

(blue) cell with established demand and high incentive
Strengthem incentive for performance and innovation: common performance and technical standards.

Says LA is between nascent and emerging on low end of incentives

Closing thoughts

• what are the quickest wins in the user experience?
• who is the principal customer for this service? States? Institutions? Households?
• Is this an embedded or stand-alone product? MOOCS could go 1 of 2 ways - as a new version of correspondence courses OR could be weave the institutional experience with what goes on outside
• How might the Feds help?
• what are the barriers to innovation holding the market back?
• How can communities like LASI and platforms like Globus add up to a commons? What steps remain before we have an enabling environment?

Mentioned a number of concerns including:

•  use of tools from private providers is a 'black box' solution - not open in any way
• are we equipping learners for the complexity of life?

break
Hearing good things about the Gooru workshop yesterday - I went to Google Apps workshop - well live-blogged by Doug Clow

Yes! can't agree more Kim Arnold :)


Session 2
Data mining and intelligent tutors
Ryan Baker & Ken Koedinger CMU
Learnlab - Pittsburgh Science of Learning Center

Ken says he will finish in 16 minutes :) then Ryan will speak.

Why LA is important?
most of what we know we are not aware of, thus ed design is flawed. Therefore need data-driven models of learners. "cognitive models" can be automatically discovered.

example - you might say you know English. But do you know what you know?

Quotes Richard Clark with picture of iceberg

Cognitive tutors: interactive support for learning by doing

Ken more interested in what is hard to learn

Using model design -> deploy -> data ->  discover ->

Using control-treatment group approach :(

Ryan Baker

Modeling engagement in ASSISTments - developed engagement assessments - perspective on engagement in literature see Fredericks et al, 2004

look at affective and behavioural engagement - developed automated engagement detectors

studied
off-task behaviour -
gaming the system eg systematic guessing, hint abuse
affect including boredom, frustration, confusion

Field Observation method
protocol designed to reduce disruption to student

• some features: observe with peripheral vision or side glances, hover over student not being observed, 20 second roundrobin ...

tried 4 different algorithms
model assessment
cross-validation for generalisability

model goodness - looked at outcomes for

• boredom
• frustration
• engaged concentration
• confusion
• off-task
• gaming

Nice but does it really matter?
Engagement and learning (ref?, 2013)

applied affect detectors to middle school students to predict whether student would attend college 6 years later? 58% college attendance

showed results - but so fast, couldnt get it down. Doug Clow did a better job of this :)

Models predict end-of-year tests and college attendance goal: use them to make a difference in these constructs



 Great comment on twitter:


Closing comment - we need to balance "The hare of intuitive design and the tortoise of cumulative science"

 see also
D'Mello et al. (2008) Automatic Detection of Learner’s Affect from Conversational Cues http://ow.ly/mAViC


Critique: Interesting ideas but has reinforced my existing views on cognitive science!  Learning is so complex we can't hold everything constant and vary one thing as we can in a laboratory. Therefore I can't see much point in the control-treatment group approaches to proving something works in learning. It also suffers from the old 'correlation does not equal causality' misconception as Kristen has pointed out above.

Concerned that LA is not making use of some of the more contemporary approaches to understanding and designing learning such as those from neuroscience, and phenomenography. Will try to write something more about this


Session 3
Plenary: Learning Analytics in Industry: Needs and Opportunities
Chair: George Siemens
Speaker: Maria Andersen (Instructure), Alfred Essa (Desire2Learn), Wayne C Grant (Intel), John Behrens (Pearson)

 George asking for perspectives

John starts with 4 major influence on data analysis:

1. formal theories of statistics
2. accelerated developments in computers and display devices
3. the challenge in many fields of more and ever larger bodies of data
4. the emphasis on quantification in an ever wider variety of disciplines

then confesses this is a quote from John Tukey and Wilk  in 1966. the more things change ...

refers to common issues between higher ed and K-12


fundamental flow about how LA happens

world -> symbol -> analysis -> interpretation -> communication

pic from Harris, Murphy & Vaisman (2013) Analyzing the analysers: an introspective survey of data scientists and their work

Alf from Desire2Learn

Started out by quoting my question from yesterday on what are the really big questions?

Which results are promising and are they reproduceable?

Eric Mazur's work on the flipped classroom - lecture mode in Physics in completely ineffective.can close gender gap with interactive teaching (at least in Physics) potentially generalisable in STEM anyway?
refs:
http://www.physics.indiana.edu/~hake/
http://ajp.aapt.org/resource/1/ajpias/v74/i2/p118_s1


carl weinman UBC - experiment - 2 groups - lecture vs interactive learning. experimental group scored twice as well as traditional. astounding result. who is reproducing that experiment?

Wayne from Intel

principles - designing whole solutions to deliver personalised, active learning experiences.

4 major pillars of activity -
products, OEM products, Intel classmates program
software,
locally relevant content, Pearson, McGraw Hill
implementation support, Intel teach program

Mentions Gartner group's hype cycle - Gartner puts LA at the top of the peak of inflated expectations. Audience puts it as rising to that. (See current version of hype cycle and technologies)

Says we have to be careful otherwise it will get killed



like


Maria Andersen  from Instructure

Is from a much smaller company hence different perspective - what we need from researchers

havent changed habits of teachers despite all the data we have been collecting and analytics. Frustrating that we create platforms for teachers, but they dont look at it. Looking for insights into how that data should be 'surfaced'

Industry want access to research and she hopes researchers will continue to publish pre-prints and, papers should include a "practical application" section to help industry practitioners.