Students, Teachers, Exams and MOOCs: Predicting and Optimizing Attainment in Web-Based Education Using a Probabilistic Graphical Model

We propose a probabilistic graphical model for predicting student attainment in web-based education. We empirically evaluate our model on a crowdsourced dataset with students and teachers; Teachers prepared lessons on various topics. Students read lessons by various teachers and then solved a multiple choice exam. Our model gets input data regarding past interactions between students and teachers and past student attainment. It then estimates abilities of students, competence of teachers and difficulty of questions, and predicts future student outcomes. We show that our model's predictions are more accurate than heuristic approaches. We also show how demographic profiles and personality traits correlate with student performance in this task. Finally, given a limited pool of teachers, we propose an approach for using information from our model to maximize the number of students passing an exam of a given difficulty, by optimally assigning teachers to students. We evaluate the potential impact of our optimization approach using a simulation based on our dataset, showing an improvement in the overall performance.

[1]  Badrul H. Khan,et al.  Web-based instruction , 1997 .

[2]  Lise Getoor,et al.  5 Probabilistic Relational Models , 2008 .

[3]  W. Gerth,et al.  Use of cost-effectiveness analysis in health-care resource allocation decision-making: how are cost-effectiveness thresholds expected to emerge? , 2004, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[4]  S. Clearwater Market-based control: a paradigm for distributed resource allocation , 1996 .

[5]  Rena M. Palloff,et al.  Lessons from the Cyberspace Classroom: The Realities of Online Teaching , 2001 .

[6]  Pushmeet Kohli,et al.  Your digital image: factors behind demographic and psychometric predictions from social network profiles , 2014, AAMAS.

[7]  Vincent Tinto,et al.  Moving Beyond Access: College Success for Low-Income, First-Generation Students. , 2008 .

[8]  L. Harasim Shift happens: online education as a new paradigm in learning , 2000, Internet High. Educ..

[9]  David C. Parkes,et al.  Dwelling on the Negative: Incentivizing Effort in Peer Prediction , 2013, HCOMP.

[10]  Peter Brusilovsky,et al.  Adaptive and Intelligent Technologies for Web-based Eduction , 1999, Künstliche Intell..

[11]  N. Hara STUDENT DISTRESS IN A WEB-BASED DISTANCE EDUCATION COURSE , 2000 .

[12]  Nikolay Archak,et al.  Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on TopCoder.com , 2010, WWW '10.

[13]  Mike Moore,et al.  Distance Education: A Systems View , 1995 .

[14]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[15]  S. Dumais,et al.  Cultural Capital, Gender, and School Success: The Role of Habitus , 2002 .

[16]  Gjergji Kasneci,et al.  Crowd IQ: measuring the intelligence of crowdsourcing platforms , 2012, WebSci '12.

[17]  Victor Lavy Evaluating the Effect of Teachers' Performance Incentives on Pupil Achievement , 2001 .

[18]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[19]  Thomas P. Minka,et al.  Gates , 2008, NIPS.

[20]  Omer Lev,et al.  Mergers and collusion in all-pay auctions and crowdsourcing contests , 2013, AAMAS.

[21]  P. Costa,et al.  NEO inventories for the NEO Personality Inventory-3 (NEO-PI-3), NEO Five-Factor Inventory-3 (NEO-FFI-3), NEO Personality Inventory-Revised (NEO PI-R) : professional manual , 2010 .

[22]  Thore Graepel,et al.  Quality Expectation-Variance Tradeoffs in Crowdsourcing Contests , 2012, AAAI.

[23]  Tom Minka,et al.  How To Grade a Test Without Knowing the Answers - A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing , 2012, ICML.

[24]  Jeffrey S. Rosenschein,et al.  Distributed multiagent resource allocation in diminishing marginal return domains , 2008, AAMAS.

[25]  R. Hambleton,et al.  Fundamentals of Item Response Theory , 1991 .

[26]  P. Costa,et al.  Revised NEO Personality Inventory (NEO-PI-R) and NEO-Five-Factor Inventory (NEO-FFI) , 1992 .

[27]  John W. Lounsbury,et al.  Predicting Academic Success: General Intelligence, "Big Five" Personality Traits, and Work Drive , 2004 .

[28]  Steven Glazerman,et al.  Evaluating Teachers: The Important Role of Value-Added , 2010 .

[29]  Pushmeet Kohli,et al.  Manifestations of user personality in website choice and behaviour on online social networks , 2013, Machine Learning.

[30]  T. Graepel,et al.  Private traits and attributes are predictable from digital records of human behavior , 2013, Proceedings of the National Academy of Sciences.

[31]  D. West Introduction to Graph Theory , 1995 .

[32]  Sue Betka,et al.  National Center for Education Evaluation and Regional Assistance , 2015 .

[33]  Linda Darling-Hammond,et al.  Evaluating Teacher Effectiveness: How Teacher Performance Assessments Can Measure and Improve Teaching. , 2010 .

[34]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[35]  Judea Pearl,et al.  Chapter 2 – BAYESIAN INFERENCE , 1988 .

[36]  Roy Williams,et al.  The ideals and reality of participating in a MOOC , 2010 .

[37]  R. C. Sprinthall Basic Statistical Analysis , 1982 .

[38]  B. de Raad,et al.  Personality in learning and education: a review , 1996 .

[39]  Gjergji Kasneci,et al.  Crowd IQ: aggregating opinions to boost performance , 2012, AAMAS.

[40]  T. Brabazon Digital Hemlock: Internet Education and the Poisoning of Teaching , 2002 .

[41]  Yoram Bachrach,et al.  Hotspotting - A Probabilistic Graphical Model For Image Object Localization Through Crowdsourcing , 2013, AAAI.

[42]  Peter Z. Schochet,et al.  Error Rates in Measuring Teacher and School Performance Based on Student Test Score Gains , 2010 .

[43]  Tom Minka,et al.  A family of algorithms for approximate Bayesian inference , 2001 .

[44]  Murray R. Barrick,et al.  THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META-ANALYSIS , 1991 .

[45]  Yoram Bachrach,et al.  Personality and Website Choice , 2012 .

[46]  Jacqueline Scott Family, Gender, and Educational Attainment in Britain: A Longitudinal Study* , 2004 .

[47]  S. Gosling,et al.  A very brief measure of the Big-Five personality domains , 2003 .

[48]  S. Urbina,et al.  Psychological testing, 7th ed. , 1997 .

[49]  Anita Sharma,et al.  Personality and Patterns of Facebook Usage , 2016 .

[50]  Liz Thomas,et al.  Improving the Retention of Students from Lower Socio-economic Groups , 2003 .