What is behind a summary-evaluation decision?

Research in psychology has reported that, among the variety of possibilities for assessment methodologies, summary evaluation offers a particularly adequate context for inferring text comprehension and topic understanding. However, grades obtained in this methodology are hard to quantify objectively. Therefore, we carried out an empirical study to analyze the decisions underlying human summary-grading behavior. The task consisted of expert evaluation of summaries produced in critically relevant contexts of summarization development, and the resulting data were modeled by means of Bayesian networks using an application called Elvira, which allows for graphically observing the predictive power (if any) of the resultant variables. Thus, in this article, we analyzed summary-evaluation decision making in a computational framework.

[1]  Betty D. Robinson,et al.  Collaborative Teaching: Reaping the Benefits , 1995 .

[2]  Peter N. Winograd Strategic difficulties in summarizing texts , 1984 .

[3]  Daniel Cassany,et al.  Reparar la escritura : didáctica de la corrección de lo escrito , 1993 .

[4]  Vania Dimitrova,et al.  STyLE-OLM: Interactive Open Learner Modelling , 2003, Int. J. Artif. Intell. Educ..

[5]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[6]  W. Kruskal,et al.  Use of Ranks in One-Criterion Variance Analysis , 1952 .

[7]  P. Spirtes,et al.  Causation, prediction, and search , 1993 .

[8]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[9]  Louis Leon Thurstone The concept of reliability. , 1931 .

[10]  Jason Catlett,et al.  On Changing Continuous Attributes into Ordered Discrete Attributes , 1991, EWSL.

[11]  Jon Williamson,et al.  Abduction, Reason, and Science: Processes of Discovery and Explanation , 2003 .

[12]  M. Stone Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[13]  Pat Langley,et al.  Induction of Selective Bayesian Classifiers , 1994, UAI.

[14]  John R. Kirby,et al.  Students' Approaches to Summarisation. , 1991 .

[15]  Asao B. Inoue Community-based assessment pedagogy , 2004 .

[16]  Wendy G. Lehnert,et al.  Plot Units and Narrative Summarization , 1981, Cogn. Sci..

[17]  S. Paris,et al.  Children's Metacognition About Reading: issues in Definition, Measurement, and Instruction , 1987 .

[18]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[19]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[20]  Michael J. Hannafin,et al.  The collaborative apprenticeship model: Situated professional development within school settings , 2006 .

[21]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[22]  Ron Kohavi,et al.  Supervised and Unsupervised Discretization of Continuous Features , 1995, ICML.

[23]  F. Bartlett,et al.  Remembering: A Study in Experimental and Social Psychology , 1932 .

[24]  Elvira: An Environment for Creating and Using Probabilistic Graphical Models , 2002, Probabilistic Graphical Models.

[25]  Randy Kerber,et al.  ChiMerge: Discretization of Numeric Attributes , 1992, AAAI.

[26]  Maria Virvou,et al.  Adding an Instructor Modelling Component to the Architecture of ITS Authoring Tools , 2001 .

[27]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[28]  Michael Pressley,et al.  Prior Knowledge Affects Text Search Success and Extraction of Information. , 1993 .

[29]  Walter Kintsch,et al.  Toward a model of text comprehension and production. , 1978 .

[30]  R. Garner Efficient Text Summarization Costs and Benefits , 1982 .

[31]  Beau Jones,et al.  Dimensions of thinking and cognitive instruction , 1990 .

[32]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[33]  Judy Kay,et al.  The justified user model: A viewable explained user model , 2003 .

[34]  Richard E. Neapolitan,et al.  Learning Bayesian networks , 2007, KDD '07.

[35]  Barbara M. Taylor,et al.  Text Structure and Children's Comprehension and Memory for Expository Material. , 1982 .

[36]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[37]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .

[38]  Wendy G. Lehnert Plot Units and Narrative Summarization , 1981 .

[39]  P. Thorndyke Cognitive structures in comprehension and memory of narrative discourse , 1977, Cognitive Psychology.

[40]  Ann L. Brown,et al.  Macrorules for summarizing texts: the development of expertise , 1983 .

[41]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[42]  Ely Kozminsky,et al.  First vs second language comprehension: some evidence from text summarizing , 1986 .

[43]  S. Thompson Social Learning Theory , 2008 .

[44]  Peter Clark,et al.  The CN2 induction algorithm , 2004, Machine Learning.

[45]  D. Rumelhart NOTES ON A SCHEMA FOR STORIES , 1975 .

[46]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[47]  Juan Luis Luque,et al.  Effects of an Intervention in Active Strategies for Text Comprehension and Recall , 2012 .

[48]  R. Garner Strategies for Reading and Studying Expository Text , 1987 .

[49]  C. Glymour The Mind's Arrows: Bayes Nets and Graphical Causal Models in Psychology , 2000 .

[50]  E. B. Page Project Essay Grade: PEG. , 2003 .

[51]  T. Bayes An essay towards solving a problem in the doctrine of chances , 2003 .

[52]  Mark T. Maybury,et al.  Advances in Automatic Text Summarization , 1999 .

[53]  Daniel Cassany Reparar l'escriptura: didàctica de la correcció de l'escrit , 1993 .

[54]  Jill Fitzgerald,et al.  Research on Revision in Writing , 1987 .

[55]  E. Hilgard,et al.  Theories of Learning , 1981 .

[56]  Carol Sherrard,et al.  Teaching students to summarize: Applying textlinguistics , 1989 .

[57]  Usama M. Fayyad,et al.  Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning , 1993, IJCAI.

[58]  L. Manelis,et al.  Analysis of expository prose and its relation to learning , 1984 .

[59]  Iraide Zipitria,et al.  Observing Lemmatization Effect in LSA Coherence and Comprehension Grading of Learner Summaries , 2006, Intelligent Tutoring Systems.

[60]  Judea Pearl,et al.  Distributed Revision of Composite Beliefs , 1987, Artif. Intell..

[61]  R. Fisher THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[62]  Peter A. Flach,et al.  Abduction and Induction , 2000 .

[63]  Jill Burstein,et al.  Automated Essay Scoring : A Cross-disciplinary Perspective , 2003 .

[64]  John Long,et al.  Summary and Recall of Text in First and Second Languages: Some Factors Contributing to Performance Differences , 1978 .

[65]  Fred Genesee,et al.  Classroom-Based Evaluation in Second Language Education , 1996 .

[66]  Lorenzo Magnani,et al.  Model-Based and Manipulative Abduction in Science , 2004 .

[67]  Peter Clark,et al.  The CN2 Induction Algorithm , 1989, Machine Learning.

[68]  Pedro Larrañaga,et al.  Feature selection in Bayesian classifiers for the prognosis of survival of cirrhotic patients treated with TIPS , 2005, J. Biomed. Informatics.

[69]  Roger C. Schank,et al.  An Integrated Understander , 1980, Am. J. Comput. Linguistics.

[70]  Marvin Minsky,et al.  Steps toward Artificial Intelligence , 1995, Proceedings of the IRE.

[71]  Solomon Eyal Shimony,et al.  A new algorithm for finding MAP assignments to belief networks , 1990, UAI.

[72]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[73]  Barbara Sherr Roswell,et al.  From Perception to Practice: The Impact of Teachers' Scoring Experience on Performance-Based Instruction and Classroom Assessment , 2000 .