Structuring, Aggregating, and Evaluating Crowdsourced Design Critique

Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a web-based system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.

[1]  Scott R. Klemmer,et al.  Learning design wisdom by augmenting physical studio critique with online self-assessment , 2012 .

[2]  Abigail Sellen,et al.  Getting the right design and the design right , 2006, CHI.

[3]  Jeffrey Heer,et al.  CommentSpace: structured support for collaborative visual analysis , 2011, CHI.

[4]  Garr Reynolds,et al.  Presentation Zen: Simple Ideas on Presentation Design and Delivery , 2008 .

[5]  D. Sadler.  Formative assessment and the design of instructional systems , 1989 .

[6]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[7]  Scott R. Klemmer,et al.  How bodies matter: five themes for interaction design , 2006, DIS '06.

[8]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[9]  Donis A. Dondis.  A primer of visual literacy , 1973 .

[10]  Terry Barrett,et al.  A Comparison of The Goals of Studio Professors Conducting Critiques and Art Education Goals for Teaching Criticism , 1988 .

[11]  Amy L. Housley Gaffney,et al.  Beyond Content, Deeper than Delivery: What Critique Feedback Reveals about Communication Expectations in Design Education , 2008 .

[12]  Elizabeth Gerber,et al.  Computer supported novice group critique , 2014, Conference on Designing Interactive Systems.

[13]  Loren Olson,et al.  CritViz: Web-Based Software Supporting Peer Critique in Large Creative Classrooms , 2013 .

[14]  Daniel L. Schwartz,et al.  Parallel prototyping leads to better design results, more divergence, and increased self-efficacy , 2010, TCHI.

[15]  Mark Guzdial,et al.  Learner-centered design: the challenge for HCI in the 21st century , 1994, INTR.

[16]  Terry Barrett,et al.  A Comparison of The Goals of Studio Professors Conducting Critiques and Art Education Goals for Teaching Criticism , 1988 .

[17]  Deanna P. Dannels,et al.  Critiquing Critiques , 2008 .

[18]  Martin Wattenberg,et al.  Designing for social data analysis , 2006, IEEE Transactions on Visualization and Computer Graphics.

[19]  Robin Williams.  The non-designer's design book, third edition , 2008 .

[20]  Avon Murphy,et al.  The Non-designer's Design Book , 1995 .

[21]  William Lidwell,et al.  Universal Principles of Design , 2003 .

[22]  Brian P. Bailey,et al.  What do you think?: a case study of benefit, expectation, and interaction in a large online critique community , 2012, CSCW.

[23]  Ann L. Brown,et al.  How people learn: Brain, mind, experience, and school. , 1999 .

[24]  Brian P. Bailey,et al.  Voyant: generating structured feedback on visual designs using a crowd of non-experts , 2014, CSCW.

[25]  Daniel L. Schwartz,et al.  Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results , 2011, CHI.

[26]  Jeffrey Heer,et al.  Strategies for crowdsourcing social data analysis , 2012, CHI.

[27]  Edmund Burke Feldman,et al.  Practical Art Criticism , 1994 .

[28]  Elizabeth Gerber,et al.  A pilot study of using crowds in the classroom , 2013, CHI.

[29]  Jennifer Marlow,et al.  From rookie to all-star: professional development in a graphic design social networking site , 2014, CSCW.

[30]  D. Schoen.  Educating the reflective practitioner , 1987 .

[31]  Austin Lee Nichols,et al.  The Good-Subject Effect: Investigating Participant Demand Characteristics , 2008, The Journal of general psychology.