Adventures in CPSAaron BruckTowns Group MeetingSeptember 25, 20071 1Goals for the summerComplete CPS projects2 different projects2 different projectsQuantification and categorization of CPS questionsQuantification and categorization of CPS questionsLinking CPS results to results on examsLinking CPS results to results on examsLook into new ideas for research (and possibly an OP)Scientific LiteracyScientific LiteracyAssessment toolsAssessment tools2What is CPS?Students use “clickers” to respond to instructor-generated questionsThese responses are stored in an online databaseStudents received grades based on the number of questions answered correctly3CategorizationWe decided to categorize the questions in the following ways:Solo vs. BuddySolo vs. BuddyDefinition vs. Algorithmic vs. ConceptualDefinition vs. Algorithmic vs. ConceptualUsing Blooms TaxonomyUsing Blooms TaxonomyUsing Smith/Nakhleh/Bretz FrameworkUsing Smith/Nakhleh/Bretz Framework1 1We also compared our analyses with those of Mazur2 to make sure we were looking for the right things.1Smith, K. C., Nakhleh, M. B., Bretz, S. L. An Expanded Framework for Analyzing General Chemistry Exams. Journal of Chemical Education. In press.2Fagen, A. P., Crouch, C. H., Mazur, E. (2002) Peer Instruction: Results from a Range of Classrooms. The Physics Teacher. 40, 206-209.4Categorization, cont.Here are the results from one of the sections (others Here are the results from one of the sections (others followed a similar trend):followed a similar trend):Blooms TaxonomyBlooms Taxonomy# of questions# of questions# Definition# Definition# Algorithmic# Algorithmic# Conceptual# ConceptualKnowledge (1)Knowledge (1)444440403 31 1Comprehension (2)Comprehension (2)4444141411111919Application (3)Application (3)27274 421212 2Analysis (4)Analysis (4)3 30 01 12 2Synthesis (5)Synthesis (5)0 00 00 00 0Evaluation (6)Evaluation (6)0 00 00 00 0totaltotal118118585836362424Blooms TaxonomyBlooms Taxonomy# of questions# of questions# solo# solo# buddy# buddyKnowledge (1)Knowledge (1)444441413 3Comprehension (2)Comprehension (2)444434341010Application (3)Application (3)272718189 9Analysis (4)Analysis (4)3 31 12 2Synthesis (5)Synthesis (5)0 00 00 0Evaluation (6)Evaluation (6)0 00 00 0totaltotal118118949424245More categorizationSmith/Nakhleh FrameworkSmith/Nakhleh FrameworkBlooms TaxonomyBlooms Taxonomy# questions# questions# Definition# Definition# A-MaMi# A-MaMi# A-MaD# A-MaD# A-MiS# A-MiS# A-Mu# A-Mu# C-E# C-E# C-P# C-P# C-I# C-I# C-O# C-OKnowledge (1)Knowledge (1)444442420 00 01 10 00 01 10 00 0Comprehension (2)Comprehension (2)444415151 10 07 70 01 116164 40 0Application (3)Application (3)27274 47 74 46 63 30 01 12 20 0Analysis (4)Analysis (4)3 30 00 00 00 01 10 02 20 00 0Synthesis (5)Synthesis (5)0 00 00 00 00 00 00 00 00 00 0Evaluation (6)Evaluation (6)0 00 00 00 00 00 00 00 00 00 0totaltotal11811861618 84 414144 41 120206 60 06Results by CategoryA 2 tailed t-test (solo/buddy) and one way A 2 tailed t-test (solo/buddy) and one way ANOVAs (all others) were performed to test for ANOVAs (all others) were performed to test for statistical differences in the datastatistical differences in the dataAnalyses showed no significant differences Analyses showed no significant differences between any of the categories and how the between any of the categories and how the students performed on the questionsstudents performed on the questionsThe only exception were the solo-buddy questions The only exception were the solo-buddy questions for one professorfor one professor7Solo/Buddy AnalysisPrompted by the unusual results, we further Prompted by the unusual results, we further investigated the solo/buddy analysisinvestigated the solo/buddy analysisWe also looked at pairs of solo/buddy questions We also looked at pairs of solo/buddy questions asked one after the other:asked one after the other:T-test results: t=-2.699 p=0.017 (significant difference)8Thats great, butWe found a significant difference between solo and buddy questionsbut is it worth anything?Our next step was to see if this apparent difference in performance due to style of question translated into better test scores on the exams.9Exam AnalysisWe compared exam questions with questions asked in class using CPS.Surprisingly, we found very few questions on the exams that directly or indirectly corresponded to CPS questions.Each exam was analyzed individually before pooling all of the data to determine any and all effects.10Exam AnalysisAll analyses showed no significant differences at the p=0.05 confidence level.11Instructor EffectsWe also ran an We also ran an analysis to check for analysis to check for any instructor effects any instructor effects that could have that could have possibly skewed the possibly skewed the data. data. Results showed no Results showed no significant differences significant differences at the p=0.05 level:at the p=0.05 level:12Is CPS better than nothing?A final analysis was performed between questions that correlated to CPS questions and those that did not.Unfortunately, no significant differences were found, though the average score was higher for CPS questions.13CPS vs. Nothing ResultsResults of ANOVA:Descriptive Statistics:Percent CorrectPercent CorrectN NMeanMeanStd. DeviationStd. DeviationStd. ErrorStd. Error1 110610663.099863.099817.2548717.254871.675941.675942 217117159.543859.543820.1381120.138111.540001.54000TotalTotal27727760.904660.904619.1326019.132601.149571.1495714ConclusionsCPS is an effective lecture tool that engages students interactively in their contentMost CPS questions are low-level questions in terms of Blooms Taxonomy and other categorization toolsStudents seem to learn content through interaction with their peers when using CPS, though this does not necessarily correlate to success on exams15What else did I do?Research QuestionsResearch Questions In the event that I need to do a project other than the In the event that I need to do a project other than the NSDL project, what avenues are available?NSDL project, what avenues are available? Could any of these ideas turn into a possible OP in the Could any of these ideas turn into a possible OP in the following months?following months?Ideas of interestIdeas of interest Scientific LiteracyScientific LiteracyWhat is the value of a textbook?What is the value of a textbook?Could other materials help?Could other materials help? AssessmentAssessmentImmediate feedback assessment technique (IFAT)Immediate feedback assessment technique (IFAT) Could it work in chemistry?Could it work in chemistry?16