Funded by the Mozilla Responsible Computer Science Challenge:
Cultivating an Ethics-Inclusive Mindset Through Role Play in Undergraduate CS Courses
Problem: Computer Scientists hold profoundly important roles in society, their work impacting lives and communities around the globe, and not always for the better. Incidents involving unethical and illegal practices in computer science (CS) are unfortunately commonplace. As such, institutions of higher education are increasingly recognizing the importance of embedding ethics in undergraduate CS curricula. Moreover, the effectiveness of active learning in instilling real-world social and ethical competencies highlights the need for interactive, non-traditional classroom activities.
Hypothesis: By requiring the exploration of problem spaces through different lenses, our research team posits that role play can be a particularly useful method for broadening student perspectives and meaningfully integrating ethics into CS classes.
Intervention: Our team is designing, piloting, and evaluating multiple role play activities (for both in-person and online classes) over the course of a one year period. We are currently in the process of evaluating Activity 1, which requires participants (stakeholders at a committee meeting) to decide whether or not to introduce autonomous buses into a community.
My Role: Researcher & Experience Designer
Time Frame: August '19 - Present (5 months and counting)
Research: Reviewed literature on role play activity design and socio-ethical issues related to autonomous vehicles (the subject matter of Activity 1), affinity mapped summer pilot feedback, and observed a similar role play activity on the Georgia Tech campus.
Design & Pilot: Developed a final prototype of Activity 1 (which included several different participant materials and an activity guide) and co-facilitated the pilot across three sections of a first-year CS seminar (150+ students per section).
Evaluation: Iteratively developed pre- and post-activity surveys for assessing the fall pilot, co-designed the assessment strategy (which included splitting our population into treatment group and control groups), conducted quantitative and qualitative analysis on the survey data.
Affinity Map of Summer Pilot Feedback
Observation of a Similar Activity (segments of my digitized notes)
Activity 1 Pilot
450+ students participated in our pilot. We split the sample into a treatment group (two sections of a first-year CS seminar) and a control group (one section of the seminar). All participants received a voluntary pre-activity survey two weeks prior to the pilot as well as a voluntary post-activity survey one week following the pilot.
Activity 1 Evaluation
The pre- and post-activity surveys contained a subset of identical questions that were designed to measure participant mindsets before and after the activity. More specifically, these questions attempted to measure how participants understood the role of a computer scientist (what skills and attributes are required of those in the profession) as well as how participants would distribute responsibility for technology's positive and negative impacts on society.
Sample Survey Questions
Quantitative Data: At the group-level, the treatment group's responses to identical quantitative questions in the pre- and post-activity surveys as a group did not change significantly. This may be due to limitations in the assessment strategy (e.g., the timing of the pre- and post activity survey distributions) and/or because mindset changes are difficult to measure in such a short time span and after only one activity.
* I am currently completing the individual-level quantitative analysis, which may reveal insights into how individual students were impacted by the activity.
Sample Quantitative Analysis (Group-Level): As you can see below, the two computer scientist attributes involving ethical and social/cultural consideration (focal attributes in our study) were the lowest scoring items in both the pre- and post-activity survey within the treatment group. While this finding does little to demonstrate the usefulness of role play for embedding ethics in CS classes, it validates one of our study's central premises: computer science undergraduates do not conceptualize their future profession with ethics, society, and culture at the forefront.
Q: Consider the following skills and attributes that a computer scientist might possess. You have 10 stones to distribute among the bins below, each of which represents a different skill or attribute. Please assign importance to each skill or attribute by distributing your stones among the bins, etc. (See above for full question)
Qualitative Data: Responses from two open-ended questions in the post-activity survey provided us with a wealth of qualitative feedback. The insights generated via the affinity map below will inform future iterations of Activity 1 as well as our subsequent role play activities.
While the group-level findings largely fail to demonstrate the effectiveness of our role play activity at cultivating ethics-inclusive mindsets among CS undergraduates, the qualitative data tells another story. Many participants found the activity to be thought-provoking, relevant to their degree, and refreshingly engaging compared to more traditional class sessions. They particularly enjoyed approaching a real world issue from multiple perspectives.
Active learning exercises that embed ethics into CS courses, like our role play activity, are not meant to be one-and-done additions to CS curricula. Rather, it is our belief and the core tenet of the Mozilla Challenge that they should be infused into degree programs systematically and regularly to maximize their potential for positive impact.
- Develop, Pilot, and Evaluate Activity 2
- Develop, Pilot, and Evaluate an online version of Activity 1