DOKK Library

What is an Algorithms Course? Survey Results of Introductory Undergraduate Algorithms Courses in the U.S.

Authors Arushi Arora Frederick Reiber Jennifer Wong-Ma Matthew Ferland Michael Luu Michael Shindler Randy Huynh Varun Nagaraj Rao

License CC-BY-4.0

    What is an Algorithms Course? Survey Results of Introductory
            Undergraduate Algorithms Courses in the U.S.
                         Michael Luu                                      Matthew Ferland∗                            Varun Nagaraj Rao∗
           University of California, Irvine                      University of Southern California                     Princeton University
                  Irvine, CA, USA                                     Los Angeles, CA, USA                              Princeton, NJ, USA

                    Arushi Arora                                           Frederick Reiber                            Jennifer Wong-Ma
                    Randy Huynh                                             Boston Universtiy                           Michael Shindler
                                                  Boston, MA, USA                 
           University of California, Irvine                                                                      University of California, Irvine
                  Irvine, CA, USA                                                                                       Irvine, CA, USA
ABSTRACT                                                                                to their wide applicability, algorithmic skills are foundational to
Algorithms courses are a core part of many CS programs, but have                        improving student programming ability [15].
received little focus in computing education, lacking statistical data                     Educators play an important role in establishing curriculum stan-
about how they are generally taught. To remedy this, we present the                     dards and ensuring graduates are competent. The ACM Computer
results of the first large-scale comprehensive survey of undergradu-                    Science Curricula 2013 [8] makes such an attempt for undergraduate
ate introductory algorithms courses at four-year institutions in the                    CS education. It provides “realistic and adoptable recommendations”
United States. Questions in the survey targeted instructor informa-                     for 18 knowledge areas, including “Algorithms and Complexity.”
tion, course concepts, the ways students are evaluated, challenges                      This curricula was formed through a combination of survey data
instructors encountered, and instructor envisioned improvements.                        and and several rounds of community feedback. However, this lacks
We received 87 responses from 34 different states, across a wide                        detailed (public) data about the content and structure of courses.
variety of 4-year institutions. The results indicate that algorithms                       It is important to understand the current content and structure
courses vary dramatically in most surveyed areas.                                       of courses for several reasons. First, it can help establish curricu-
                                                                                        lum standards like those published by the ACM [1, 2, 8]. Second,
CCS CONCEPTS                                                                            the complexity of transferring course credits between institutions
                                                                                        can be reduced when content is aligned and prerequisites are well-
• Social and professional topics → Computer science educa-
                                                                                        calibrated [3]. And third, it can help inform future curriculum revi-
                                                                                        sions and education research, such as the prioritization of topics
                                                                                        for inclusion in a concept inventory (CI), like one created for data
                                                                                        structures [10]. For more about Computer Science CIs, see [14].
algorithms education, survey, curricula                                                    We set out to answer two research questions - What is the cur-
ACM Reference Format:                                                                   rent content and structure of introductory algorithms courses? and
Michael Luu, Matthew Ferland, Varun Nagaraj Rao, Arushi Arora, Randy                    What associations and trends emerge from the analysis of the content
Huynh, Frederick Reiber, Jennifer Wong-Ma, and Michael Shindler. 2023.                  and structure of algorithms courses? To answer these questions, we
What is an Algorithms Course? Survey Results of Introductory Under-                     have conducted the first large-scale comprehensive survey of in-
graduate Algorithms Courses in the U.S.. In Proceedings of the 54th ACM                 troductory algorithms courses at institutions in the U.S. that offer
Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023),
                                                                                        a four-year undergraduate degree in CS or a closely related field
March 15–18, 2023, Toronto, ON, Canada. ACM, New York, NY, USA, 7 pages.
                                                                                        (e.g. software design, information technology). We sent the survey
                                                                                        to 411 faculty from across all 50 states who have recently taught
                                                                                        such a course. We received 87 responses. The rest of the paper is
                                                                                        structured as follows - we discuss data collection and survey design
The study of the design and analysis of algorithms is fundamental                       (Section 3), we present quantitative results of our survey (Section 4)
to undergraduate computer science (CS) education [8]. In addition                       and we analyze our results to obtain additional data-driven insights
∗ Equal   Contribution
                                                                                        (Section 5). Overall we see major divergences in course structure
                                                                                        and content
                           This work is licensed under a Creative Commons Attribution
                           International 4.0 License.
                                                                                        2   RELATED WORK
SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada                                     Prior work on algorithms education has primarily focused on identi-
© 2023 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9431-4/23/03.                                                       fying student misconceptions. Farghally et al. [4] study misconcep-                                                 tions across a list of algorithm analysis topics, including asymptotic
SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada                                                                                   Michael Luu et al.

analysis and recursive analysis, among others. Shindler et al. [12]              In the second part, based on the core topics from the ACM Com-
perform a replication study of [19] that targeted misconceptions              puter Science Curricula 2013 [8], we asked where the topic is first
about dynamic programming. Özdener [9]’s study identifies stu-                taught (i.e., prerequisite course, the algorithms course in question,
dents’ misconceptions about time-efficiency of algorithms, while              other elective or required courses, nowhere in the curriculum, un-
Velázquez-Iturbide [16]’s work addresses misconceptions about                 sure, or other). Multiple core topics under Basic Analysis were com-
optimization problems and their corresponding algorithms.                     bined into the topic “Asymptotic Analysis” to simplify and shorten
   Other research related to algorithms concerned effective teach-            the survey length. Similarly, we included an abridged version of the
ing and evaluation strategies [5, 11, 18] and means to incorporate            Proof Techniques section from the Discrete Structures knowledge
responsible-computing content into the course structure [6].                  area to identify how proofs were integrated into the course. The
   More relevant to our work is the study conducted by Hertz [7].             “Basic Automata, Computability and Complexity” and “Advanced
Their survey was focused on CS 1 and CS 2 course topics, whereas              Data Structures, Algorithms, and Analysis” sections were preceded
our survey concerned all algorithms course topics. Similar to our             by a question on whether the instructor taught any topics that
own work, they found significant divergence between courses.                  could fall under either section and if not, the section was skipped.
                                                                                 Finally, we asked three open ended questions, on issues encoun-
                                                                              tered, desired course changes, and any other comments.
3.1 Data Collection                                                           3.3    Threats to Validity
The first step of our data collection process entailed creating a (non-       The most significant threat to the validity of our survey is in the
comprehensive) list of academic institutions to include in the study.         selection process of courses and institutions. In the first wave, we
We only included universities in the United States that had a 4-year          selected the initial universities ourselves. As such, the first institu-
computer science (or related) program. We also ensured that each              tions we reached out to were more often well-known universities.
of the 50 states was represented, at least to some extent.                    Our second and third waves addressed this problem by attempting
    Once our list of institutions had been constructed, we began ex-          to be almost fully comprehensive in previously uncovered states
amining the catalog descriptions of undergraduate courses at each             along with a few other states. However, it should be noted that this
institution in order to identify a suitable course. If the course title in-   means we do not have a perfectly random sample. Secondly, while
dicated the class was introductory and solely an algorithms course            “properly titled” introductory courses would always be surveyed,
(e.g., “Design of Algorithms,” “Analysis of Algorithms,” “Fundamen-           programs that didn’t have such a course were only surveyed if they
tal Algorithms,” etc.), we included it. Otherwise, if there was no            included a topic from “Algorithmic Strategies.” This criterion, how-
such course, we would include another course with “algorithm(s)”              ever, may cause some confirmation-bias, since this pre-supposes
in the title so long as there was evidence that at least one topic            that algorithms courses need to cover these strategies.
from the ACM’s “Algorithmic Strategies” was included. This condi-                Another threat comes from the six adjunct respondents. While
tion was most often applied to “Data Structures and Algorithms”               adjuncts likely know the course they are teaching very well, it is
courses, which were often a terminal algorithmic course.                      possible that they are less familiar with the general curriculum of
    After creating lists of undergraduate algorithms courses, for each        the institution as a whole. A survey by the TIAA Institute indicates
course, we compiled a list of up to three instructors who had taught          around 23% of adjuncts have jobs outside of academia, and 26%
it within the past 2 years. To find the instructors for each course, we       teach at multiple institutions [17]. As such, this could lead to certain
applied a variety of methods. The most common way was through                 information being omitted or misrepresented.
looking at publicly viewable course schedules which list instructors’            Finally, there are threats related to doing surveys. All of the
contact information, though other sources were employed, such                 data is given voluntarily by those who responded, which can bias
as course websites. After finalizing the list of emails, we sent an           the sample. There can be errors when filling out surveys. For ours
invitation to participate in the survey to all instructors in that email      particularly, we allowed the final two sections to be skipped if the
list. Each survey respondent was offered a $25 Amazon gift card               instructor indicated they did not teach any topic in the section.
for completing the survey.                                                    It could be that in these cases, the topics are taught in different
                                                                              locations at these universities. There is also the issue of sample size;
3.2    Survey Design                                                          we only received responses from 87 instructors.
When designing the survey, our goal was to gain insight into both
the course structure and the topics typically taught. The survey              4     RESULTS
contained two main parts. The first, consisting of three sections,            Our initial list had 615 higher-education institutions. Of these, 495
focused on the organization of the course. These sections ask about           had a 4-year computer-science adjacent program, and 373 of those
general course details, such as grading breakdowns, dedicated class           had an applicable algorithms course. From this, we were able to
section times, programming assignments, written assignments, and              send out emails to 411 instructors from 302 institutions whose
in-class assessments (exams and quizzes).                                     contact information we were able to find. We were able to gather
   One of our motivations behind these questions was to determine             responses from 87 instructors across 79 institutions in 34 states.
emphasis on the theoretical components versus applied components              57 of these responses were from doctoral universities with 36 be-
of the courses. As such, we asked questions about both the content            ing from an R1 university, 12 from an R2 university, and 9 from a
and number of assignments.                                                    doctoral/professional university as defined by the Basic Carnegie
Survey Results of Introductory Undergraduate Algorithms Courses in the U.S.                                          SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada

Classification [13]. 14 of these responses were from master’s degree-
granting institutions, and 16 were from baccalaureate colleges.
These rates mirror the ACM 2013 curricula survey [8]. 70 of these
institutions used a semester system, seven used a quarter system,
and two classes had a term length of 12-13 weeks. As answered by
respondents, of the distinct courses, 70 were upper-division courses,
11 were lower-division, and 2 were indicated as both (by different
respondents teaching the same course).

4.1     Instructor Background
We found that over 90% of instructors were professors, comprising
31 assistant professors, 25 associate professors, and 24 full profes-
sors. 6 respondents were adjunct faculty, and one was a graduate
student teacher. We also asked about their involvement in teaching
in comparison to research. We found that only 11% of instructors
considered themselves more focused on research, with 54% being
more teaching-oriented. Additionally, we categorized the respon-
dents’ research areas into theoretical and non-theoretical1 . Of the
82 that did any amount of research, we classified 39 of them as
doing theoretic research.
                                                                                                 Figure 1: Grading criteria for course categories
4.2     Course Structure
4.2.1 General Course Details. Participants were asked to describe
                                                                                        21% had assignments that involved plotting the algorithms with
the grading criteria of their courses. We provided categories for
                                                                                        data (eg: the time elapsed at various input sizes).
participation, programming assignments, in-class assessments (ex-
ams, quizzes, etc), and written homework assignments2 (Figure 1).
                                                                                        4.3     Course Topics
Among these categories, participation and programming assign-
ments typically accounted for the lowest portion of the total course                    In figure 2, we present graphs of all of the results from this part of
grade, with nearly 29% of respondents weighting programming                             the survey. We use the written part of this section to note what we
assignments at 0%, and over 55% of respondents weighting partic-                        believe are the highlights.
ipation at 0%. In-class assessments, in contrast, were usually the                      4.3.1 Algorithmic Strategies. This first part of the Course Topics
highest portion of the course grade, with over 26% of instructors                       section deals with topics concerning algorithm design techniques.
noting that they weighted it at over 60% of the total grade.                            Of the 79 responses3 , we observe that nearly all of the responses
   For the “other” category, responses varied. There were presenta-                     cover divide-and-conquer and greedy algorithms with one and four
tions, student-made class notes, digital quizzes, reading assignments                   responses respectively stating the topic is covered in a different
(with a flipped-classroom), take-home exams, and some logistical                        course. The same is true for dynamic programming, with only seven
complications due to different grading systems.                                         responses stating they do not cover it. The other topics were taught
4.2.2 Evaluation of Student Mastery. Among the 81 instructors                           in between 24% and 58% of classes.
who gave non-programming homework assignments, the average                              4.3.2 Fundamental Data Structures and Algorithms (DSA). Multiple
number of assignments was 8.1, with many indicating that they                           topics related to data structures, like binary search, numerical algo-
required proof writing, algorithm design, complexity analysis, and                      rithms, binary trees, hash tables, and sorting algorithms, are taught
algorithm tracing. Similarly, for the 85 instructors who gave in-class                  in a prerequisite course for 45 to 60% of responses. Moreover, graph
assessments, an average of around 4.3 were given, again with the                        topics, such as graph representations as well as breadth- and depth-
same five areas as the homework.                                                        first search, are taught in over 50% and over 60% of the courses
                                                                                        respectively. Notably, shortest-path and minimum-spanning tree
4.2.3 Programming Assignments. Of the 66 respondents (76%) with
                                                                                        algorithms were taught by over 80% of the respondents.
programming assignments, they assigned on average 5.2 through-
out the academic term. There was significant variation among                            4.3.3 Proof Techniques. We split the proof techniques section into
which languages were accepted - Java (89%), Python (47%), C++                           two main categories: basic logic (ie: notions of implication, equiva-
(36%), and C (13%). 13% accepted any language. 91% of instructors                       lence, converse, inverse), and mathematical proof techniques. Proof
indicated their assignments including having students implement                         techniques were taught in algorithms by only 11 instructors (13%),
algorithms, 82% had assignments where they design their own, and                        and only 1 taught elementary logic. Both topics were covered in pre-
                                                                                        vious courses by 47 respondents (54%), with 2 indicating only proofs
1 We considered theoretical to be any research involving algorithms, data structures,   were taught in a perquisite, and 7 (8%) indicating that logic was
theory of computation, complexity, or other heavily mathematically-oriented fields.
2 One respondent did not use a weighted grading system. They instead had specific       3 Due to an early error in the survey, we did not receive responses from 8 instructors
requirements for each grade. They gave >60% to homework, quizzes, and participation     for this section and the Fundamental Data Structures and Algorithms section.
SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada                                                              Michael Luu et al.

                         (a) Algorithmic Strategies                    (b) Fundamental Data Structures and Algorithms

                              (c) Basic Analysis                                    (d) Proof Techniques

           (e) Basic Automata Computability and Complexity          (f) Advanced Data Structures Algorithms and Analysis

         Figure 2: Distribution of where Sub-Topics are taught as defined by ACM Computer Science Curricula 2013 [8]
Survey Results of Introductory Undergraduate Algorithms Courses in the U.S.                                      SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada

                                                                                     Theme       Description
                                                                                     1           Difficulties regarding students’ insufficient background, par-
                                                                                                 ticularly in these areas:
                                                                                                   A) Mathematical and proof-writing skills
                                                                                                   B) Programming and implementation skills
                                                                                     2           Difficulties in creating problems for in-class assessments and
                                                                                                 homework assignments, especially those with answers that
                                                                                                 cannot be easily found online
                                                                                     3           Difficulties with illustrating relevance and real-world applica-
                                                                                                 tions for algorithms
                                                                                     4           Difficulties with a particular algorithms topic that students
                                                                                                 struggled with
Figure 3: Breakdown of themes of 80 responses for Q1                                 5           Difficulties in having students adequately apply algorithm
(left)4 and 46 responses for Q2 (right)                                                          design techniques to new problems
                                                                                     6           More programming assignments and focus on implementation
previously taught but not proofs. Most other instructors indicated                               of algorithms
that it was taught in a required (but non-prerequisite) course.                      7           Cover less topics, move topics to other courses (existing or
                                                                                                 new), or replace topics with other topics
4.3.4 Basic Analysis. The basic analysis section of our survey con-                  8           Ensure prerequisite courses cover necessary knowledge
tained several categories, such as asymptotic analysis and recur-                    9           More emphasis on proofs and mathematical aspects
rence relations. The majority of respondents (over 50% or higher                     10          Add lab/recitation section to the course for more chances to
for each topic) indicated all eight topics as being covered in their                             practice problems
own algorithms courses; around 10%-20% of respondents noted that                    Table 1: Themes in instructors’ difficulties with teaching
these basic analysis topics were covered in a prerequisite course.                  algorithms (Q1) and in changes instructors’ would like to
                                                                                    implement, assuming full control over course (Q2)
4.3.5 Basic Automata, Computability, and Complexity. Automata
and complexity theory was skipped by 52% of instructors surveyed.
However, the other 48% cover at least P vs NP, 45% cover NP-
completeness, and 10% also cover the halting problem. Only 2% to 4%
of instructors covered other complexity and theory of computation                   5 ANALYSIS
topics, but those topics appeared elsewhere in the curriculum in an
elective, prerequisite, or required course.                                         5.1 Implications of Instructors’ Background
                                                                                    As previously noted, 39 respondents reported performing research
4.3.6 Advanced Data Structures, Algorithms, and Analysis. Topics                    in theoretical computer science or mathematical fields while 47
in advanced DSA and analysis were skipped by 47% of instruc-                        were involved in other kinds of research or none at all. For the
tors. Out of the remaining 53% of instructors, graph theoretical                    37 of those 39 theory researchers who gave written homework
algorithms such as topological sort and finding strongly connected                  assignments, 33 (89%) indicated that they require students to write
components were common with 73% of instructors who answered                         proofs, all 37 required algorithm runtime analysis, 35 (95%) required
this section covering it (38% overall). Network flow is also fairly                 designing algorithms, and 22 (59%) required tracing algorithms. For
common, covered by 64% of this section’s respondents (34% overall).                 the 44 instructors involved in other fields of research who gave
Balanced tree-related algorithms were usually covered in prereq-                    homework, 28 (64%) required proofs, 39 (89%) required algorithm
uisite courses. Instructors who answered this section indicate that                 runtime analysis, 37 (84%) required algorithm design, and 35 (82%)
the remaining topics from this section are either typically covered                 required tracing algorithms.
by an elective course or not covered in the curriculum at all.                         Notably, those involved in theoretical research required proofs
                                                                                    more often and tracing algorithms less often than those not involved
4.4     Free Response Questions                                                     in theoretical research, suggesting some difference in emphasis on
In the final section of the survey, we asked respondents three op-                  the mathematics of algorithms versus the implementation. This is
tional free-response questions about difficulties encountered in                    also reflected in the in-class assessments, sporting similar trends to
teaching their course (Q1), changes they would like to implement                    the data regarding assignments. Although, it is significant to note
assuming they had full control over their course (Q2), and any other                that only 61% and 49% of theory and non-theory researching instruc-
final thoughts they believed would be important for us to know.                     tors respectively expect students to write proofs in assessments.
We found commonalities among the responses and categorized                          Furthermore, the grading criteria for programming assignments,
them into common themes with Q1 and Q2 each having 5 different                      written assignments, and exams did not significantly differ between
themes (with 2 sub-themes for Q1), listed in Table 1. Percentage                    instructors involved in theoretical and non-theoretical research.
breakdowns of these themes are presented in Figure 3.

4 The stacked bar graph represents responses showing Theme 1 and subthemes 1A and   section showcasing the percentage of responses that did not fall under 1A or 1B, often
1B (responses may have exhibited either or both as shown) with the remaining blue   due to plainly stating that prerequisite knowledge was lacking.
SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada                                                                                                   Michael Luu et al.

5.2      Themes from Free Response Questions                                                  the topics). Given these observations, it is possible that the “typical”
When teaching algorithms, by far the most common issue instruc-                               DSA topics concern tree traversal and graph algorithms. Further-
tors reported facing was insufficient background on prerequisite                              more, 𝑂 (𝑛 log 𝑛) sorting algorithms are the only DSA topic that has
concepts, with 54 of the 80 (68%) responses to Q1 exhibiting this                             any moderate associations with any Algorithmic Strategic topics,
(Theme 1). 40 out of those 54 (74%) made references to students’                              most notably having an Cramér’s V score of 0.34 with divide-and-
weak foundation in mathematics and proof-writing. Only nine re-                               conquer, suggesting 𝑂 (𝑛 log 𝑛) sorting algorithms may be covered
sponses (17%) expressed concern with a lack of programming skills                             to serve as examples for algorithmic paradigms. For the advanced
to properly implement algorithms (the final 10 responses (19%) not                            algorithms topics, they have moderate intra-associations and weak
elaborating on the prior knowledge students were lacking).                                    inter-associations, possibly implying these topics are rarely covered
   Another concern raised was the assessment of students’ algo-                               together. Balanced trees, graph-theoretic algorithms, and network
rithm design skills (Themes 4 and 7). Instructors frequently com-                             flow, which show up in a higher percent of algorithms/prerequisite
mented on the difficulty in creating assessments and assignments                              courses, have less association with the other advanced topics.
for students to apply their knowledge in designing algorithms. A                                 We did not find any high associations between institution data
key difficulty is to come up with novel problems, unique from those                           and instructor’s research area and our survey variables, suggesting
discussed online. Another is that some students are unable to apply                           that algorithms course structure and topics are not particularly
algorithm design beyond problems encountered in class.                                        dependent institution type or professor research background.
   Respondents also noted the abstract nature inherent in learning
these topics as a struggle. Themes 5 and 6 illustrate this as in-                             6   FUTURE WORK
structors note students’ struggles in recognizing the importance of                           A more focused survey could expand upon the themes we derived.
algorithms and their relevance with real-world applications. Some                             Other research could look for and apply solutions to these issues.
instructors listed particular algorithmic concepts that students                              Finally, conducting surveys in other areas can further increase our
had trouble with as well, most commonly dynamic programming,                                  understanding of the current state of CS education.
greedy algorithms, and NP-completeness.
   Of the 46 responses we collected for Q2, we witness two cat-                               7   CONCLUSION
egories of themes: those expressing desires to add to the course
                                                                                              In this study, we found that while instructors often use the term
in some way, like adding more material, emphasizing particular
                                                                                              “algorithms” to refer to a specific course, the actual course being
topics, or providing more opportunities to apply what they learned
                                                                                              referenced has a large variation between academic institutions: it
(Themes 6, 9, 10) and those which seek to address the difficulty of
                                                                                              can be oriented around mathematics, programming, or somewhere
the course by reducing the amount of material covered or ensuring
                                                                                              in between. Even within these divisions, instructors often select
prerequisite courses cover the concepts needed (Themes 7 and 8).
                                                                                              very different subsets of topics. In addition, instructors regularly
                                                                                              feel students are unprepared for their courses and have a variety of
5.3      Associations Among Survey Variables and                                              new directions they wish to take it - whether to focus on certain
         Institution Data                                                                     course topics or to make the class more or less mathematical.
We looked to see if there were any associations between the survey                               As educators and researchers, we need to take into account
variables and additional data we collected on the institutions we                             the diversity in how algorithms courses are taught. Future studies
surveyed5 . We also used our previous categorization of instructors’                          should keep this in mind, and either make efforts to be applicable
research in algorithms or non-algorithms adjacent research as a                               to a variety of courses or be tailored to a specific version of the
variable. We obtained these associations by calculating the Cramér’s                          course. When interacting with new transfer students or graduate
V measure for each pair of variables6 .                                                       students, we should be careful not to make assumptions about the
   We found that there are some moderate associations between                                 topics covered in prior algorithms courses. Finally, as educators,
variables from the same sections of the survey. We first see this with                        we should make use of this heterogeneity to pull elements from
the variables from the “Evaluation of Student Mastery” sections                               other courses into our own to continue to improve and refine.
where question types on the homework and exams were correlated
with one another. Likewise, for the Fundamental DSA topic sec-                                ACKNOWLEDGEMENTS
tion, there are moderate associations between quadratic sorting                               We are extremely grateful to the 87 respondents who filled out the
algorithms, hash tables, binary trees, and heaps, having Cramér’s V                           survey. We are grateful to Michael Dillencourt for helpful com-
values in the range 0.18 and 0.55. On the other hand, graph represen-                         ments with early drafts of the survey. We would also like to thank
tations, depth- and breadth-first search, shortest-path algorithms,                           an anonymous reviewer from a different paper that one author pub-
and minimum spanning tree algorithms, the most common topics                                  lished at CCSC-Southwest 2022, which helped to highlight the need
from the DSA section, have Cramér’s V measures less than 0.13 with                            for this work. Varun Nagaraj Rao was supported in part by NSF
the other topics (and associations between 0.13 and 0.78 amongst                              awards #1916153, #1956435, and #1943584. Matthew Ferland was
5 This includes the institution’s Basic Carnegie Classification, its research activity, and   supported in part by Simons Investigator Award for fundamental
whether or not the institution was public/private and religious/nonsectarian.                 and curiosity-driven research and NSF grant CCF-1815254. We also
6 We note that these associations values do not indicate the kinds of relationships
                                                                                              thank UCI’s Academic Senate Council on Research, Computing and
the variables have and whether variables are positively or negatively correlated with
one another in one manner. We provide interpretations of these associations with the          Libraries (CORCL) for the research fund that enabled us to provide
intention of yielding additional potential avenues of consideration and future study.         gift cards as incentives for participants.
Survey Results of Introductory Undergraduate Algorithms Courses in the U.S.                                             SIGCSE 2023, March 15–18, 2023, Toronto, ON, Canada

REFERENCES                                                                                      Computing Machinery, New York, NY, USA, 111–119.
 [1] ACM. 2005. Computing Curricula 2005.                 3291279.3339404
     assets/education/curricula-recommendations/cc2005-march06final.pdf                  [11]   Michael Shindler, Matt Ferland, Aaron Cote, and Olivera Grujic. 2020. Experience
 [2] ACM. 2020. Computing Curricula 2020.                 report: preemptive final exams for computer science theory classes. Journal of
     assets/education/curricula-recommendations/cc2020.pdf                                      Computing Sciences in Colleges 35, 10 (2020), 9–14.
 [3] Philip E Agre. 1999. Information technology in higher education: The “global        [12]   Michael Shindler, Natalia Pinpin, Mia Markovic, Frederick Reiber, Jee Hoon
     academic village” and intellectual standardization. On the Horizon 7, 5 (1999),            Kim, Giles Pierre Nunez Carlos, Mine Dogucu, Mark Hong, Michael Luu, Brian
     8–11.                                                                                      Anderson, Aaron Cote, Matthew Ferland, Palak Jain, Tyler LaBonte, Leena Mathur,
 [4] Mohammed F Farghally, Kyu Han Koh, Jeremy V Ernst, and Clifford A Shaffer.                 Ryan Moreno, and Ryan Sakuma. 2022. Student misconceptions of dynamic
     2017. Towards a concept inventory for algorithm analysis topics. In Proceedings            programming: a replication study. Computer Science Education 32, 3 (2022),
     of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education.                  288–312.
     ACM, Seattle, WA, USA, 207–212.                                                     [13]   Lee S Shulman. 2001. The Carnegie classification of institutions of higher educa-
 [5] Mohammed Fawzi Seddik Farghally. 2016. Visualizing algorithm analysis topics.              tion.
     Ph. D. Dissertation. Virginia Tech.                                                 [14]   Cynthia Taylor, Michael Clancy, Kevin C Webb, Daniel Zingaro, Cynthia Lee,
 [6] Kathi Fisler, Sorelle Friedler, Kevin Lin, and Suresh Venkatasubramanian. 2022.            and Leo Porter. 2020. The practical details of building a CS concept inventory. In
     Approaches for Weaving Responsible Computing into Data Structures and Algo-                Proceedings of the 51st ACM Technical Symposium on Computer Science Education.
     rithms Courses. In Proceedings of the 53rd ACM Technical Symposium on Computer             ACM, Portland, OR, USA, 372–378.
     Science Education V. 2. ACM, Providence, RI, USA, 1049–1050.                        [15]   Pinar Mihci Türker and Ferhat Kadir Pala. 2020. The effect of algorithm education
 [7] Matthew Hertz. 2010. What do "CS1" and "CS2" mean? Investigating differences               on students’ computer programming self-efficacy perceptions and computational
     in the early courses. In Proceedings of the 41st ACM technical symposium on                thinking skills. International Journal of Computer Science Education in Schools 3,
     Computer science education. ACM, Milwaukee, WI, USA, 199–203.                              3 (2020), 19–32.
 [8] The Joint Task Force on Computing Curricula Association for Computing Ma-           [16]   J Ángel Velázquez-Iturbide. 2019. Students’ Misconceptions of Optimization Algo-
     chinery (ACM) IEEE Computer Society. 2013. Computer Science Curricula 2013.                rithms. In Proceedings of the 2019 ACM Conference on Innovation and Technology                                                 in Computer Science Education. ACM, Aberdeen, UK, 464–470.
 [9] Nesrin Özdener. 2008. A comparison of the misconceptions about the time-            [17]   Paul Yakoboski. 2018. Adjunct faculty: Who they are and what is their experience.
     efficiency of algorithms by various profiles of computer-programming students.      [18]   Jeffrey Young and Eric Walkingshaw. 2018. A domain analysis of data structure
     Computers & Education 51, 3 (2008), 1094–1102.                                             and algorithm explanations in the wild. In Proceedings of the 49th ACM Technical
[10] Leo Porter, Daniel Zingaro, Soohyun Nam Liao, Cynthia Taylor, Kevin C. Webb,               Symposium on Computer Science Education. ACM, Baltimore, MD, USA, 870–875.
     Cynthia Lee, and Michael Clancy. 2019. BDSI: A Validated Concept Inventory for      [19]   Shamama Zehra, Aishwarya Ramanathan, Larry Yueli Zhang, and Daniel Zingaro.
     Basic Data Structures. In Proceedings of the 2019 ACM Conference on International          2018. Student misconceptions of dynamic programming. In Proceedings of the
     Computing Education Research (Toronto ON, Canada) (ICER ’19). Association for              49th ACM technical symposium on Computer Science Education. ACM, Baltimore,
                                                                                                MD, USA, 556–561.