How would you feel if someone could check what you had written at any point during your completion of an assessment?
What if they could also see the amount of words you’ve typed, pasted and removed, how long you spent working on it, when you started and finished, the percentage that had been pasted, the browser you used, the country you worked in and even more?
This is the kind of data that the platform Cadmus collects. The University of Sydney has been “piloting” Cadmus in the Business School since 2021.
Cadmus is an online assessment platform resembling a word processor — complete with a text editor, notes section, and resources. The benefits for students include the availability of preset text styles, auto-saving, automatic referencing styles, and pop-ups that advise students about rephrasing when they paste content (all but the final of these benefits are available to some extent in Microsoft Word).
In exchange for these benefits, students are subjected to constant monitoring for the duration of their assignment.
***
A USyd spokesperson said that, “Cadmus has been approved for piloting in assessment and teaching by the ‘eTools’ sub-committee of the University Executive Education Committee, following rigorous checks including privacy, cybersecurity and accessibility.”
The program was first approved by the eTools sub-committee, part of the University Executive Education Committee, in April of 2021. By July, it received endorsement and approval from the Deputy Vice-Chancellor (Education) and the Office of General Council. By the next semester, the Business School had started their trial of Cadmus.
The eTools sub-committee was scheduled to close off their consideration of the platform in November of 2022. It got postponed to their December meeting. Except, their December meeting didn’t end up happening.
Eight months later, after Honi asked the University about their use of Cadmus, the sub-committee is now scheduled to discuss their wrap up of the piloting process at their next meeting later this month. Cadmus is being used in the Business School, primarily for early core units and English language support units, as part of a semester-by-semester agreement. Since March of 2023, USyd has been listed fourth in the “Trusted globally by leaders in Higher Education” section on the Cadmus homepage.
***
The University previously trialled Cadmus in 2016, before deciding against its use in 2017. At the time, students raised concerns about “whether universities will be notified of how long students spend writing their assessments, and how close to submission deadlines they begin writing.” This information is available in a Cadmus Assessment Report.
In September of 2021, Cadmus tweeted that USyd was using the platform to “deliver supportive and meaningful assessments (in-semester and exams) in @sydney_business and @ArtSS_Sydney.” There appears to be no public mention from the University about their use of the program.
The next public mention is instead found in the President’s Report in the Week 12, Semester 2 2021 edition of Honi. When asked at the time, a USyd spokesperson described it as a “very different version” of Cadmus, saying that they were “undertaking a trial of a product Cadmus has developed to provide a scaffolded approach to assignment writing.”
This focus appears to have shifted over time as a USyd spokesperson told Honi in August of 2023 that, “The Business School is currently trialling Cadmus, seeking to improve learning outcomes for students through ‘authentic assessments’ that more directly relate to their future careers.”
Both the University and Cadmus connect use of the platform to the idea of “authentic assessment”. Authentic assessments are considered to be tasks that are related to scenarios that students are likely to encounter when working in their fields, such as accounting students using systems typically used in businesses or medical students working with actors in role-play scenarios. It is unclear why using Cadmus is necessary for such assessments. When the platform itself is aimed at universities not businesses, it is unclear how students are gaining real-world skills from its use. Even if it was necessary, whilst programs that allow businesses to monitor their workers digitally continue to be developed, subjecting students to such extensive data collection during their studies is worryingly dystopic. This also bears a risk of function creep, where this data may be used for wider purposes than it was collected for, with the potential to lead to increased surveillance and monitoring.
***
Cadmus produces three kinds of views of the data that is collected — Insights, Assessment Reports, and Activity Reports. Given the vagueness of their Privacy Policy, the guides for teachers about using each of these views provide the clearest explanation of what data is collected about students.
Insights provides a de-identified view of the progression of students in the task. This includes how many students have started and the spread of their progression with the task. After submission, a summary report is available including the average time spent and collated feedback from students. Whilst this information may be useful to teachers aiming to improve the quality of their assessments, this could easily be collected by directly surveying or consulting students, without the need for an additional platform.
Assessment Reports provide more specificity, through a subject-wide snapshot of data about the completion of the assignment, separated by student. A sample Cadmus Assessment Report shows the email, time spent, number of sessions, grade, word count, the total words added/pasted/removed, the number of pastes, percentage pasted, the most common paste source, the times of first access and save, and final save and submission, the most common country it was completed it, countries outside of Australia and the number of resources accessed for each student completing the task.
Activity Reports are a focused view of the data collected about an individual student. The report includes sections for the data about words, resources, sessions and pastes. The pastes section provides a collection of all the content pasted into the document, including when it was pasted, and the source of the text — in the video guides, this appears to be limited to external, Microsoft Word or Google Docs.
***
At the bottom of their guide explaining the above insights, it says that, “Cadmus analytics are designed to help you understand student behaviour and to inform Academic Integrity investigations. Cadmus does not flag plagiarism or indicate whether any form of academic misconduct has taken place.”
This is a distinction that Cadmus seems eager to make.
The version of Cadmus used circa 2016-2017 seemed to focus on keystroke analysis to uncover cases of contract cheating (a distinguishing point from other edtech programs like Turnitin, who were better suited for cases of plagiarism). Whilst Cadmus has since removed their keystroke logging, their platform still collects an extensive amount of data about student submissions, with more available upon request.
This includes PDF copies of the submission from requested times during its completion. Rather than tracking the actual keys typed, Cadmus seems to be able to instead produce the resulting content from the saved versions (likely produced from the continuous autosave — one of the features promoted to students as a benefit). When Cadmus were asked how this was possible, Honi was told a particularly vague answer: “If an academic integrity investigation is being conducted, pdf versions are available on request to support the investigation team’s enquiries.”
Jess Ashman, Head of Learning at Cadmus, told Honi that “Originally, Cadmus was developed as an academic integrity product, designed to detect contract cheating. Following extensive consultation with students, academics and universities, Cadmus evolved into a product that optimises student learning and academic integrity, rather than catching cheating.”
“As a result of this evolution, Cadmus now supports students to understand the purpose of an assessment task, scaffolds their learning throughout the assessment, and uses analytics to proactively nudge students down the path of integrity, ultimately decreasing the likelihood of academic misconduct and accidental plagiarism.”
Despite this apparent shift, much of the data collected appears to still be connected to academic honesty. This also appears to be reflected in the public coverage of Cadmus. The Australian Trade and Investment Commission’s EdTech Directory describes how Cadmus works with universities to solve their biggest problems including “academic integrity, student retention, remote learning and online exams.” Startupdaily described Cadmus as using “sophisticated learner analytics that detect the authenticity of a student’s work, including the use of ChatGPT and other AI platforms.” Two showcases in the Business School’s 2022 Learning and Teaching Forum, and a blog post from the Disruptive Innovations in Business Education Research Group appear to have discussed the usage of Cadmus for academic honesty purposes. This suggests that whether they intend it or not, Cadmus is still considered to be — or at least, can be used as — an academic integrity tool.
***
The argument that underpins Cadmus’ transformation is that they do not make the conclusions about academic integrity, instead it is the University using the platform that does — by using their data.
Despite this claim, in the “Getting started for teachers” video series and other guide documentation, teachers are advised of various phenomena that may indicate academic integrity issues. These include a lack of editing activity, not using resources, or a change in browser versions. At one point, teachers are told that “when a student works authentically, we should see an almost equal ratio of words added to words removed” without any evidence for this claim. When Honi asked Cadmus about these statements, rather than being conclusions Ashman said that, “These are suggestions in order to assist the teacher to consider all possible options.”
Many of these supposed indicators are easily explainable by very mundane, yet potentially alternative student behaviours when completing an assessment. A concern about minimal editing could indicate transcription of a purchased response or the use of generative AI, or it could indicate having handwritten a draft, running out of time and only having time for one draft, focussing on writing a polished version to avoid needing to edit, working in a Word document whilst drafting, or a variety of other things. These are all behaviours that are not innately acts of academic misconduct.
Even more so, students may find it necessary to work in a Word document (or on another word processor) in order to deal with the limitations of the platform. A student using Cadmus as part of the unit “OLES2210: Succeeding in a Post-Crisis World” told Honi that they used a Word document to overcome the inability to work offline. When they pasted their work into Cadmus, they were met with a pop-up advising them to rephrase the newly added text — work that they had just written themselves. Cadmus only works offline if it has been first opened whilst connected to the internet. This then relies on students not closing the window, restarting their computer, accidentally clicking back or a variety of other technological mishaps that would lead to a loss of work before it could be saved by connecting to the internet again.
The idea that the browser version you use to complete an assignment could be used as an indicator of academic misconduct is also questionable. Whilst it could indicate multiple users working on an assignment, it equally can suggest that a student is moving between devices to work on their task. The laptop and desktop I’ve used for working on this article are almost certainly using different browser versions, and yet, I have written this all by myself.
The ways that these trends transform the varied practices of students completing their assignment into data that could be used in an academic honesty investigation is problematic. The ongoing management and retention of this data poses further concerns as it is unclear how it will be used or stored in future. When asked about this, Ashman said that, “Cadmus does not retain student data. Cadmus is a processor of student data on behalf of the university. It is the university who decides on the retention policy which is often in line with state legislation.”
***
After an approximately two year trial period, Honi will be waiting to see the conclusions of the eTools committee when they meet later this month. Between collecting vast amounts of data and the ability to view the contents of the document, using Cadmus requires students to agree to their work being surveilled. Amidst the perennial concerns of privacy and data security, students shouldn’t have to use platforms that track even more of their lives.
If you have used Cadmus and would like to share your experience with Honi, please use our anonymous tip form or email [email protected]