“Honey, I tringed your assignments”

“You were getting so stressed about them and we never seem now to have any fun time. So I got our AI to sort them out.“

There is a great deal of excitable discussion into what percentage of students (both young and old) are using ChatGPT and similar LLM tools – and how much they are using them – to “sort out” their assignments. The general consensus among knowledgeable people is that a lot of this is happening, but with the technology changing so fast it is impossible for peer-reviewed articles in learned journals to keep up with the topic as it develops. A brave attempt was made by Chuhan Xu in 2024 [1]. You can read a distillation of the most relevant parts to this post in [2].

So where do academic faculty look to find expert guidance? Universities, with some help from sector-level agencies, national bodies and international organisations (such as OECD this year [9]), are updating their guidelines on LLMs and academic integrity, but at a glacial and occasional pace (little sign of even annual updates) and with a general reassuring tone to outsiders (funders, students, parents etc) of “nothing to worry about here”, while being privately concerned about how effective such guidelines are when “everybody is doing it”.

And even more worryingly, does guidance on academic integrity just foster students to work harder to circumvent any guidelines?

There are many postings on LinkedIn and other social media channels for professionals but a great lack of practical information focussed on specific topics – such as actual assignments in bachelor or master programmes.

I hope to change that FAST. We have an assessment emergency all around us.

As a ground-clearning exercise I got Manus to produce a report Misuse of LLMs in University Assignments: A Literature Review [3] based totally on peer-reviewed papers published in 2024 or this year (anything earlier is pointless now).

Most people are trying to wish the emergency away with platitudes on “ethics” or simplistic vague suggestions, mostly untested: “authentic assessment” (doesn’t help – check it out), “lots of vivas” (accessibility issues and staff time/cost issues), “process-based assessment” (easy to game), “move all students back to the classroom” (yeah, right).

To me, there seems to be a complete lack of fundamental concepts underpinning the analyses. My doctorate was in mathematical logic so I am prone to such theorising. More recently, much work of mine has been on “time” aspects of online learning – and “time” seems like a great basis from which to start looking at the concepts. It also fits well with “study hours” (ECTS, CATS, credit hours, guided learning hours) and other traditional metrics for the amount of stuyding (including working on assignments) that students are working on learning tasks.

(Note to logicians: you will look in vain here for any conceptual advances in temporal logic, but you might be intrigued by “time of the third kind” and its relevance to microlearning and micro-credentials.)

I propose that we coin the neologism verb “tring” to mean:

create an artefact in such a way that it cannot be feasibly distinguished from a human-created artefact, using help from AI systems so as to substantially reduce the effort that the creator has to put in compared with others not using AI.

(As should be obvious, the derivation is from Turing, as per the famous Turing Test to distinguish a human from a robot.)

There is the weasel word “substantial” in there. Based on my own experience as a researcher and from observation of student behaviour I posit “substantial” to mean “saving at least 3/4 of the effort”, or in other words “four times as fast” – “tring factor 4”.

An assignment is “tringable” if AI-expert users can produce it with a small fraction of the effort that less privileged users have to use and it cannot be feasibly distinguished from human-created submissions.

There is an issue over what “feasible” means – this has both financial and ethical aspects:

• Additional effort on checking assigments cannot go beyond a certain level (quite low in these days of stressed university budgets in many advanced economies).
• Automated systems cannot produce so many mis-diagnoses as to upset or put off the majority of students – often paying customers in many jurisdictions. The challenge is still substantial [4].
• Assignment specifications and submission processes cannot be so distorted due to the needs of anti-AI checking that they lose sight of the learning outcomes of the module and the wider set of skills and attitudes the university is trying to inculcate.
• Assignment specifications, disaggregation (e.g. bibliography, first draft, merged submission) and submission processes cannot disadvantage whole groups of the student population (such as students working in other than their first language, or deaf students, or students who suffer stress with viva examinations).

An assignment is “narrowly tringable” if an AI-assisted submission is not detected when anonymous submission takes place but when the student author is known to the system the submission appears to be substantially better than or just different from the student’s usual submissions (cue for learner analytics here).

An assignment specification is “broadly tringable” if it is not feasible to reliably differentiate submissions from a student’s own work even with knowledge of students and their prior work.

One hypothesis is that every narrowly tringable assignment is broadly tringable – that is what the students I overheard on the train believed.

The “tring factor” (analogous with warp factor) is the reciprocal of the fraction of effort the student needs to put in using LLMs compared with the expected amount of work. Thus if a tringed assignment takes 1/4 of the usual (non-AI-assisted) time, the tring factor is four; for a tenth of the time the tring factor is ten.

A “strongly tringable” assigment is one where no more than 10% of the expected work has to be undertaken by the human, i.e. a tring factor of 10. At that high a factor, all bets are off and assessment becomes severely distorted in favour of the AI-capable students.

It turns out the the tringability of an assignment depends crucially on the process of submission and the required format of the submitted artefact. For example a requirement to submit a styled Word document with citations and bibliography (as should be standard at university level, one assumes) using Microsoft Reference Manager will have a lower tring factor than one where citations use Mendeley or EndNote, or one submitted using Open Office. Indeed, the Word-manipulating libraries that LLMs commonly use cannot even produce actual footnotes, having to simulate them using superscripts [5] – an easy give-away unless assignments can be submitted via PDF or printed on paper. More surprisingly Google Docs, often felt to be more modern than Word, is no better at this than Word. [6] In contrast Open Office is much “better” for tringing since it has much stronger programmatic support. [6]

Thus an immediate lesson is that vagueness in assignment format, naively felt useful not to disadvantage poorer students, in fact opens the door to tringing. It is also sad philosophically that the open software seems to be more susceptible to facilitating tringing.

In typical university fashion (alleviate the symptoms, don’t cure the disease) attention is immediately drawn (in IT departments) to detection of tringing. (Perish the thought that we should reconceptualise the assignment process.) Sadly, such techniques are of limited value and generate many false positives. [4] The wide range of LLMs now available to students as well as many rewriting tools and the ability of LLMs to match students’ own writing styles means that such approaches are of increasingly limited value. Hence, universities are now turning to asserting ethical standards (good luck with that). In addition, and more usefully in our view, there is a move towards reconstructing assignments to be more tring-proof but without major increases in assignment burden on instructors and students. We support that, but it is not easy, for some of the reasons we have outlined.

However, there is some scope for a different set of detection tools. When an LLM with the ability to manipulate Word documents (not all can) creates a Word document, it tends to leave traces buried in the XML metadata of the document that can be detected later. [5] Most LLMs that can manipulate Word do so via the python-docx library in the Python language and this leaves clear traces of its use not usually found in user-generated Word documents.

Despite that, there are sophisticated XML editors that can be used to remove such traces [7] – noting that these are hard to use by non-expert users. Sadly, there are also much simpler approaches that can produce similar results. For example manually opening the document in Word, making a few trivial changes and then saving again seems to remove much of the evidence. Similarly, conversions to RTF or Open Document Format are likely to achieve similar “useful” results.

The ever-helpful LLMs also can produce recommendations on how to produce large referenced documents in an interactive way. [8] Using these techniques it is likely that the typical UK MSc dissertation is now in seriously tringable range. From now on it will be useful to check them also for evidence of tring activity.

There is another, trickier aspect to tringability. This is when the AI tools allow a student to not just reduce the time but the level of the intellectual effort put into the assignment. For example, attempts to tring assignments often mean that one spends less time with manual searches and brain-straining summaries and much more time doing Word rewriting and systematic reformatting (useful skills but probably at a lower ISCED level).

As a note for a later topic, the naive idea that Search=Good and Summarise=Bad does not stand up well. Now that Google has AI Search as standard, how ethical is Search? In addition LLMs are very useful for producing refined search techniques e.g. using the Search Operators few faculty and students ever get their heads around. Speaking personally, my knowledge of global use of VLEs in universities took a big step up in functionality when I asked ChatGPT to advise how best to search for Canvas, Brightspace, Blackboard and Moodle sites in schools and universities. (Spoiler: searching for the words does not take you that far.)

This is a delicate issue but active in those countries where substantially greater numbers of students are now studying at university than was the case ten years ago.

We need a new verb for this. As a point to ponder and feedback I suggest “disk” – derived from de-ISCED – meaning to reduce the peak intellectual level needed to complete an assignment – e.g. from masters to bachelor level. Some might suggest “kirk” (after Kirkpatrick levels). But this discussion is for another time. Like Online Educa Berlin next week.

References

1. Xu, C. (2024). A Systematic Review of the Potential Influencing Factors for ChatGPT-Assisted Education. International Journal of Technology-Enhanced Education (IJTEE), 3(1), 1-19. https://doi.org/10.4018/IJTEE.339189
2. Analysis of Xu (2024) with particular reference to Assignments
3. Misuse of LLMs in University Assignments: A Literature Review
4. The State of the Art in Detecting LLM-Generated Text in Academic Assignments
5. Analysis of LLM Capabilities for Microsoft Word Document Manipulation
6. LLM Document Generation: How ODT and Google Docs Compare to Word
7. XML Edit tools for Word documents
8. Producing Large, Well-Referenced University-Style Word Documents with AI Support