Turnitin's AI Detection Capabilities
Can Turnitin detect AI-generative text?
On April 5, 2023, Turnitin released its AI writing detection capabilities in Turnitin Feedback Studio (TFS). As of now, HKUST has access to these capabilities as part of our current license agreement with them.
Turnitin claims to be 98% confident in its ability to detect AI-generated text based on data that was collected and verified in its AI innovation lab. However, CEI's recent testing has revealed that the detection results were not fool proof.
How does Turnitin detect AI-generated writing?
When a paper is submitted to Turnitin, the submission is first broken down into segments of five to ten sentences, which are then overlapped to ensure each sentence is viewed in the proper context.
The segments are then scored by the AI detection model, which assigns each sentence a score between 0 and 1 to determine whether it was written by a human or AI. A score of 0 means it was likely not generated by AI, while a score of 1 means the sentence was entirely generated by AI.
The average scores of all the segments are then used to generate an overall prediction of how much of the submission is believed to have been generated by AI.
What are the issues or limitations of Turnitin’s AI writing detection capabilities?
- The current model only detects content from the GPT-3 and GPT-3.5 language models.
- The current model only analyses standard grammatical sentences and does not include other types of writing structures, such as lists or bullet points.
- The detection rate of AI-generated writing may be influenced by the sequence of the paragraphs.
- The presence of matched text in the Similarity Report does not necessarily indicate that the text is of human origin, as it could be generated by AI.
Due to the issues or limitations that may lead to potentially unreliable results, CEI does not currently recommend using Turnitin's AI writing detection capabilities as a conclusive measure of misconduct or as a tool for punishment.
