Turnitin's AI Detection Capabilities
How has it developed since its launch in April 2023?
Turnitin released its AI writing detection capabilities in Turnitin Feedback Studio (TFS) in April 2023. CEI has been evaluating these new capabilities and monitoring their development since then. As a result of our recent evaluation, and as suggested by current literature (example: here and here), CEI considers Turnitin to be an unreliable tool for AI detection.
In our recent evaluation, we conducted a two-part assessment of Turnitin's AI detection capabilities. In the first part, we utilized two AI tools, QuillBot and ChatGPT 3.5 in POE, to paraphrase sample text taken from a book and a Wikipedia page. Our findings indicate that Turnitin can partially detect paraphrased text from ChatGPT 3.5 but fails to detect anything from QuillBot.
In the second part of the evaluation, we prompted the generative AI tools, ChatGPT 3.5 in POE and Bing Chat, to generate new text. Our findings indicate that Turnitin's AI detection capabilities can detect most of the generated text from both tools. We repeated this process with three additional new texts to verify consistency. The detection results consistently showed 100% detection for ChatGPT 3.5 generated text in all four cases. However, the detection results for Bing Chat generated text varied, with values such as 50%, 77%, and 100% detected.
It is important to note that the detection model only processes long-form writing. The model appears to automatically exclude the detection of text in point form. So, the results may be influenced by the length of individual paragraphs and the format of the text.
While there seems to be some improvement in Turnitin's detection capabilities, considering the unsatisfactory results in the first part of our evaluation and the mixed results in the second part with a GPT-4 based AI tool, CEI does not recommend relying on Turnitin's AI writing detection capabilities as a conclusive measure of misconduct or as a tool for punishment.
