The AI Compilatio content detector allows to make the difference between human texts and texts generated by Artificial Intelligences, thus verifying the authenticity of written documents.
How do you measure the reliability of the Compilatio system for detecting texts written by AI?
The effectiveness of AI content detection relies on both:
- Detection engine capacity to label "AI" or "human" each of the extracts from consistent text presented to it (texte homogeneous = text entirely generated by AI or text entirely written by human).
- Compilatio analysis capability to identify in a heterogeneous document passages attributable to a human or AI author (heterogeneous text = text containing a mixture of text written by a human and text generated by AI).
Note : Performance measurements in this article are valid for the Compilatio AI text detection system version 4.2.1, in use since October 15, 2024.
The reliability of the AI text detection engine
The role of the detection engine: labelling short text extracts as "AI" or "human
The Compilatio detection engine uses a language model (an artificial intelligence specialized in language processing) specifically trained to determine whether a text is similar to an AI or human production.
This "detection engine" receives texts from unknown sources and determines, according to the writing style which are similar to texts written by humans and those written by artificial intelligence.
Detection engine reliability measures
To get a comprehensive look at reliability, we need to measure several indicators: precision, recall and accuracy. For a better understanding of how these indicators are calculated, consult the following articles: "Precision and recall" and "Accuracy and precision".
-
Precision of Compilatio detection engine: 98,5%
Precision = the engine's ability to make no mistakes.
This means that out of 20 items identified by the engine as "text written by an AI", 19 are indeed generated by an AI and 1 is actually written by a human. -
Recall of the Compilatio detection engine: 92.75%
Recall = the engine's ability to remember.
This means that out of 20 AI passages to be identified, 19 are correctly found and 1 is not. -
Accuracy of the Compilatio detection engine: 95,75%
Accuracy = the ability to label texts correctly (human or AI).
This means that out of 20 passages to be labeled (human or AI), 19 were correctly labeled.
These measurements were carried out on 2,000 texts of a few sentences each in French, Italian, Spanish and English. The sample consisted of 1,000 texts written by humans and 1,000 texts written by Artificial Intelligence.
The reliability of Compilatio analysis
Compilatio's task when analyzing a document is not simply to judge whether texts are 100% AI or 100% human (as the AI detection engine does).
The role of analysis is to identify and quantify passages likely to be written by an AI or a human in a text containing a mixture from both sources (heterogeneous document).
|
Compilatio's final performance in detecting texts written by AIs is 9/10. |
This means that in a document containing 10 passages to be labeled (human or AI), 9 are correctly labeled.
The illustration below is representative of the level of efficiency measured, for the system currently offered by Compilatio:
Precautions to be taken regarding efficiency measures
The statistics provided describe the overall performance service on a large number of documents representing student work.
In practice, the sources (AI or human) of some passages/documents may be perfectly identified, and others less so. Remember that AI detection relies on the recognition of stylistic characteristics typical of texts written by an AI; it may happen that a human has a style similar to that of an artificial intelligence.
No AI detector can be 100% reliable.
The reliability rates communicated by AI detectors are not comparable! It is difficult to objectively compare the reliability of AI detection tools if test environments differ.
Measurements can change depending on a number of factors: corpus source and language, number of documents tested, generative AI model used, origin of human documents, company commitment not to manipulate results, etc.
It is important to remember that Compilatio tools provide indications on suspicious passages. It is always up to the examiner to interpret this information to validate or impute potential fraud. If in doubt, carry out a closer examination of the student's knowledge of the suspect passages.
To find out whether our solutions are adapted to your needs, please contact our advisors.
Is the Compilatio AI Detector multilingual?
Yes, the Compilatio AI Detector is multilingual. It can identify AI content in several languages: French, English, Spanish, Italian, Portuguese and other languages from all over the world!
Reliability measurements have been carried out on texts in several languages.
Will the Compilatio AI detector keep pace with ongoing AI advances?
Read the answer here: https://support.compilatio.net/hc/en-us/articles/17435773405329-Will-the-Compilatio-AI-detector-keep-pace-with-ongoing-AI-advances-
|
* Reminder: |
This article has been automatically translated. Do you have any suggestions for improvement? Contact us.