How to Tell Whether a Student Is Using ChatGPT
For the uninitiated, ChatGPT is a chatbot built on top of the GPT3 large language model from OpenAI. Since it came out in late November it has rapidly gained millions of users, hitting the 1 million user mark in 5 days, arguably the fastest product to ever do so.
It can answer questions and generate essays that are remarkably well written. It’s flexible and can also follow directions like “write with the quality of a 10th grader” or “include 3 grammatical mistakes.” And it is now being used frequently by students to cheat on essays. The proof of this is all over twitter, but nothing solidified the risk to me quite like the comments section in this tiktok.
If you are not yet sold, you just need to see it to believe it.
Now if you are a teacher that is already worried about suspicious essays, then you are in luck because a few good Samaritan data scientists have posted open tools on Hugging Face that can be useful at detecting whether text came from ChatGPT.
Take this example. I asked ChatGPT to “write a 500 word essay on the Napoleonic revolution”. [Note that ChatGPT often gives different answers each time you prompt it with the same question. So it is not enough to just input the question and check if students wrote the same answer.]
I then inputted the first sentence of the essay and the tool positively identified the text as coming from ChatGPT and gave a confidence score. The accuracy of the model increases with more text, so, despite the example, this is best used on larger essays when a situation might already be suspicious.
Now these tools are not perfect and there will be ways to get around it. So these need to be used alongside other tools and systems for preventing further plagiarism.
I’d also bet that existing paid plagiarism checkers will start to include tools for detecting AI usage. But given that plagiarism checkers were already imperfect before this new problem, I don’t suspect this will ever be fully solved. So, frankly I don’t know what to say about the long-term implications of ChatGPT (and it’s equivalents like Chinchilla) for Education. I think the tweet below does a good job of laying out some likely macro predictions for the next year, but it doesn’t touch on the Educational impact.
There’s of course a rosy way of seeing this as just becoming another tool for learning and productivity like Google or Wikipedia. A strong proponent of this view is Noah Smith who I think lays out a compelling future for how this could look.
From this outlook, it will just become standard for students to use tools liek ChatGPT for all assignments, and so they end up getting graded on how effectively they can leverage the tool rather than problem solve by themselves. In a weird way, this is sort of already happening in that homework now is basically just assessing how good students are at using Google to find the answer.
My best way to think about this is to ask myself what I would do if I was still teaching. That answer is easy, I’d stop grading homework, get rid of all out-of-class graded essays, and go back to purely grading based on written, in-class essays and assignments. I really do think that there is value in developing cognitive skills for solving problems without google, stackoverflow, or chatgpt.
So my question to all of you teachers is…what will you do?
