90%

AI is writing 90% of the code I was in charge of

AI-text detection tools are really easy to fool

A recent crop of AI systems claiming to detect AI-generated text perform poorly—and it doesn’t take much to get past them.

Artificial Writing and Automated Detection

Artificial intelligence (AI) tools are increasingly used for written deliverables. This has created demand for distinguishing human-generated text from AI-generated text at scale, e.g., ensuring assignments were completed by students, product reviews written by actual customers, etc. A decision-maker aiming to implement a detector in practice must consider two key statistics: the False Negative Rate (FNR), which corresponds to the proportion of AI-generated text that is falsely classified as human, and the False Positive Rate (FPR), which corresponds to the proportion of human-written text that is falsely classified as AI-generated. We evaluate three leading commercial detectors—Pangram, OriginalityAI, GPTZero—and an open-source one —RoBERTa—on their performance in minimizing these statistics using a large corpus spanning genres, lengths, and models. Commercial detectors outperform open-source, with Pangram achieving near-zero FNR and FPR rates that remain robust across models, threshold rules, ultra-short passages, "stubs" (≤ 50 words) and ’humanizer’ tools. A decision-maker may weight one type of error (Type I vs. Type II) as more important than the other. To account for such a preference, we introduce a framework where the decision-maker sets a policy cap—a detector-independent metric reflecting tolerance for false positives or negatives. We show that Pangram is the only tool to satisfy a strict cap (FPR ≤ 0.005) without sacrificing accuracy. This framework is especially relevant given the uncertainty surrounding how AI may be used at different stages of writing, where certain uses may be encouraged (e.g., grammar correction) but may be difficult to separate from other uses.

I pulled ~90,000 Reddit posts about what makes writing "sound like AI" to determine the biggest AI-slop giveaways (Part 2)

674 votes, 228 comments. The majority of people can instantly tell when writing is generated by AI. For those who don't intend to get into the weeds…

How easy is it to spot AI writing?

With AI writing becoming increasingly commonplace, how can we spot it and what difference does it make?

God Damn AI is making me dumb

It's so god damn tempting to use AI to write. Whether it is articles, code, or documents. I feel like using AI is diminishing my ability to write myself. ...

Your AI Use Is Breaking My Brain

AI writing is impossible to avoid, is making everything sound the same, and is driving us crazy.

What I learned running an adversarial test on an AI text detector

If you get your bot to rhyme / Pangram will misclassify it many a time

Please Use AI

Be sure to use AI when making

How to write well with AI

Why people who pledge never to write with AI are telling on themselves

AI-Detectors Biased Against Non-Native English Writers | Stanford HAI

Don’t put faith in detectors that are “unreliable and easily gamed,” says scholar.

Scary New Report: Just 8 Percent of People Check AI Answers

It's time to talk with your people about best practices for using artificial intelligence.

Writing With AI Is Harder Than You Think

It takes rigor, judgment, and willingness to be told your work isn't good enough.