What Is a Profanity Filter? Word Lists vs AI Moderation
A list of bad words gets you started and then blocks a town in England. Here is how profanity filters work, where word lists break, and what AI moderation adds.
Word-list filter vs AI moderation
|
Word-list filter |
AI moderation | |
|---|---|---|
|
How it decides |
Matches text against a list of words or patterns |
A model scores the whole message for harm |
|
Catches |
Known words and simple misspellings |
Insults, threats and harassment with no swear words |
|
False positives |
Common without word boundaries (Scunthorpe) |
Fewer, but tuned by a confidence threshold |
|
Speed |
Microseconds, runs anywhere |
Adds a model call before delivery |
|
Languages |
One list per language you maintain |
Depends on the model; check coverage |
|
Explainability |
Easy: this word matched |
Harder: a score and a category |
|
Best for |
Masking swear words, banned terms, usernames |
Toxicity, hate and harassment in open chat |
What is a profanity filter?
A profanity filter checks text people write (chat messages, comments, usernames) for offensive words and acts before anyone else sees it. It usually does one of three things:
Mask: replace the word with asterisks and deliver the rest of the message.
Block: stop the message and tell the sender.
Flag: deliver or hold the message and send it to a human moderator.
Profanity is the narrowest slice of content moderation. A message can be hostile without a single swear word, and a friendly message can be full of them. That gap is why most chat apps now pair a word list with AI moderation.
How profanity filters work
1. Normalize
People dodge filters, so clean the text first: lowercase it, strip accents, map leetspeak (3 to e, @ to a), collapse repeated letters and map look-alike characters from other scripts, such as Cyrillic letters that look Latin.
2. Match
Compare whole words against a block list, with an allowlist for exceptions. Regular expressions catch variants, such as a word with optional letters.
3. Score (optional)
An AI model rates the message for toxicity, harassment or hate, and returns a category and a confidence score. You pick the threshold.
4. Act
Mask, block or flag based on what matched and how confident the model is. Log the decision so moderators can review it and users can appeal.
The Scunthorpe problem
In 1996, AOL's filter stopped people in Scunthorpe, England, from creating accounts, because the town's name contains a slur. Filters that replace "ass" have turned "classic" into "clbuttic". These false positives are called the Scunthorpe problem.
The cause is matching inside words. Fixes:
Match whole words after splitting on spaces and punctuation.
Keep an allowlist for place names, surnames and technical terms.
Mask the matched word only, never the surrounding text.
Let an AI model handle the cases where context decides.
There's a trade-off. Whole-word matching misses "h e c k" and words glued together. Filters that try to catch everything end up blocking everyone.
Rule of thumb: use a word list for words you never want shown, and an AI model for behavior you never want tolerated. Neither does the other's job well.
Example: a whole-word filter with masking
const BLOCKED = new Set(['darn', 'heck']); // placeholders: load your real list
const LEET = { 0: 'o', 1: 'i', 3: 'e', 4: 'a', 5: 's', 7: 't', '@': 'a', $: 's' };
function variants(token) {
const base = token
.normalize('NFKD').replace(/[\u0300-\u036f]/g, '') // strip accents
.toLowerCase()
.replace(/[0-9@$]/g, (c) => LEET[c] ?? c);
return [base, base.replace(/(.)\1+/g, '$1')]; // "heeeck" -> "heck"
}
function mask(text) {
// Check whole tokens only, so "Scunthorpe" is never searched inside.
return text.replace(/[\p{L}\p{N}@$]+/gu, (token) =>
variants(token).some((v) => BLOCKED.has(v)) ? '*'.repeat(token.length) : token
);
}
mask('What the h3ck, Scunthorpe?'); // "What the ****, Scunthorpe?"
Where profanity filters are used
Kids, education and family apps: strict lists and blocking, since a single slip is a support ticket.
Gaming and live event chat: fast masking for thousands of messages a minute, with AI moderation for harassment.
Marketplaces and dating: profanity rules alongside filters for phone numbers and emails that move users off the platform.
Customer support: protect agents from abusive messages and flag the conversation.
Usernames and profiles: check display names once, at signup, where false positives are easy to fix.
Calls and live captions: run the same filter on transcripts and captions from voice and video calls.
Limits and pitfalls
Evasion never stops. Spaces, emoji, images of text and new slang get past lists. Update them and watch what gets flagged.
Context changes meaning. "This game is f***ing great" isn't an attack. Swear-word counts are a poor proxy for harm.
Language and culture. A word that is offensive in one language or region is normal in another. Reclaimed terms make this harder.
Latency. AI checks before delivery add time. Holding a message as pending until it's approved keeps harmful content off screen, at the cost of a short delay.
Over-masking annoys people. Tell senders what was filtered, and give moderators a way to approve false positives.
Profanity filtering and moderation in CometChat
If you're building chat into your app, CometChat moderation checks messages as they're sent, before other users see them.
Rules for profanity, toxicity, contact details and media
CometChat's AI Moderation runs rules you set in the dashboard or through the Moderation API. A default profanity rule blocks offensive language using a predefined keyword list, and you can add your own keyword and regex lists. AI rules cover toxicity, hate and harassment, spam, scams and more, plus filters for phone numbers and email addresses and checks on images and video. Each rule blocks a message or flags it for review. In the SDK, a message is
PENDINGuntil moderation marks itAPPROVEDorDISAPPROVEDthroughonMessageModerated. We covered the older Profanity Filter and Data Masking extensions, which mask words as asterisks, in this product update. If you still use them, disable them before creating moderation rules so messages aren't processed twice.
Frequently asked questions
How does a profanity filter work?
It normalizes the text, compares words against a block list, and then masks, blocks or flags the message. Many filters also send the message to an AI model that scores it for toxicity or harassment.
What is the Scunthorpe problem?
It's when a filter blocks innocent text because an offensive string appears inside a longer word, like the English town of Scunthorpe. Matching whole words and keeping an allowlist prevents most cases.
Is AI better than a word list for filtering profanity?
For harassment and hate, yes, because it reads context. For masking specific words, a list is faster and easier to explain. Most chat apps use both.
Should a profanity filter mask or block messages?
Mask when the rest of the message is fine and the audience just shouldn't see the word. Block or flag when the message itself is abusive.
Can people get around profanity filters?
Yes, with spacing, symbols, look-alike characters and new slang. Normalizing text, updating lists and adding AI moderation and user reporting keeps up with most of it.