Text Cleaner

What "clean text" actually means

Text that looks fine on screen is often dirty underneath. Copying a paragraph out of a PDF, a Word document, a Slack message or a webpage drags along baggage you can't see: non-breaking spaces (U+00A0) masquerading as ordinary spaces, curly "smart" quotes that break code and CSV files, trailing whitespace at the end of every line, and a scattering of zero-width characters that pad the string with invisible bytes. Cleaning normalizes all of that back to plain, predictable text — the kind that diffs cleanly, pastes into a terminal without surprises, and counts the way you expect.

The fixes fall into four buckets, and which ones matter depends entirely on where the text is headed. Whitespace and line endings matter for code and data. Smart-quote and dash normalization matters for anything that touches a compiler, a shell, or a regular expression. Invisible-character stripping matters for security review and for text you suspect was copied from a rich source. Knowing which problem you actually have saves you from over-cleaning.

The four problems, and when each one bites

Most pasted-from-rich-source text carries at least two of these at once.

Why zero-width characters are worth taking seriously

Zero-width and invisible characters render as nothing, but they are real codepoints that occupy bytes and change how a string compares. A zero-width space (U+200B) wedged into a coupon code, a username, or an API key makes two visually identical strings unequal — and the bug is nearly impossible to spot by eye. The Unicode Standard defines these as default-ignorable formatting characters; see the character database at unicode.org. A byte-order mark (U+FEFF) at the very start of a file is the classic offender: it's invisible, but it breaks JSON parsers, prepends junk to the first CSV header, and shows up as a stray glyph in some editors.

Stripping them is also a mild security hygiene step. Invisible characters have been used to hide watermarks in shared text and, in some attacks, to smuggle confusable content. If you've pasted text from an unknown source and something downstream is behaving oddly, running it through a clean pass that removes zero-width codepoints is a fast first diagnosis.

What cleaning is not — and how to avoid over-cleaning

• It is not encoding conversion. Cleaning normalizes characters; it does not transcode UTF-16 to UTF-8 or fix mojibake. Garbled-encoding damage happens before the text reaches you and a clean pass won't reverse it.
• Don't collapse whitespace blindly inside code. Leading indentation in Python or YAML is load-bearing — flatten it and you change the program. Clean line endings and trailing spaces there, but leave indentation alone.
• Smart-quote removal is one-directional and lossy by design: you're choosing straight quotes on purpose. If you're publishing prose for humans, you may actually want the curly quotes — keep them for the web, strip them for code and data.
• If you only need to change letter casing or count what's left, that's a different job — reach for Text Case or Word Count instead of cleaning.
• To confirm a clean pass changed exactly what you intended and nothing else, paste before-and-after into Text Diff and read the delta.