diff line endings crlf unicode

Why Every Line Shows as Changed: CRLF, Unicode and Other Invisible Diffs

When two files look identical but a diff says otherwise: line endings, byte order marks, non-breaking spaces, Unicode normalization and trailing newlines — how to spot each and how to produce a patch.

· Mohammed Aquib Ansari

The most frustrating diff I’ve debugged was a postgresql.conf that “hadn’t changed”, yet every single line showed as different. The two files looked identical in every editor. The cause was line endings — and once I started looking, I found a whole family of differences you can’t see.

This guide is about those, and about getting a usable patch out of a diff. I use the text diff tool for all of it; it runs in the browser, so config files and logs stay on my machine.

CRLF vs LF

Windows ends lines with two characters, carriage return + line feed (\r\n, CRLF). Linux and macOS use just the line feed (\n, LF). A file that went through a Windows editor, or through Git with core.autocrlf=true, can come back with every line ending changed.

Paste two such versions of the same config into the text diff and, instead of a wall of red, it says:

The texts differ only in line endings (left: CRLF, right: LF). Git's core.autocrlf or an editor setting usually causes this.

“Ignore line endings” is on by default so you see the real changes. Turn it off when the endings are the bug — a shell script with CRLF endings fails on Linux with /bin/bash^M: bad interpreter, and that ^M is the carriage return.

To fix it at the source, add a .gitattributes:

* text=auto eol=lf
*.cmd text eol=crlf

The invisible characters

When the line endings match and the diff still disagrees, it’s usually one of these:

  • Byte order mark (U+FEFF). Some Windows tools put three bytes at the start of UTF-8 files. Invisible in editors; breaks shebang lines, the first header of a CSV, and JSON parsers that don’t expect it.
  • Non-breaking space (U+00A0). Arrives when you copy from Word, Confluence, Slack or a web page. Looks like a space; isn’t one. YAML and many shells treat it as part of the value.
  • Zero-width space (U+200B). Copied from web pages and chat apps. Truly invisible, and it makes "apikey" and "api​key" different strings.
  • Tabs vs spaces. Matters for Makefiles and YAML.
  • A missing final newline. Git shows \ No newline at end of file; POSIX tools may ignore the last line.

The tool counts each of these on both sides and tells you, for example, that a non-breaking space appears once on the left and not at all on the right. “Show whitespace” draws spaces as ·, tabs as → and non-breaking spaces as ⍽, so you can see where they are.

Unicode normalization

This one surprised me most. The letter é can be stored two ways: as one code point (U+00E9) or as e followed by a combining acute accent (U+0301). They render identically. macOS file systems historically used the second form for file names; most other places use the first. A string comparison says they’re different, and so does a naive diff.

The tool detects when two texts differ only by normalization and suggests “Normalize Unicode (NFC)”, which converts both sides to the composed form before comparing. In code, str.normalize('NFC') in JavaScript or string.Normalize() in .NET does the same.

Line, word or character?

  • Lines for code and config. Changed lines are paired and the changed words inside them highlighted, so a one-character edit in a long line is obvious.
  • Words for prose — contracts, docs, release notes.
  • Characters for single values: tokens, hashes, IDs.

Getting a patch

In line mode, “Download .patch” saves a unified diff. For a small config change it looks like this:

--- a/db.conf
+++ b/db.conf
@@ -1,3 +1,4 @@
-host=db1
+host=db2
 port=5432
 user=app
+sslmode=require

git apply changes.patch or patch -p1 < changes.patch applies it — the a/ and b/ prefixes are why -p1 is needed with patch. The export is tested by applying it back to the original text. The patch honours the options you chose, so if you ignored line endings, the patch won’t contain line-ending changes either.

When not to use a text diff

For JSON, a text diff is the wrong tool: reordered keys and reordered arrays make it useless. Use the JSON diff, which compares structure. And if you’re hunting for a pattern across the differences, the regex tester is the next stop.