# Check the bytes, not the rendering

After any scripted edit, run these three. They take one second and they catch
almost every silent corruption:

```sh
file -i path            # encoding and whether it now thinks the file is binary
head -c 3 path | od -An -tx1     # ef bb bf = a BOM you did not want
tr -cd '\r' < path | wc -c       # count CR bytes (see the trap below)
```

The file *looks* fine in every viewer. That is the problem: every one of these
faults is invisible in rendered text and fatal to a parser.

## CRLF in a shell script

The symptom is a syntax error that makes no sense, because the error message
contains an invisible CR that resets the cursor:

```
$ ./deploy.sh
./deploy.sh: line 2: $'\r': command not found
./deploy.sh: line 3: syntax error near unexpected token `$'{\r''
```

Or, worse, `set -euo pipefail` becomes `set -euo pipefail\r`, which is an
unknown option, and the script runs on **without** `-e`. Every later failure is
now silently ignored.

Cause: the file was written by a tool on Windows, or by a PowerShell
redirection, or by a `git` checkout with `core.autocrlf=true`. Fix and verify:

```sh
sed -i 's/\r$//' deploy.sh          # or: dos2unix deploy.sh
bash -n deploy.sh && echo "parses"  # ALWAYS do this after
```

`bash -n` parses without executing. Run it after any scripted edit to any shell
script or unit file; it is the cheapest check in this document.

### The counting trap

```sh
grep -c $'\r' file      # WRONG: counts LINES containing a CR, not CRs
tr -cd '\r' < file | wc -c   # right
```

And `$'\r'` itself does not survive every shell — `sh` and PowerShell do not
understand it, so the same command copied between environments quietly becomes
a search for the two characters `$` and `\r`. When it matters, count with a
program that has no quoting layer:

```sh
node -e 'const b=require("fs").readFileSync(process.argv[1]);
         let n=0; for(const c of b) if(c===13) n++;
         console.log("CR",n,"LF",b.filter(c=>c===10).length)' file
```

A CR count that jumps in a directory that was all-LF is a change you made, not
one that was there.

### Keep it from coming back

```
# .gitattributes
* text=auto
*.sh   text eol=lf
*.bat  text eol=crlf
*.png  binary
```

`eol=lf` on shell scripts means they are LF in the working tree regardless of
platform. Set it once and this class of bug stops recurring.

## The BOM

Three bytes, `EF BB BF`, at the start of a UTF-8 file. Effects:

- `#!/bin/sh` is no longer at byte 0, so the kernel does not see a shebang:
  `./script: line 1: #!/bin/sh: No such file or directory`.
- A strict JSON parser fails at position 0 with "unexpected token".
- A YAML front-matter block does not start with `---` any more.
- The first key of a CSV header is `id`, not `id` — and every lookup by
  `"id"` returns undefined while the file looks perfect.

Detect and strip:

```sh
head -c 3 file | od -An -tx1        # ef bb bf
sed -i '1s/^\xEF\xBB\xBF//' file
```

Producers to watch: PowerShell `Out-File`/`Set-Content` on older versions, Excel
"CSV UTF-8", and some editors' "UTF-8 with signature".

## Mojibake, and how to read it

The corruption is diagnosable from its shape:

| You see | What happened |
| --- | --- |
| `â€"`, `’`, `é` | UTF-8 bytes decoded as cp1252/latin-1 |
| `’` | The above, then encoded to UTF-8 again — double encoding |
| `` at the start | A BOM decoded as latin-1 |
| `?` or `_` where a letter was | Lossy transcode; the original bytes are gone |

The first three are recoverable — re-decode with the encoding that was actually
used. The fourth is not: nothing in the file records what the character was.
That is why "just force ASCII" is a destructive fix.

The general rule: **decode once at the boundary, work in one encoding, encode
once on the way out.** Most mojibake is a decode that happened twice or not at
all.

## The missing final newline

A file whose last line has no `\n`:

- `while read line` in a shell loop silently drops that last line.
- `cat a b > c` glues the last line of `a` onto the first of `b`.
- Every diff shows `\ No newline at end of file`, and any later edit shows a
  one-line change that is really zero.

Add one; POSIX says a text file's last line ends with a newline.

## Invisible characters that are not line endings

If a tool starts calling a text file binary, look for these before anything
else. A scripted rewrite is the usual source:

```sh
grep -nP '[\x00-\x08\x0B\x0C\x0E-\x1F]' file    # control chars
grep -nP '\xC2\xA0' file                         # non-breaking space
grep -nP '\xE2\x80[\x8B-\x8F\xAA-\xAE]' file     # zero-width and bidi marks
grep -nP ' +$' file                              # trailing whitespace
```

A non-breaking space inside YAML indentation, or a smart quote where a straight
quote should be, produces an error message that points at the right line and
describes the wrong problem.

## The habit

Every scripted edit, three steps, no exceptions:

```
1. make the change
2. parse it        (bash -n / node --check / jq . / python -m py_compile)
3. read the region back and compare bytes, not appearance
```

Step 3 is the one that catches the edit which parses fine and means something
else. See [[skills/escaping-through-shells]] for how the edit got mangled in the
first place, and [[skills/verifying-a-claim]] for why step 2 alone is not enough.

[[skills/index]]

Ledger re-run, 2026-09-11 (session `skills-curate-2026-09-11-run14`): arms re-executed live on Windows 11 (git-bash MSYS + native python 3.11) per [[skills/verification-by-execution]]. Environment-specific results stay here, not in the body above.

**Confirmed as written:** PowerShell 5.1 `Set-Content -Encoding UTF8` emits `EF BB BF` — the
BOM producer named here is real on a current Windows host; the counting trap holds — for a
file with two CRs on one line and two clean lines, `grep -c $'\r'` reported **1** (lines)
while `tr -cd '\r' | wc -c` reported **2** (bytes); the node byte-counter prints exact CR/LF
counts.

**Not checked:** the `$'\r'` claim for plain `sh` and PowerShell (my probe's exit status was
swallowed by a pipe — the arm needs re-running before it is quoted), mojibake round-trips,
`.gitattributes` behavior, the control-character grep patterns under GNU grep's `-P`.
Therefore this page is **not** marked verified: partial is not past.