ThisCalculator

Everyday

Determine File Encoding Mac Command Line

Updated · Standard formula, shown below

Determine File Encoding Mac Command Line

To determine a file’s encoding from the Mac command line, use the built-in file command, which prints the detected encoding along with the file type. Simply type file -I filename (or file --mime-encoding filename) to see the character encoding, such as utf-8 or iso-8859-1.

Why Encoding Matters

Text files store characters as bytes, and the encoding defines how those bytes map to characters. If you open a file with the wrong encoding, you’ll see garbled text (mojibake) like é instead of é. Knowing the encoding is essential for converting files, debugging display issues, and ensuring data integrity when processing files across different systems.

The file Command

The file command is the quickest way to check encoding. It examines the file’s content and reports the MIME type and encoding.

  • file -I filename – prints the MIME type and encoding (e.g., text/plain; charset=utf-8).
  • file -b --mime-encoding filename – prints only the encoding (e.g., utf-8).
  • file filename – without options, it shows the file type and sometimes the encoding.

For example, if you have a file named data.csv, running file -I data.csv might output data.csv: text/plain; charset=utf-8. This tells you the file is UTF-8 encoded.

How file Detects Encoding

The file command uses a series of tests: it checks for a byte order mark (BOM), looks for valid UTF-8 sequences, and then tries other common encodings like ISO-8859-1 or ASCII. It’s heuristic, so it can be fooled by short or ambiguous files. For example, a file containing only ASCII characters will be reported as us-ascii, which is a subset of UTF-8.

Using iconv to Convert

Once you know the encoding, you can convert the file to another encoding using iconv. The syntax is:

iconv -f original_encoding -t target_encoding input_file > output_file

For instance, to convert a file from ISO-8859-1 to UTF-8:

iconv -f ISO-8859-1 -t UTF-8 input.txt > output.txt

To list all supported encodings, run iconv -l. This is helpful when you need to find the exact name for a less common encoding.

When file Fails: Manual Inspection

If file reports unknown-8bit or gives an unexpected result, you can inspect the raw bytes yourself with hexdump or xxd.

  • xxd filename | head – shows the first few bytes in hexadecimal.
  • hexdump -C filename | head – similar but with a different format.

Look for patterns: a UTF-8 BOM appears as EF BB BF at the start. UTF-16 files often start with FF FE (little-endian) or FE FF (big-endian). If you see many bytes with the high bit set (values 80 to FF) that don’t form valid UTF-8, the file is likely a legacy single-byte encoding like Windows-1252.

Common Pitfalls

  • BOM confusion: Some tools expect a BOM, others don’t. file may report utf-8 even when a BOM is present, but the BOM is technically part of the encoding.
  • Ambiguous encodings: Files with only ASCII characters are valid in many encodings, so file will report us-ascii. That’s fine for most purposes.
  • Wrong iconv name: Use iconv -l to get the exact name; aliases like UTF8 may not work.
  • Overwriting the original: iconv writes to stdout, so redirect to a new file. To replace the original, use a temp file and then move it.

Alternatives to file

If you prefer a GUI, TextEdit can reveal encoding when opening a file (choose “Open” and look at the bottom). Or use a hex editor like Hex Fiend to inspect bytes visually. For scripting, python3 can also detect encodings with the chardet library, but that’s not built-in.

Frequently Asked Questions

How do I check the encoding of multiple files at once?

Use a loop in the terminal: for f in *.txt; do echo -n "$f: "; file -b --mime-encoding "$f"; done. This prints each filename and its encoding.

What is the difference between UTF-8 and UTF-16?

UTF-8 uses 1 to 4 bytes per character and is ASCII-compatible, making it the standard for web and Unix. UTF-16 uses 2 or 4 bytes and is common in Windows and Java. For most text files, UTF-8 is preferred due to smaller size and compatibility.

Why does file say “unknown-8bit”?

This means the file contains bytes with the high bit set, but they don’t form valid UTF-8 sequences, and the program couldn’t identify the encoding. You may need to inspect the bytes manually or try converting with iconv using a likely encoding like ISO-8859-1.

Can I use file on files without extensions?

Yes, file ignores extensions and examines the content. It works on any file, regardless of its name.

Is there a way to see the encoding in Finder?

Finder doesn’t show encoding by default. You can use the Terminal commands above, or install a Quick Look plugin, but the command line is the most reliable method.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.