Page History: TextFileEncodingDetector project
Compare Page Revisions
Page Revision: 2010-01-21 09:34
There's an awkward situation on Windows machines (and, I suspect, more generally) - text files, and text-based files like CSV files, etc, can be saved in any number of encodings: windows codepages, less-common encodings such as EBCDIC, and more modern encodings like UTF-8 and UTF-16.
The newer Unicode formats have a standard for "self-describing" the encoding, in the form of a Byte Order Mark, but this is often not present, and in fact actively discouraged by the unicode consortium, in the case of UTF-8.
For UTF-8 in particular, this poses a problem because UTF-8 encoding looks a whole lot like ASCII/ANSI/Windows-1252/Latin-1, a family of related encodings commonly used and confused on Windows systems and nowadays globally.
The "Correct" thing to do, when presented with a text file, is to:
- Check for a BOM, indicating a Unicode file of some specific type
- If not found, ask the user what encoding was used.
Or at least, this is the opinion of many developers, see
this stack overflow question and the linked seminal rant by Joel Spolsky,
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).
Now in the real world, most users don't know what encoding their files use, and on windows machines in western and particularly english-speaking countries, the number of options commonly encountered is quite limited:
- Windows-1252 (a superset of Latin-1, which itself is a superset of US-ASCII)
- UTF-8, with or without BOM
- UTF-16, LE or BE, with or without BOM
Automatically determining which of these a text file uses is, 99% of the time, quite straightforward, but I couldn't find any libraries that do it - nothing in the .Net framework, no usable code snippets online (beyond trivial BOM detection), simply no easy way to do it.
So there it is. A simple class that automatically detects with of these encodings a file probably uses, for when your users don't have a clue. If they do get a choice, please please get them to use Unicode or UTF-8 with BOM! It makes things sooo much easier...
(as usual, not cleaned up, contact me for details)