ColdFusion MX has brought discord into my world. Why, you may ask? Because it silently, and stealthily took away our standard ISO-8859-1 encoding, and replaced it with UTF-8.

For the majority of web applications by and for Americans, the question of encodings will never become an issue. UTF-8 was developed as a theoretically universal way to encode asian (and other) characters. UTF-8 was officially defined as an internet standard in RFC-2044 in 1996. This was later superseded by RFC-2279 in 1998.

The main concept behind UTF-8 was that the first 128 characters of it match up exactly with the 7-bit US-ASCII standard that has been used for decades by UNIX, Dos, and (more recently) Windows and Macintosh programmers. Thus, an old US-ASCII file is still valid when read as UTF-8.

Unfortunately, between the development of US-ASCII and UTF-8, there were several other competing standards for other encodings. Some of these have since become recognized standards (such as ISO-8859-1, or Standard Latin). Others have languished in a technological backwater, or been supplanted by the other standards (such as windows-1252).

Which brings us back to ColdFusion MX, and it's somewhat unwanted UTF-8 encoding. Since we are forced to support Netscape 4.79 on Windows NT 4 systems, the support for UTF-8 is far buggier on our clients than it would be if we were using, say Mozilla 1.6 and Windows XP or Linux 2.6

ColdFusion does a wonderful job of reading UTF-8, but our users are having a hard time rendering UTF-8. The solution that has eventually presented itself is to use ColdFusion to translate the UTF-8 Characters into HTML entities, which allows them to render correctly on all of the browsers I tested them on.

Here is a sample code snippet to convert a UTF-8 String into something that all browsers will render correctly:

<cfset destString = utf8_encoded_string>

<cfloop from="1" to="#len(utf8_encoded_string)#" index="i">

  <cfset val=asc(mid(utf8_encoded_string,i,1))>

  <cfif val gt 127>

    <cfset destString=replace(destString, chr(val),"&###val#;")>

  </cfif>

</cfloop>