Unicode

Unicode
Name	Unicode
Released	October 1991
Status	Current
Based on	ASCII, ISO/IEC 8859
Extended from	UCS-2
Extended to	UTF-8, UTF-16, UTF-32
Standard	ISO/IEC 10646

Unicode. It is a computing industry standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. Developed and maintained by the Unicode Consortium, its goal is to eventually replace older, limited character sets with a single, universal standard. The standard provides a unique number, called a code point, for every character, regardless of platform, program, or language, enabling seamless international text communication.

Overview

The fundamental purpose is to create a single, unified character set that encompasses all characters used for written human communication. This solves the long-standing problems of incompatibility between different national and vendor-specific encoding standards like Shift JIS for Japanese or Windows-1252 for Western European languages. By assigning a unique identifier to each character, from the Latin letter 'A' to a complex Cuneiform sign, it allows text data to be interchanged globally without corruption. The scope includes not only modern scripts like Arabic and Devanagari, but also historical scripts, mathematical symbols, and a vast array of emoji.

History and development

Work began in the late 1980s, driven by engineers from Apple Inc., Xerox, and other technology companies who recognized the limitations of existing encodings. A pivotal meeting at Xerox PARC in 1987, involving Joe Becker from Xerox and Lee Collins and Mark Davis from Apple Inc., laid the initial groundwork. The first official version was published in October 1991, following collaboration with experts from the International Organization for Standardization working on the parallel ISO/IEC 10646 standard. Key figures in its ongoing development include co-founders Joe Becker and Lee Collins, and long-time president Mark Davis.

The project was a direct response to the chaotic state of character encoding, exemplified by the hundreds of incompatible code pages used by systems like those from IBM and Microsoft. Early versions focused on unifying major modern scripts, but the standard has continually expanded. A significant milestone was the merger with the ISO/IEC 10646 project, ensuring a single universal character set. The creation of efficient transformation formats, most notably UTF-8, by Ken Thompson and Rob Pike, was crucial for its practical adoption on the Internet.

Technical description

At its core, it defines a codespace divided into 17 planes, each comprising 65,536 code points. The most commonly used characters, including those in the Basic Latin block and the Bopomofo phonetic system, reside in the first plane, known as the Basic Multilingual Plane. Characters are logically grouped into blocks, such as the Cyrillic block or the Georgian block, for organizational purposes. The standard does not define glyphs or visual appearance, but rather the abstract characters; font files, like those from Adobe Systems or Microsoft, provide the visual representations.

To represent these code points in computer memory or data streams, several encoding forms are specified. UTF-8 has become the dominant encoding for the World Wide Web and Unix-like systems due to its backward compatibility with ASCII. UTF-16 is commonly used in operating systems like Microsoft Windows and the Java (programming language) platform, while UTF-32 offers a fixed-width format. The standard also includes extensive specifications for character properties, normalization forms, and bidirectional text algorithms for scripts like Hebrew.

Adoption and impact

Adoption has been widespread and transformative, becoming the foundational text encoding for modern computing and global communication. It is mandated by core internet standards like HTML and XML, as defined by the World Wide Web Consortium. All major operating systems, including macOS, Microsoft Windows, Linux distributions, and Android (operating system), now use it natively. Major software companies like Google, Microsoft, and Adobe Systems build their products around it, ensuring consistent text display and processing worldwide.

Its impact on global communication and software internationalization cannot be overstated. It enabled the true localization of software for global markets and is the backbone of the modern Internet, allowing a webpage from South Korea to display correctly on a device in Brazil. The inclusion of emoji, originally from Japanese mobile carriers like NTT Docomo, has created a new, near-universal pictographic language. It has also revitalized academic and cultural work on historical and minority scripts by providing them with digital representation.

Versions and standards

New versions are released regularly by the Unicode Consortium, with each version adding support for more characters, scripts, and features. Major releases, such as Unicode 3.0 which added the Ethiopic script, or Unicode 6.0 which significantly expanded emoji support, mark important expansions. The development process involves extensive proposals and review by committees, with input from national bodies, corporations, and linguistic experts. Each version is synchronized with a corresponding amendment to the international standard ISO/IEC 10646.

The standard is published as a large book and online, containing the complete code charts, character properties, and detailed annexes. Important auxiliary specifications include the Unicode Collation Algorithm for sorting text and the Common Locale Data Repository for formatting dates and numbers. The consortium also maintains the Unicode Character Database, a machine-readable file that is the definitive source for all character properties and is integral to the functioning of modern operating systems and programming languages like Python (programming language).

Category:Character sets Category:Computing standards Category:Text encoding