Weird symbols on Mac

Question

Since I've had my mac, I keep seeing this weird symbol. Till today, it had been only in the place of bullet points in OpenOffice.org. The first pictures shows this in a .doc file created on a Windows system.

I thought nothing of it - just an annoyance. It appears no matter what the font. Real bullets appear if I delete the text and insert a bulletted list using the toolbar.

Then, today I noticed in in iTunes - which seemed strange.

Image 3 is a zoom of the character. It says on it: Private Use E000 F8FF.

What is it (unicode related?), and how do I get the bullets working properly?

Edit: The plot thickens... If I boot in Safe Mode, the symbols look like little snap boards like you'd have at the beginning of filming a scene in a film...

Solution 1

(Err, yes, this answer is way too long. And there's no happy ending for the bullets! Highlights in bold...)

The images from the question are defined in Apple's Last Resort font, which holds 236 different funny symbols. I guess the name says it all, but according to Wikipedia:

LastResort is a Mac OS font that is invisible to the end user, but is used by the system to display glyphs that are not available in any other font. The symbols provided by LastResort place glyphs into categories based on their location in the Unicode system and provide a hint to the user about which font or script is required to view the unavailable characters.

And Apple explains:

Examplar glyphs were chosen in a number of ways. Almost all of the Brahmic scripts show the initial consonant ka. Latin uses the letter A because it's the first letter, and because in each Latin block there is a letter A so they can be easily differentiated. Greek and Cyrillic use their last letters, omega and ya, because they are so distinctive. Most other alphabets and syllabaries use their initial letter where distinctive.

(I like the Unicode BMP Fallback font, like used in Firefox, much better as it shows the exact 4-character Unicode code.)

So, your Mac does not know what to display, and uses the Last Resort font to provide some information.

My first guess was: Microsoft Office uses some proprietary symbol font, in which it uses character codes from the private use area (PUA) to define what the bullets look like. (Or maybe the author has some odd font installed from which some funny bullet has been used.) Your Mac neither knows that font, nor has any other font in which the same Unicode character code happens to be defined. And even if another font did define some character for that code, it would not help either, as characters from the private use areas may by definition have a totally different meaning in different fonts. Installing Office on the Mac might also include the font, probably making the bullets show fine in OpenOffice.org as well. (In fact, installing a trial of Office might already install the missing proprietary font.)

Well, no.

While the above would be true for any sane usage of Unicode, some further investigation learns that in old applications Microsoft uses the range U+F020..U+F0FF to display symbols using another font. When displaying characters from that range, it then automatically switches to another font for those characters. Some organisation named SIL International figured out:

One of the mysteries of text formatted with symbol fonts (at least, in certain Microsoft applications) is that characters appear to be encoded in terms of 8-bit code points even if a document is otherwise encoded in Unicode. When U+F021 was inserted into WordPad from the clipboard, not only did WordPad (more precisely, the Rich Edit control) apply the Wingdings font, it seems that it also changed the code point to 0x21. When the character was reformatted to a non-symbol font, this became U+0021.

Or, as Microsoft explains it, for Microsoft Platform Software Development Kit-January 2000 Edition:

Richedit 4.1 maps the range of characters between U+F020 and U+F0FF in the PUA to symbol fonts. Therefore, when you map any character in this range, Richedit 4.1 shows the symbol character instead of the end-user-defined character (EUDC).

I think that meanwhile Microsoft has added the symbols to many Microsoft versions of fonts, to allow more recent Microsoft software to display those characters without switching fonts. For example, U+F020 shows a bullet in most fonts, but not in Arial on your Mac. Installing a Microsoft-version of Arial might help. But that might surely get you into other problems when you're using Arabic...

I doubt any non-Microsoft software will handle the above exceptions.

Still, in general:

One way to determine the name of the font: when printing from Word to PDF one can choose to include the font (or: the subset of characters that are used) in the PDF, to ensure it prints fine on systems that do not have that font installed. So, peeking into the PDF's properties might then reveal the name of that font. And maybe simply selecting the character in OpenOffice.org will already reveal its name in the font list. (However, given the automatic displaying of symbols like described above, both methods will probably not work for bullets at all.)

One way to determine the exact character code: copy it (for Pub Quiz , copy it from the auto-suggest in the search of iTunes) and paste into some Unicode code converter. This reveals that the Pub Quiz character is U+E047, which could be some odd double-quote. But well, as this is from a private use area, and we don't know which font the developers had in mind when they typed that name in the iPhone app-store, I guess only the developers can tell us what they hoped it would look like...

(Fileformat.info has a neat utility to show the character using all fonts on your computer.)

Solution 2

I only can answer the "what is it" portion of your question.

The text around the side, "Private Use" and "e000" and "f8ff", are referring a Unicode character set - specially, the Private Use set:

The UCS includes 137,468 code points for private use. This means these code points can be assigned characters with specific properties by individuals, organizations and software vendors outside the ISO and Unicode Consortium. A Private Use Area (PUA) is one of several ranges which are reserved for private use. For this range, the Unicode standard does not specify any characters.

The Basic Multilingual Plane includes a PUA in the range from U+E000 to U+F8FF (57344–63743). Plane Fifteen (U+F0000 to U+FFFFD), and Plane Sixteen (U+100000 to U+10FFFD) are completely reserved for private use as well.

The use of the PUA was a concept inherited from certain Asian encoding systems. These systems had private use areas to encode Japanese Gaiji (rare personal name characters) in application-specific ways. One example of usage of the Private Use Area is Apple's usage of U+F8FF for the Apple logo.

According to Brian Tiemann at Grotto11, these show up when the character's undefined:

Look at that. It's squares, like always, for undisplayable characters. But now that it's all vector graphics, and because the Arabic font sets apparently aren't done yet (as they keep completing point releases of OS X, they keep adding character-set packs that fill out these blocks of letters), it's squares with cool information in them. The Unicode range that Arabic occupies, plus a central symbol to tell you what will eventually go there.

This is what it looks like when Apple isn't quite done with something.

So our curiosity was running rampant now; we switched into Unicode hex input mode (hold down Option and type four-digit hex numbers) and started entering values, to see what the ranges were like and what they were assigned to, and what symbols they had:

Mmm-mm. Isn't that insane? I love it. (And especially that Dr. Seuss-looking "Private Use" one.) Apple even makes unimplemented features look cool. These squares sit at the beginning of each block of assigned characters and define what that block is going to be, and if the characters haven't all been finished yet, they all show up as that generic identifier.

Wish I could tell you why it's being displayed, though...

Solution 3

It sounds like a font cache corruption issue. If you are running Leopard (10.5) you can try running sudo atsutil databases -remove in the Terminal to remove the font caches on your computer. A GUI alternative is FontNuke.

Solution 4

I found the likely font for the U+E047 character using the Characters Palette. Most likely the app designer wanted to use the left pointing finger dingbat from Adobe Pi Std Regular.

View more solutions

Comments

[Dustin]

Okay, sounds like some interesting issues, since Safe Mode isn't display it correctly... (By the way, which version of MOSX?) What Font is the character suppose to be? Go to Applications --> Font Book Highlight the font in question, File --> Validate Font, and let us know what it says... Please note, that MS Word, maybe not reporting that the font doesn't exist, and silently substituting a different font.

[Rich Bradshaw]

It's meant to be Arial - though changing the font doesn't matter. I think it's the bullet point that MS Word inserts that's getting screwed up. Snow Leopard and Leopard show problems on various files.

[Dustin]

But you don't know what Font MS Word is substituting... So Highlight all the fonts in Font Book, and choose Resolve Duplicates, and then Validate Font... Any bad fonts should be then manually disabled or removed...

[Rich Bradshaw]

Hmm, didn't work - good thinking though...

[Rich Bradshaw]

That's pretty cool! Why is it there though! Wonder if I'll ever know :(

[Rich Bradshaw]

Why wouldn't you just link to support.apple.com/kb/HT1564, rather than copying and pasting the content into your blog?

[Arjan]

For those on a Mac: the Unicode input is hidden between the input sources for languages. On an English Mac OS X that would be somewhere between Ukrainian and Vietnamese.

[Richard]

I've seen the Private Use icon displayed on my iPhone (in Tweetie) in place of what I assume are emoji - the emoticons that can be enabled by using certain apps. I'm also clueless as to why they're there though as I've seen the emoticons before.

[Arjan]

Search the iTunes app store for Pub Quiz, for one super specific example? Or copy this: "Pub Quiz ". But why would this be an odd situation? Microsoft Office apparently uses characters from private (vendor-specific) ranges as bullets. And the developers of the Pub Quiz just copied some odd character into the description field of the app store.

[Benjamin Dobson]

So the Office includes special characters for bullets (and presumably some other things) which the open source version, of course, doesn't. The pub quiz app is a bit weird. At least this isn't a problem on the machine.

[Dustin]

Because I do go through and add content to the page as I find it. Yes, the Safe Boot page is about 75-80% of a cut and paste, but I have added some content... But I do expand, and improve on the content, for example, take a look at the Time Machine page.. schollnick.net/wordpress/systems-administration/… . And third, the blog is really intended as a info-dump for myself...

[Rich Bradshaw]

Fair enough - just seems a little spammy to not link to the original content.

[Dustin]

Which is why I replied with all the relevant information in the original comment, and only mentioned additional information on the blog entry...

[Jouni K. Seppänen]

Did you reboot after doing that? The OnyX utility is good for cleaning all sorts of caches.

[Arjan]

Not done yet... Bullet Point Characters Incorrect user.services.openoffice.org/en/forum/… for some this works: I've had success using the Font Replacement Table, in Tools»Options»OpenOffice.org»Fonts. Enable "Apply replacement table", select "Symbol" in the lefthand FONT drop down, select "OpenSymbol" in the righthand "Replace with" dropdown. Press the checkmark to the right to add the substitution to the table. Make sure you check the Always box. Once set up, opening/saving in MSWord 97/XP format preserves the bullet characters in both directions.

[Rich Bradshaw]

Yeah, but the point is that it's nicer and more useful to link to the apple support page, rather than your copied and pasted version

[Rich Bradshaw]

Hmm, good point.Since then I've reinstalled everything, which fixed it. Weird!