PHP output showing little black diamonds with a question mark

Question

I'm writing a php program that pulls from a database source. Some of the varchars have quotes that are displaying as black diamonds with a question mark in them (�, REPLACEMENT CHARACTER, I assume from Microsoft Word text).

How can I use php to strip these characters out?

Solution 1

I also faced this � issue. Meanwhile I ran into three cases where it happened:

1. substr()

   I was using substr() on a UTF8 string which cut UTF8 characters, thus the cut chars could not be displayed correctly. Use mb_substr($utfstring, 0, 10, 'utf-8'); instead. Credits

2. htmlspecialchars()

   Another problem was using htmlspecialchars() on a UTF8 string. The fix is to use: htmlspecialchars($utfstring, ENT_QUOTES, 'UTF-8');

3. preg_replace()

   Lastly I found out that preg_replace() can lead to problems with UTF. The code $string = preg_replace('/[^A-Za-z0-9ÄäÜüÖöß]/', ' ', $string); for example transformed the UTF string "F(×)=2×-3" into "F � 2� ". The fix is to use mb_ereg_replace() instead.

I hope this additional information will help to get rid of such problems.

Solution 2

This is a charset issue. As such, it can have gone wrong on many different levels, but most likely, the strings in your database are utf-8 encoded, and you are presenting them as iso-8859-1. Or the other way around.

The proper way to fix this problem, is to get your character-sets straight. The simplest strategy, since you're using PHP, is to use iso-8859-1 throughout your application. To do this, you must ensure that:

• All PHP source-files are saved as iso-8859-1 (Not to be confused with cp-1252).
• Your web-server is configured to serve files with charset=iso-8859-1
• Alternatively, you can override the webservers settings from within the PHP-document, using header.
• In addition, you may insert a meta-tag in you HTML, that specifies the same thing, but this isn't strictly needed.
• You may also specify the accept-charset attribute on your <form> elements.
• Database tables are defined with encoding as latin1
• The database connection between PHP to and database is set to latin1

If you already have data in your database, you should be aware that they are probably messed up already. If you are not already in production phase, just wipe it all and start over. Otherwise you'll have to do some data cleanup.

A note on meta-tags, since everybody misunderstands what they are:

When a web-server serves a file (A HTML-document), it sends some information, that isn't presented directly in the browser. This is known as HTTP-headers. One such header, is the Content-Type header, which specifies the mimetype of the file (Eg. text/html) as well as the encoding (aka charset). While most webservers will send a Content-Type header with charset info, it's optional. If it isn't present, the browser will instead interpret any meta-tags with http-equiv="Content-Type". It's important to realise that the meta-tag is only interpreted if the webserver doesn't send the header. In practice this means that it's only used if the page is saved to disk and then opened from there.

This page has a very good explanation of these things.

Solution 3

As mentioned in earlier answers, it is happening because your text has been written to the database in iso-8859-1 encoding, or any other format.

So you just need to convert the data to utf8 before outputting it.

$text = “string from database”;
$text = utf8_encode($text);
echo $text;

Solution 4

To make sure your MYSQL connection is set to UTF-8 (or latin1, depending on what you're using), you can do this to:

$con = mysql_connect("localhost","username","password");    
mysql_set_charset('utf8',$con);

or use this to check what charset you are using:

$con = mysql_connect("localhost","username","password");   
$charset = mysql_client_encoding($con);
echo "The current character set is: $charset\n"; 

More info here: http://php.net/manual/en/function.mysql-set-charset.php

Solution 5

Just Paste This Code In Starting to The Top of Page.

<?php
header("Content-Type: text/html; charset=ISO-8859-1");
?>

View more solutions

Comments

[troelskn]

The problem with this suggestion is that most likely the data is a mix of different charsets at this point. If you don't know exactly what went wrong, it just becomes even messier, if you just throw some random fixes in here and there.

[Daniel Cassidy]

I agree. I edited my post somewhat to reflect that this solution isn't a substitute for knowing what you're doing. However, I've come to the conclusion that most developers are either incapable of understanding this issue, or just don't care. It seems to come up at least once a month where I work.

[troelskn]

That's pretty much my observation too. For what I care, they reap as they sow. But you're probably right; Chances are that his data is indeed cp-1252 .. At least some of it is.

[contrid]

This was very useful and solved my quotes encoding issue in data coming from a remote MySQL database, thank you!

[Ren]

That was exactly the problem I was facing. Didn't know about the mb string functions.

[unixmiah]

@ptwiggerl this helped a lot.

[drtechno]

the above code fixes my table. but I would recommend commenting the update statements so you can see first if it is going to fix the issue.

[Rafael Moni]

I migrated an website to another server and I faced this problem, mysql_set_charset('utf8', $con); solved it!

[ppovoski]

Please elaborate on this answer.

[sixstring]

I tried a bunch of solutions to the same issue. This one was immediately effective with the least effort

[Mike Casan Ballester]

It happened also for strtolower function. All functions concerned in PHP manual

[treyBake]

warning about the $text = section: this will change all question marks within the string, not just the diamond

[rk_programmer]

this is the function that allows you to remove the special character and returns you the utf8 standard of character google.com/…

[Rog]

This worked with fractions that were not displayed correctly.

[CT Hall]

Please include a brief explanation of what the code does.

[Harshil Kaneria]

This php Code To Allow the character set of "ISO-8859-1" and in this character set this symbol � is shown as a character.

[quantme]

In my opinion, these should be an accepted answer; this is the only method that worked for me, I tried all of it.

[sanjeev shetty]

This is awesome, it worked for me, tried utf8_encode and ut8_decode also- did not work. But this solution worked in my case. Thank you.

[TomoMiha]

For mysqli it is this line: mysqli_set_charset($conn, 'utf8');

[EthanNAustin]

The perfect answer worked like a charm thank you! Adding a meta tag to the HTML code did not help me, BTW.