How to remove all non printable characters in a string?
Solution 1
7 bit ASCII?
If your Tardis just landed in 1963, and you just want the 7 bit printable ASCII chars, you can rip out everything from 0-31 and 127-255 with this:
$string = preg_replace('/[\x00-\x1F\x7F-\xFF]/', '', $string);
It matches anything in range 0-31, 127-255 and removes it.
8 bit extended ASCII?
You fell into a Hot Tub Time Machine, and you're back in the eighties. If you've got some form of 8 bit ASCII, then you might want to keep the chars in range 128-255. An easy adjustment - just look for 0-31 and 127
$string = preg_replace('/[\x00-\x1F\x7F]/', '', $string);
UTF-8?
Ah, welcome back to the 21st century. If you have a UTF-8 encoded string, then the /u
modifier can be used on the regex
$string = preg_replace('/[\x00-\x1F\x7F]/u', '', $string);
This just removes 0-31 and 127. This works in ASCII and UTF-8 because both share the same control set range (as noted by mgutt below). Strictly speaking, this would work without the /u
modifier. But it makes life easier if you want to remove other chars...
If you're dealing with Unicode, there are potentially many non-printing elements, but let's consider a simple one: NO-BREAK SPACE (U+00A0)
In a UTF-8 string, this would be encoded as 0xC2A0
. You could look for and remove that specific sequence, but with the /u
modifier in place, you can simply add \xA0
to the character class:
$string = preg_replace('/[\x00-\x1F\x7F\xA0]/u', '', $string);
Addendum: What about str_replace?
preg_replace is pretty efficient, but if you're doing this operation a lot, you could build an array of chars you want to remove, and use str_replace as noted by mgutt below, e.g.
//build an array we can re-use across several operations
$badchar=array(
// control characters
chr(0), chr(1), chr(2), chr(3), chr(4), chr(5), chr(6), chr(7), chr(8), chr(9), chr(10),
chr(11), chr(12), chr(13), chr(14), chr(15), chr(16), chr(17), chr(18), chr(19), chr(20),
chr(21), chr(22), chr(23), chr(24), chr(25), chr(26), chr(27), chr(28), chr(29), chr(30),
chr(31),
// non-printing characters
chr(127)
);
//replace the unwanted chars
$str2 = str_replace($badchar, '', $str);
Intuitively, this seems like it would be fast, but it's not always the case, you should definitely benchmark to see if it saves you anything. I did some benchmarks across a variety string lengths with random data, and this pattern emerged using php 7.0.12
2 chars str_replace 5.3439ms preg_replace 2.9919ms preg_replace is 44.01% faster
4 chars str_replace 6.0701ms preg_replace 1.4119ms preg_replace is 76.74% faster
8 chars str_replace 5.8119ms preg_replace 2.0721ms preg_replace is 64.35% faster
16 chars str_replace 6.0401ms preg_replace 2.1980ms preg_replace is 63.61% faster
32 chars str_replace 6.0320ms preg_replace 2.6770ms preg_replace is 55.62% faster
64 chars str_replace 7.4198ms preg_replace 4.4160ms preg_replace is 40.48% faster
128 chars str_replace 12.7239ms preg_replace 7.5412ms preg_replace is 40.73% faster
256 chars str_replace 19.8820ms preg_replace 17.1330ms preg_replace is 13.83% faster
512 chars str_replace 34.3399ms preg_replace 34.0221ms preg_replace is 0.93% faster
1024 chars str_replace 57.1141ms preg_replace 67.0300ms str_replace is 14.79% faster
2048 chars str_replace 94.7111ms preg_replace 123.3189ms str_replace is 23.20% faster
4096 chars str_replace 227.7029ms preg_replace 258.3771ms str_replace is 11.87% faster
8192 chars str_replace 506.3410ms preg_replace 555.6269ms str_replace is 8.87% faster
16384 chars str_replace 1116.8811ms preg_replace 1098.0589ms preg_replace is 1.69% faster
32768 chars str_replace 2299.3128ms preg_replace 2222.8632ms preg_replace is 3.32% faster
The timings themselves are for 10000 iterations, but what's more interesting is the relative differences. Up to 512 chars, I was seeing preg_replace alway win. In the 1-8kb range, str_replace had a marginal edge.
I thought it was interesting result, so including it here. The important thing is not to take this result and use it to decide which method to use, but to benchmark against your own data and then decide.
Solution 2
Many of the other answers here do not take into account unicode characters (e.g. öäüßйȝîûηыეமிᚉ⠛ ). In this case you can use the following:
$string = preg_replace('/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F]/u', '', $string);
There's a strange class of characters in the range \x80-\x9F
(Just above the 7-bit ASCII range of characters) that are technically control characters, but over time have been misused for printable characters. If you don't have any problems with these, then you can use:
$string = preg_replace('/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]/u', '', $string);
If you wish to also strip line feeds, carriage returns, tabs, non-breaking spaces, and soft-hyphens, you can use:
$string = preg_replace('/[\x00-\x1F\x7F-\xA0\xAD]/u', '', $string);
Note that you must use single quotes for the above examples.
If you wish to strip everything except basic printable ASCII characters (all the example characters above will be stripped) you can use:
$string = preg_replace( '/[^[:print:]]/', '',$string);
For reference see http://www.fileformat.info/info/charset/UTF-8/list.htm
Solution 3
Starting with PHP 5.2, we also have access to filter_var, which I have not seen any mention of so thought I'd throw it out there. To use filter_var to strip non-printable characters < 32 and > 127, you can do:
Filter ASCII characters below 32
$string = filter_var($input, FILTER_UNSAFE_RAW, FILTER_FLAG_STRIP_LOW);
Filter ASCII characters above 127
$string = filter_var($input, FILTER_UNSAFE_RAW, FILTER_FLAG_STRIP_HIGH);
Strip both:
$string = filter_var($input, FILTER_UNSAFE_RAW, FILTER_FLAG_STRIP_LOW|FILTER_FLAG_STRIP_HIGH);
You can also html-encode low characters (newline, tab, etc.) while stripping high:
$string = filter_var($input, FILTER_UNSAFE_RAW, FILTER_FLAG_ENCODE_LOW|FILTER_FLAG_STRIP_HIGH);
There are also options for stripping HTML, sanitizing e-mails and URLs, etc. So, lots of options for sanitization (strip out data) and even validation (return false if not valid rather than silently stripping).
Sanitization: http://php.net/manual/en/filter.filters.sanitize.php
Validation: http://php.net/manual/en/filter.filters.validate.php
However, there is still the problem, that the FILTER_FLAG_STRIP_LOW will strip out newline and carriage returns, which for a textarea are completely valid characters...so some of the Regex answers, I guess, are still necessary at times, e.g. after reviewing this thread, I plan to do this for textareas:
$string = preg_replace( '/[^[:print:]\r\n]/', '',$input);
This seems more readable than a number of the regexes that stripped out by numeric range.
Solution 4
you can use character classes
/[[:cntrl:]]+/
Solution 5
All of the solutions work partially, and even below probably does not cover all of the cases. My issue was in trying to insert a string into a utf8 mysql table. The string (and its bytes) all conformed to utf8, but had several bad sequences. I assume that most of them were control or formatting.
function clean_string($string) {
$s = trim($string);
$s = iconv("UTF-8", "UTF-8//IGNORE", $s); // drop all non utf-8 characters
// this is some bad utf-8 byte sequence that makes mysql complain - control and formatting i think
$s = preg_replace('/(?>[\x00-\x1F]|\xC2[\x80-\x9F]|\xE2[\x80-\x8F]{2}|\xE2\x80[\xA4-\xA8]|\xE2\x81[\x9F-\xAF])/', ' ', $s);
$s = preg_replace('/\s+/', ' ', $s); // reduce all multiple whitespace to a single space
return $s;
}
To further exacerbate the problem is the table vs. server vs. connection vs. rendering of the content, as talked about a little here
Related videos on Youtube
Stewart Robinson
Updated on January 26, 2022Comments
-
Stewart Robinson about 2 years
I imagine I need to remove chars 0-31 and 127.
Is there a function or piece of code to do this efficiently?
-
Stewart Robinson almost 15 yearsdoesn't this require me to use ereg though?
-
Nick over 13 yearsIf you need to consider a newline safe, change the expression to this (inversely search for printables): preg_replace(/[^\x0A\x20-\x7E]/,'',$string);
-
Dalin over 12 yearsThis also strips line feeds, carriage returns, and UTF8 characters.
-
Hazard almost 12 yearsYour regexp handles UTF8 characters fine; but it strips non-UTF8 "special" characters; like ç, ü and ö.
'/[\x00-\x1F\x80-\xC0]/u'
leaves them intact; but also division (F7) and multiplication (D7) sign. -
Mathias Bynens over 11 years@Dalin There is no such thing as an “UTF-8 character”. There are Unicode symbols/characters, and UTF-8 is an encoding that can represent all of them. You meant to say this doesn’t work for characters outside of the ASCII character set.
-
Mathias Bynens over 11 years@Dalin There is no such thing as an “UTF-8 character”. There are Unicode symbols/characters, and UTF-8 is an encoding that can represent all of them. You meant to say this strips characters outside of the ASCII range as well.
-
Dalin about 11 years@Hazar yes you are correct \x80-\xFF stripped out too much, but \x80-\xC0 is still too restrictive. This would miss other printable characters like ©£±. For reference see utf8-chartable.de
-
Rolf almost 11 yearsEats up Arabic characters :)
-
Peter Olson almost 11 yearsIf you need to match a unicode character above \xFF, use \x{####}
-
Mubashar over 10 yearsyou missed \x7F (127) which is a non-printable character
-
Josh Ribakoff over 10 yearsThe third example with :print: behaves differently on different machines. It worked on localhost, but didn't strip the same characters on our live server. The first example stripped regular numbers from my string on localhost.
-
Dalin over 10 years@JoshRibakoff I don't see how [:print:] could show different results on different machines, it is a POSIX standard: en.wikipedia.org/wiki/Regular_expression#Character_classes also I don't see how the first example could strip regular numbers, you'll need to give more info.
-
Josh Ribakoff over 10 yearsNot sure what more info to give. Perhaps some server setting differed such as the locale. I don't know where to start to debug it, I fixed it using a whitelist of allowed characters which was kind of a pain but ended up getting the job done.
-
Gajus about 10 yearsThis well remove characters like quotes, brackets, etc. Those are certainly printable characters.
-
Ayman Hussein about 10 yearsthis will remove Arabic letters, bad solution.
-
kimbarcelona about 10 yearsHi is there a way that it can preserve new lines? I'm using it and it actually deletes special characters from my string but my string is for example 20 lines. The output is now one-line (All 20 lines were combined).
-
kliron almost 9 years is an encoding, not a character. The solution above is only intended to work on ASCII characters.
-
Korri about 8 yearsThe only one that passes all my unit tests, awesome!
-
krishna almost 8 yearsthis is wonderful! it saved my life, messed up while printing Arabic characters, worked like champ :)
-
Tim Malone almost 8 years@Dalin I noticed at least the second option doesn't work with double quotes, but it does with single quotes. Do you know why this is?
-
Dalin over 7 years@TimMalone because PHP will expand those character sequences: php.net/manual/en/… so the regex won't see the range that you're trying to tell it about.
-
Bell over 7 yearsWhat about 7F? Should it not be
\x7F-\x9F
? -
Dalin over 7 years@Bell Good catch. Fixed.
-
Aaron Esau over 7 yearsDoes it preserve spaces?
-
kliron over 7 yearsA space is 32 (0x20), so yes.
-
mgutt about 7 yearsSorry, but this answer is completely wrong, see mine: stackoverflow.com/a/42058165/318765
-
mgutt about 7 yearsWhy do you want to remove the euro sign
\x80
and all the other printable characters?! Look my answer: stackoverflow.com/a/42058165/318765 -
mgutt about 7 yearsThis answer is wrong, too. See: stackoverflow.com/a/42058165/318765
-
kliron about 7 years@mgutt I've clarified the answer. See also interesting benchmark result on str_replace.
-
mgutt about 7 yearsRemove the deletion of 128-255. There does not exist something like a "7-bit extended ascii table". The only 128-255 control set I know is the one in UTF-8 and this should not be touched as it could contain (in Windows) the euro sign and other characters as stated in my answer. P.S I verified your benchmark.
preg_replace
is faster. -
mgutt about 7 yearsP.S. maybe you think about adding
chr(160) (NO-BREAK SPACE)
andchr(173) (SOFT HYPHEN)
. They are non-printable, too. -
Dalin about 7 years@mgutt
\x80
isn't the euro in unicode,\x20AC
is.\x80
is the euro in some other encodings but in Unicode it's technically a control character. If you want to leave it in, go for it. -
mgutt about 7 years@Dalin Read my answer. I provided sources were you can see the behaviour. It seems to be a backwards compatibility to CP-1252. Test it by yourself through entering
€ € €
on this website: mothereff.in/html-entities I see three euro signs. -
Patrizio Bekerle about 7 years
$string = preg_replace('/[\x00-\x1F\x7F\xA0]/u', '', $string);
worked perfectly to sanitize data fromiptcparse
, thank you! -
Marcio Mazzucato almost 7 yearsIt works perfect for me! I added just
/u
for UTF-8 chars. Could you please explain what the first part(?!\n)
does? -
John over 6 yearsI just tried a lot, i tried every encoding function available in PHP from regex to mb_ to htmlspecialchars etc. Nothing removed control characters, thanks for investing the work.
-
MingalevME about 6 years\xE2\x80[\xA4-\xA8] (or 226.128.[164-168]) - is wrong, the sequence include next printable symbols: Unicode Character 'ONE DOT LEADER' (U+2024), Unicode Character 'TWO DOT LEADER' (U+2025), Unicode Character 'HORIZONTAL ELLIPSIS' (U+2026), Unicode Character 'HYPHENATION POINT' (U+2027). And only one non-printable: Unicode Character 'LINE SEPARATOR' (U+2028). Next one is non-printable too: Unicode Character 'PARAGRAPH SEPARATOR' (U+2029). So replace the sequence with: \xE2\x80[\xA8-\xA9] \xE2\x80[\xA8-\xA9] to remove LINE SEPARATOR and PARAGRAPH SEPARATOR.
-
giorgio79 almost 6 yearsDidn't work with LSEP, for example:
print(preg_replace('/[\x00-\x1F\x7F\xA0]/u', '', " An"));
Dalin's answer worked with [:print:]... Stackoverflow is stripping LSEP it seems... -
ChristoKiwi over 5 yearsThis lib converts UTF-8 accented characters and UTF-8 emoticons to "?" symbols. Fairly serious issue unfortunately.
-
dmnc almost 5 yearsI need to remove every space and nbsp (160), so this
$string = preg_replace('/\s+/u', '', $string);
is enough for me... -
Joe Black almost 5 yearsThis is the best solution I could find so far, but I laso had to add
$s = preg_replace('/(\xF0\x9F[\x00-\xFF][\x00-\xFF])/', ' ', $s);
because of all the emoji characters were messing up mysql -
Mubashar about 4 yearsabove answer was a compliment to original answer which only adds up "delete" character.
-
azerto00 over 3 yearsPerfect ! I was looking for a way to remove unicode 'useless' characters and keep the important one (letters including accent, numbers, special chars) . Thanks for the answer and the documentation link
-
Robert over 2 yearsThis can be useful when only pure words are needed. For example, for a search engine on the page and an index in the database. Parentheses, periods and commas are then unnecessary.
-
HoldOffHunger over 2 yearsThis fails on most whitespace, which renders as a space in browsers but gets stripped out entirely here. Demo of Code Above Stripping Out Whitespace
-
user1432181 over 2 years:print is too restrictive and will loose euro and pound signs among other things. Just use
/[\x00-\x1F\x80-\xC0]/u
-
Volomike about 2 yearsWhy would I want 127, which is DEL ? Wouldn't it be better as
[\x00-\x1F\x7F-\xFF]
to remove 127 to 255 instead of 128 to 255 ? -
StanE about 2 yearsIf you want to check binary data, like a file, you have to remove the "u" modifier from the UTF-8 solution. The documentation says that simply nothing will be matched, but the function seems to return a completely empty result instead. If I'm not wrong, then use the modifier for UTF encoded texts and remove it if using with other binary data, like files.
-
Avatar about 2 yearsUnfortunately the "bad utf-8" Regex above also removes line breaks!