"Illegal Byte sequence" error while using shell commands in mac bash terminal
16,195
It seems that a UTF-8 locale is causing Illegal byte sequence
.
Instead say:
LC_CTYPE=C your_command
man locale
says:
These environment variables affect each locale categories for all
locale-aware programs:
LC_CTYPE
Character classification and case conversion.
Related videos on Youtube
Author by
Abhineet Prasad
Indie Game Developer/Designer, currently working on Hypercasual games, Co-founder at Dastan Games Projects: Creator of Upcoming Game of the year Award winning title "The Light Inside Us" https://itunes.apple.com/app/id1013478481 Other Games :Number Rain, Steps & Dragons, Ice Slide, Hit & Run, Super Soccer ** ** Previous Experience: Game developer at Kiwi Inc. Senior Technology Associate at Morgan Stanley @abhineetprasad (twitter)
Updated on June 20, 2022Comments
-
Abhineet Prasad almost 2 years
Getting "illegal byte sequence" error while trying to extract non English characters from a large file in MacOS bash shell. This is the script that I am trying to use:
sed 's/[][a-z,0-9,A-Z,!@#\$%^&*(){}":/_-|. -][\;''=?]*//g' < $1 >Abhineet_extract1.txt; sed 's/\(.\)/\1\ /g' <Abhineet_extract1.txt | sort | uniq |tr -d '\n' >&1; rm Abhineet_extract1.txt;
and here is the error that I am getting:
uniq: stdin: Illegal byte sequence
'+?
-
Abhineet Prasad over 10 yearsThanks for your help. The error is gone but the output now contains only '+? characters. I was feeding the output from a sqlite query to the script. I formatted the output to csv and then my script started working.
-
devnull over 10 yearsIt's not very clear what you're saying. Please update your question instead.
-
TMin almost 6 yearsI ran into this issue while deleting some sensitive data from my git history using
git filter-branch --tree-filter "find . -type f -exec sed -i -e 's/originalpassword/newpassword/g' {} \;"
and it worked like a charm -
MarcusJ over 5 yearsSo, is this an issue with Sort? does FreeBSD still suffer from this problem? or should I just file a bug with Apple and tell them to update their ancient ass bin/utils?