Python shlex.split(), ignore single quotes

Question

How, in Python, can I use shlex.split() or similar to split strings, preserving only double quotes? For example, if the input is "hello, world" is what 'i say' then the output would be ["hello, world", "is", "what", "'i", "say'"].

Solution 1

import shlex

def newSplit(value):
    lex = shlex.shlex(value)
    lex.quotes = '"'
    lex.whitespace_split = True
    lex.commenters = ''
    return list(lex)

print newSplit('''This string has "some double quotes" and 'some single quotes'.''')

Solution 2

You can use shlex.quotes to control which characters will be considered string quotes. You'll need to modify shlex.wordchars as well, to keep the ' with the i and the say.

import shlex

input = '"hello, world" is what \'i say\''
lexer = shlex.shlex(input)
lexer.quotes = '"'
lexer.wordchars += '\''

output = list(lexer)
# ['"hello, world"', 'is', 'what', "'i", "say'"]

Comments

[Matt Ball]

Only negligibly so, and the lex.commenters bit is actually something that my answer doesn't do. +1 for a different way to git-r-dun.

[tekknolagi]

actually, this one failed on something else I was writing, and Peter's worked. thank you though!

[Matt Ball]

Out of curiosity, what input did it fail on?

[Peter Lyons]

I started with the source code to the shlex.split function from the python source and just tweaked it with the list of quotes characters.

[Pykler]

It took me a while, but this is slightly different than the normal shlex.split which passes posix ... shlex.shlex(value, posix=True)

[user14717]

If the escaped quote is in the quoted section, is there a way forward? Say, s = "1 'K\^o, Suzuk\'e'" to split into [1, "K\^o, Suzuk\'e"]