Log Parsing with Regex
10,536
Solution 1
You need to make your group 7
optional by adding a ?
. Use the following regex:
^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(\S+) (\S+)\s*(\S+)?\s*" (\d{3}) (\S+)
↑
See the DEMO
Solution 2
import re
HOST = r'^(?P<host>.*?)'
SPACE = r'\s'
IDENTITY = r'\S+'
USER = r'\S+'
TIME = r'(?P<time>\[.*?\])'
REQUEST = r'\"(?P<request>.*?)\"'
STATUS = r'(?P<status>\d{3})'
SIZE = r'(?P<size>\S+)'
REGEX = HOST+SPACE+IDENTITY+SPACE+USER+SPACE+TIME+SPACE+REQUEST+SPACE+STATUS+SPACE+SIZE+SPACE
def parser(log_line):
match = re.search(REGEX,log_line)
return ( (match.group('host'),
match.group('time'),
match.group('request') ,
match.group('status') ,
match.group('size')
)
)
logLine = """180.76.15.30 - - [24/Mar/2017:19:37:57 +0000] "GET /shop/page/32/?count=15&orderby=title&add_to_wishlist=4846 HTTP/1.1" 404 10202 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"""
result = parser(logLine)
print(result)
RESULT
('180.76.15.30', '[24/Mar/2017:19:37:57 +0000]', 'GET /shop/page/32/?count=15&orderby=title&add_to_wishlist=4846 HTTP/1.1', '404', '10202')
Author by
Rocky Ray
Updated on June 04, 2022Comments
-
Rocky Ray almost 2 years
I'm trying to parse an Apache Log with regex using Python and assign it to separate variables.
ACCESS_LOG_PATTERN = '^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] "(\S+) (\S+)\s*(\S+)\s*" (\d{3}) (\S+)' logLine='127.0.0.1 - - [01/Jul/1995:00:00:01 -0400] "GET /images/launch-logo.gif HTTP/1.0" 200 1839'
I will parse and group it into the following variable:
match = re.search(APACHE_ACCESS_LOG_PATTERN, logLine) host = match.group(1) client_identd = match.group(2) user_id = match.group(3) date_time = match.group(4) method = match.group(5) endpoint = match.group(6) protocol = match.group(7) response_code = int(match.group(8)) content_size = match.group(9)
The regex pattern is working fine for the log line, but the parsing/regex match fails for the following case:
'127.0.0.1 - - [01/Jul/1995:00:00:01 -0400] "GET /" 200 1839' '127.0.0.1 - - [01/Jul/1995:00:00:01 -0400] "GET / " 200 1839'
How do I fix this?