Python-Scapy or the like-How can I create an HTTP GET request at the packet level

Question

I am a moderate programmer, just getting into network programming.

As an attempt to improve my understanding of networks in general, I am trying to perform several basic HTTP actions from the packet level. My question is this: How might I use a library such as SCAPY to build an HTTP GET request and assosciated items at the packet level? I realise this may sound odd, but I can't seem to find any information detailing it, and my own attempts with PAROS and Ethereal have been... Less than satisfactory.

Thanks for any offered help!

Trimiert

Solution 1

If you want to do a full three-way handshake, you'll have to do it manually.

Start with your SYN packet:

>>> syn = IP(dst='www.google.com') / TCP(dport=80, flags='S')
>>> syn
<IP  frag=0 proto=tcp dst=Net('www.google.com') |<TCP  dport=www flags=S |>>

Then receive the SYN-ACK packet from the server, sr1 works. Then send your HTTP GET request:

>>> syn_ack = sr1(syn)
Begin emission:
Finished to send 1 packets.
*
Received 1 packets, got 1 answers, remaining 0 packets

>>> syn_ack
<IP  version=4L ihl=5L tos=0x0 len=44 id=424 flags= frag=0L ttl=55 proto=tcp chksum=0x2caa src=74.125.226.148 dst=10.20.30.40 options=[] |<TCP  sport=www dport=ftp_data seq=3833491143 ack=1 dataofs=6L reserved=0L flags=SA window=5720 chksum=0xd8b6 urgptr=0 options=[('MSS', 1430)] |<Padding  load='\x00\x00' |>>>

Then set your TCP sequence and ack numbers and send the GET:

getStr = 'GET / HTTP/1.1\r\nHost: www.google.com\r\n\r\n'
request = IP(dst='www.google.com') / TCP(dport=80, sport=syn_ack[TCP].dport,
             seq=syn_ack[TCP].ack, ack=syn_ack[TCP].seq + 1, flags='A') / getStr
reply = sr1(request)

Solution 2

FTR, as of Scapy 2.4.3, dissection of HTTP packets was implemented, among a util called "TCP_client" to do the 3 handshake automatically.

While it's not as teaching as the above answer, it doesn't hurt to have a look: https://scapy.readthedocs.io/en/latest/layers/http.html#use-scapy-to-send-receive-http-1-x

load_layer("http")
req = HTTP()/HTTPRequest(
    Accept_Encoding=b'gzip, deflate',
    Cache_Control=b'no-cache',
    Connection=b'keep-alive',
    Host=b'www.secdev.org',
    Pragma=b'no-cache'
)
a = TCP_client.tcplink(HTTP, "www.secdev.org", 80)
answser = a.sr1(req)
a.close()

Comments

[orlp]

I would suggest to do this in a lower-level language than Python, like C/C++.

[nmichaels]

@nightcracker: I strongly disagree. Scapy lets you build and send packets very easily without worrying about any of the stuff a C or C++ programmer needs. If you're interested in how the networks work, Scapy is the way to go. If you want to write an actual server, then maybe a different language is more appropriate. And I say this as someone who writes C code for network devices for a living.

[orlp]

@OP: Listen to what nmichaels says. I'm just a 16 year old with an (unbased) opinion about everything.

[Trimiert]

Thank you very much! By far the best information I've seen yet. And yes, I did plan to do the three-way shake myself, but that was as much as I had. Thank you!

[Trimiert]

Yes, I did. But just sending it like that didn't seem to work, and doesn't give me any idea of how it works at the packet level.

[fkl]

Is there are way to capture an http packet using sniff and latter insert http headers into it?

[nmichaels]

@fayyazkl: That's the kind of thing that should be its own question.

[Thomas Wagenaar]

How does one print the anwser of this get request? I'm having trouble doing so!

[nmichaels]

@ThomasW: You can print reply directly to dump the raw packet data, or repr(reply) to get something that looks like what you'd type into Scapy to make the same packet.

[Rag]

Note that when you get the SYN/ACK back from the server, your kernel's TCP/IP stack may freak out at the rogue connection and RST the connection. You may need to block RSTs. stackoverflow.com/questions/9058052/…

[User366]

Does not work for me either. Just tries to receive forever.

[sahil shekhawat]

The question asks how to create a http get request at packet level and this answer clarifies this. The fact that you need to do tcp handshake is assumed. Don't think there is anything wrong with the response. Although, people who don't know about tcp handshake must refer to the correct answer.