debugging ASA firewall rules (with or without ASDM)

Question

Is there any way to debug ASA firewall rule application? I have created 2 simple access rules: allow any ICMP and allow any UDP.
The first one works, I can ping. The udp doesn't work. Running a trace (simulated packet) in ASDM shows that the packet is dropped by the implicit reject rule, but I don't understand why does it not match my any to any UDP rule? Can I enable logging of rule evaluation?

Here's the piece of configuration which I think is relevant (sorry, not a Cisco expert, using ASDM):

access-list Split-tunnel-ACL standard permit 10.65.0.0 255.255.0.0 
access-list outside_access_in extended permit icmp any any 
access-list outside_access_in remark test
access-list outside_access_in extended permit udp host x.x.x.x host y.y.y.y

I also try any any instead of x.x.x.x and y.y.y.y no different. Packet trace says that packet is dropped by implicit deny rule on the access checking stage. The icmp rule is working.

More data:

    Result of the command: "packet-tracer input  outside udp x.x.x.x    5060    y.y.y.y 5060 detailed"

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 2
Type: ACCESS-LIST
Subtype: 
Result: DROP
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0xad31d370, priority=111, domain=permit, deny=true
    hits=28380, user_data=0x0, cs_id=0x0, flags=0x4000, protocol=0
    src ip/id=0.0.0.0, mask=0.0.0.0, port=0
    dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0
    input_ifc=outside, output_ifc=outside

Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

---

Result of the command: "show access-list"

access-list cached ACL log flows: total 0, denied 0 (deny-flow-max 4096)
            alert-interval 300
access-list Split-tunnel-ACL; 1 elements; name hash: 0xaa04f5f3
access-list Split-tunnel-ACL line 1 standard permit xxx.xx5.0.0 255.255.0.0 (hitcnt=6240) 0x9439a34b 
access-list outside_access_in; 2 elements; name hash: 0x6892a938
access-list outside_access_in line 1 extended permit icmp any any (hitcnt=0) 0x71af81e1 
access-list outside_access_in line 2 remark test
access-list outside_access_in line 3 extended permit udp host x.x.x.x host y.y.y.y (hitcnt=0) 0x9fbf7dc7 
access-list inside_nat0_outbound; 4 elements; name hash: 0x467c8ce4
access-list inside_nat0_outbound line 1 extended permit ip object City-network object Remote-mgmt-pool 0x1c53e4c4 
  access-list inside_nat0_outbound line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 192.168.2.0 255.255.255.248 (hitcnt=0) 0x1c53e4c4 
access-list inside_nat0_outbound line 2 extended permit ip object City-network object City2-network 0x278c6c43 
  access-list inside_nat0_outbound line 2 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx2.0.0 255.255.0.0 (hitcnt=0) 0x278c6c43 
access-list inside_nat0_outbound line 3 extended permit ip object City-network object City1-network 0x2b77c336 
  access-list inside_nat0_outbound line 3 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx1.0.0 255.255.0.0 (hitcnt=0) 0x2b77c336 
access-list inside_nat0_outbound line 4 extended permit ip object City-network object City3-network 0x9fdd4c28 
  access-list inside_nat0_outbound line 4 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx5.0.0 255.255.0.0 (hitcnt=0) 0x9fdd4c28 
access-list outside_cryptomap; 1 elements; name hash: 0x39bea18f
access-list outside_cryptomap line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 object City1-network 0x12693b9a 
  access-list outside_cryptomap line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx1.0.0 255.255.0.0 (hitcnt=265) 0x12693b9a 
access-list inside_nat_outbound; 1 elements; name hash: 0xb64b365a
access-list inside_nat_outbound line 1 extended permit tcp object City-network any eq smtp 0x4c753adf 
  access-list inside_nat_outbound line 1 extended permit tcp xxx.xx5.0.0 255.255.0.0 any eq smtp (hitcnt=0) 0x4c753adf 
access-list outside_cryptomap_1; 1 elements; name hash: 0x759febfa
access-list outside_cryptomap_1 line 1 extended permit ip object City-network object City-network 0x4b257004 
  access-list outside_cryptomap_1 line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx5.0.0 255.255.0.0 (hitcnt=0) 0x4b257004 
access-list outside_cryptomap_2; 1 elements; name hash: 0x4e1c27f3
access-list outside_cryptomap_2 line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 object City4-network 0xa82be620 
  access-list outside_cryptomap_2 line 1 extended permit ip xxx.xx5.0.0 255.255.0.0 xxx.xx3.0.0 255.255.0.0 (hitcnt=25) 0xa82be620 

Solution

Your tracer is coming back with input_ifc=outside, output_ifc=outside because it has no other routing information for the destination address, and your outside_access_in ACL has hit counts of 0 on both entries; ICMP is not working, at least, not via this ACL.

Definitely need to see that NAT rule (and the associated ACL if it's a policy NAT).

Is it using a dedicated address for this, or the firewall's interface address? It's not getting this far yet, but, we'll also want to confirm that the correct routing information is there to get traffic to the post-destination address, too; this will be automatic if the server's in the same subnet as the firewall's inside interface.

Comments

[ravi yarlagadda]

Debug logging doesn't go that far - it would become quickly out of control with more than a few rules. Can you show us the relevant parts of config and the packet tracer output?

[user3824502]

@Shane I tried updating the question with the details you asked for, let me know if I failed )

[ravi yarlagadda]

Some more details that would be helpful: the exact output of packet-tracer, doesn't need to be the full command but the whole "dropped" section would be great; any and all NAT rules for the inside and outside interfaces (lines starting with "nat" or "static"); the relevant output of the show access-list command. Basically, lots more of the config. I know it's a pain to scrub all the sensitive data, but it's hard to get a good idea of what the issue is without it.

[user3824502]

Added some more details. There is also a NAT rule which rewrites y.y.y.y into a local address but as far as I understand we are failing well before the NAT part.

[user3824502]

icmp is definitely working and I saw the hit count go up on that rule. I think it resets itself after a while. I agree that it is weird that both interfaces show as outside, but why would it not know where the dest add... oh. Hold on.

[user3824502]

So I understand now why it doesn't work: I'm assigned 2 separate networks by the ISP, one is for the firewall itself and the other for the ips that I can assign to my devices. I have to somehow tell ASA about the second network so that it knows to forward these packets to the inside, but how do I do that?

[user3824502]

So am I right that it doesn't know about the 2nd subnet assigned to me or not? Do I need to do something special or NAT should be enough? My NAT rule is simply "nat (inside,outside) source static any any destination static BorderNet-Outside BorderNet-Inside" where BorderNet-Outside is the y.y.y.y address and BorderNet-Inside is on the very simple inside network.

[ravi yarlagadda]

If it receives packets on the interface (ie, the ISP is correctly routing them to your firewall) then it will have no problem with translating or routing them. (but, do they route to your firewall, or are they expecting you to route the "device address" traffic to a different gateway address?) Since it's clearly not functioning correctly even with just a packet capture, the issue is in your config, and you can cross the ISP routing bridge when you get to it. The NAT config is important, and I'd like to ask once more that you provide it.

[ravi yarlagadda]

Sorry, didn't see your most recent post. Your representation of the static command doesn't have the interfaces, which is important. But, if it is what you say it is then your inside and outside IPs are backwards. It should look like this: static (inside,outside) 20.5.5.5 10.5.5.5 netmask 255.255.255.255 (where 20.5.5.5 is the public address and 10.5.5.5 is the internal address). Please, post the actual config info with the sensitive details changed - you wouldn't be asking for help here if you were 100% certain that that part of the config is correct.

[user3824502]

Shane, thank you very much, nat rule was in fact reversed: i wasn't even checking it because I assumed that it is failing before it even would consider NAT rules. Connection is going through now (but is dropped by SIP inspection...so I might be back here soon :)