Palo Alto: Resolving URL Category in Decryption Policy When Multiple URLs are Behind the Same IP

<< Issue >>
Problem happens when there are multiple web services behind the same IP, as with Google who hosts all its services (such as Drive, Translate, Search engine, Google+, Maps, Play, Gmail, Calendar etc.) behind the same group of IP addresses.

In cases where DNS resolves both www.google.com and www.drive.google.com in same IP address (eg. 173.194.78.189), hosts will use the same IP for both google.com and drive.google.com. So, if the first session traffic is to www.google.com, the local cache will map 173.194.78.189 to “search-engines”. Then, if the next host goes to www.drive.google.com using the same destination IP, the URL category will be resolved in “search-engines” instead of “online-personal-storage”.

If there is decryption policy that is set to decrypt only “online-personal-storage” category, this combination of traffic will not by hit and thus real drive.google.com data will not be decrypted.


<< Details >>
When troubleshooting SSL decryption related issues, a good starting point is to understand how decryption mechanism works in terms of URL categorization. In order to establish a secure SSL tunnel, the client and server perform a certain method of authentication. The client usually authenticates server’s identity based on its certificate. HTTPS connection is always initiated by the client who first resolves server’s URL and then sends a Client Hello towards the resolved IP address. The client then waits for response from server side, which should include its certificate.

In order to resolve proper URL category and determine whether or not to decrypt certain SLL traffic, the Palo Alto Networks firewall relies on the Common Name (CN) field of certificate received from the server. So, URL categorization is based on what is found in CN field. The resolved URL category is then mapped to the destination IP of intercepted packet sent from client side. In order to speed up the process of resolving URL category, the firewall stores each URL to the destination IP mapping in its local cache memory. So, the next time there is SSL traffic to the same destination, it will be resolved in the URL category already stored in local cache file.  Having said that, mechanism of URL categorization for purpose of decryption looks like the following:

1. Client Hello message is intercepted by the firewall
2. Firewall determines packet’s destination IP
3. Firewall compares that destination IP with the list of IP to URL category mapping from its local cache memory
4. If the same IP is in the list, the URL category is then taken from local cache memory
5. If there is no match with local cache, the firewall waits for a response from the server to take a look in the server certificate's CN field
6. URL resolution is done based on CN field, and that category is mapped to Server’s IP and added to the list in local cache memory for future use

<< Resolution >>
In PAN-OS 6.0 a new method of resolving URL category for purpose of decryption was introduced. This new method is not based on the server's certificate CN field but on the SNI value of client's HTTP Hello message. Using this method ensures that under each circumstance, the Palo Alto Networks firewall will be able to properly resolve the URL category of upstream traffic and, with that information, engage right decryption policy.


Reference:
Resolving URL Category in Decryption Policy When Multiple URLs are Behind the Same IP
https://live.paloaltonetworks.com/docs/DOC-7235