Cowboy Denny Posted November 16, 2022 Share Posted November 16, 2022 The BIG-IP is a 'full' proxy. This means there are 2 separate and independent connections that are managed by the BIG-IP. We refer to those as the Clientside (incoming traffic) and Serverside (outgoing traffic). Whether the traffic originates on your external or internal side is irrelevant to a BIG-IP. It is where the connection originates and hits that virtual server (clientside) and exits the BIG-IP (serverside). So, in most cases to track an issue or resolve a question you need the traffic for both connections and then be able to align that traffic. To do that there are 2 things that will help. One is referred to as the 'p' flag. This will instruct the BIG-IP to catch the flow on both sides of the BIG-IP. The other is to drop the capture into Wireshark and look at the F5 Ethernet headers. To gather those you need the interface modifier :nnn. When using a 'p' flag in the capture syntax, and running the capture on the BIG-IP, it will instruct the BIG-IP to capture the traffic on both sides of the BIG-IP. The syntax is fairly simple to construct. For example, you want to track the traffic for a specific virtual server and its pool members, and it would look something like this... tcpdump -nni 0.0:nnnp -s0 --f5 ssl host 10.17.199.72 or host 10.17.21.59 or host 10.17.21.61 or host 10.17.26.87 or 10.17.24.74 or 10.17.24.75 -vw /var/tmp/tcpdump_VS-POOL_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap Lets break that down... -s0 is an Unlimited Snaplen. SnapLen, Snap Length, or snapshot length is the amount of data for each frame that is actually captured by the network capturing tool and stored into the CaptureFile. This will provide the most data. -nn Don’t convert host addresses to names. This is used to avoid DNS lookups. i 0.0 Capture the traffic on interface 0.0 which tells the BIG-IP to use 'any' interface to gather this traffic on. :nnnp Here you see the 'p' flag and what we call "full noise" by the use of the 'nnn'. This will create the information for the F5 Ethernet Trailers and the ‘p’ the traffic on both sides of the proxy. ‘host’ The IP of the virtual server or source IP. If the virtual server destination is 0.0.0.0:0 (referred to as any:any) then the source IP is what will be needed. ‘port’ the specific port used by the virtual server. This helps to reduce the size of the capture. A virtual IP address can be used many times but must be used in conjunction with a port number to form a websocket. So, if you are using 192.168.1.1:443 for ssl and then using it as 192.168.1.1:161 for SNMP and 192.168.1.1:25 for SMTP you would want to filter out the traffic you do not need to look at by specifying the port. -v will add verbosity and provide and screen counter so you see if packets are being caught, how many, and how fast. -w this will send it to the file location. /var/tmp/hostname the path to the location and the file name. You can name it what you want. But remember it is a Linux system and there are format rules. The file does not need to be created before the capture is started as the system will do it for you. .pcap is the file type. (.cap is still used but is not quite as effective and pcapng is the newest form. In most cases all will work) Important: When using the 'p' flag simply typing Ctrl + c will not stop some background functions of the 'p' flag and this can cause some issues. To completely stop it you can kill the PID or simply stop it by typing... killall tcpdump Note: There are some times in which a 'p' flag cannot be used, such as when HTTP/2 and some other protocols are in play. In that case this article should help you deal with those circumstances...K87524842: The tcpdump 😛 option does not capture peer traffic on some types of flows Note: Beginning in Wireshark 2.6.0, the f5ethtrailer dissector is built into the utility. To display TMM information in Wireshark 2.6.0 and later, navigate to Analyze > Enabled Protocols and search for f5ethtrailer. Click the options to enable the F5 Ethernet trailer. If you are using a Wireshark version before 2.6.0, F5 recommends upgrading your Wireshark to the latest version to make use of this feature. You can also use the TMM information to filter the dump using some additional F5 details. For example, the following Wireshark filter string shows traffic to and from TMM0 on Slot1: f5ethtrailer.slot == 1 and f5ethtrailer.tmm == 0 A list of all F5 filters appears in Wireshark within Filter Expression. Pre-REQ for SSL tmsh modify sys db tcpdump.sslprovider value enable Identify Client IP Address Identify Virtual Server IP Address Identify Pool Members BEGIN the capture If you know the client ip address: tcpdump -ni 0.0:nnnp -s0 --f5 ssl host [client ip address] -w /var/tmp/api-qa_tcpdump_client_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap Otherwise, filter for the virtual ip and pool member ip(s): tcpdump -ni 0.0:nnn -s0 --f5 ssl host [virtual server ip] or host [pool member ip] or host [pool member ip] -w /var/tmp/api-qa_tcpdump_VS_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap When you open the capture file... How to Filter HTTP Traffic in Wireshark Filtering HTTP traffic in Wireshark is a fairly trivial task but it does require the use of a few different filters to get the whole picture. Many people think the http filter is enough, but you end up missing the handshake and termination packets. Wireshark HTTP Protocol Filter To display packets using the HTTP protocol you can enter the following filter in the Display Filter Toolbar: http You’ll notice that all the packets in the list show HTTP for the protocol. The unfortunate thing is that this filter isn’t showing the whole picture. You’re missing the setup handshakes and termination tcp packets. To display all the HTTP traffic you need to use the following protocol and port display filter: tcp.dstport == 443 Now you’ll see all the packets related to your browsing of any HTTPS sites you browsed while capturing. Filtering HTTP Traffic to and from Specific IP Address in Wireshark If you want to filter for all HTTP traffic exchanged with a specific you can use the “and” operator. If, for example, you wanted to see all HTTPS traffic related to a server with an IP of 10.17.199.72 you could use the following filter: tcp.dstport == 443 and ip.addr == 10.17.199.72 Notice only packets with 10.17.199.72 in either the source or destination columns is shown. You can also use the OR or || operators to create an “either this or that” filter. tcp.dstport == 443 || ip.addr == 10.17.199.72 Wireshark HTTP Method Filter If you want to dig into your HTTP traffic you can filter for things like GET, PUT, POST, DELETE, HEAD, OPTIONS, CONNECT, and TRACE. To filter for these methods use the following filter syntax: http.request.method == "requestmethod" For example, if you wanted to filter for just the GET requests, enter the following filter in the Display Filter toolbar: http.request.method == "GET" Now you’re left with all of the GET requests for assets from the website. Viewing HTTP Packet Information in Wireshark Working with the GET Method Filter displayed above, click on a packet in the Packet List Pane and then look at the information in the Packet Details Pane. Expand the Hypertext Transfer Protocol detail: Wireshark HTTP Response Filter One of the many valuable bits of information in a HTTP conversation is the response. This is the code a website returns that tells the status of the asset that was requested. You’ve probably seen things like Error 404 (Not Found) and 403 (Forbidden). These are HTTP responses and only a couple of the many that exist. To filter for all responses enter the following display filter: http.response Notice to the right of the protocol version information there is a column of numbers. These are your response codes. If you see 200 in this example which means the HTTP request was successful. To filter for a specific response, such as a HTTP 200 (OK), HTTP 301 (Moved Permanently), or HTTP 404 (Not Found) use the following display filter: http.response.code == 200 Change 200 to another code to search for that code. Follow the Full HTTP Stream to Match Get Requests with Responses A very handy feature of Wireshark is the ability to view streams in a human readable format from beginning to end. To this, pick a HTTP protocol packet such as the packet containing the 200 response that we saw earlier and right click on it. Click on Follow -> HTTP Stream. You’ll now be presented with a window that shows the entire stream including the GET (red) and HTTP/1.1 200 OK (Blue) As you can see, there is a lot to HTTP traffic and just filtering for the HTTP protocol doesn’t cut it. If you really want to put the whole picture together when troubleshooting problems with accessing websites you have to take a multi-pronged approach. Link to comment Share on other sites More sharing options...
Cowboy Denny Posted November 30, 2022 Author Share Posted November 30, 2022 Capturing encrypted traffic on the F5 SSL DUMP Pre-REQ SSL Profile Session Cache should be enabled (verify) Make sure ssl profile has cache setting enabled tmsh list ltm profile client-ssl ssl.client.app-wildcard-model cache-size ltm profile client-ssl ssl.client.app-wildcard-model { cache-size 262144 <—DEFAULT but can be 262,144 sessions to 4,194,304 sessions } tmsh list ltm profile client-ssl ssl.client.app-wildcard-model cache-timeout ltm profile client-ssl ssl.client.app-wildcard-model { cache-timeout 3600 (seconds) <—DEFAULT } Quote Cache Timeout setting specifies the timeout value in seconds of the SSL session cache entries. The default cache timeout is 3,600 seconds. Note: The cache entries are removed automatically when restarting TMM or reloading the configuration. The timeout of the cache entry is not refreshed when the client connects using the same session ID. When the timeout expires the entry is eligible for removal but may remain until the memory is reclaimed. Note: If you make changes to the SSL profile, and want the changes implemented immediately for all SSL connections that use the modified SSL profile. To clear the SSL session cache for a client-ssl profile, change the cache-timeout value to 0, then change it back to the previous value. tmsh modify ltm profile client-ssl ssl.client.app-wildcard-model cache-timeout 0 tmsh modify ltm profile client-ssl ssl.client.app-wildcard-model cache-timeout 3600 Viewing the SSL session cache tmsh show ltm profile client-ssl ssl.client.app-wildcard-model | grep 'Session Cache' -A 5 Session Cache Current Entries 173 Hits 1.5M Lookups 1.5M Overflows 0 Invalidations 370.1K You must also know the source <client_IP_addr> INSTRUCTIONS Decrypting SSL traffic using the SSL::sessionsecret iRules when SSL session cache feature is enabled in the SSL profile, you can use the following procedure to decrypt SSL traffic using the SSL::sessionsecret iRules command. To do so, perform the following steps: Impact of procedure: Performing the following procedure should not have a negative impact on your system. Run this command to validate you have cache enabled on ssl profile (can not be set to 0): tmsh list ltm profile client-ssl ssl.client.openshift-wildcard-model cache-size Go to Local Traffic > iRules > iRules List Select Create and define an iRule using a SSL::sessionsecret iRules command syntax similar to the following: Note: In the following example, <datagroup_name> is the IP addresses of the remote client accessing the BIG-IP virtual server. when CLIENTSSL_HANDSHAKE { if { [IP::addr [getfield [IP::client_addr] "%" 1] equals <datagroup_name>] } { log local0. "[TCP::client_port] :: RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]" } } when SERVERSSL_HANDSHAKE { # Check if client IP is <client_IP_addr>, in any route domain if { [IP::addr [getfield [IP::client_addr] "%" 1] equals <datagroup_name>] } { log local0. "[TCP::client_port] :: RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]" } } Note: To dump a server-side session key, use SERVERSSL_HANDSHAKE instead of CLIENTSSL_HANDSHAKE. Associate the iRule with the virtual server. From the BIG-IP command line, run the tcpdump command on the BIG-IP system. Example: tcpdump -ni 0.0:nnn -s0 --f5 ssl host 198.74.89.51 or host 10.46.69.31 -w /var/tmp/app.hosangit.com_tcpdump_VS_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap virtual server: 209.59.154.97 pool: 10.46.69.31 From the client system, access the virtual server. Stop the tcpdump command. Download the generated tcpdump file. Perform the Producing a Master Secret log file and decrypting the SSL session procedure. Producing a Master Secret log file and decrypting the SSL session using either of the following methods: You can produce a Master Secret log file and use the file to decrypt the SSL session Producing a Master Secret log file using the sed command Producing a Master Secret log file manually 1. Producing a Master Secret log file using the sed command On the BIG-IP command line, enter the following command: sed -e 's/^.*\(RSA Session-ID\)/\1/;tx;d;:x' /var/log/ltm > /var/tmp/sessionsecrets.pms Note: Run this command exactly as shown. Download the /var/tmp/sessionsecrets.pms file to your local drive. On the client system, open the Wireshark application. Go to Edit > Preferences > Protocols > SSL. Note: In the latest version of Wireshark (3.x and higher) navigate to Edit > Preferences > Protocols > TLS. For (Pre)-Master-Secret log filename, select Browse and locate the session key file you created. Note: If you are using an older version of Wireshark, you may need to manually enter the file information. Select OK. Open the packet capture. The selected SSL sessions are now decrypted. 2. Producing a Master Secret log file manually On the BIG-IP system, locate the SSL session keys in the /var/log/ltm log file. They appear similar to the following example: info tmm1[19365]: Rule /Common/Decrypt_session <CLIENTSSL_HANDSHAKE>: 50194 :: RSA Session-ID:2b14674e4bb505ab97a63ab8fe47cf6f827aa34964871338dc3b5bcc034bd91c Master-Key:7bfcb98c9725de0da25eda2110937550d9097039912ad422c877e20b43d8741572d9467365dac6a6a3e4cc19ec608bfd Note: The previous example represents a single log line. Copy the last section of the log lines, starting with the R in RSA. On the client system, open a text editor such as Notepad. Paste the string into the text editor and ensure each line of the file has the following syntax: RSA Session-ID:<session_id> Master-Key:<master_key> Note: Make sure to carefully check the syntax. If you vary the syntax, Wireshark cannot recognize the file. Save the file. To use the same filename as the previous instructions, name it sessionsecrets.pms Open the Wireshark application. Go to Edit > Preferences > Protocols > SSL. Note: In the latest version of Wireshark (3.x and higher) navigate to Edit > Preferences > Protocols > TLS. For (Pre)-Master-Secret log filename, select Browse and locate the session key file you created. Note: If you are using an older version of Wireshark, you may need to manually enter the file information. Select OK. Open the packet capture. The selected SSL sessions are now decrypted. Link to comment Share on other sites More sharing options...
Recommended Posts