tcpdump virtual server and pool members

[Cowboy Denny]

We get asked all the time to take a tcpdump of the app on the F5 since they always think its the F5 (which maybe 1% of the time it is)

So this is how I do it (which may be the wrong way but it works for me)

Step 1: Enable ssl decrypt (ONLY IF TROUBLESHOOTING SSL ISSUES)

It is often necessary to create a decrypted capture in order to track down an issue while troubleshooting.  DB Variable tcpdump.sslprovider has been introduced and will cause the LTM to save TLS Master Secret, Client Random and Server Random data to the end of each TLS Packet.  In order to leverage the new functionality, add --f5 ssl to the tcpdump flags.

tmsh modify sys db tcpdump.sslprovider value enable

Important NOTICE: When you perform a tcpdump capture with tcpdump.sslprovider enabled, understand that the TLS master secret will be written to the tcpdump capture itself. Be careful with whom you share the capture file.  Using the "ssl" option captures additional information related to the SSL/TLS connections, such as master secrets. This enables some packet capture analysis tools to decrypt the SSL/TLS payload in the captured packets. Use only as needed for troubleshooting purposes, and handle captured data with caution.
 

Step 2: Create datagroup with Source IP's and/or Subnets

To much traffic to capture if you don't limit/filter down what you are capturing and you don't ever want to fill up your /shared partition because bad things happen.

Easiest way is to have a , seperated txt file with you IP Addresses or subnets..  below is an example of the different txt files.  Since datagroup can only be IP Address, String or Integer (never combined) then which ever one you are building (in my case it was IP Address) you have an example below.  Note every entry has key, value (note address you don't need a value but you can) and key and value are seperated in your text file with := 

Address

# cat ext_dg_address.txt
host 192.168.1.1,
host 192.168.1.2 := "host 2",
network 192.168.2.0/24,
network 192.168.3.0 mask 255.255.255.0 := "network 3",
network 192.168.4.0 prefixlen 24,

IMPORT using: tmsh create /sys file data-group ext_dg_address_file separator ":=" source-path file:/var/tmp/ext_dg_address.txt type ip

String

# cat ext_dg_string.txt
"name1" := "value1",
"name2" := "value2",
"name3" := "value3",

IMPORT using: tmsh create /sys file data-group ext_dg_string_file separator ":=" source-path file:/var/tmp/ext_dg_string.txt type string

Integer

# cat ext_dg_integer.txt
1 := "test 1",
2 := "test 2",
3 := "test 3",

IMPORT using: tmsh create /sys file data-group ext_dg_integer_file separator ":=" source-path file:/var/tmp/ext_dg_integer.txt type integer

Step 3: Create an iRule to decrypt the data

First you need to identify if you are using ssl cache (if its enabled) which more than likely it is.  when you run the tmsh command below and you get a cache-size more than 0 then cache is enabled.

tmsh list ltm profile client-ssl <ssl client profile> cache-size
ltm profile client-ssl <ssl client profile> {
    cache-size 262144 <—DEFAULT but can be 262,144 sessions to 4,194,304 sessions
}

Your iRule should look like this (based on the datagroup you created above for address named ext_dg_address_file)

when CLIENTSSL_HANDSHAKE {
    if {[class match [getfield [IP::client_addr] "%" 1] equals ext_dg_address_file] } { 
        log local0. "CLIENT_Side_IP:TCP source port: [IP::client_addr]:[TCP::remote_port]"
        log local0. "CLIENT_RANDOM [SSL::clientrandom] [SSL::sessionsecret]"
        log local0. "RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]"
        }
}
when SERVERSSL_HANDSHAKE {
    if {[class match [getfield [IP::client_addr] "%" 1] equals ext_dg_address_file] } { 
        log local0. "CLIENT_Side_IP:TCP source port: [IP::client_addr]:[TCP::remote_port]"
        log local0. "CLIENT_RANDOM [SSL::clientrandom] [SSL::sessionsecret]"
        log local0. "RSA Session-ID:[SSL::sessionid] Master-Key:[SSL::sessionsecret]"
        }
}

 

Now you are prepped for Implementation/Testing.

Is your application tester ready to begin testing the application that you would like to capture an error or the traffic?  If yes, then proceed

Add your newly created iRule to the Virtual Server you want to decrypt and capture traffic

Then Start your capture

Run tcpdump (remove -f5 ssl from commands below if you don't need the SSL traffic decrypted)

tcpdump -ni 0.0:nnn -s0 --f5 ssl host [virtual server IP] or host [pool member 1 IP] or host [pool member 2 IP] -W 10 -C 100 -w /var/tmp/app_tcpdump_VS_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap

-W 10 (means rotate up to 10 files)

-C 100 (means each file is up to 100MB in size)

-w write output of capture to that file

Alternative1 command for Step2 that should show packets on the screen while also sending to the .pcap file

tcpdump -ni 0.0:nnn -s0 --f5 ssl host [virtual server IP] or host [pool member 1 IP] or host [pool member 2 IP] -U -w - | tee /var/tmp/app_tcpdump_VSandMEMBERS_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pcap | tcpdump -r -

Alternative2, an additional command for Step2 that could be used to see packets on the screen while also writing to a pcap file

tail -F -n+0 $pcapfile | tcpdump -r -

 

AFTER you capture the error and press CTRL + C to break out of tcpdump you have a few things left to do

capture the pre-master shared keys by running

sed -e 's/^.*\(RSA Session-ID\)/\1/;tx;d;:x' /var/log/ltm > /var/tmp/sessionsecrets_$(date +%d_%b_%H_%M_%S)_$HOSTNAME.pms

 Download the .pms file you just created

 

NEXT merge all the pcap files since you should have no more than 10 pcap files that end with pcap0, pcap1, pcap2, etc.. and you merge these files by running a command like this

mergecap -w /var/tmp/app_tcpdump_VS_10_Nov_20_53_17__server1.hosangit.com.pcap $(ls /var/tmp/app_tcpdump_VS_10_Nov_20_53_17__server1.hosangit.com.pcap*|xargs)

Download the .pcap file you just created

 

Final step on the F5 is to remove the iRule you created above from the Virtual Server.

 

HOW TO VIEW Capture in Wireshark

Open the .pcap you downloaded and click on Wireshark - Preferences - Protocols and scroll down until you find TLS

Now while you are on TLS you should see (Pre)-Master-Secret log filename click Browse to the right of that and click on the .pms file you downloaded and click OK