There are times where you will need to troubleshoot a network issue and have to go into the packet detail level to analyse it. To capture the packets, you will usually run
tcpdump either at the source or destination, or even both. However, this gets tricky if you are using containers to run your application or microservice, which are deployed to a Kubernetes cluster such as Amazon EKS (Elastic Kubernetes Service), and if the containers might not have
This blog post will show you how you can capture the container packets from the EKS worker nodes.
Here, we assume your worker nodes to have
tcpdump installed. To verify, SSH into your worker node and run the
tcpdump --version command.
root@ip-10-x-x-93 ~]# tcpdump --version
tcpdump version 4.9.2
libpcap version 1.5.3
OpenSSL 1.0.2k-fips 26 Jan 2017
Identify the worker node of your pod
The first step is to find out which worker node your pod is running on so that we will know where to run
tcpdump later on. In the example below, I only have one worker node and the pod I’m interested in is
kieranyio@Kierans-MBP-2 ~ % kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-7f99dd46c4-s9777 2/2 Running 0 3m45s 10.x.x.132 ip-10-x-x-93.ap-southeast-1.compute.internal <none> <none>
nginx-deployment-test-5dd7b48477-rlv77 1/1 Running 0 2m17s 10.x.x.31 ip-10-x-x-93.ap-southeast-1.compute.internal <none> <none>
Note: In Kubernetes, a pod is a group of one or more containers. You will need to know the name of the pod where your container is.
Identify the container name
If you have multiple containers running in a pod, you will need to find the name of the container that you want to capture the packets from. You can do so by running the command
kubectl describe pod <pod name> and viewing the Containers property. This also works for pods with a single container.
kieranyio@Kierans-MBP-2 ~ % kubectl describe pod nginx-deployment-7f99dd46c4-s9777
Start Time: Wed, 10 May 2023 21:05:43 +0800
Controlled By: ReplicaSet/nginx-deployment-7f99dd46c4
Container ID: containerd://c3786d2d0a0a1f7fce9005e4006e4453466efa7aa0f116c7ead01ba47dc6a130
Image ID: docker.io/library/nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
Host Port: 0/TCP
Started: Wed, 10 May 2023 21:05:44 +0800
Restart Count: 0
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jn7d5 (ro)
Container ID: containerd://43fc84423c1beff9b118498c7f302561999bdb4c22724bba81de877f601a74f2
Image ID: docker.io/library/busybox@sha256:b5d6fe0712636ceb7430189de28819e195e8966372edfc2d9409d79402a0dc16
Host Port: <none>
Identify the container network interface on the worker node
Once we have the container name, we will get a shell to the container by running the command
kubectl exec -i -t <pod name> -c <container name> -- /bin/bash. While inside the container, run the command
cat /sys/class/net/eth0/iflink to retrieve the network interface index.
kieranyio@Kierans-MBP-2 ~ % kubectl exec -i -t nginx-deployment-7f99dd46c4-s9777 -c nginx -- /bin/bash
root@nginx-deployment-7f99dd46c4-s9777:/# cat /sys/class/net/eth0/iflink
With the network interface index, we can locate the network interface on the worker node by SSH-ing into it and running
ip link | grep <network interface index>.
[root@ip-10-x-x-93 ~]# ip link | grep 8
8: enid16b4c965eb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP mode DEFAULT group default
link/ether ea:dc:dc:d8:xx:xx brd ff:ff:ff:ff:ff:ff link-netns cni-c0b05ee1-6ef2-68ea-d1c3-c2d1c5355612
Capture packets from the network interface
Now that we have all the necessary information, we can start running the command
tcpdump -i <network interface> on the worker node to capture the packets. If you want to save it to a file, add the argument
-w <file name>.pcap. To stop the run, press Ctrl + c.
[root@ip-10-x-x-93 ~]# tcpdump -i enid16b4c965eb -w network.pcap
tcpdump: listening on enid16b4c965eb, link-type EN10MB (Ethernet), capture size 262144 bytes
^C128 packets captured
128 packets received by filter
0 packets dropped by kernel
Analyse and view packets on Wireshark
There are multiple ways to extract the pcap file out of the worker node. One way is to upload it to an S3 bucket and download the file from console or via CLI. After extracting, you can use a Wireshark to view and analyse the packets.
Note: For demo purpose, our nginx is using HTTP protocol and that is why we are able to see the content in the packet. In normal circumstances, HTTPS is used and we won’t be able to view the content as it is encrypted.
That’s it. You now know how to capture the container packets from EKS worker nodes.