Capturing container packets from EKS worker nodes using tcpdump

4 min readMay 13, 2023

There are times where you will need to troubleshoot a network issue and have to go into the packet detail level to analyse it. To capture the packets, you will usually run tcpdump either at the source or destination, or even both. However, this gets tricky if you are using containers to run your application or microservice, which are deployed to a Kubernetes cluster such as Amazon EKS (Elastic Kubernetes Service), and if the containers might not have tcpdump installed.

This blog post will show you how you can capture the container packets from the EKS worker nodes.

Here, we assume your worker nodes to have tcpdump installed. To verify, SSH into your worker node and run the tcpdump --version command.

root@ip-10-x-x-93 ~]# tcpdump --version
tcpdump version 4.9.2
libpcap version 1.5.3
OpenSSL 1.0.2k-fips  26 Jan 2017

Identify the worker node of your pod

The first step is to find out which worker node your pod is running on so that we will know where to run tcpdump later on. In the example below, I only have one worker node and the pod I’m interested in is nginx-deployment-7f99dd46c4-s9777.

kieranyio@Kierans-MBP-2 ~ % kubectl get pods -o wide
NAME                                     READY   STATUS    RESTARTS   AGE     IP             NODE                                             NOMINATED NODE   READINESS GATES
nginx-deployment-7f99dd46c4-s9777        2/2     Running   0          3m45s   10.x.x.132   ip-10-x-x-93.ap-southeast-1.compute.internal   <none>           <none>
nginx-deployment-test-5dd7b48477-rlv77   1/1     Running   0          2m17s   10.x.x.31    ip-10-x-x-93.ap-southeast-1.compute.internal   <none>           <none>

Note: In Kubernetes, a pod is a group of one or more containers. You will need to know the name of the pod where your container is.

Identify the container name

If you have multiple containers running in a pod, you will need to find the name of the container that you want to capture the packets from. You can do so by running the command kubectl describe pod <pod name> and viewing the Containers property. This also works for pods with a single container.

kieranyio@Kierans-MBP-2 ~ % kubectl describe pod nginx-deployment-7f99dd46c4-s9777 
Name:         nginx-deployment-7f99dd46c4-s9777
Namespace:    default
Priority:     0
Node:         ip-10-x-x-93.ap-southeast-1.compute.internal/10.x.x.93
Start Time:   Wed, 10 May 2023 21:05:43 +0800
Labels:       app=nginx
              pod-template-hash=7f99dd46c4
Annotations:  <none>
Status:       Running
IP:           10.x.x.132
IPs:
  IP:           10.x.x.132
Controlled By:  ReplicaSet/nginx-deployment-7f99dd46c4
Containers:
  nginx:
    Container ID:   containerd://c3786d2d0a0a1f7fce9005e4006e4453466efa7aa0f116c7ead01ba47dc6a130
    Image:          nginx:1.14.2
    Image ID:       docker.io/library/nginx@sha256:f7988fb6c02e0ce69257d9bd9cf37ae20a60f1df7563c3a2a6abe24160306b8d
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 10 May 2023 21:05:44 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jn7d5 (ro)
  busybox:
    Container ID:  containerd://43fc84423c1beff9b118498c7f302561999bdb4c22724bba81de877f601a74f2
    Image:         busybox
    Image ID:      docker.io/library/busybox@sha256:b5d6fe0712636ceb7430189de28819e195e8966372edfc2d9409d79402a0dc16
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      sleep 9999999
...

Identify the container network interface on the worker node

Once we have the container name, we will get a shell to the container by running the command kubectl exec -i -t <pod name> -c <container name> -- /bin/bash. While inside the container, run the command cat /sys/class/net/eth0/iflink to retrieve the network interface index.

kieranyio@Kierans-MBP-2 ~ % kubectl exec -i -t nginx-deployment-7f99dd46c4-s9777 -c nginx -- /bin/bash
root@nginx-deployment-7f99dd46c4-s9777:/# cat /sys/class/net/eth0/iflink
8
root@nginx-deployment-7f99dd46c4-s9777:/#

With the network interface index, we can locate the network interface on the worker node by SSH-ing into it and running ip link | grep <network interface index>.

[root@ip-10-x-x-93 ~]# ip link | grep 8
...
8: enid16b4c965eb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP mode DEFAULT group default
    link/ether ea:dc:dc:d8:xx:xx brd ff:ff:ff:ff:ff:ff link-netns cni-c0b05ee1-6ef2-68ea-d1c3-c2d1c5355612
...
[root@ip-10-x-x-93 ~]#

Capture packets from the network interface

Now that we have all the necessary information, we can start running the command tcpdump -i <network interface> on the worker node to capture the packets. If you want to save it to a file, add the argument -w <file name>.pcap. To stop the run, press Ctrl + c.

[root@ip-10-x-x-93 ~]# tcpdump -i enid16b4c965eb -w network.pcap
tcpdump: listening on enid16b4c965eb, link-type EN10MB (Ethernet), capture size 262144 bytes
^C128 packets captured
128 packets received by filter
0 packets dropped by kernel
[root@ip-10-x-x-93 ~]#

Analyse and view packets on Wireshark

There are multiple ways to extract the pcap file out of the worker node. One way is to upload it to an S3 bucket and download the file from console or via CLI. After extracting, you can use a Wireshark to view and analyse the packets.

Note: For demo purpose, our nginx is using HTTP protocol and that is why we are able to see the content in the packet. In normal circumstances, HTTPS is used and we won’t be able to view the content as it is encrypted.

That’s it. You now know how to capture the container packets from EKS worker nodes.