3 Ways to retrieve values from awscli json output

The AWS CLI (AWS Command Line Interface) is a tool provided by AWS to manage your resources and it is commonly used as part of the DevOps process. For example, assuming a role, baking an image, taking a snapshot, uploading files to s3, increasing auto scaling capacity limit, and the list can go on and on.

By default, when using AWS CLI to run commands, the output will be in json format. There are times where you need to retrieve certain values from the output and use it for your next command but the system does not have the binary that you want to use due to various reasons (e.g. security). In this blog post, we will explore some of the methods that you can use to retrieve values from json .

To be consistent, the AWS CLI command we will be using for all of the examples is aws ec2 describe-network-interfaces --network-interface-ids eni-0c53811a076216ede with the output below and we will be retrieving the PrivateIpAddress value (i.e. 172.31.46.30).

{
    "NetworkInterfaces": [
        {
            "AvailabilityZone": "ap-southeast-1b",
            "Description": "test",
            "Groups": [
                {
                    "GroupName": "private",
                    "GroupId": "sg-12345678910111213"
                }
            ],
            "InterfaceType": "interface",
            "Ipv6Addresses": [],
            "MacAddress": "06:d1:de:87:c0:b0",
            "NetworkInterfaceId": "eni-0c53811a076216ede",
            "OwnerId": "012345678910",
            "PrivateDnsName": "ip-172-31-46-30.ap-southeast-1.compute.internal",
            "PrivateIpAddress": "172.31.46.30",
            "PrivateIpAddresses": [
                {
                    "Primary": true,
                    "PrivateDnsName": "ip-172-31-46-30.ap-southeast-1.compute.internal",
                    "PrivateIpAddress": "172.31.46.30"
                }
            ],
            "RequesterManaged": false,
            "SourceDestCheck": true,
            "Status": "available",
            "SubnetId": "subnet-12345678",
            "TagSet": [],
            "VpcId": "vpc-12345678"
        }
    ]
}

1. AWS CLI query

By itself, the latest AWS CLI has already provided you with a powerful tool to retrieve values by using the --query parameter and it is based on JMESPath syntax to filter out the results. This parameter is available in version 1 and 2 of the AWS CLI.

Example: aws2 ec2 describe-network-interfaces --network-interface-ids eni-0c53811a076216ede --query 'NetworkInterfaces[0].PrivateIpAddress' --output text

Since the PrivateIpAddress is in string format, we have added the --output text parameter as we want to exclude the quotes in the result. Otherwise, quotes will be there and it is not desirable.

Documentation: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html#cli-usage-filter-client-side

2. jq

If you are not comfortable with AWS CLI query or you are looking for a multi purpose tool, jq is another tool that is commonly used to retrieve values in the CLI space. It is easy to use and fairly straight forward. You can also use it to retrieve value from other sources other than AWS CLI output. However, this binary may not always exist in the system, especially for a highly regulated industry. You will have to install it separately.

Example: aws ec2 describe-network-interfaces --network-interface-ids eni-0c53811a076216ede | jq -r ".NetworkInterfaces[0].PrivateIpAddress"

jq example

Here, we are using the -r option as we want to retrieve the raw string. Similar to AWS CLI query, if -r is omitted, jq will print out the value with quotes.

Documentation: https://stedolan.github.io/jq/manual/

3. python

Many operating systems have python installed by default. If you are not keen on installing additional binary, python may be a good choice for you. However, it’s a little bit more complex compared to other methods as you will be doing “python programming” via CLI and it also takes a little bit longer (by milliseconds) to process.

Example: aws ec2 describe-network-interfaces --network-interface-ids eni-0c53811a076216ede | /usr/bin/python -c 'import json,sys;obj=json.load(sys.stdin);print(obj["NetworkInterfaces"][0]["PrivateIpAddress"])'

In the example above, I am using python2 to execute the command. The same also works for python3. You can replace /usr/bin/python with your own python binary path. To retrieve the desired value, simply replace the key(s) of the obj variable like how you would retrieve the value of a particular dictionary key in python.