Difference between revisions of "Jq"

From Christoph's Personal Wiki
Jump to: navigation, search
(See also)
 
(9 intermediate revisions by the same user not shown)
Line 87: Line 87:
 
  $ echo '{ "packet_loss": [ {"ips": "10.0.0.10 10.0.0.11 10.0.0.12", "node-17": "3/3" }] }' | jq -r '[.packet_loss[] | .ips] | .[]'
 
  $ echo '{ "packet_loss": [ {"ips": "10.0.0.10 10.0.0.11 10.0.0.12", "node-17": "3/3" }] }' | jq -r '[.packet_loss[] | .ips] | .[]'
 
  10.0.0.10 10.0.0.11 10.0.0.12
 
  10.0.0.10 10.0.0.11 10.0.0.12
 +
 +
===Practical example===
 +
 +
Here is how to print out all the [[:Category:OpenStack|OpenStack]] compute nodes in my example environment:
 +
 +
<pre>
 +
#!/bin/bash
 +
# AUTHOR: Christoph Champ <christoph.champ@gmail.com>
 +
# Requires jq 1.5+
 +
JQ=$(which jq)
 +
 +
OS_AUTH_URL=http://1.2.3.4:5000/v2.0/
 +
OS_TENANT_NAME=admin
 +
OS_USERNAME=admin
 +
OS_PASSWORD=admin
 +
 +
INFO=$(curl -sXPOST "${OS_AUTH_URL}/tokens" \
 +
        -H "Content-Type: application/json" \
 +
        -d "{\"auth\":{\"tenantName\":\"$OS_TENANT_NAME\",\"passwordCredentials\":\
 +
        {\"username\":\"$OS_USERNAME\",\"password\":\"$OS_PASSWORD\"}}}" | \
 +
        ${JQ} -crM '[.access.token.id + "," + (.access.serviceCatalog[] | select(.name == "nova") | .endpoints[].publicURL)] | .[]')
 +
 +
TOKEN=${INFO%%,*}
 +
NOVA_ENDPOINT=${INFO#*,}
 +
 +
IGNORE_ZONES="internal|nova"
 +
 +
raw=$(curl -s -H "X-Auth-Token: ${TOKEN}" "${NOVA_ENDPOINT}/os-availability-zone/detail" | \
 +
    ${JQ} -crM '[.availabilityZoneInfo[].zoneName] | .[]' | \
 +
    grep -vE "(${IGNORE_ZONES})" | tr '\n' ',')
 +
 +
IFS=',' read -r -a zones <<< "${raw%,}"
 +
 +
for zone in "${zones[@]}"; do
 +
    raw=($(curl -s -H "X-Auth-Token: ${TOKEN}" "${NOVA_ENDPOINT}/os-availability-zone/detail" | \
 +
        ${JQ} --arg zone "$zone" '[.availabilityZoneInfo[] | select(.zoneName==$zone) | .hosts|keys] | .[]' | \
 +
        tr -d '[]",' | sed '/^$/d' | tr '\n' ',' | tr -d ' '))
 +
 +
    IFS=',' read -r -a nodes <<< "${raw%,}"
 +
    for node in "${nodes[@]}"; do
 +
        echo "node: $zone $node"
 +
    done
 +
done
 +
</pre>
 +
 +
Running the above script produces the following output:
 +
<pre>
 +
node: az1 node-1.example.com
 +
node: az1 node-2.example.com
 +
node: az2 node-3.example.com
 +
node: az2 node-4.example.com
 +
</pre>
 +
 +
===Append to JSON===
 +
 +
* Example of how to append key/values to an already existing JSON structure:
 +
<pre>
 +
$ cat foo.json
 +
{ "name": "bob", "age": 30 }
 +
 +
$ cat foo.json | jq 'to_entries'
 +
[
 +
  {
 +
    "key": "name",
 +
    "value": "bob"
 +
  },
 +
  {
 +
    "key": "age",
 +
    "value": 30
 +
  }
 +
]
 +
 +
$ cat foo.json | BEARERTOKEN="Bearer abc123" jq 'to_entries | . + [{"key":"routes","value":[{"path":"api/v1","url":"http://example.com","headers":[{"name":"Authorization","content":env.BEARERTOKEN}]}]}] | from_entries'
 +
{
 +
  "name": "bob",
 +
  "age": 30,
 +
  "routes": [
 +
    {
 +
      "path": "api/v1",
 +
      "url": "http://example.com",
 +
      "headers": [
 +
        {
 +
          "name": "Authorization",
 +
          "content": "Bearer abc123"
 +
        }
 +
      ]
 +
    }
 +
  ]
 +
}
 +
</pre>
 +
 +
; Update a specific nested value in a JSON file
 +
<pre>
 +
$ export NEW_URL="https://172.x.x.x:6443"
 +
$ jq --arg new_url "${NEW_URL}" '(.resources[] | select(.type == "rke_cluster") | .instances[].attributes.api_server_url) |= $new_url' foo.json
 +
{
 +
  "resources": [
 +
    {
 +
      "module": "module.rancher",
 +
      "type": "rke_cluster",
 +
      "instances": [
 +
        {
 +
          "attributes": {
 +
            "api_server_url": "https://172.x.x.x:6443",
 +
            "foo": "bar"
 +
          }
 +
        }
 +
      ]
 +
    }
 +
  ]
 +
}
 +
</pre>
 +
 +
==Miscellaneous==
 +
 +
* Get the randomly generated [[Rancher]] admin password, as created by the <code>rancher2</code> [[Terraform]] provider:
 +
$ jq -crM '.resources[] | select(.provider == "module.rancher.provider.rancher2.bootstrap") | {instances: .instances[]|.attributes.current_password} | .[]' terraform.tfstate
 +
 +
* <code>kubectl-neat</code>: Easily copy a [[Kubernetes]] certificate secret to another namespace:
 +
<pre>
 +
$ SOURCE_NAMESPACE=<update-me>
 +
$ DESTINATION_NAMESPACE=<update-me>
 +
$ kubectl -n ${SOURCE_NAMESPACE} get secret kafka-client-credentials -o json |\
 +
    kubectl neat |\
 +
    jq 'del(.metadata["namespace"])' |\
 +
    kubectl apply -n ${DESTINATION_NAMESPACE} -f -
 +
</pre>
 +
 +
==See also==
 +
* [https://github.com/mikefarah/yq yq] &mdash; a portable command-line YAML, JSON, XML, CSV and properties processor.
 +
* [https://github.com/simeji/jid jid] &mdash; a JSON incremental digger.
 +
* [https://github.com/jrockway/kubectl-jq kubectl-jq] &mdash; kubectl plugin that works like <code>kubectl get</code> but runs everything through a JQ program you provide
  
 
==External links==
 
==External links==
 
*[https://stedolan.github.io/jq/ Official website]
 
*[https://stedolan.github.io/jq/ Official website]
 +
*[https://starkandwayne.com/blog/bash-for-loop-over-json-array-using-jq/ Bash For Look Over JSON Array Using Jq]
 +
*[https://jsonnet.org/ Jsonnet] &mdash; an extension of JSON
  
 
[[Category:Linux Command Line Tools]]
 
[[Category:Linux Command Line Tools]]

Latest revision as of 22:14, 25 January 2023

jq is a lightweight and flexible command-line JSON processor. jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep, and friends let you play with text.

Example usage

$ cat azones.json
{
    "availabilityZoneInfo": [
        {
            "hosts": {
                "node-1.example.com": {
                    "nova-compute": {
                        "active": true,
                        "available": true
                    }
                },
                "node-2.example.com": {
                    "nova-compute": {
                        "active": true,
                        "available": true
                    }
                }
            },
            "zoneName": "az1",
            "zoneState": {
                "available": true
            }
        },
        {
            "hosts": {
                "node-3.example.com": {
                    "nova-compute": {
                        "active": true,
                        "available": true
                    }
                },
                "node-4.example.com": {
                    "nova-compute": {
                        "active": true,
                        "available": true
                    }
                }
            },
            "zoneName": "az2",
            "zoneState": {
                "available": true
            }
        }
    ]
}
  • Capture just the availability zone names:
$ cat azones.json | jq '[.availabilityZoneInfo[] | .zoneName]'
[
  "az1",
  "az2"
]

Or, for compact instead of pretty-printed output:

$ cat azones3.json | jq -c '[.availabilityZoneInfo[] | .zoneName]'
["az1","az2"]
  • Capture just the hostname (e.g., "node-1.example.com") key for availability zone "az1":
$ cat azones.json | jq '[.availabilityZoneInfo[] | select(.zoneName == "az1") | {hosts: .hosts|keys}]'
[
  {
    "hosts": [
      "node-1.example.com",
      "node-2.example.com"
    ]
  }
]

Or, for a more script-friendly output:

$ cat azones.json | jq -cM '[.availabilityZoneInfo[] | select(.zoneName == "az1") | {hosts: .hosts|keys}]' | sed -e 's/["}\[]//g;s/\]//g;s/{hosts://g;s/,/ /g'
#~OR~
$ foo=($(cat azones3.json | jq -cM '[.availabilityZoneInfo[] | select(.zoneName == "az1") | {hosts: .hosts|keys}]' | sed -e 's/["}\[]//g;s/\]//g;s/{hosts://g;s/,/ /g'))
$ echo ${foo[0]} #=> node-1.example.com
  • Get just the raw values:
$ echo '{ "packet_loss": [ {"ips": "10.0.0.10 10.0.0.11 10.0.0.12", "node-17": "3/3" }] }' | jq -r '[.packet_loss[] | .ips] | .[]'
10.0.0.10 10.0.0.11 10.0.0.12

Practical example

Here is how to print out all the OpenStack compute nodes in my example environment:

#!/bin/bash
# AUTHOR: Christoph Champ <christoph.champ@gmail.com>
# Requires jq 1.5+
JQ=$(which jq)

OS_AUTH_URL=http://1.2.3.4:5000/v2.0/
OS_TENANT_NAME=admin
OS_USERNAME=admin
OS_PASSWORD=admin

INFO=$(curl -sXPOST "${OS_AUTH_URL}/tokens" \
        -H "Content-Type: application/json" \
        -d "{\"auth\":{\"tenantName\":\"$OS_TENANT_NAME\",\"passwordCredentials\":\
        {\"username\":\"$OS_USERNAME\",\"password\":\"$OS_PASSWORD\"}}}" | \
        ${JQ} -crM '[.access.token.id + "," + (.access.serviceCatalog[] | select(.name == "nova") | .endpoints[].publicURL)] | .[]')

TOKEN=${INFO%%,*}
NOVA_ENDPOINT=${INFO#*,}

IGNORE_ZONES="internal|nova"

raw=$(curl -s -H "X-Auth-Token: ${TOKEN}" "${NOVA_ENDPOINT}/os-availability-zone/detail" | \
    ${JQ} -crM '[.availabilityZoneInfo[].zoneName] | .[]' | \
    grep -vE "(${IGNORE_ZONES})" | tr '\n' ',')

IFS=',' read -r -a zones <<< "${raw%,}"

for zone in "${zones[@]}"; do
    raw=($(curl -s -H "X-Auth-Token: ${TOKEN}" "${NOVA_ENDPOINT}/os-availability-zone/detail" | \
        ${JQ} --arg zone "$zone" '[.availabilityZoneInfo[] | select(.zoneName==$zone) | .hosts|keys] | .[]' | \
        tr -d '[]",' | sed '/^$/d' | tr '\n' ',' | tr -d ' '))

    IFS=',' read -r -a nodes <<< "${raw%,}"
    for node in "${nodes[@]}"; do
        echo "node: $zone $node"
    done
done

Running the above script produces the following output:

node: az1 node-1.example.com
node: az1 node-2.example.com
node: az2 node-3.example.com
node: az2 node-4.example.com

Append to JSON

  • Example of how to append key/values to an already existing JSON structure:
$ cat foo.json 
{ "name": "bob", "age": 30 }

$ cat foo.json | jq 'to_entries'
[
  {
    "key": "name",
    "value": "bob"
  },
  {
    "key": "age",
    "value": 30
  }
]

$ cat foo.json | BEARERTOKEN="Bearer abc123" jq 'to_entries | . + [{"key":"routes","value":[{"path":"api/v1","url":"http://example.com","headers":[{"name":"Authorization","content":env.BEARERTOKEN}]}]}] | from_entries'
{
  "name": "bob",
  "age": 30,
  "routes": [
    {
      "path": "api/v1",
      "url": "http://example.com",
      "headers": [
        {
          "name": "Authorization",
          "content": "Bearer abc123"
        }
      ]
    }
  ]
}
Update a specific nested value in a JSON file
$ export NEW_URL="https://172.x.x.x:6443"
$ jq --arg new_url "${NEW_URL}" '(.resources[] | select(.type == "rke_cluster") | .instances[].attributes.api_server_url) |= $new_url' foo.json
{
  "resources": [
    {
      "module": "module.rancher",
      "type": "rke_cluster",
      "instances": [
        {
          "attributes": {
            "api_server_url": "https://172.x.x.x:6443",
            "foo": "bar"
          }
        }
      ]
    }
  ]
}

Miscellaneous

  • Get the randomly generated Rancher admin password, as created by the rancher2 Terraform provider:
$ jq -crM '.resources[] | select(.provider == "module.rancher.provider.rancher2.bootstrap") | {instances: .instances[]|.attributes.current_password} | .[]' terraform.tfstate
  • kubectl-neat: Easily copy a Kubernetes certificate secret to another namespace:
$ SOURCE_NAMESPACE=<update-me>
$ DESTINATION_NAMESPACE=<update-me>
$ kubectl -n ${SOURCE_NAMESPACE} get secret kafka-client-credentials -o json |\
    kubectl neat |\
    jq 'del(.metadata["namespace"])' |\
    kubectl apply -n ${DESTINATION_NAMESPACE} -f -

See also

  • yq — a portable command-line YAML, JSON, XML, CSV and properties processor.
  • jid — a JSON incremental digger.
  • kubectl-jq — kubectl plugin that works like kubectl get but runs everything through a JQ program you provide

External links