my-docs/Linux/Advanced-Administration/02-jq.md

1. Overview

jq is a lightweight and powerful command-line tool for parsing, filtering, transforming, and formatting JSON data.

In DevOps workflows, jq is commonly used to:

• Analyze logs (Docker, Kubernetes, application logs)
• Filter observability data (metrics/events in JSON format)
• Debug CI/CD pipelines
• Process API responses (AWS, GitHub, Terraform outputs)
• Transform JSON for automation scripts

It is essentially the “grep + awk + sed” equivalent for JSON.

---

2. Installation

Linux (Debian/Ubuntu)

sudo apt-get update
sudo apt-get install jq

RHEL/CentOS

sudo yum install jq

macOS

brew install jq

Verify installation

jq --version

---

3. Basic Syntax

jq '<filter>' file.json

Or pipe input:

cat file.json | jq '<filter>'

---

4. Core Concepts

4.1 Identity filter

Returns input as-is:

jq '.'

4.2 Access fields

jq '.name'
jq '.user.id'

4.3 Arrays

jq '.items[]'

4.4 Pretty print

jq '.'

---

5. Filtering Logs (DevOps Use Case)

Example log entry

{
  "level": "error",
  "service": "auth",
  "message": "invalid credentials",
  "status": 401,
  "timestamp": "2026-04-15T10:00:00Z"
}

Filter only errors

jq 'select(.level == "error")'

Filter by service

jq 'select(.service == "auth")'

Extract specific fields

jq '{time: .timestamp, msg: .message}'

---

6. Working with Arrays (Common in Logs)

Example: multiple log entries

Count entries

jq 'length'

Filter array elements

jq '.[] | select(.status >= 500)'

Extract fields from array

jq '.[] | {service, status, message}'

---

7. Kubernetes Logs with jq

Example:

kubectl logs pod-name -n default | jq

Filter error logs

kubectl logs pod-name | jq 'select(.level=="error")'

Extract container metadata logs

kubectl logs pod-name | jq '{time, container, message}'

---

8. Docker Logs with jq

Streaming logs

docker logs container_name | jq

Filter failures

docker logs container_name | jq 'select(.status != "success")'

---

9. AWS / Cloud Logs (JSON-based)

Example CloudWatch JSON logs

aws logs filter-log-events --log-group-name my-app | jq

Extract messages only

... | jq '.events[].message'

Filter by keyword

... | jq '.events[] | select(.message | contains("ERROR"))'

---

10. Transforming JSON (Automation Use Cases)

Rename fields

jq '{userId: .id, username: .name}'

Add computed fields

jq '. + {isActive: true}'

Build new structure

jq '{users: [.[] | {id, name}]}'

---

11. Advanced Filtering

Logical conditions

jq 'select(.status == 200 and .service == "api")'

Regex matching

jq 'select(.message | test("timeout|failed"))'

Sorting

jq 'sort_by(.timestamp)'

Unique values

jq 'unique_by(.service)'

---

12. Aggregations (DevOps Analytics)

Count by status

jq 'group_by(.status) | map({status: .[0].status, count: length})'

Error rate estimation

jq 'map(select(.status >= 400)) | length'

---

13. Formatting Output for Humans

Compact JSON

jq -c '.'

Raw output (no quotes)

jq -r '.message'

Tabular-like output

jq -r '[.timestamp, .level, .message] | @tsv'

---

14. Debugging Pipelines

Validate JSON

jq empty file.json

Highlight structure

jq '. | type'

Pretty inspect nested structures

jq 'paths'

---

15. DevOps Best Practices

1. Always validate JSON first

jq empty

2. Use -c in pipelines

Reduces log noise:

jq -c '.'

3. Use -r for scripting

jq -r '.field'

4. Combine with grep when needed

grep ERROR app.log | jq

5. Avoid unnecessary formatting in CI/CD

Keep output machine-readable.

---

16. Common Patterns Cheat Sheet

Task	Command
Pretty print	jq '.'
Filter by field	jq 'select(.field=="value")'
Extract field	jq '.field'
Array iteration	jq '.[]'
Count items	jq 'length'
Convert to text	jq -r '.field'
Compact output	jq -c '.'

---

17. Real DevOps Example Pipeline

Analyze application logs

cat app.log | jq -c 'select(.level=="error") | {time, service, message}'

Kubernetes debugging

kubectl logs my-pod | jq -c 'select(.status>=500)'

CI/CD artifact inspection

cat terraform-output.json | jq '.outputs'