added jq documents
This commit is contained in:
386
Linux/Advanced-Administration/02-jq.md
Normal file
386
Linux/Advanced-Administration/02-jq.md
Normal file
@@ -0,0 +1,386 @@
|
||||
## 1. Overview
|
||||
|
||||
jq is a lightweight and powerful command-line tool for parsing, filtering, transforming, and formatting JSON data.
|
||||
|
||||
In DevOps workflows, `jq` is commonly used to:
|
||||
|
||||
* Analyze logs (Docker, Kubernetes, application logs)
|
||||
* Filter observability data (metrics/events in JSON format)
|
||||
* Debug CI/CD pipelines
|
||||
* Process API responses (AWS, GitHub, Terraform outputs)
|
||||
* Transform JSON for automation scripts
|
||||
|
||||
It is essentially the “grep + awk + sed” equivalent for JSON.
|
||||
|
||||
---
|
||||
|
||||
## 2. Installation
|
||||
|
||||
### Linux (Debian/Ubuntu)
|
||||
|
||||
```bash
|
||||
sudo apt-get update
|
||||
sudo apt-get install jq
|
||||
```
|
||||
|
||||
### RHEL/CentOS
|
||||
|
||||
```bash
|
||||
sudo yum install jq
|
||||
```
|
||||
|
||||
### macOS
|
||||
|
||||
```bash
|
||||
brew install jq
|
||||
```
|
||||
|
||||
### Verify installation
|
||||
|
||||
```bash
|
||||
jq --version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Basic Syntax
|
||||
|
||||
```bash
|
||||
jq '<filter>' file.json
|
||||
```
|
||||
|
||||
Or pipe input:
|
||||
|
||||
```bash
|
||||
cat file.json | jq '<filter>'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Core Concepts
|
||||
|
||||
### 4.1 Identity filter
|
||||
|
||||
Returns input as-is:
|
||||
|
||||
```bash
|
||||
jq '.'
|
||||
```
|
||||
|
||||
### 4.2 Access fields
|
||||
|
||||
```bash
|
||||
jq '.name'
|
||||
jq '.user.id'
|
||||
```
|
||||
|
||||
### 4.3 Arrays
|
||||
|
||||
```bash
|
||||
jq '.items[]'
|
||||
```
|
||||
|
||||
### 4.4 Pretty print
|
||||
|
||||
```bash
|
||||
jq '.'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Filtering Logs (DevOps Use Case)
|
||||
|
||||
### Example log entry
|
||||
|
||||
```json
|
||||
{
|
||||
"level": "error",
|
||||
"service": "auth",
|
||||
"message": "invalid credentials",
|
||||
"status": 401,
|
||||
"timestamp": "2026-04-15T10:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Filter only errors
|
||||
|
||||
```bash
|
||||
jq 'select(.level == "error")'
|
||||
```
|
||||
|
||||
### Filter by service
|
||||
|
||||
```bash
|
||||
jq 'select(.service == "auth")'
|
||||
```
|
||||
|
||||
### Extract specific fields
|
||||
|
||||
```bash
|
||||
jq '{time: .timestamp, msg: .message}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Working with Arrays (Common in Logs)
|
||||
|
||||
### Example: multiple log entries
|
||||
|
||||
### Count entries
|
||||
|
||||
```bash
|
||||
jq 'length'
|
||||
```
|
||||
|
||||
### Filter array elements
|
||||
|
||||
```bash
|
||||
jq '.[] | select(.status >= 500)'
|
||||
```
|
||||
|
||||
### Extract fields from array
|
||||
|
||||
```bash
|
||||
jq '.[] | {service, status, message}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Kubernetes Logs with jq
|
||||
|
||||
### Example:
|
||||
|
||||
```bash
|
||||
kubectl logs pod-name -n default | jq
|
||||
```
|
||||
|
||||
### Filter error logs
|
||||
|
||||
```bash
|
||||
kubectl logs pod-name | jq 'select(.level=="error")'
|
||||
```
|
||||
|
||||
### Extract container metadata logs
|
||||
|
||||
```bash
|
||||
kubectl logs pod-name | jq '{time, container, message}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Docker Logs with jq
|
||||
|
||||
### Streaming logs
|
||||
|
||||
```bash
|
||||
docker logs container_name | jq
|
||||
```
|
||||
|
||||
### Filter failures
|
||||
|
||||
```bash
|
||||
docker logs container_name | jq 'select(.status != "success")'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. AWS / Cloud Logs (JSON-based)
|
||||
|
||||
### Example CloudWatch JSON logs
|
||||
|
||||
```bash
|
||||
aws logs filter-log-events --log-group-name my-app | jq
|
||||
```
|
||||
|
||||
### Extract messages only
|
||||
|
||||
```bash
|
||||
... | jq '.events[].message'
|
||||
```
|
||||
|
||||
### Filter by keyword
|
||||
|
||||
```bash
|
||||
... | jq '.events[] | select(.message | contains("ERROR"))'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Transforming JSON (Automation Use Cases)
|
||||
|
||||
### Rename fields
|
||||
|
||||
```bash
|
||||
jq '{userId: .id, username: .name}'
|
||||
```
|
||||
|
||||
### Add computed fields
|
||||
|
||||
```bash
|
||||
jq '. + {isActive: true}'
|
||||
```
|
||||
|
||||
### Build new structure
|
||||
|
||||
```bash
|
||||
jq '{users: [.[] | {id, name}]}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Advanced Filtering
|
||||
|
||||
### Logical conditions
|
||||
|
||||
```bash
|
||||
jq 'select(.status == 200 and .service == "api")'
|
||||
```
|
||||
|
||||
### Regex matching
|
||||
|
||||
```bash
|
||||
jq 'select(.message | test("timeout|failed"))'
|
||||
```
|
||||
|
||||
### Sorting
|
||||
|
||||
```bash
|
||||
jq 'sort_by(.timestamp)'
|
||||
```
|
||||
|
||||
### Unique values
|
||||
|
||||
```bash
|
||||
jq 'unique_by(.service)'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Aggregations (DevOps Analytics)
|
||||
|
||||
### Count by status
|
||||
|
||||
```bash
|
||||
jq 'group_by(.status) | map({status: .[0].status, count: length})'
|
||||
```
|
||||
|
||||
### Error rate estimation
|
||||
|
||||
```bash
|
||||
jq 'map(select(.status >= 400)) | length'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13. Formatting Output for Humans
|
||||
|
||||
### Compact JSON
|
||||
|
||||
```bash
|
||||
jq -c '.'
|
||||
```
|
||||
|
||||
### Raw output (no quotes)
|
||||
|
||||
```bash
|
||||
jq -r '.message'
|
||||
```
|
||||
|
||||
### Tabular-like output
|
||||
|
||||
```bash
|
||||
jq -r '[.timestamp, .level, .message] | @tsv'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Debugging Pipelines
|
||||
|
||||
### Validate JSON
|
||||
|
||||
```bash
|
||||
jq empty file.json
|
||||
```
|
||||
|
||||
### Highlight structure
|
||||
|
||||
```bash
|
||||
jq '. | type'
|
||||
```
|
||||
|
||||
### Pretty inspect nested structures
|
||||
|
||||
```bash
|
||||
jq 'paths'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15. DevOps Best Practices
|
||||
|
||||
### 1. Always validate JSON first
|
||||
|
||||
```bash
|
||||
jq empty
|
||||
```
|
||||
|
||||
### 2. Use `-c` in pipelines
|
||||
|
||||
Reduces log noise:
|
||||
|
||||
```bash
|
||||
jq -c '.'
|
||||
```
|
||||
|
||||
### 3. Use `-r` for scripting
|
||||
|
||||
```bash
|
||||
jq -r '.field'
|
||||
```
|
||||
|
||||
### 4. Combine with grep when needed
|
||||
|
||||
```bash
|
||||
grep ERROR app.log | jq
|
||||
```
|
||||
|
||||
### 5. Avoid unnecessary formatting in CI/CD
|
||||
|
||||
Keep output machine-readable.
|
||||
|
||||
---
|
||||
|
||||
## 16. Common Patterns Cheat Sheet
|
||||
|
||||
| Task | Command |
|
||||
| --------------- | ------------------------------ |
|
||||
| Pretty print | `jq '.'` |
|
||||
| Filter by field | `jq 'select(.field=="value")'` |
|
||||
| Extract field | `jq '.field'` |
|
||||
| Array iteration | `jq '.[]'` |
|
||||
| Count items | `jq 'length'` |
|
||||
| Convert to text | `jq -r '.field'` |
|
||||
| Compact output | `jq -c '.'` |
|
||||
|
||||
---
|
||||
|
||||
## 17. Real DevOps Example Pipeline
|
||||
|
||||
### Analyze application logs
|
||||
|
||||
```bash
|
||||
cat app.log | jq -c 'select(.level=="error") | {time, service, message}'
|
||||
```
|
||||
|
||||
### Kubernetes debugging
|
||||
|
||||
```bash
|
||||
kubectl logs my-pod | jq -c 'select(.status>=500)'
|
||||
```
|
||||
|
||||
### CI/CD artifact inspection
|
||||
|
||||
```bash
|
||||
cat terraform-output.json | jq '.outputs'
|
||||
```
|
||||
|
||||
@@ -1,3 +1,32 @@
|
||||
```bash
|
||||
sudo apt install bind9
|
||||
```
|
||||
|
||||
|
||||
|
||||
```conf
|
||||
options {
|
||||
directory "/var/cache/bind";
|
||||
|
||||
forwarders {
|
||||
192.168.1.10;
|
||||
8.8.8.8;
|
||||
1.1.1.1;
|
||||
};
|
||||
|
||||
|
||||
dnssec-validation no;
|
||||
|
||||
#listen-on { any; };
|
||||
# listen-on-v6 { any; };
|
||||
|
||||
listen-on port 53 { 127.0.0.1; };
|
||||
listen-on-v6 { none; };
|
||||
forward only;
|
||||
allow-query { any; };
|
||||
recursion yes;
|
||||
allow-recursion { any; };
|
||||
};
|
||||
|
||||
|
||||
```
|
||||
Reference in New Issue
Block a user