160 lines
3.5 KiB
Markdown
160 lines
3.5 KiB
Markdown
# ELK Stack Overview (DevOps Notes)
|
||
|
||
## What is ELK?
|
||
|
||
**ELK** stands for:
|
||
|
||
* **Elasticsearch**
|
||
* **Logstash**
|
||
* **Kibana**
|
||
|
||
The ELK Stack is a powerful platform used for **log management, monitoring, data analysis, and observability**. It is widely used in DevOps for **centralized logging, troubleshooting, and performance monitoring**.
|
||
|
||
---
|
||
|
||
## Core Components
|
||
|
||
### 1. Elasticsearch
|
||
|
||
* Distributed, REST-based **search and analytics engine**
|
||
* Used for **storing, indexing, and searching logs and metrics**
|
||
* Built on Apache Lucene
|
||
* Highly scalable and fast for full-text search
|
||
|
||
**Key Responsibilities:**
|
||
|
||
* Store logs and events
|
||
* Index data for fast search
|
||
* Support aggregations and analytics
|
||
|
||
---
|
||
|
||
### 2. Logstash
|
||
|
||
* **Data processing pipeline**
|
||
* Ingests data from multiple sources
|
||
* Transforms, parses, enriches, and forwards data
|
||
|
||
**Pipeline Stages:**
|
||
|
||
```
|
||
Input → Filter → Output
|
||
```
|
||
|
||
**Examples of filters:**
|
||
|
||
* grok (parse logs)
|
||
* mutate (modify fields)
|
||
* date (timestamp handling)
|
||
* geoip (location enrichment)
|
||
|
||
---
|
||
|
||
### 3. Kibana
|
||
|
||
* **Visualization and analytics UI**
|
||
* Connects directly to Elasticsearch
|
||
* Used for:
|
||
|
||
* Dashboards
|
||
* Log exploration
|
||
* Metrics visualization
|
||
* Alerts and reporting
|
||
|
||
---
|
||
|
||
## Beats (Data Shippers)
|
||
|
||
**Beats** are lightweight agents installed on servers to collect and send data to Elasticsearch or Logstash.
|
||
|
||
Common Beats:
|
||
|
||
* **Filebeat** – collects log files
|
||
* **Metricbeat** – system and service metrics (CPU, memory, disk)
|
||
* **Heartbeat** – uptime and availability monitoring
|
||
* **Packetbeat** – network traffic analysis
|
||
* **Auditbeat** – security and audit data
|
||
|
||
**Role:**
|
||
|
||
* Data collection
|
||
* Minimal resource usage
|
||
* Sends data to Logstash or directly to Elasticsearch
|
||
|
||
---
|
||
|
||
## Fluentd
|
||
|
||
* **Cloud-native log aggregator and processor**
|
||
* Alternative to Logstash
|
||
* Common in Kubernetes environments
|
||
|
||
**Responsibilities:**
|
||
|
||
* Collect logs from multiple sources
|
||
* Enrich and transform data
|
||
* Route logs to multiple destinations (Elasticsearch, S3, Kafka)
|
||
|
||
---
|
||
|
||
## Typical ELK Architecture
|
||
|
||
```
|
||
Server / Application
|
||
↓
|
||
Filebeat
|
||
↓
|
||
Logstash
|
||
↓
|
||
Elasticsearch
|
||
↓
|
||
Kibana
|
||
```
|
||
|
||
> Note: In some setups, Beats can send data **directly to Elasticsearch** (Logstash optional).
|
||
|
||
---
|
||
|
||
## Database Concepts vs Elasticsearch Concepts
|
||
|
||
| Traditional Database | Elasticsearch |
|
||
| -------------------- | -------------------------- |
|
||
| Database | Index |
|
||
| Schema | Mapping |
|
||
| Table | Index (Type is deprecated) |
|
||
| Column | Field |
|
||
| Row | Document |
|
||
| Primary Key | Document ID |
|
||
|
||
> ⚠️ **Note:** `Type` is deprecated in modern Elasticsearch versions (7+).
|
||
|
||
---
|
||
|
||
## Elasticsearch Data Model
|
||
|
||
* **Index**: Logical namespace for documents
|
||
* **Document**: JSON object containing data
|
||
* **Field**: Key-value pair in a document
|
||
* **Mapping**: Defines field types and structure
|
||
|
||
---
|
||
|
||
## Why ELK in DevOps?
|
||
|
||
* Centralized logging
|
||
* Faster incident response
|
||
* Debugging distributed systems
|
||
* Monitoring infrastructure and applications
|
||
* Security analysis (SIEM use cases)
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
* **Elasticsearch** → Storage & search engine
|
||
* **Logstash / Fluentd** → Data processing & enrichment
|
||
* **Beats** → Lightweight data collectors
|
||
* **Kibana** → Visualization & dashboards
|
||
|
||
The ELK Stack enables DevOps teams to **observe, analyze, and troubleshoot systems at scale**.
|