ELK Doc

Monitoring & Logging/ELK/01-Information.md

@@ -0,0 +1,159 @@
+# ELK Stack Overview (DevOps Notes)
+
+## What is ELK?
+
+**ELK** stands for:
+
+* **Elasticsearch**
+* **Logstash**
+* **Kibana**
+
+The ELK Stack is a powerful platform used for **log management, monitoring, data analysis, and observability**. It is widely used in DevOps for **centralized logging, troubleshooting, and performance monitoring**.
+
+---
+
+## Core Components
+
+### 1. Elasticsearch
+
+* Distributed, REST-based **search and analytics engine**
+* Used for **storing, indexing, and searching logs and metrics**
+* Built on Apache Lucene
+* Highly scalable and fast for full-text search
+
+**Key Responsibilities:**
+
+* Store logs and events
+* Index data for fast search
+* Support aggregations and analytics
+
+---
+
+### 2. Logstash
+
+* **Data processing pipeline**
+* Ingests data from multiple sources
+* Transforms, parses, enriches, and forwards data
+
+**Pipeline Stages:**
+
+```
+Input → Filter → Output
+```
+
+**Examples of filters:**
+
+* grok (parse logs)
+* mutate (modify fields)
+* date (timestamp handling)
+* geoip (location enrichment)
+
+---
+
+### 3. Kibana
+
+* **Visualization and analytics UI**
+* Connects directly to Elasticsearch
+* Used for:
+
+  * Dashboards
+  * Log exploration
+  * Metrics visualization
+  * Alerts and reporting
+
+---
+
+## Beats (Data Shippers)
+
+**Beats** are lightweight agents installed on servers to collect and send data to Elasticsearch or Logstash.
+
+Common Beats:
+
+* **Filebeat** – collects log files
+* **Metricbeat** – system and service metrics (CPU, memory, disk)
+* **Heartbeat** – uptime and availability monitoring
+* **Packetbeat** – network traffic analysis
+* **Auditbeat** – security and audit data
+
+**Role:**
+
+* Data collection
+* Minimal resource usage
+* Sends data to Logstash or directly to Elasticsearch
+
+---
+
+## Fluentd
+
+* **Cloud-native log aggregator and processor**
+* Alternative to Logstash
+* Common in Kubernetes environments
+
+**Responsibilities:**
+
+* Collect logs from multiple sources
+* Enrich and transform data
+* Route logs to multiple destinations (Elasticsearch, S3, Kafka)
+
+---
+
+## Typical ELK Architecture
+
+```
+Server / Application
+        ↓
+     Filebeat
+        ↓
+     Logstash
+        ↓
+   Elasticsearch
+        ↓
+      Kibana
+```
+
+> Note: In some setups, Beats can send data **directly to Elasticsearch** (Logstash optional).
+
+---
+
+## Database Concepts vs Elasticsearch Concepts
+
+| Traditional Database | Elasticsearch              |
+| -------------------- | -------------------------- |
+| Database             | Index                      |
+| Schema               | Mapping                    |
+| Table                | Index (Type is deprecated) |
+| Column               | Field                      |
+| Row                  | Document                   |
+| Primary Key          | Document ID                |
+
+> ⚠️ **Note:** `Type` is deprecated in modern Elasticsearch versions (7+).
+
+---
+
+## Elasticsearch Data Model
+
+* **Index**: Logical namespace for documents
+* **Document**: JSON object containing data
+* **Field**: Key-value pair in a document
+* **Mapping**: Defines field types and structure
+
+---
+
+## Why ELK in DevOps?
+
+* Centralized logging
+* Faster incident response
+* Debugging distributed systems
+* Monitoring infrastructure and applications
+* Security analysis (SIEM use cases)
+
+---
+
+## Summary
+
+* **Elasticsearch** → Storage & search engine
+* **Logstash / Fluentd** → Data processing & enrichment
+* **Beats** → Lightweight data collectors
+* **Kibana** → Visualization & dashboards
+
+The ELK Stack enables DevOps teams to **observe, analyze, and troubleshoot systems at scale**.