my-docs/Monitoring-Logging/ELK/01-Information.md

ELK Stack Overview (DevOps Notes)

What is ELK?

ELK stands for:

• Elasticsearch
• Logstash
• Kibana

The ELK Stack is a powerful platform used for log management, monitoring, data analysis, and observability. It is widely used in DevOps for centralized logging, troubleshooting, and performance monitoring.

---

Core Components

1. Elasticsearch

• Distributed, REST-based search and analytics engine
• Used for storing, indexing, and searching logs and metrics
• Built on Apache Lucene
• Highly scalable and fast for full-text search

Key Responsibilities:

• Store logs and events
• Index data for fast search
• Support aggregations and analytics

---

2. Logstash

• Data processing pipeline
• Ingests data from multiple sources
• Transforms, parses, enriches, and forwards data

Pipeline Stages:

Input → Filter → Output

Examples of filters:

• grok (parse logs)
• mutate (modify fields)
• date (timestamp handling)
• geoip (location enrichment)

---

3. Kibana

• Visualization and analytics UI

• Connects directly to Elasticsearch

• Used for:

  • Dashboards
  • Log exploration
  • Metrics visualization
  • Alerts and reporting

---

Beats (Data Shippers)

Beats are lightweight agents installed on servers to collect and send data to Elasticsearch or Logstash.

Common Beats:

• Filebeat – collects log files
• Metricbeat – system and service metrics (CPU, memory, disk)
• Heartbeat – uptime and availability monitoring
• Packetbeat – network traffic analysis
• Auditbeat – security and audit data

Role:

• Data collection
• Minimal resource usage
• Sends data to Logstash or directly to Elasticsearch

---

Fluentd

• Cloud-native log aggregator and processor
• Alternative to Logstash
• Common in Kubernetes environments

Responsibilities:

• Collect logs from multiple sources
• Enrich and transform data
• Route logs to multiple destinations (Elasticsearch, S3, Kafka)

---

Typical ELK Architecture

Server / Application
        ↓
     Filebeat
        ↓
     Logstash
        ↓
   Elasticsearch
        ↓
      Kibana

Note: In some setups, Beats can send data directly to Elasticsearch (Logstash optional).

---

Database Concepts vs Elasticsearch Concepts

Traditional Database	Elasticsearch
Database	Index
Schema	Mapping
Table	Index (Type is deprecated)
Column	Field
Row	Document
Primary Key	Document ID

⚠️ Note: Type is deprecated in modern Elasticsearch versions (7+).

---

Elasticsearch Data Model

• Index: Logical namespace for documents
• Document: JSON object containing data
• Field: Key-value pair in a document
• Mapping: Defines field types and structure

---

Why ELK in DevOps?

• Centralized logging
• Faster incident response
• Debugging distributed systems
• Monitoring infrastructure and applications
• Security analysis (SIEM use cases)

---

Summary

• Elasticsearch → Storage & search engine
• Logstash / Fluentd → Data processing & enrichment
• Beats → Lightweight data collectors
• Kibana → Visualization & dashboards

The ELK Stack enables DevOps teams to observe, analyze, and troubleshoot systems at scale.