diff --git a/AWS/1-Information.md b/AWS/1-Information.md new file mode 100644 index 0000000..fb01b56 --- /dev/null +++ b/AWS/1-Information.md @@ -0,0 +1,209 @@ +# AWS Core Services Overview + +## Compute & Container Services + +**EC2 (Elastic Compute Cloud)** + +* Infrastructure as a Service (IaaS) +* Provides virtual machines (instances) +* Storage options: + + * **EBS** (Elastic Block Store): High-performance block storage attached to a single instance + * **EFS** (Elastic File System): Network file system that can be mounted by multiple instances +* Requires user management of OS, patching, and scaling + +**ECS (Elastic Container Service)** + +* AWS-managed container orchestration service +* Supports Docker containers +* Deployment options: + + 1. **EC2 Launch Type** – you manage EC2 instances + 2. **Fargate Launch Type** – serverless, AWS manages infrastructure + +**ECR (Elastic Container Registry)** + +* Fully managed Docker container image registry +* Used to store, manage, and deploy container images for ECS and EKS + +**EKS (Elastic Kubernetes Service)** + +* Managed Kubernetes service +* AWS manages the Kubernetes control plane +* Worker nodes can run on EC2 or Fargate + +**AWS Lambda** + +* Serverless compute service +* Event-driven execution +* Maximum execution time: **15 minutes** +* No server management required +* Common use cases: APIs, background jobs, automation + +--- + +## Messaging & Integration + +**SQS (Simple Queue Service)** + +* Fully managed message queue service +* Used for decoupling and scaling distributed systems +* Supports Standard and FIFO queues + +--- + +## Databases + +**RDS (Relational Database Service)** + +* Managed relational databases (MySQL, PostgreSQL, Oracle, SQL Server, MariaDB, Aurora) +* Typically deployed in **private subnets** +* High availability using Multi-AZ +* Automated backups, patching, and scaling + +**DynamoDB** + +* Fully managed NoSQL key-value and document database +* Serverless, auto-scaling, and highly available +* Low latency and global replication support + +--- + +## Networking & Traffic Management + +**VPC (Virtual Private Cloud)** + +* Isolated virtual network in AWS +* Uses CIDR ranges for IP addressing + +**Subnets** + +* **Public Subnet**: Has a route to the Internet Gateway +* **Private Subnet**: No direct internet access + +**Internet Gateway (IGW)** + +* Enables inbound and outbound internet access for public subnets + +**NAT Gateway** + +* Placed in a public subnet +* Allows **outbound-only** internet access for private subnet resources +* Cannot receive inbound connections + +**Route 53** + +* Managed DNS service +* Supports domain registration, routing policies, and health checks + +--- + +## Load Balancing + +**ELB (Elastic Load Balancing)** + +* Distributes traffic across multiple targets + +**ALB (Application Load Balancer)** + +* Layer 7 (Application layer) +* Supports HTTP/HTTPS routing rules +* Can route traffic to: + + * EC2 + * ECS + * Lambda + * IP addresses + +--- + +## Security & Identity + +**IAM (Identity and Access Management)** + +* Manages users, groups, roles, and permissions +* Global AWS service + +**IAM Roles** + +* Used by AWS services to access other AWS resources securely + +**IAM Reports** + +* **Credential Report**: Shows credential status for all users +* **Access Advisor**: Shows last-used service permissions + +**Security Groups** + +* Stateful virtual firewalls for AWS resources +* Control inbound and outbound traffic +* Attached to EC2, ALB, RDS, ECS, etc. + +--- + +## Monitoring & Logging + +**CloudWatch** + +* Monitoring and observability service +* Collects metrics, logs, and events +* Used for alarms, dashboards, and automation + +--- + +## AWS Global Infrastructure + +**Region** + +* Geographic area containing multiple Availability Zones + +**Availability Zone (AZ)** + +* One or more isolated data centers within a region + +**Global Services** + +* IAM +* Route 53 +* CloudFront +* AWS WAF + +**Regional Services** + +* EC2 +* ECS +* EKS +* RDS +* Lambda + +--- + +## IP Addressing + +**Private IP** + +* Assigned from VPC CIDR range +* Used for internal communication + +**Public IP** + +* Assigned automatically to EC2 instances in public subnets +* Released when instance is stopped + +**Elastic IP (EIP)** + +* Static public IPv4 address +* Remains allocated even if the instance stops +* Used for failover and stable endpoints + +--- + +## Database Networking Best Practices + +* RDS instances should run in **private subnets** +* Access options: + + * EC2 in the same VPC + * Bastion host + * VPN or Direct Connect +* NAT Gateway can be used for outbound access (updates, patches) diff --git a/Code-Management/Git/main.md b/Code-Management/Git/main.md index bcd3f12..c9b2326 100644 --- a/Code-Management/Git/main.md +++ b/Code-Management/Git/main.md @@ -1,161 +1,523 @@ -# Git Commands Guide +# Git Commands Guide (DevOps-Oriented) -## Getting Started with Git +## 1. Installation and Setup -### 1. Installing Git +### Install Git -Before you begin, ensure Git is installed on your machine. You can download it from [git-scm.com](https://git-scm.com/). +Download and install Git from: +[https://git-scm.com/](https://git-scm.com/) -### 2. Check Git Installation +Linux (Debian/Ubuntu): -To verify that Git is installed, run: +```bash +sudo apt update && sudo apt install git -y +``` + +RHEL/CentOS: + +```bash +sudo yum install git -y +``` + +macOS (Homebrew): + +```bash +brew install git +``` + +### Verify Installation ```bash git --version ``` -### 3. Configure Git User Information +### Configure User Identity -Set up your name and email address, which will be used for your commits: +Git uses this information for commits: ```bash git config --global user.name "Your Name" git config --global user.email "your.email@example.com" ``` -## Configuring Git to Use a Custom SSH Key - -If you need to use a specific SSH key for your Git operations, you can configure Git as follows: +Check configuration: ```bash -git config --add --local core.sshCommand 'ssh -i ' +git config --list ``` -For Clone With Custom SSH Key Use: +Configuration scopes: + +* `--system`: All users +* `--global`: Current user +* `--local`: Repository only + +--- + +## 2. SSH Key Configuration + +### Generate SSH Key + ```bash -git -c core.sshCommand="ssh -i " clone host:repo +ssh-keygen -t ed25519 -C "your.email@example.com" ``` +Start SSH agent and add key: -*Replace `` with the actual path to your SSH key file.* +```bash +eval "$(ssh-agent -s)" +ssh-add ~/.ssh/id_ed25519 +``` -## Creating and Managing a Local Git Repository +### Use Custom SSH Key (Per Repository) -### 1. Initialize a Git Repository +```bash +git config --local core.sshCommand "ssh -i " +``` -Start by creating a new Git repository in your local project directory: +Clone with custom SSH key: + +```bash +git -c core.sshCommand="ssh -i " clone git@host:repo.git +``` + +--- + +## 3. Initialize Repository + +Create a new Git repository: ```bash git init -b main ``` -*The `-b main` flag sets the default branch name to "main".* +Existing repository: -### 2. Add Files and Commit Changes +```bash +git init +``` -Next, stage all your files and create your initial commit: +--- + +## 4. Basic Workflow + +### Stage and Commit Changes + +Stage all changes: ```bash git add -A -git commit -m "Initial Commit" ``` -*The `git add -A` command stages all changes, while the `git commit` command records those changes with a descriptive message.* - -### 3. Connect to a Remote Repository - -Now, link your local repository to a remote GitHub repository: +Commit changes: ```bash -git remote add origin +git commit -m "Initial commit" ``` -*Replace `` with the URL of your GitHub repository.* - -### 4. Push Changes to GitHub - -Finally, push your initial commit to the remote repository: +### Connect Local Repository to Remote ```bash -git push origin main +git remote add origin ``` -## Common Git Commands for Beginners +Verify: -### 1. Check the Status of Your Repository +```bash +git remote -v +``` -To see which changes are staged, unstaged, or untracked: +### Push to Remote + +First push: + +```bash +git push -u origin main +``` + +Subsequent pushes: + +```bash +git push +``` + +--- + +## 5. Repository Status and History + +### Check Repository Status ```bash git status ``` -### 2. View Commit History - -To view the commit history of your repository: +### View Commit History ```bash git log ``` -*You can press `q` to exit the log view.* +Common options: -### 3. Viewing Changes +```bash +git log --oneline +git log --graph --oneline --all +git log -p +git log -3 +``` -To see changes made to files before staging them: +### View File Changes + +Unstaged changes: ```bash git diff ``` -### 4. Staging Individual Files - -If you want to stage specific files instead of all changes: +Staged changes: ```bash -git add +git diff --staged ``` -*Replace `` with the name of the file you wish to stage.* - -### 5. Undoing Changes - -To unstage a file that you added by mistake: +Compare branches: ```bash -git reset +git diff main..dev ``` -To discard changes in a file and revert it to the last committed state: +--- + +## 6. File Operations + +### Stage Specific Files ```bash -git checkout -- +git add ``` -### 6. Cloning a Repository - -If you want to create a copy of an existing remote repository: +### Unstage Files ```bash -git clone +git reset ``` -*Replace `` with the URL of the repository you want to clone.* +### Discard Local Changes -### 7. Creating a New Branch +```bash +git checkout -- +``` -To create a new branch for development: +Restore using modern command: + +```bash +git restore +``` + +### Rename File + +```bash +git mv old-name new-name +``` + +### Remove File + +```bash +git rm +``` + +Remove but keep locally: + +```bash +git rm --cached +``` + +--- + +## 7. Branch Management + +### Create and Switch Branch ```bash git checkout -b ``` -*Replace `` with your desired branch name.* +Modern alternative: -### 8. Merging Branches +```bash +git switch -c +``` -To merge changes from another branch into your current branch: +### List Branches + +```bash +git branch +git branch -a +git branch -v +``` + +### Delete Branch + +```bash +git branch -d +``` + +Force delete: + +```bash +git branch -D +``` + +### Rename Branch + +```bash +git branch -m old-name new-name +``` + +--- + +## 8. Merging and Rebasing + +### Merge Branch ```bash git merge ``` + +Merge types: + +* Fast-forward +* Three-way merge (creates merge commit) + +### Rebase (Linear History) + +```bash +git rebase main +``` + +Abort rebase: + +```bash +git rebase --abort +``` + +Continue rebase: + +```bash +git rebase --continue +``` + +--- + +## 9. Remote Operations + +### List Remotes + +```bash +git remote +git remote -v +``` + +### Show Remote Details + +```bash +git remote show origin +``` + +### Fetch Changes + +```bash +git fetch +git fetch --all +``` + +### Pull Changes + +Fetch + merge: + +```bash +git pull +``` + +Rebase instead of merge: + +```bash +git pull --rebase +``` + +--- + +## 10. Commit Management + +### Amend Last Commit + +```bash +git commit --amend +``` + +### Show Commit Details + +```bash +git show +``` + +### Revert Commit (Safe for Shared Branches) + +```bash +git revert +``` + +### Reset Commit (Use with Caution) + +Soft reset: + +```bash +git reset --soft HEAD~1 +``` + +Mixed reset: + +```bash +git reset HEAD~1 +``` + +Hard reset: + +```bash +git reset --hard HEAD~1 +``` + +--- + +## 11. Stash (Temporary Changes) + +Save work without committing: + +```bash +git stash +``` + +List stashes: + +```bash +git stash list +``` + +Apply stash: + +```bash +git stash apply +``` + +Pop stash: + +```bash +git stash pop +``` + +--- + +## 12. Tags (Releases) + +Create tag: + +```bash +git tag v1.0.0 +``` + +Annotated tag: + +```bash +git tag -a v1.0.0 -m "Release v1.0.0" +``` + +Push tags: + +```bash +git push origin --tags +``` + +--- + +## 13. .gitignore + +Create `.gitignore`: + +```bash +touch .gitignore +``` + +Example: + +``` +.env +node_modules/ +*.log +``` + +Apply after commit: + +```bash +git rm -r --cached . +git add . +git commit -m "Apply gitignore" +``` + +--- + +## 14. Useful Configuration and Aliases + +Change default editor: + +```bash +git config --global core.editor "vim" +``` + +Create aliases: + +```bash +git config --global alias.st status +git config --global alias.co checkout +git config --global alias.cm commit +git config --global alias.br branch +``` + +--- + +## 15. Troubleshooting and Recovery + +Undo last commit but keep changes: + +```bash +git reset --soft HEAD~1 +``` + +Recover deleted branch: + +```bash +git reflog +git checkout -b +``` + +Fix detached HEAD: + +```bash +git checkout main +``` + +--- + +## 16. Clone Repository + +Clone via SSH: + +```bash +git clone git@github.com:user/repo.git +``` + +Clone specific branch: + +```bash +git clone -b +``` + diff --git a/Containerization-Orchestration/Kubernetes/1-Information.md b/Containerization-Orchestration/Kubernetes/1-Information.md index 869e7ba..988e894 100755 --- a/Containerization-Orchestration/Kubernetes/1-Information.md +++ b/Containerization-Orchestration/Kubernetes/1-Information.md @@ -1,87 +1,357 @@ -# 🚒 Kubernetes (K8s) Documentation +# Kubernetes (K8s) – Technical Documentation -## 🌐 Overview -**Kubernetes (K8s)** is an open-source container orchestration platform designed to automate the deployment, scaling, and operation of containerized applications. +## 1. Overview + +**Kubernetes (K8s)** is an open-source container orchestration platform that automates the deployment, scaling, networking, and lifecycle management of containerized applications. It provides declarative configuration and self-healing capabilities to maintain the desired state of workloads. + +Kubernetes follows a **control plane / worker node** architecture and is designed to run reliably at scale. --- -## 🧠 Control Plane (CP) -The **Control Plane** is the core management component of a Kubernetes cluster. It makes global decisions about the cluster (e.g., scheduling) and maintains the desired state of the cluster by managing workloads and directing communication within the system. +## 2. Kubernetes Architecture -> πŸ’‘ **Note:** By default, the Control Plane does not directly manage or run application containers. +A Kubernetes cluster consists of: -### πŸ”‘ Key Components of the Control Plane - -- **API Server (`kube-apiserver`)** - Exposes the Kubernetes API and serves as the cluster's entry point. It handles communication between internal components and external clients. - -- **Scheduler (`kube-scheduler`)** - Assigns workloads (e.g., Pods) to nodes based on resource availability and defined policies. - -- **Controller Manager (`kube-controller-manager`)** - Runs controllers that monitor and regulate the cluster's state, such as the Node Controller and Replication Controller. - -- **etcd** - A consistent and highly available key-value store that stores all cluster data, configurations, and state. This is the "database" of Kubernetes. +* **Control Plane nodes** – manage cluster state +* **Worker nodes** – run application workloads --- -## 🧱 Worker Nodes -**Worker nodes** are the machines where containerized applications run. Each node contains essential components for managing containers. +## 3. Control Plane -### πŸ”§ Key Components of a Worker Node +The **Control Plane** is responsible for managing the overall cluster state. It does not normally run application workloads. -- **Kubelet** - An agent that ensures containers run as specified in their Pod definitions. It communicates with the Control Plane to execute assigned tasks. +### 3.1 Control Plane Components -- **Kube Proxy** - Maintains network rules and manages routing for communication within the cluster and with external systems. +#### kube-apiserver + +* Entry point to the Kubernetes cluster +* Exposes the Kubernetes REST API +* Validates requests and persists state to etcd +* All components communicate through the API server + +#### etcd + +* Distributed, consistent key-value store +* Stores all cluster state and configuration +* Uses the **Raft consensus algorithm** +* Requires an **odd number of members (3, 5, …)** to maintain quorum +* Minimum recommended production setup: **3 etcd members** + +#### kube-scheduler + +* Assigns Pods to nodes +* Makes scheduling decisions based on: + + * Resource requests and limits + * Node affinity / anti-affinity + * Taints and tolerations + * Pod affinity rules + +#### kube-controller-manager + +* Runs multiple controllers, including: + + * Node Controller + * ReplicaSet Controller + * Deployment Controller + * Job Controller +* Ensures the actual cluster state matches the desired state --- -## πŸ”„ Data Flow -- **Kubelet** and **Kube Proxy** on each worker node interact with the **API Server** to perform operations and update resource states. -- The **Scheduler** selects suitable nodes for pod placement based on available resources. -- The **Controller Manager** ensures the actual state of the cluster matches the desired state. +## 4. Worker Nodes + +Worker nodes run application containers and system workloads. + +### 4.1 Worker Node Components + +#### kubelet + +* Node agent running on each worker +* Responsibilities: + + * Register the node with the API server + * Create and manage Pods + * Monitor Pod and container health + * Report node and Pod status + * Manage DaemonSet Pods +* Communicates with the container runtime via CRI + +#### kube-proxy + +* Handles networking and service routing +* Maintains iptables or IPVS rules +* Enables Service abstraction and load balancing +* Usually runs as a **DaemonSet** + +#### Container Runtime + +* Responsible for running containers +* Must be **CRI-compliant** +* Common runtimes: + + * containerd (recommended) + * CRI-O --- -## πŸ› οΈ Administration Tools +## 5. Container Runtime Interface (CRI) -- **`kubeadm`** - A command-line tool to bootstrap and configure Kubernetes clusters. It streamlines the setup of both the Control Plane and worker nodes. +**CRI (Container Runtime Interface)** is a Kubernetes API that allows kubelet to communicate with container runtimes. -- **`kubectl`** - The CLI for interacting with the Kubernetes API. It's used to deploy apps, inspect cluster resources, and manage configurations. +Important clarification: + +* CRI is **not a registry** +* It is an interface between kubelet and the container runtime --- -## 🧩 Kubernetes Version Compatibility +## 6. Cluster Networking & DNS -### Kubernetes and Container Runtimes +### 6.1 CoreDNS -- **Kubernetes ≀ 1.23** - βœ… Compatible with **Docker** as the default container runtime. +* Kubernetes internal DNS service +* Runs as a **Deployment** (not DaemonSet in modern clusters) +* Provides service discovery inside the cluster -- **Kubernetes 1.24 – 1.25** - ❌ Docker is **not supported** directly. Use `containerd` or another CRI-compliant runtime. +#### Default cluster domain -- **Kubernetes β‰₯ 1.25** - ⚠️ Docker may be installed on the system but must be used **indirectly** through `containerd` or another supported CRI. +``` +cluster.local +``` + +#### DNS formats + +* Service: + +``` +..svc.cluster.local +``` + +* Pod: + +``` +..pod.cluster.local +``` --- -## πŸ‘₯ Kubernetes Roles +## 7. Administration Tools -- **Control Plane (Manager)** - Requires an **odd number** of nodes for high availability (e.g., 1, 3, 5, ...). This ensures quorum in distributed consensus. +### kubeadm -- **Worker (none)** - These nodes run application workloads and do not participate in control decisions. +* Tool for bootstrapping Kubernetes clusters +* Used to initialize control plane and join worker nodes +### kubectl -image pull policy in kubernetes: +* Command-line interface to interact with Kubernetes API +* Used for deployment, debugging, inspection, and administration +### Lens -example of all work loads: -https://k8s-examples.container-solutions.com/ \ No newline at end of file +* Client-side GUI for Kubernetes +* Requires kubeconfig access + +### Kubernetes Dashboard + +* Server-side web UI +* Runs inside the cluster +* Requires RBAC configuration for access + +--- + +## 8. Kubernetes Version & Runtime Compatibility + +| Kubernetes Version | Docker Support | +| ------------------ | -------------------------------------------- | +| ≀ 1.23 | Docker supported via dockershim | +| 1.24+ | Docker shim removed | +| 1.25+ | Docker only usable indirectly via containerd | + +**Recommendation:** Use `containerd` directly. + +--- + +## 9. Node Roles & High Availability + +### Control Plane + +* Requires **odd number of nodes** (1, 3, 5…) +* Necessary for etcd quorum and fault tolerance + +### Worker Nodes + +* Can scale horizontally without restrictions +* Do not participate in control decisions + +--- + +## 10. Pod Lifecycle Hooks + +### postStart + +* Executed immediately after container creation +* Runs asynchronously with container startup +* Failure causes container restart + +### preStop + +* Executed before container termination +* Commonly used for graceful shutdown +* Kubernetes waits for completion (within termination grace period) + +--- + +## 11. Static Pods + +* Managed directly by kubelet +* Defined via local manifest files +* Do **not** require API server scheduling +* Commonly used for core components: + + * kube-apiserver + * etcd + * kube-controller-manager + +--- + +## 12. Workload Types + +Common Kubernetes workloads: + +* Deployment +* ReplicaSet +* StatefulSet +* DaemonSet +* Job +* CronJob + +Examples: +[https://k8s-examples.container-solutions.com/](https://k8s-examples.container-solutions.com/) + +--- + +## 13. Scheduling Behavior + +Pod scheduling is **skipped** for: + +* **DaemonSet Pods** +* **Static Pods** + +These are directly bound to nodes. + +--- + +## 14. Scaling + +### Horizontal Scaling + +* Adjust replica count +* Manual or automatic + +### Vertical Scaling + +* Adjust CPU and memory resources +* Requires Pod restart + +--- + +## 15. Autoscaling Components + +### Horizontal Pod Autoscaler (HPA) + +* Scales replicas based on: + + * CPU + * Memory + * Custom metrics + +### Vertical Pod Autoscaler (VPA) + +Components: + +1. **Recommender** – calculates resource recommendations +2. **Updater** – evicts Pods if needed +3. **Admission Controller** – applies recommendations at Pod creation + +### Cluster Autoscaler (CA) + +* Scales worker nodes up/down +* Integrates with cloud providers or node groups + +--- + +## 16. Resource Management + +### ResourceQuota + +* Limits total resource usage per namespace +* Controls CPU, memory, object count, etc. + +### LimitRange + +* Sets default and maximum limits per Pod or container +* Applies at namespace level + +--- + +## 17. Finalizers + +* Prevent resource deletion until cleanup is complete +* Common use cases: + + * External resource cleanup + * Storage detachment +* Object remains in `Terminating` state until finalizer is removed + +--- + +## 18. Deployment Update Strategies + +### Recreate + +* Terminates old Pods before creating new ones +* Causes downtime + +### RollingUpdate + +* Gradual replacement +* Zero or minimal downtime +* Default for Deployments + +### Blue-Green Deployment + +* Two environments (blue and green) +* Traffic switched after validation + +### Canary Deployment + +* Gradual traffic increase to new version +* Used for risk reduction + +### A/B Testing + +* Traffic split between versions +* Used for experimentation + +### Shadow Testing + +* New version receives production traffic without user impact +* Used for performance and behavior analysis + +--- + +## 19. Services + +### Service + +* Provides stable networking and load balancing +* Uses label selectors to target Pods + +### Headless Service + +* No virtual IP +* Direct Pod DNS resolution +* Commonly used with StatefulSets (e.g., databases) diff --git a/Linux/Basic Administration/30-diff.md b/Linux/Basic Administration/30-diff.md new file mode 100644 index 0000000..5fd7847 --- /dev/null +++ b/Linux/Basic Administration/30-diff.md @@ -0,0 +1,190 @@ +# diff Command Reference + +The `diff` command is a standard Unix/Linux utility used to compare files line by line. It is commonly used in development, DevOps, and system administration to identify changes between configuration files, source code, logs, or generated outputs. + +--- + +## 1. Basic File Comparison + +Compare two files line by line: + +```bash +diff file1 file2 +``` + +### Output Behavior + +* Shows only the lines that differ +* Uses symbols to indicate changes: + + * `<` line from `file1` + * `>` line from `file2` + * `c` change + * `a` addition + * `d` deletion + +--- + +## 2. Side-by-Side Comparison + +Display files next to each other: + +```bash +diff -y file1 file2 +``` + +### Notes + +* Useful for human-readable comparison +* Differences are shown in two columns +* Change indicators appear in the middle + +Limit output width: + +```bash +diff -y --width=120 file1 file2 +``` + +Suppress common lines: + +```bash +diff -y --suppress-common-lines file1 file2 +``` + +--- + +## 3. Unified Diff Format (Most Common) + +Generate a unified diff: + +```bash +diff -u file1 file2 +``` + +### Why Unified Diff + +* Standard format used by Git, patch, and code reviews +* Shows context before and after changes +* Easier to read and apply + +Example output markers: + +* `+` added lines +* `-` removed lines +* `@@` line numbers and context + +--- + +## 4. Save Diff Output to a File + +Redirect diff output to a file: + +```bash +diff -u file1 file2 > different.diff +``` + +Common use cases: + +* Code reviews +* Patch creation +* Change tracking +* CI/CD artifact storage + +--- + +## 5. Recursive Directory Comparison + +Compare directories: + +```bash +diff -r dir1 dir2 +``` + +Unified recursive diff: + +```bash +diff -ru dir1 dir2 +``` + +Ignore missing files: + +```bash +diff -rq dir1 dir2 +``` + +--- + +## 6. Ignore Differences + +Ignore whitespace: + +```bash +diff -w file1 file2 +``` + +Ignore blank lines: + +```bash +diff -B file1 file2 +``` + +Ignore case differences: + +```bash +diff -i file1 file2 +``` + +--- + +## 7. Apply a Diff as a Patch + +Create a patch: + +```bash +diff -u oldfile newfile > change.patch +``` + +Apply patch: + +```bash +patch < change.patch +``` + +Dry-run patch: + +```bash +patch --dry-run < change.patch +``` + +--- + +## 8. diff vs Git diff + +| diff | git diff | +| -------------------- | ------------------------------- | +| Compares any files | Compares Git-tracked files | +| Works without Git | Requires Git repository | +| Produces patch files | Integrated with version control | + +--- + +## 9. Best Practices + +* Use `-u` format for readability and compatibility +* Store diff files with `.diff` or `.patch` extensions +* Avoid committing generated diff files unless required +* Use `diff` for system configuration audits +* Use `git diff` inside Git repositories + +--- + +## 10. Quick Reference + +```bash +diff file1 file2 +diff -y file1 file2 +diff -u file1 file2 +diff -ru dir1 dir2 +diff -u file1 file2 > change.diff +``` + diff --git a/Linux/Basic Administration/31-date.md b/Linux/Basic Administration/31-date.md new file mode 100644 index 0000000..415224b --- /dev/null +++ b/Linux/Basic Administration/31-date.md @@ -0,0 +1,243 @@ +Here is a cleaned up, expanded, and more production-ready version of your **date and time manipulation** document, written from a DevOps / Linux system administration perspective and keeping it concise and accurate. + +--- + +# Essential Date and Time Manipulation Commands + +This document covers common shell commands for working with dates and times. These commands are frequently used in scripting, logging, monitoring, automation, and system administration tasks. + +--- + +## 1. The `date` Command Overview + +The `date` command is a standard Unix/Linux utility used to: + +* Display the current system date and time +* Format timestamps +* Convert between human-readable dates and Unix epoch time +* Perform date arithmetic + +Check system time: + +```bash +date +``` + +--- + +## 2. Creating Epoch Timestamps + +### Current Epoch Time (Seconds) + +```bash +date +%s +``` + +Output example: + +``` +1737154200 +``` + +### Current Epoch Time (Milliseconds) + +```bash +date +%s%3N +``` + +Notes: + +* `%s` β†’ seconds since Unix epoch +* `%3N` β†’ milliseconds (GNU date only) + +--- + +### Convert Specific Date to Epoch (Milliseconds) + +Convert a human-readable date to epoch time: + +```bash +date -d "2026-01-13 14:31:26" +%s%3N +``` + +Common use cases: + +* Log correlation +* API timestamps +* CI/CD pipeline timing +* Monitoring and alerting + +--- + +## 3. Formatting Current Date and Time + +Custom output formats are widely used in scripts and logs. + +### Common Date Formats + +| Command | Description | Example Output | +| --------------------------- | --------------------- | ---------------------------- | +| `date` | Default system format | Wed Jan 17 10:30:00 UTC 2026 | +| `date +'%Y-%m-%d %H:%M:%S'` | ISO-like format | 2026-01-17 10:30:00 | +| `date +'%Y-%m-%d'` | Date only | 2026-01-17 | +| `date +'%H:%M:%S'` | Time only | 10:30:00 | +| `date +%s` | Epoch (seconds) | 1737154200 | + +### Common Format Specifiers + +| Specifier | Meaning | +| --------- | --------------- | +| `%Y` | Year (4 digits) | +| `%m` | Month (01–12) | +| `%d` | Day (01–31) | +| `%H` | Hour (00–23) | +| `%M` | Minute (00–59) | +| `%S` | Second (00–60) | +| `%N` | Nanoseconds | + +--- + +## 4. Converting Epoch to Human-Readable Time + +Convert epoch seconds to readable format: + +```bash +date -d @1737154200 +``` + +Using a variable: + +```bash +EPOCH_TIME=1737154200 +date -d @"$EPOCH_TIME" +``` + +Formatted output: + +```bash +date -d @"$EPOCH_TIME" '+%Y-%m-%d %H:%M:%S' +``` + +--- + +## 5. Date Arithmetic + +GNU `date` supports flexible date calculations. + +### Relative Dates + +| Command | Description | +| ----------------------- | ----------------------- | +| `date -d "yesterday"` | Previous day | +| `date -d "tomorrow"` | Next day | +| `date -d "2 days ago"` | Two days in the past | +| `date -d "+1 hour"` | One hour from now | +| `date -d "+30 minutes"` | Thirty minutes from now | +| `date -d "+1 week"` | One week from now | +| `date -d "2 weeks ago"` | Two weeks ago | + +--- + +## 6. Date Arithmetic with Formatting + +Example: date 7 days from now: + +```bash +date -d "+7 days" '+%Y-%m-%d' +``` + +Example: timestamp 15 minutes ago (epoch): + +```bash +date -d "-15 minutes" +%s +``` + +--- + +## 7. Working with UTC and Time Zones + +Display current UTC time: + +```bash +date -u +``` + +Format UTC time: + +```bash +date -u '+%Y-%m-%d %H:%M:%S' +``` + +Convert date in a specific timezone: + +```bash +TZ=UTC date +TZ=America/New_York date +``` + +--- + +## 8. Script-Friendly Usage Examples + +Add timestamp to a log entry: + +```bash +echo "$(date '+%Y-%m-%d %H:%M:%S') Application started" +``` + +Generate a timestamped filename: + +```bash +backup_$(date +%Y%m%d_%H%M%S).tar.gz +``` + +Measure execution time: + +```bash +start=$(date +%s) +# command here +end=$(date +%s) +echo "Duration: $((end - start)) seconds" +``` + +--- + +## 9. macOS Compatibility Notes + +macOS uses BSD `date`, which differs from GNU `date`. + +Example difference: + +```bash +# GNU (Linux) +date -d "yesterday" + +# BSD (macOS) +date -v -1d +``` + +Install GNU date on macOS: + +```bash +brew install coreutils +gdate -d "yesterday" +``` + +--- + +## 10. Best Practices + +* Use UTC for logs and distributed systems +* Store timestamps as epoch values when possible +* Format dates only at display time +* Avoid locale-dependent formats in scripts +* Be aware of GNU vs BSD `date` differences + +--- + +If you want, I can: + +* Add cron-specific date examples +* Add log rotation and backup use cases +* Provide a quick-reference cheat sheet +* Add cross-platform date handling strategies diff --git a/Monitoring & Logging/ELK/01-Information.md b/Monitoring & Logging/ELK/01-Information.md new file mode 100644 index 0000000..f80a490 --- /dev/null +++ b/Monitoring & Logging/ELK/01-Information.md @@ -0,0 +1,159 @@ +# ELK Stack Overview (DevOps Notes) + +## What is ELK? + +**ELK** stands for: + +* **Elasticsearch** +* **Logstash** +* **Kibana** + +The ELK Stack is a powerful platform used for **log management, monitoring, data analysis, and observability**. It is widely used in DevOps for **centralized logging, troubleshooting, and performance monitoring**. + +--- + +## Core Components + +### 1. Elasticsearch + +* Distributed, REST-based **search and analytics engine** +* Used for **storing, indexing, and searching logs and metrics** +* Built on Apache Lucene +* Highly scalable and fast for full-text search + +**Key Responsibilities:** + +* Store logs and events +* Index data for fast search +* Support aggregations and analytics + +--- + +### 2. Logstash + +* **Data processing pipeline** +* Ingests data from multiple sources +* Transforms, parses, enriches, and forwards data + +**Pipeline Stages:** + +``` +Input β†’ Filter β†’ Output +``` + +**Examples of filters:** + +* grok (parse logs) +* mutate (modify fields) +* date (timestamp handling) +* geoip (location enrichment) + +--- + +### 3. Kibana + +* **Visualization and analytics UI** +* Connects directly to Elasticsearch +* Used for: + + * Dashboards + * Log exploration + * Metrics visualization + * Alerts and reporting + +--- + +## Beats (Data Shippers) + +**Beats** are lightweight agents installed on servers to collect and send data to Elasticsearch or Logstash. + +Common Beats: + +* **Filebeat** – collects log files +* **Metricbeat** – system and service metrics (CPU, memory, disk) +* **Heartbeat** – uptime and availability monitoring +* **Packetbeat** – network traffic analysis +* **Auditbeat** – security and audit data + +**Role:** + +* Data collection +* Minimal resource usage +* Sends data to Logstash or directly to Elasticsearch + +--- + +## Fluentd + +* **Cloud-native log aggregator and processor** +* Alternative to Logstash +* Common in Kubernetes environments + +**Responsibilities:** + +* Collect logs from multiple sources +* Enrich and transform data +* Route logs to multiple destinations (Elasticsearch, S3, Kafka) + +--- + +## Typical ELK Architecture + +``` +Server / Application + ↓ + Filebeat + ↓ + Logstash + ↓ + Elasticsearch + ↓ + Kibana +``` + +> Note: In some setups, Beats can send data **directly to Elasticsearch** (Logstash optional). + +--- + +## Database Concepts vs Elasticsearch Concepts + +| Traditional Database | Elasticsearch | +| -------------------- | -------------------------- | +| Database | Index | +| Schema | Mapping | +| Table | Index (Type is deprecated) | +| Column | Field | +| Row | Document | +| Primary Key | Document ID | + +> ⚠️ **Note:** `Type` is deprecated in modern Elasticsearch versions (7+). + +--- + +## Elasticsearch Data Model + +* **Index**: Logical namespace for documents +* **Document**: JSON object containing data +* **Field**: Key-value pair in a document +* **Mapping**: Defines field types and structure + +--- + +## Why ELK in DevOps? + +* Centralized logging +* Faster incident response +* Debugging distributed systems +* Monitoring infrastructure and applications +* Security analysis (SIEM use cases) + +--- + +## Summary + +* **Elasticsearch** β†’ Storage & search engine +* **Logstash / Fluentd** β†’ Data processing & enrichment +* **Beats** β†’ Lightweight data collectors +* **Kibana** β†’ Visualization & dashboards + +The ELK Stack enables DevOps teams to **observe, analyze, and troubleshoot systems at scale**. diff --git a/Monitoring & Logging/ELK/02-Node-Types.md b/Monitoring & Logging/ELK/02-Node-Types.md new file mode 100644 index 0000000..639a552 --- /dev/null +++ b/Monitoring & Logging/ELK/02-Node-Types.md @@ -0,0 +1,220 @@ +# ELK Node Types +## Overview + +The ELK Stack (Elasticsearch, Logstash, Kibana) is commonly deployed using multiple node types (roles) to ensure scalability, performance, and resilience. This document outlines the main node types used in production-grade ELK deployments from a DevOps perspective. + +--- + +## 1. Elasticsearch Node Types + +Elasticsearch nodes can be assigned one or more roles. In production environments, roles are usually separated for stability and performance. + +### 1.1 Master Node (Dedicated Master) + +**Purpose:** Cluster coordination and management + +Responsibilities: + +* Manages cluster state +* Controls shard allocation +* Handles node joins and failures + +Best Practices: + +* Deploy 3 dedicated master nodes (odd number for quorum) +* Do not assign data or ingest roles +* Require minimal CPU and disk, but stable memory + +Configuration: + +```yaml +node.roles: [ master ] +``` + +--- + +### 1.2 Data Nodes + +**Purpose:** Store data and execute search and indexing operations + +#### a. Hot Data Node + +* Handles recent and high-traffic data +* Requires fast SSD storage +* Heavy indexing and querying workload + +```yaml +node.roles: [ data_hot ] +``` + +#### b. Warm Data Node + +* Stores less frequently accessed data +* Moderate CPU and disk requirements + +```yaml +node.roles: [ data_warm ] +``` + +#### c. Cold Data Node + +* Stores rarely accessed data +* Optimized for cost efficiency + +```yaml +node.roles: [ data_cold ] +``` + +#### d. Frozen Data Node + +* Archival data with searchable snapshots +* Minimal local storage requirements + +```yaml +node.roles: [ data_frozen ] +``` + +--- + +### 1.3 Coordinating Node + +**Purpose:** Query routing and result aggregation + +Characteristics: + +* No data storage +* No master role +* Acts as a load balancer for search requests + +Use Case: + +* Kibana and client applications connect to coordinating nodes + +```yaml +node.roles: [ ] +``` + +--- + +### 1.4 Ingest Node + +**Purpose:** Data preprocessing before indexing + +Responsibilities: + +* Executes ingest pipelines +* Performs grok parsing, enrichment, geoip, and transformations +* Reduces load on data nodes + +```yaml +node.roles: [ ingest ] +``` + +--- + +### 1.5 Machine Learning Node + +**Purpose:** Run machine learning jobs + +Use Cases: + +* Anomaly detection +* Advanced analytics + +```yaml +node.roles: [ ml ] +``` + +--- + +### 1.6 Transform Node + +**Purpose:** Data transformation and aggregation + +Use Cases: + +* Pivot and latest transforms +* Pre-aggregated indices + +```yaml +node.roles: [ transform ] +``` + +--- + +## 2. Logstash Node Types + +Logstash does not use formal roles but is deployed based on function. + +### 2.1 Ingest / Collector Nodes + +* Receive data from Beats, syslog, Kafka, etc. +* Minimal processing + +### 2.2 Processing Nodes + +* Perform heavy parsing and enrichment +* CPU-intensive workloads + +### 2.3 Output Nodes + +* Focused on reliable delivery to Elasticsearch + +--- + +## 3. Kibana Node Types + +### 3.1 Kibana Server Node + +* Provides UI and REST API +* Stateless and horizontally scalable + +### 3.2 Reporting / Task Manager Node + +* Handles scheduled tasks and reporting +* Often separated in large deployments + +--- + +## 4. Beats and Agents (Edge Nodes) + +Although not part of the core ELK stack, Beats are critical for data collection. + +Common Beats: + +* Filebeat: Log collection +* Metricbeat: System and service metrics +* Auditbeat: Security events +* Heartbeat: Uptime and endpoint monitoring + +--- + +## 5. Typical Production Architectures + +### Small Cluster + +* 3 nodes with combined roles (master, data, ingest) + +### Medium to Large Cluster + +* 3 Dedicated Master Nodes +* Hot, Warm, and Cold Data Nodes +* Dedicated Ingest Nodes +* Coordinating Nodes +* Optional ML and Transform Nodes + +--- + +## 6. Node Role Summary + +| Node Type | Purpose | +| --------------------------- | ---------------------------- | +| Master | Cluster coordination | +| Data (Hot/Warm/Cold/Frozen) | Data storage and querying | +| Coordinating | Query routing | +| Ingest | Data preprocessing | +| ML | Anomaly detection | +| Transform | Data aggregation | +| Logstash | Data pipeline | +| Kibana | Visualization and management | + diff --git a/Vagrant/Commands.md b/Vagrant/Commands.md new file mode 100644 index 0000000..0eb6282 --- /dev/null +++ b/Vagrant/Commands.md @@ -0,0 +1,159 @@ +# Vagrant Installation and Operations Guide (Debian/Ubuntu) + +## 1. Install Vagrant on Debian/Ubuntu + +Add the HashiCorp GPG key and repository, then install Vagrant: + +```bash +wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg +echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list +sudo apt update && sudo apt install vagrant +``` + +--- + +## 2. Project Setup + +### Initialize a Vagrant Environment + +```bash +vagrant init +``` + +### Initialize with a Specific Base Box + +```bash +vagrant init ubuntu/focal64 +``` + +--- + +## 3. Lifecycle Management + +### Start and Provision the VM + +```bash +vagrant up +``` + +### Stop the VM (Graceful Shutdown) + +```bash +vagrant halt +``` + +### Force Stop (Power Off) + +```bash +vagrant halt -f +``` + +### Reboot the VM + +```bash +vagrant reload +``` + +### Reload and Re-Provision + +```bash +vagrant reload --provision +``` + +--- + +## 4. Provisioning + +### Run Provisioners Without Restarting + +```bash +vagrant provision +``` + +--- + +## 5. Box Operations + +### List Installed Boxes + +```bash +vagrant box list +``` + +### Add a Box + +```bash +vagrant box add ubuntu/focal64 +``` + +### Remove a Box + +```bash +vagrant box remove ubuntu/focal64 +``` + +--- + +## 6. VM Management and Access + +### SSH into the Machine + +```bash +vagrant ssh +``` + +### Get SSH Configuration + +```bash +vagrant ssh-config +``` + +### Check VM Status + +```bash +vagrant status +``` + +### Show Machine Information + +```bash +vagrant info +``` + +--- + +## 7. Cleanup + +### Destroy a VM + +```bash +vagrant destroy +``` + +### Destroy Without Confirmation + +```bash +vagrant destroy -f +``` + +--- + +## 8. Synced Folder and Debugging + +### View Global VM List + +```bash +vagrant global-status +``` + +### Prune Stale Global Entries + +```bash +vagrant global-status --prune +``` + +### Enable Debug Output + +```bash +vagrant up --debug +``` diff --git a/Vagrant/Network.md b/Vagrant/Network.md new file mode 100644 index 0000000..71d54b1 --- /dev/null +++ b/Vagrant/Network.md @@ -0,0 +1,110 @@ +# Vagrant Networking Configuration Guide (DevOps Oriented) + +This document provides a structured reference for configuring Vagrant virtual machine networking, focusing on private, public, and forwarded port configurations commonly used in DevOps workflows. + +--- + +## 1. Types of Networks + +Vagrant supports multiple networking modes. This guide focuses on two primary types: + +### 1.1 Private Network + +A private network allows the VM to communicate only with the host or other VMs on the same private network. Often used for internal service communication or NAT-style layouts. + +**NAT-based private network:** + +```ruby +config.vm.network "private_network", type: "dhcp" +``` + +**Static IP private network:** + +```ruby +config.vm.network "private_network", ip: "192.168.50.4" +``` + +**Static IP with manual configuration (no auto-config):** + +```ruby +config.vm.network "private_network", ip: "192.168.50.4", auto_config: false +``` + +**IPv6 private network:** + +```ruby +config.vm.network "private_network", ip: "fde4:8dba:82e1::c4" +``` + +--- + +### 1.2 Public Network + +A public network bridges the VM directly to the physical network, making it appear as a full-fledged device on the LAN, similar to the host. + +**Basic public network:** + +```ruby +config.vm.network "public_network" +``` + +**Use DHCP-assigned default route:** + +```ruby +config.vm.network "public_network", use_dhcp_assigned_default_route: true +``` + +**Static IP assignment:** + +```ruby +config.vm.network "public_network", ip: "192.168.0.17" +``` + +**Specify network bridge interface:** + +```ruby +config.vm.network "public_network", bridge: "en1: Wi-Fi (AirPort)" +``` + +**Multiple bridge options:** + +```ruby +config.vm.network "public_network", bridge: [ + "en1: Wi-Fi (AirPort)", + "en6: Broadcom NetXtreme Gigabit Ethernet Controller", +] +``` + +--- + +## 2. Port Forwarding + +Port forwarding maps ports from the host machine to the guest VM, allowing external access to services running inside the VM. + +### 2.1 Basic Port Forwarding + +```ruby +config.vm.network "forwarded_port", guest: 80, host: 8080 +``` + +### 2.2 Port Forwarding with Protocol Selection + +```ruby +config.vm.network "forwarded_port", guest: 2003, host: 12003, protocol: "tcp" +config.vm.network "forwarded_port", guest: 2003, host: 12003, protocol: "udp" +``` + +### 2.3 Auto Correcting Port Conflicts + +```ruby +config.vm.network "forwarded_port", guest: 80, host: 8080, auto_correct: true +``` + +### 2.4 Define a Usable Host Port Range + +Useful when Vagrant must auto-correct ports within a controlled range. + +```ruby +config.vm.usable_port_range = 8000..8999 +``` + diff --git a/Vagrant/Vagrantfile.md b/Vagrant/Vagrantfile.md new file mode 100644 index 0000000..d8fbc05 --- /dev/null +++ b/Vagrant/Vagrantfile.md @@ -0,0 +1,245 @@ + +# **Vagrant Configuration Reference for DevOps Engineers** + +This document provides a practical collection of Vagrant configurations commonly used in DevOps workflows. Each example includes explanations and recommended usage patterns. + +--- + +# **1. Basic Ubuntu VM** + +### **Vagrantfile** + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = "ubuntu/focal64" + + config.vm.network "private_network", ip: "192.168.56.10" + + config.vm.provider "virtualbox" do |vb| + vb.memory = "1024" + vb.cpus = 1 + end +end +``` + +### **Summary** + +* Defines a simple Ubuntu VM. +* Adds private network access. +* Sets CPU and memory limits for VirtualBox. + +--- + +# **2. VM with Shell Provisioning** + +### **Vagrantfile** + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = "ubuntu/jammy64" + + config.vm.provision "shell", inline: <<-SHELL + apt update + apt install -y nginx + systemctl enable nginx + SHELL +end +``` + +### **Summary** + +* Runs a shell script automatically on VM boot. +* Installs and enables Nginx. + +--- + +# **3. Multi-Machine Environment (Web + Database)** + +### **Vagrantfile** + +```ruby +Vagrant.configure("2") do |config| + + config.vm.define "web" do |web| + web.vm.box = "ubuntu/focal64" + web.vm.hostname = "web" + web.vm.network "private_network", ip: "192.168.56.11" + + web.vm.provision "shell", inline: <<-SHELL + apt update + apt install -y apache2 + SHELL + end + + config.vm.define "db" do |db| + db.vm.box = "ubuntu/focal64" + db.vm.hostname = "db" + db.vm.network "private_network", ip: "192.168.56.12" + + db.vm.provision "shell", inline: <<-SHELL + apt update + apt install -y mysql-server + SHELL + end + +end +``` + +### **Summary** + +* Creates two VMs. +* Web VM runs Apache. +* DB VM runs MySQL. +* Local network enables app-to-database communication. + +--- + +# **4. Using Ansible for Provisioning** + +### **Vagrantfile** + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = "ubuntu/focal64" + + config.vm.provision "ansible" do |ansible| + ansible.playbook = "playbook.yml" + ansible.inventory_path = "hosts" + end +end +``` + +### **Summary** + +* Leverages Ansible for idempotent provisioning. +* Requires a playbook and inventory file. + +--- + +# **5. Docker Provider Example** + +### **Vagrantfile** + +```ruby +Vagrant.configure("2") do |config| + config.vm.provider "docker" do |d| + d.image = "nginx:latest" + d.remains_running = true + d.ports = ["8080:80"] + end +end +``` + +### **Summary** + +* Uses Docker engine instead of a VM. +* Runs Nginx container accessible on host port 8080. + +--- + +# **6. Synced Folder Example** + +### **Vagrantfile** + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = "ubuntu/focal64" + + config.vm.synced_folder "./app", "/var/www/app", type: "virtualbox" + + config.vm.provision "shell", inline: <<-SHELL + apt update + apt install -y nodejs npm + SHELL +end +``` + +### **Summary** + +* Syncs host source code into VM. +* Suitable for development environments where code updates must sync immediately. + +--- + +# **7. Kubernetes Single-Node Cluster (kubeadm)** + +### **Vagrantfile** + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = "ubuntu/focal64" + config.vm.hostname = "k8s-master" + + config.vm.provider "virtualbox" do |vb| + vb.cpus = 2 + vb.memory = 4096 + end + + config.vm.provision "shell", path: "install-k8s.sh" +end +``` + +### **Summary** + +* Boots a VM with enough resources for Kubernetes. +* Runs external script containing Kubernetes installation steps. + +--- + +# **8. Port Forwarding with Multiple Shell Provisioners and VirtualBox Limits** + +### **Vagrantfile** + +```ruby +Vagrant.configure("2") do |config| + config.vm.box = "ubuntu/focal64" + config.vm.hostname = "webserver" + + # Port forwarding (host 8080 β†’ guest 80) + config.vm.network "forwarded_port", guest: 80, host: 8080 + + # VirtualBox hardware limits + config.vm.provider "virtualbox" do |vb| + vb.memory = "2048" + vb.cpus = 2 + end + + # Provisioner 1: package install + config.vm.provision "shell", inline: <<-SHELL + apt update + apt install -y nginx curl + SHELL + + # Provisioner 2: configuration and service restart + config.vm.provision "shell", inline: <<-SHELL + echo "Hello from Vagrant" > /var/www/html/index.html + systemctl restart nginx + SHELL +end +``` + +### **Summary** + +* Demonstrates forwarded ports for local development. +* Runs two shell provisioners in sequence. +* Applies VirtualBox memory and CPU constraints. + +--- + +# **Additional Recommendations** + +### **General Best Practices** + +* Use `config.ssh.insert_key = false` for shared team environments. +* Install recommended plugins: + + * vagrant-vbguest + * vagrant-hostmanager +* Pin specific box versions for reproducibility. +* Store provisioning scripts in separate files for maintainability. + +### **Resource Allocation Guidelines** + +* Web servers: 1–2 GB RAM, 1–2 CPUs +* Kubernetes nodes: 4 GB+ RAM, 2 CPUs+ +* Database nodes: 2–4 GB RAM minimum diff --git a/Web-Servers/Nginx/13-openssl.md b/Web-Servers/Nginx/13-openssl.md new file mode 100644 index 0000000..9ab776c --- /dev/null +++ b/Web-Servers/Nginx/13-openssl.md @@ -0,0 +1,239 @@ +# OpenSSL Command Reference for Self-Signed Certificate Generation + +This document explains the OpenSSL command-line tool and provides a structured, DevOps-friendly guide for generating self-signed SSL/TLS certificates. These certificates are commonly used for internal services, development environments, testing, or private infrastructure components such as Gitea, Jenkins, internal APIs, or Kubernetes ingress controllers. + +--- + +## 1. What is OpenSSL? + +**OpenSSL** is a widely used, open-source cryptographic toolkit that implements the SSL and TLS protocols. It provides utilities for: + +* Generating private and public key pairs +* Creating Certificate Signing Requests (CSRs) +* Issuing and managing X.509 certificates +* Encrypting and decrypting data +* Inspecting and troubleshooting TLS connections + +OpenSSL is available on most Linux distributions by default and is commonly used in DevOps and SRE workflows. + +Check installed version: + +```bash +openssl version +``` + +--- + +## 2. When to Use Self-Signed Certificates + +Self-signed certificates are suitable for: + +* Internal services +* Development and staging environments +* Private networks +* Lab and proof-of-concept setups + +They are **not recommended for public-facing production systems**, because they are not trusted by browsers or external clients without manual trust configuration. + +--- + +## 3. Generating a Self-Signed Certificate (Example: Gitea) + +This process consists of three logical steps: + +1. Generate a private key +2. Create a Certificate Signing Request (CSR) +3. Sign the CSR using the same key to produce a self-signed certificate + +--- + +## 4. Step 1: Generate a Private Key + +Generate a 2048-bit RSA private key: + +```bash +openssl genrsa -out gitea.key 2048 +``` + +### Explanation + +| Component | Description | +| ---------------- | -------------------------------------------------- | +| `genrsa` | Generates an RSA private key | +| `-out gitea.key` | Output file for the private key | +| `2048` | Key size in bits (2048 is the recommended minimum) | + +Security note: + +* The private key must be kept secret +* Restrict file permissions: + +```bash +chmod 600 gitea.key +``` + +--- + +## 5. Step 2: Create a Certificate Signing Request (CSR) + +Generate a CSR using the private key: + +```bash +openssl req -new -key gitea.key -out gitea.csr +``` + +### Explanation + +| Component | Description | +| ---------------- | ---------------------------------------- | +| `req` | Certificate request management command | +| `-new` | Creates a new CSR | +| `-key gitea.key` | Private key used to generate the request | +| `-out gitea.csr` | Output file for the CSR | + +### Interactive Fields + +During execution, OpenSSL prompts for certificate metadata: + +* Country Name (C) +* State or Province (ST) +* Locality (L) +* Organization (O) +* Organizational Unit (OU) +* **Common Name (CN)** + +Important: + +* The **Common Name (CN)** should match the service hostname or domain name + Example: + + ``` + gitea.example.com + ``` + +For automation or CI/CD pipelines, this step can be made non-interactive (see section 8). + +--- + +## 6. Step 3: Create the Self-Signed Certificate + +Sign the CSR using the same private key: + +```bash +openssl x509 -req -in gitea.csr -signkey gitea.key -out gitea.crt +``` + +### Explanation + +| Component | Description | +| -------------------- | ------------------------------------------ | +| `x509` | X.509 certificate management | +| `-req` | Indicates the input is a CSR | +| `-in gitea.csr` | Input CSR file | +| `-signkey gitea.key` | Signs the certificate with the private key | +| `-out gitea.crt` | Output certificate file | + +Default behavior: + +* Certificate is valid immediately +* Validity is typically 30 days unless specified + +To set validity explicitly (example: 365 days): + +```bash +openssl x509 -req -days 365 -in gitea.csr -signkey gitea.key -out gitea.crt +``` + +--- + +## 7. Verify the Certificate + +Inspect certificate details: + +```bash +openssl x509 -in gitea.crt -text -noout +``` + +Verify key and certificate match: + +```bash +openssl rsa -noout -modulus -in gitea.key | openssl md5 +openssl x509 -noout -modulus -in gitea.crt | openssl md5 +``` + +Matching hashes confirm correctness. + +--- + +## 8. Non-Interactive Certificate Generation (Recommended for Automation) + +Generate key and certificate in one command: + +```bash +openssl req -x509 -newkey rsa:2048 \ +-keyout gitea.key \ +-out gitea.crt \ +-days 365 \ +-nodes \ +-subj "/C=US/ST=State/L=City/O=Company/OU=DevOps/CN=gitea.example.com" +``` + +Options explained: + +* `-x509`: Generate a self-signed certificate +* `-newkey`: Create a new private key +* `-nodes`: Do not encrypt the private key (required for services) +* `-subj`: Certificate subject (non-interactive) + +This approach is ideal for: + +* CI/CD pipelines +* Docker images +* Infrastructure as Code workflows + +--- + +## 9. Subject Alternative Name (SAN) Consideration + +Modern TLS clients require **SAN** instead of CN. + +Example OpenSSL config snippet: + +``` +subjectAltName = DNS:gitea.example.com,DNS:gitea.internal +``` + +Without SAN: + +* Browsers may reject the certificate +* TLS warnings may appear even if CN matches + +--- + +## 10. Summary of Generated Files + +| File | Description | +| ----------- | ---------------------------------------------------- | +| `gitea.key` | Private key (keep secure, server-side only) | +| `gitea.csr` | Certificate Signing Request (optional after signing) | +| `gitea.crt` | Self-signed certificate presented to clients | + +--- + +## 11. Best Practices + +* Never commit private keys to Git +* Store secrets securely (Vault, SSM, Kubernetes secrets) +* Rotate certificates regularly +* Use Let’s Encrypt or internal CA for production +* Prefer SAN over CN +* Automate certificate generation where possible + +--- + +If you want, I can: + +* Add an OpenSSL SAN configuration example +* Provide Kubernetes, Nginx, or Gitea-specific TLS configs +* Convert this into a reusable internal PKI guide +* Add troubleshooting and common TLS errors