radin/my-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gitlab cache added runner config for s3

Browse Source
This commit is contained in:
RadinPirouz
2025-10-04 12:40:39 +03:30
parent 83114643c8
commit 5bb174079e
1 changed files with 48 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
Code-Management/Gitlab/Gitlab-Cache.md
View File

@@ -1,21 +1,21 @@
# GitLab CI/CD Cache Guide # GitLab CI/CD Cache Guide
GitLab provides a caching system for jobs to speed up pipelines by reusing dependencies like `node_modules`, Ruby gems, and other build artifacts. This guide covers how to configure and use caching effectively. GitLab CI/CD provides a caching mechanism to speed up pipeline execution by reusing dependencies and build artifacts such as `node_modules`, Ruby gems, or compiled binaries. This guide covers how to configure, optimize, and manage caching effectively.
--- ---
## Key Features ## Key Features of GitLab Cache
* **Cache Storage**: Data is saved in the GitLab Runner and can also be stored in external storage such as S3, MinIO, GCS, etc. * **Cache Storage**: Cache data is stored in the GitLab Runner and can optionally be persisted in external storage such as S3, MinIO, Google Cloud Storage (GCS), or Azure Blob Storage.
* **Branch-Specific Caching**: Maintain separate caches per branch, job, or stage. * **Branch-Specific Caching**: Create independent caches per branch, job, or pipeline stage to avoid conflicts.
* **Fallbacks**: Jobs can pull cache from other branches if the current branch does not have cache. * **Fallbacks**: Jobs can retrieve cache from other branches if the current branch has no cache yet.
* **Simple Cache**: Uses file locks and directory caching for efficiency. * **Simple & Efficient**: Uses file locks and directory caching to maximize efficiency without complex configurations.
--- ---
## Basic Cache Example ## Basic Cache Example
Cache dependencies like Ruby gems and Yarn modules: A typical caching configuration for Ruby gems and Yarn modules:
```yaml ```yaml
test-job: test-job:
@@ -39,13 +39,34 @@ test-job:
``` ```
**Behavior**: **Behavior**:
If the lock file (`Gemfile.lock` or `yarn.lock`) changes, the cache is updated automatically. If the lock file (`Gemfile.lock` or `yarn.lock`) changes, the cache is updated automatically, ensuring dependencies remain consistent with the project.
--- ---
## Branch Cache with Fallbacks ## Configuring GitLab Runner to Use S3 Cache
When creating a new branch, it may not have a cache yet. You can pull from the default branch (e.g., `main` or `master`) or a global default cache: You can configure the GitLab Runner to save cache to S3 storage as follows:
```toml
[[runners]]
[runners.cache]
Type = "s3"
[runners.cache.s3]
ServerAddress = "s3.example.com"
AccessKey = "access-key"
SecretKey = "secret-key"
BucketName = "runner"
Insecure = true # true = http, false = https
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
```
---
## Branch-Specific Cache with Fallbacks
When a new branch does not yet have a cache, it can use fallback caches from other branches:
```yaml ```yaml
test-job: test-job:
@@ -67,7 +88,7 @@ test-job:
## Default Cache Configuration ## Default Cache Configuration
Define a global cache for all jobs and override it for specific jobs as needed: Define a global cache and override it per job as necessary:
```yaml ```yaml
default: default:
@@ -82,7 +103,7 @@ default:
job: job:
cache: cache:
<<: *global_cache <<: *global_cache
policy: pull # Override policy for this job policy: pull # Overrides global policy for this job
``` ```
--- ---
@@ -90,22 +111,22 @@ job:
## Cache Policies ## Cache Policies
| Policy | Description | | Policy | Description |
| --------- | ------------------------------------------------------ | | --------- | ------------------------------------------------- |
| pull | Only fetch the cache, do not update it. | | pull | Fetches cache without updating it |
| pull-push | Fetch the cache and update it after the job completes. | | pull-push | Fetches cache and updates it after job completion |
--- ---
## Cache Examples ## Cache Examples
* **Per Job & Branch** **Per Job & Branch**:
```yaml ```yaml
cache: cache:
key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" key: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
``` ```
* **Per Stage & Branch** **Per Stage & Branch**:
```yaml ```yaml
cache: cache:
@@ -116,7 +137,7 @@ cache:
## Conditional Cache Policy ## Conditional Cache Policy
Update cache only on the default branch; otherwise, just pull: Update cache only on the default branch while other branches only fetch:
```yaml ```yaml
conditional-policy: conditional-policy:
@@ -142,7 +163,7 @@ conditional-policy:
## Cache Based on Lock Files ## Cache Based on Lock Files
Update cache automatically when `package-lock.json` changes: Automatically update cache when lock files change, e.g., `package-lock.json`:
```yaml ```yaml
default: default:
@@ -154,16 +175,15 @@ default:
- .npm/ - .npm/
``` ```
**Tip**: This ensures your npm modules are always synced with the lock file. **Tip**: Using lock files ensures that cached dependencies are always consistent with the project’s declared versions.
--- ---
## Summary ## Best Practices Summary
* Use **branch-specific keys** for independent caches. * Use **branch-specific keys** for independent caches to avoid conflicts.
* Use **fallback_keys** to share cache between branches. * Use **fallback_keys** to share caches across branches.
* **Policies** control whether cache is updated or only fetched. * Use **cache policies** to control whether cache is updated or only fetched.
* Use **lock files** to trigger cache updates automatically. * Leverage **lock files** to trigger cache updates automatically.
Caching effectively can dramatically reduce build times and maintain clean, efficient pipelines.
Proper caching can dramatically reduce build times, improve pipeline efficiency, and ensure consistent builds across branches and environments.