Add Template to deploy forgejo.
This template allows deploying a forgejo en either Scaleway or Hetzner (untested) without much knowledge about them. It DOES require knowledge about Terragrunt and ansible. A wizard of sorts is provided but it will not guarantee success without some knowledge about the underlying technology.
This commit is contained in:
parent
a9f546f92a
commit
822e42dbb8
48 changed files with 6846 additions and 2 deletions
574
docs/OPERATIONS.md
Normal file
574
docs/OPERATIONS.md
Normal file
|
|
@ -0,0 +1,574 @@
|
|||
# Operations Guide - Forgejo Self-Hosting
|
||||
|
||||
Complete guide for day-to-day operations and maintenance.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Daily Operations](#daily-operations)
|
||||
2. [Backup & Recovery](#backup--recovery)
|
||||
3. [Updates & Upgrades](#updates--upgrades)
|
||||
4. [Monitoring](#monitoring)
|
||||
5. [Troubleshooting](#troubleshooting)
|
||||
6. [Security Operations](#security-operations)
|
||||
7. [Scaling](#scaling)
|
||||
8. [Disaster Recovery](#disaster-recovery)
|
||||
|
||||
## Daily Operations
|
||||
|
||||
### Health Checks
|
||||
|
||||
```bash
|
||||
# Quick health check
|
||||
make health
|
||||
|
||||
# Full status
|
||||
make status
|
||||
|
||||
# View recent logs
|
||||
make logs | tail -100
|
||||
```
|
||||
|
||||
### User Management
|
||||
|
||||
```bash
|
||||
# SSH into server
|
||||
make ssh
|
||||
|
||||
# List users
|
||||
docker exec --user git forgejo forgejo admin user list
|
||||
|
||||
# Create user
|
||||
docker exec --user git forgejo forgejo admin user create \
|
||||
--username newuser \
|
||||
--password 'SecureP@ssw0rd' \
|
||||
--email user@example.com
|
||||
|
||||
# Change user password
|
||||
docker exec --user git forgejo forgejo admin user change-password \
|
||||
--username user \
|
||||
--password 'NewP@ssw0rd'
|
||||
|
||||
# Make user admin
|
||||
docker exec --user git forgejo forgejo admin user change-admin \
|
||||
--username user \
|
||||
--admin
|
||||
```
|
||||
|
||||
### Repository Management
|
||||
|
||||
```bash
|
||||
# Sync mirrors
|
||||
docker exec --user git forgejo forgejo admin repo-sync-releases
|
||||
|
||||
# Rebuild indexes
|
||||
docker exec --user git forgejo forgejo admin regenerate hooks
|
||||
docker exec --user git forgejo forgejo admin regenerate keys
|
||||
```
|
||||
|
||||
## Backup & Recovery
|
||||
|
||||
### Manual Backup
|
||||
|
||||
```bash
|
||||
# Create immediate backup
|
||||
make backup
|
||||
|
||||
# Backup with S3 upload
|
||||
make backup-to-s3
|
||||
|
||||
# Verify backup
|
||||
make ssh
|
||||
ls -lh /opt/forgejo/backups/
|
||||
```
|
||||
|
||||
### Automated Backups
|
||||
|
||||
Backups run automatically at 2 AM daily.
|
||||
|
||||
**Check backup status:**
|
||||
```bash
|
||||
make ssh
|
||||
tail -f /var/log/forgejo-backup.log
|
||||
```
|
||||
|
||||
**Modify schedule:**
|
||||
```yaml
|
||||
# In inventory or vars
|
||||
forgejo_backup_schedule: "0 3 * * *" # 3 AM daily
|
||||
```
|
||||
|
||||
### Restore from Backup
|
||||
|
||||
```bash
|
||||
# List available backups
|
||||
make ssh
|
||||
ls /opt/forgejo/backups/
|
||||
|
||||
# Restore (timestamp format: 20240115T120000)
|
||||
make restore
|
||||
# Enter backup timestamp when prompted
|
||||
|
||||
# Force restore without confirmation
|
||||
make restore-force
|
||||
|
||||
# Restore from S3
|
||||
make restore-from-s3
|
||||
```
|
||||
|
||||
### Backup Verification
|
||||
|
||||
```bash
|
||||
# Test restore in separate environment
|
||||
make ssh
|
||||
cd /opt/forgejo/backups
|
||||
|
||||
# Verify database backup
|
||||
gunzip -c database-TIMESTAMP.sql.gz | head -100
|
||||
|
||||
# Verify repositories backup
|
||||
tar -tzf repositories-TIMESTAMP.tar.gz | head -20
|
||||
```
|
||||
|
||||
## Updates & Upgrades
|
||||
|
||||
### Update Forgejo
|
||||
|
||||
```bash
|
||||
# Standard update (includes backup)
|
||||
make update
|
||||
|
||||
# Update without backup (not recommended)
|
||||
make update-no-backup
|
||||
```
|
||||
|
||||
**Update process:**
|
||||
1. Creates pre-update backup
|
||||
2. Stops Forgejo service
|
||||
3. Pulls new Docker image
|
||||
4. Starts service
|
||||
5. Runs database migrations
|
||||
6. Verifies health
|
||||
|
||||
### Update System Packages
|
||||
|
||||
```bash
|
||||
make ssh
|
||||
sudo apt update
|
||||
sudo apt upgrade -y
|
||||
sudo reboot # If kernel updated
|
||||
```
|
||||
|
||||
### Update Infrastructure
|
||||
|
||||
```bash
|
||||
# Review changes
|
||||
make terraform-plan PROVIDER=scaleway
|
||||
|
||||
# Apply updates
|
||||
make terraform-apply PROVIDER=scaleway
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Log Management
|
||||
|
||||
```bash
|
||||
# Forgejo logs
|
||||
make logs
|
||||
|
||||
# Follow logs (real-time)
|
||||
make logs | tail -f
|
||||
|
||||
# Nginx access logs
|
||||
make logs-nginx
|
||||
|
||||
# System logs
|
||||
make ssh
|
||||
sudo journalctl -u forgejo -f
|
||||
|
||||
# Docker logs
|
||||
docker logs forgejo --tail 100 -f
|
||||
```
|
||||
|
||||
### Performance Monitoring
|
||||
|
||||
```bash
|
||||
# Check resource usage
|
||||
make ssh
|
||||
|
||||
# CPU and memory
|
||||
htop
|
||||
|
||||
# Disk usage
|
||||
df -h
|
||||
du -sh /opt/forgejo/*
|
||||
|
||||
# Docker stats
|
||||
docker stats forgejo
|
||||
|
||||
# PostgreSQL status
|
||||
sudo systemctl status postgresql
|
||||
```
|
||||
|
||||
### Database Monitoring
|
||||
|
||||
```bash
|
||||
make ssh
|
||||
|
||||
# Connect to PostgreSQL
|
||||
sudo -u postgres psql forgejo
|
||||
|
||||
# Check database size
|
||||
SELECT pg_size_pretty(pg_database_size('forgejo'));
|
||||
|
||||
# Check table sizes
|
||||
SELECT
|
||||
schemaname,
|
||||
tablename,
|
||||
pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
|
||||
FROM pg_tables
|
||||
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
|
||||
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
|
||||
LIMIT 10;
|
||||
|
||||
# Active connections
|
||||
SELECT count(*) FROM pg_stat_activity;
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service Won't Start
|
||||
|
||||
```bash
|
||||
# Check Docker status
|
||||
make ssh
|
||||
docker ps -a
|
||||
docker logs forgejo
|
||||
|
||||
# Check system resources
|
||||
free -h
|
||||
df -h
|
||||
|
||||
# Restart service
|
||||
make restart
|
||||
|
||||
# Full rebuild if needed
|
||||
docker compose down
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
```bash
|
||||
make ssh
|
||||
|
||||
# Check PostgreSQL
|
||||
sudo systemctl status postgresql
|
||||
|
||||
# Check connections
|
||||
sudo -u postgres psql -c "SELECT * FROM pg_stat_activity;"
|
||||
|
||||
# Restart PostgreSQL
|
||||
sudo systemctl restart postgresql
|
||||
|
||||
# Check logs
|
||||
sudo tail -f /var/log/postgresql/postgresql-16-main.log
|
||||
```
|
||||
|
||||
### SSL Certificate Issues
|
||||
|
||||
```bash
|
||||
make ssh
|
||||
|
||||
# Check certificate status
|
||||
sudo certbot certificates
|
||||
|
||||
# Renew manually
|
||||
sudo certbot renew --force-renewal
|
||||
|
||||
# Check Nginx configuration
|
||||
sudo nginx -t
|
||||
sudo systemctl restart nginx
|
||||
|
||||
# Check certificate files
|
||||
sudo ls -la /etc/letsencrypt/live/git.yourdomain.com/
|
||||
```
|
||||
|
||||
### Performance Issues
|
||||
|
||||
```bash
|
||||
# Check slow queries
|
||||
make ssh
|
||||
sudo -u postgres psql forgejo
|
||||
|
||||
# Enable slow query logging
|
||||
ALTER SYSTEM SET log_min_duration_statement = 1000;
|
||||
SELECT pg_reload_conf();
|
||||
|
||||
# View slow queries
|
||||
sudo tail -f /var/log/postgresql/postgresql-16-main.log
|
||||
|
||||
# Vacuum database
|
||||
sudo -u postgres vacuumdb --analyze forgejo
|
||||
```
|
||||
|
||||
### Disk Space Issues
|
||||
|
||||
```bash
|
||||
make ssh
|
||||
|
||||
# Check space
|
||||
df -h
|
||||
|
||||
# Find large directories
|
||||
du -h /opt/forgejo | sort -rh | head -20
|
||||
|
||||
# Clean Docker
|
||||
docker system prune -a
|
||||
|
||||
# Clean old backups
|
||||
find /opt/forgejo/backups -type f -mtime +30 -delete
|
||||
|
||||
# Clean logs
|
||||
sudo journalctl --vacuum-time=7d
|
||||
```
|
||||
|
||||
## Security Operations
|
||||
|
||||
### Security Audit
|
||||
|
||||
```bash
|
||||
make ssh
|
||||
|
||||
# Check for security updates
|
||||
sudo apt update
|
||||
sudo apt list --upgradable
|
||||
|
||||
# Review firewall rules
|
||||
sudo ufw status verbose
|
||||
|
||||
# Check open ports
|
||||
sudo netstat -tulpn
|
||||
|
||||
# Review SSH configuration
|
||||
cat /etc/ssh/sshd_config
|
||||
|
||||
# Check failed login attempts
|
||||
sudo grep "Failed password" /var/log/auth.log | tail -20
|
||||
```
|
||||
|
||||
### Rotate Secrets
|
||||
|
||||
```bash
|
||||
# Generate new secrets
|
||||
openssl rand -base64 48
|
||||
|
||||
# Update vault
|
||||
make ansible-vault-edit
|
||||
|
||||
# Redeploy with new secrets
|
||||
make deploy
|
||||
```
|
||||
|
||||
### SSL Certificate Renewal
|
||||
|
||||
```bash
|
||||
# Auto-renewal is configured, but to force:
|
||||
make ssh
|
||||
sudo certbot renew --force-renewal
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
### Security Updates
|
||||
|
||||
```bash
|
||||
# Enable automatic security updates (already configured)
|
||||
make ssh
|
||||
sudo cat /etc/apt/apt.conf.d/50unattended-upgrades
|
||||
|
||||
# Check update history
|
||||
sudo cat /var/log/unattended-upgrades/unattended-upgrades.log
|
||||
```
|
||||
|
||||
## Scaling
|
||||
|
||||
### Vertical Scaling (More Resources)
|
||||
|
||||
```bash
|
||||
# Update instance type
|
||||
vim terraform/scaleway/compute/terraform.tfvars
|
||||
# Change: instance_type = "DEV1-L" # 8GB RAM
|
||||
|
||||
make terraform-apply PROVIDER=scaleway
|
||||
|
||||
# Service restarts automatically
|
||||
```
|
||||
|
||||
### Database Optimization
|
||||
|
||||
```bash
|
||||
make ssh
|
||||
sudo -u postgres psql forgejo
|
||||
|
||||
# Optimize settings
|
||||
ALTER SYSTEM SET shared_buffers = '512MB';
|
||||
ALTER SYSTEM SET effective_cache_size = '2GB';
|
||||
ALTER SYSTEM SET maintenance_work_mem = '256MB';
|
||||
|
||||
SELECT pg_reload_conf();
|
||||
```
|
||||
|
||||
### Add Storage Volume
|
||||
|
||||
```bash
|
||||
# In inventory
|
||||
forgejo_use_external_volume: true
|
||||
forgejo_volume_device: /dev/sdb
|
||||
|
||||
# Redeploy
|
||||
make deploy-tags TAGS=volume
|
||||
```
|
||||
|
||||
## Disaster Recovery
|
||||
|
||||
### Complete Failure Recovery
|
||||
|
||||
```bash
|
||||
# 1. Create new infrastructure
|
||||
make terraform-apply PROVIDER=scaleway
|
||||
|
||||
# 2. Update inventory with new IP
|
||||
vim ansible/inventory/production/hosts.yml
|
||||
|
||||
# 3. Update DNS records
|
||||
# Point domain to new IP
|
||||
|
||||
# 4. Deploy with restore
|
||||
# First, copy backup timestamp
|
||||
make ssh
|
||||
ls /opt/forgejo/backups/ # Note timestamp
|
||||
|
||||
# Then restore
|
||||
make restore
|
||||
# Enter timestamp when prompted
|
||||
|
||||
# 5. Verify
|
||||
make health
|
||||
```
|
||||
|
||||
### Backup Storage Migration
|
||||
|
||||
```bash
|
||||
# Copy backups to new location
|
||||
make ssh
|
||||
rsync -avz /opt/forgejo/backups/ /new/backup/location/
|
||||
|
||||
# Update backup configuration
|
||||
vim ansible/playbooks/vars/main.yml
|
||||
# forgejo_backup_path: /new/backup/location
|
||||
|
||||
make deploy-tags TAGS=backup
|
||||
```
|
||||
|
||||
### Database Recovery from Corruption
|
||||
|
||||
```bash
|
||||
make ssh
|
||||
|
||||
# Stop Forgejo
|
||||
docker compose down
|
||||
|
||||
# Backup corrupted database
|
||||
sudo -u postgres pg_dump forgejo > /tmp/forgejo-corrupted.sql
|
||||
|
||||
# Drop and recreate
|
||||
sudo -u postgres dropdb forgejo
|
||||
sudo -u postgres createdb forgejo
|
||||
|
||||
# Restore from latest backup
|
||||
gunzip -c /opt/forgejo/backups/database-TIMESTAMP.sql.gz | \
|
||||
sudo -u postgres psql forgejo
|
||||
|
||||
# Restart Forgejo
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Daily Checklist
|
||||
|
||||
- [ ] Check service health: `make health`
|
||||
- [ ] Review logs for errors: `make logs | grep -i error`
|
||||
- [ ] Verify backups ran: `ls -lt /opt/forgejo/backups/ | head -5`
|
||||
- [ ] Check disk space: `df -h`
|
||||
|
||||
### Weekly Checklist
|
||||
|
||||
- [ ] Review security logs: `sudo grep "Failed" /var/log/auth.log`
|
||||
- [ ] Check for updates: `sudo apt update && apt list --upgradable`
|
||||
- [ ] Test backup restore (in dev environment)
|
||||
- [ ] Review performance metrics
|
||||
- [ ] Check SSL certificate expiry
|
||||
|
||||
### Monthly Checklist
|
||||
|
||||
- [ ] Full security audit
|
||||
- [ ] Review and update firewall rules
|
||||
- [ ] Rotate secrets if needed
|
||||
- [ ] Review and optimize database
|
||||
- [ ] Update documentation
|
||||
- [ ] Test disaster recovery procedures
|
||||
|
||||
## Emergency Contacts
|
||||
|
||||
Keep these handy:
|
||||
|
||||
```bash
|
||||
# Quick recovery commands
|
||||
make health # Check status
|
||||
make restart # Restart service
|
||||
make logs # View logs
|
||||
make backup # Create backup
|
||||
make restore # Restore from backup
|
||||
make ssh # SSH access
|
||||
|
||||
# Emergency rollback
|
||||
make update # Includes backup
|
||||
# If issues: make restore # Roll back
|
||||
```
|
||||
|
||||
## Useful Scripts
|
||||
|
||||
### Health Check Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Save as: health-check.sh
|
||||
|
||||
echo "=== Forgejo Health Check ==="
|
||||
echo "Date: $(date)"
|
||||
echo ""
|
||||
|
||||
echo "1. Service Status:"
|
||||
docker ps | grep forgejo
|
||||
|
||||
echo ""
|
||||
echo "2. Disk Space:"
|
||||
df -h | grep -E "Filesystem|/dev/vda1|/dev/sda1"
|
||||
|
||||
echo ""
|
||||
echo "3. Memory Usage:"
|
||||
free -h
|
||||
|
||||
echo ""
|
||||
echo "4. Latest Backup:"
|
||||
ls -lth /opt/forgejo/backups/*.tar.gz | head -1
|
||||
|
||||
echo ""
|
||||
echo "5. HTTP Status:"
|
||||
curl -s -o /dev/null -w "%{http_code}" http://localhost:3000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Remember**: Always test procedures in a development environment first!
|
||||
Loading…
Add table
Add a link
Reference in a new issue