forgejo-autohebergement/docs/OPERATIONS.md

Operations Guide - Forgejo Self-Hosting

Complete guide for day-to-day operations and maintenance.

Table of Contents

1. Daily Operations
2. Backup & Recovery
3. Updates & Upgrades
4. Monitoring
5. Troubleshooting
6. Security Operations
7. Scaling
8. Disaster Recovery

Daily Operations

Health Checks

# Quick health check
make health

# Full status
make status

# View recent logs
make logs | tail -100

User Management

# SSH into server
make ssh

# List users
docker exec --user git forgejo forgejo admin user list

# Create user
docker exec --user git forgejo forgejo admin user create \
  --username newuser \
  --password 'SecureP@ssw0rd' \
  --email user@example.com

# Change user password
docker exec --user git forgejo forgejo admin user change-password \
  --username user \
  --password 'NewP@ssw0rd'

# Make user admin
docker exec --user git forgejo forgejo admin user change-admin \
  --username user \
  --admin

Repository Management

# Sync mirrors
docker exec --user git forgejo forgejo admin repo-sync-releases

# Rebuild indexes
docker exec --user git forgejo forgejo admin regenerate hooks
docker exec --user git forgejo forgejo admin regenerate keys

Backup & Recovery

Manual Backup

# Create immediate backup
make backup

# Backup with S3 upload
make backup-to-s3

# Verify backup
make ssh
ls -lh /opt/forgejo/backups/

Automated Backups

Backups run automatically at 2 AM daily.

Check backup status:

make ssh
tail -f /var/log/forgejo-backup.log

Modify schedule:

# In inventory or vars
forgejo_backup_schedule: "0 3 * * *"  # 3 AM daily

Restore from Backup

# List available backups
make ssh
ls /opt/forgejo/backups/

# Restore (timestamp format: 20240115T120000)
make restore
# Enter backup timestamp when prompted

# Force restore without confirmation
make restore-force

# Restore from S3
make restore-from-s3

Backup Verification

# Test restore in separate environment
make ssh
cd /opt/forgejo/backups

# Verify database backup
gunzip -c database-TIMESTAMP.sql.gz | head -100

# Verify repositories backup
tar -tzf repositories-TIMESTAMP.tar.gz | head -20

Updates & Upgrades

Update Forgejo

# Standard update (includes backup)
make update

# Update without backup (not recommended)
make update-no-backup

Update process:

1. Creates pre-update backup
2. Stops Forgejo service
3. Pulls new Docker image
4. Starts service
5. Runs database migrations
6. Verifies health

Update System Packages

make ssh
sudo apt update
sudo apt upgrade -y
sudo reboot  # If kernel updated

Update Infrastructure

# Review changes
make terraform-plan PROVIDER=scaleway

# Apply updates
make terraform-apply PROVIDER=scaleway

Monitoring

Log Management

# Forgejo logs
make logs

# Follow logs (real-time)
make logs | tail -f

# Nginx access logs
make logs-nginx

# System logs
make ssh
sudo journalctl -u forgejo -f

# Docker logs
docker logs forgejo --tail 100 -f

Performance Monitoring

# Check resource usage
make ssh

# CPU and memory
htop

# Disk usage
df -h
du -sh /opt/forgejo/*

# Docker stats
docker stats forgejo

# PostgreSQL status
sudo systemctl status postgresql

Database Monitoring

make ssh

# Connect to PostgreSQL
sudo -u postgres psql forgejo

# Check database size
SELECT pg_size_pretty(pg_database_size('forgejo'));

# Check table sizes
SELECT 
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;

# Active connections
SELECT count(*) FROM pg_stat_activity;

Troubleshooting

Service Won't Start

# Check Docker status
make ssh
docker ps -a
docker logs forgejo

# Check system resources
free -h
df -h

# Restart service
make restart

# Full rebuild if needed
docker compose down
docker compose up -d

Database Connection Issues

make ssh

# Check PostgreSQL
sudo systemctl status postgresql

# Check connections
sudo -u postgres psql -c "SELECT * FROM pg_stat_activity;"

# Restart PostgreSQL
sudo systemctl restart postgresql

# Check logs
sudo tail -f /var/log/postgresql/postgresql-16-main.log

SSL Certificate Issues

make ssh

# Check certificate status
sudo certbot certificates

# Renew manually
sudo certbot renew --force-renewal

# Check Nginx configuration
sudo nginx -t
sudo systemctl restart nginx

# Check certificate files
sudo ls -la /etc/letsencrypt/live/git.yourdomain.com/

Performance Issues

# Check slow queries
make ssh
sudo -u postgres psql forgejo

# Enable slow query logging
ALTER SYSTEM SET log_min_duration_statement = 1000;
SELECT pg_reload_conf();

# View slow queries
sudo tail -f /var/log/postgresql/postgresql-16-main.log

# Vacuum database
sudo -u postgres vacuumdb --analyze forgejo

Disk Space Issues

make ssh

# Check space
df -h

# Find large directories
du -h /opt/forgejo | sort -rh | head -20

# Clean Docker
docker system prune -a

# Clean old backups
find /opt/forgejo/backups -type f -mtime +30 -delete

# Clean logs
sudo journalctl --vacuum-time=7d

Security Operations

Security Audit

make ssh

# Check for security updates
sudo apt update
sudo apt list --upgradable

# Review firewall rules
sudo ufw status verbose

# Check open ports
sudo netstat -tulpn

# Review SSH configuration
cat /etc/ssh/sshd_config

# Check failed login attempts
sudo grep "Failed password" /var/log/auth.log | tail -20

Rotate Secrets

# Generate new secrets
openssl rand -base64 48

# Update vault
make ansible-vault-edit

# Redeploy with new secrets
make deploy

SSL Certificate Renewal

# Auto-renewal is configured, but to force:
make ssh
sudo certbot renew --force-renewal
sudo systemctl reload nginx

Security Updates

# Enable automatic security updates (already configured)
make ssh
sudo cat /etc/apt/apt.conf.d/50unattended-upgrades

# Check update history
sudo cat /var/log/unattended-upgrades/unattended-upgrades.log

Scaling

Vertical Scaling (More Resources)

# Update instance type
vim terraform/scaleway/compute/terraform.tfvars
# Change: instance_type = "DEV1-L"  # 8GB RAM

make terraform-apply PROVIDER=scaleway

# Service restarts automatically

Database Optimization

make ssh
sudo -u postgres psql forgejo

# Optimize settings
ALTER SYSTEM SET shared_buffers = '512MB';
ALTER SYSTEM SET effective_cache_size = '2GB';
ALTER SYSTEM SET maintenance_work_mem = '256MB';

SELECT pg_reload_conf();

Add Storage Volume

# In inventory
forgejo_use_external_volume: true
forgejo_volume_device: /dev/sdb

# Redeploy
make deploy-tags TAGS=volume

Disaster Recovery

Complete Failure Recovery

# 1. Create new infrastructure
make terraform-apply PROVIDER=scaleway

# 2. Update inventory with new IP
vim ansible/inventory/production/hosts.yml

# 3. Update DNS records
# Point domain to new IP

# 4. Deploy with restore
# First, copy backup timestamp
make ssh
ls /opt/forgejo/backups/  # Note timestamp

# Then restore
make restore
# Enter timestamp when prompted

# 5. Verify
make health

Backup Storage Migration

# Copy backups to new location
make ssh
rsync -avz /opt/forgejo/backups/ /new/backup/location/

# Update backup configuration
vim ansible/playbooks/vars/main.yml
# forgejo_backup_path: /new/backup/location

make deploy-tags TAGS=backup

Database Recovery from Corruption

make ssh

# Stop Forgejo
docker compose down

# Backup corrupted database
sudo -u postgres pg_dump forgejo > /tmp/forgejo-corrupted.sql

# Drop and recreate
sudo -u postgres dropdb forgejo
sudo -u postgres createdb forgejo

# Restore from latest backup
gunzip -c /opt/forgejo/backups/database-TIMESTAMP.sql.gz | \
  sudo -u postgres psql forgejo

# Restart Forgejo
docker compose up -d

Best Practices

Daily Checklist

• Check service health: make health
• Review logs for errors: make logs | grep -i error
• Verify backups ran: ls -lt /opt/forgejo/backups/ | head -5
• Check disk space: df -h

Weekly Checklist

• Review security logs: sudo grep "Failed" /var/log/auth.log
• Check for updates: sudo apt update && apt list --upgradable
• Test backup restore (in dev environment)
• Review performance metrics
• Check SSL certificate expiry

Monthly Checklist

• Full security audit
• Review and update firewall rules
• Rotate secrets if needed
• Review and optimize database
• Update documentation
• Test disaster recovery procedures

Emergency Contacts

Keep these handy:

# Quick recovery commands
make health              # Check status
make restart             # Restart service
make logs                # View logs
make backup              # Create backup
make restore             # Restore from backup
make ssh                 # SSH access

# Emergency rollback
make update              # Includes backup
# If issues: make restore  # Roll back

Useful Scripts

Health Check Script

#!/bin/bash
# Save as: health-check.sh

echo "=== Forgejo Health Check ==="
echo "Date: $(date)"
echo ""

echo "1. Service Status:"
docker ps | grep forgejo

echo ""
echo "2. Disk Space:"
df -h | grep -E "Filesystem|/dev/vda1|/dev/sda1"

echo ""
echo "3. Memory Usage:"
free -h

echo ""
echo "4. Latest Backup:"
ls -lth /opt/forgejo/backups/*.tar.gz | head -1

echo ""
echo "5. HTTP Status:"
curl -s -o /dev/null -w "%{http_code}" http://localhost:3000

---

Remember: Always test procedures in a development environment first!