# Operations Guide - Forgejo Self-Hosting

Complete guide for day-to-day operations and maintenance.

## Table of Contents

1. [Daily Operations](#daily-operations)
2. [Backup & Recovery](#backup--recovery)
3. [Updates & Upgrades](#updates--upgrades)
4. [Monitoring](#monitoring)
5. [Troubleshooting](#troubleshooting)
6. [Security Operations](#security-operations)
7. [Scaling](#scaling)
8. [Disaster Recovery](#disaster-recovery)

## Daily Operations

### Health Checks

```bash
# Quick health check
make health

# Full status
make status

# View recent logs
make logs | tail -100
```

### User Management

```bash
# SSH into server
make ssh

# List users
docker exec --user git forgejo forgejo admin user list

# Create user
docker exec --user git forgejo forgejo admin user create \
  --username newuser \
  --password 'SecureP@ssw0rd' \
  --email user@example.com

# Change user password
docker exec --user git forgejo forgejo admin user change-password \
  --username user \
  --password 'NewP@ssw0rd'

# Make user admin
docker exec --user git forgejo forgejo admin user change-admin \
  --username user \
  --admin
```

### Repository Management

```bash
# Sync mirrors
docker exec --user git forgejo forgejo admin repo-sync-releases

# Rebuild indexes
docker exec --user git forgejo forgejo admin regenerate hooks
docker exec --user git forgejo forgejo admin regenerate keys
```

## Backup & Recovery

### Manual Backup

```bash
# Create immediate backup
make backup

# Backup with S3 upload
make backup-to-s3

# Verify backup
make ssh
ls -lh /opt/forgejo/backups/
```

### Automated Backups

Backups run automatically at 2 AM daily.

**Check backup status:**
```bash
make ssh
tail -f /var/log/forgejo-backup.log
```

**Modify schedule:**
```yaml
# In inventory or vars
forgejo_backup_schedule: "0 3 * * *"  # 3 AM daily
```

### Restore from Backup

```bash
# List available backups
make ssh
ls /opt/forgejo/backups/

# Restore (timestamp format: 20240115T120000)
make restore
# Enter backup timestamp when prompted

# Force restore without confirmation
make restore-force

# Restore from S3
make restore-from-s3
```

### Backup Verification

```bash
# Test restore in separate environment
make ssh
cd /opt/forgejo/backups

# Verify database backup
gunzip -c database-TIMESTAMP.sql.gz | head -100

# Verify repositories backup
tar -tzf repositories-TIMESTAMP.tar.gz | head -20
```

## Updates & Upgrades

### Update Forgejo

```bash
# Standard update (includes backup)
make update

# Update without backup (not recommended)
make update-no-backup
```

**Update process:**
1. Creates pre-update backup
2. Stops Forgejo service
3. Pulls new Docker image
4. Starts service
5. Runs database migrations
6. Verifies health

### Update System Packages

```bash
make ssh
sudo apt update
sudo apt upgrade -y
sudo reboot  # If kernel updated
```

### Update Infrastructure

```bash
# Review changes
make terraform-plan PROVIDER=scaleway

# Apply updates
make terraform-apply PROVIDER=scaleway
```

## Monitoring

### Log Management

```bash
# Forgejo logs
make logs

# Follow logs (real-time)
make logs | tail -f

# Nginx access logs
make logs-nginx

# System logs
make ssh
sudo journalctl -u forgejo -f

# Docker logs
docker logs forgejo --tail 100 -f
```

### Performance Monitoring

```bash
# Check resource usage
make ssh

# CPU and memory
htop

# Disk usage
df -h
du -sh /opt/forgejo/*

# Docker stats
docker stats forgejo

# PostgreSQL status
sudo systemctl status postgresql
```

### Database Monitoring

```bash
make ssh

# Connect to PostgreSQL
sudo -u postgres psql forgejo

# Check database size
SELECT pg_size_pretty(pg_database_size('forgejo'));

# Check table sizes
SELECT 
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;

# Active connections
SELECT count(*) FROM pg_stat_activity;
```

## Troubleshooting

### Service Won't Start

```bash
# Check Docker status
make ssh
docker ps -a
docker logs forgejo

# Check system resources
free -h
df -h

# Restart service
make restart

# Full rebuild if needed
docker compose down
docker compose up -d
```

### Database Connection Issues

```bash
make ssh

# Check PostgreSQL
sudo systemctl status postgresql

# Check connections
sudo -u postgres psql -c "SELECT * FROM pg_stat_activity;"

# Restart PostgreSQL
sudo systemctl restart postgresql

# Check logs
sudo tail -f /var/log/postgresql/postgresql-16-main.log
```

### SSL Certificate Issues

```bash
make ssh

# Check certificate status
sudo certbot certificates

# Renew manually
sudo certbot renew --force-renewal

# Check Nginx configuration
sudo nginx -t
sudo systemctl restart nginx

# Check certificate files
sudo ls -la /etc/letsencrypt/live/git.yourdomain.com/
```

### Performance Issues

```bash
# Check slow queries
make ssh
sudo -u postgres psql forgejo

# Enable slow query logging
ALTER SYSTEM SET log_min_duration_statement = 1000;
SELECT pg_reload_conf();

# View slow queries
sudo tail -f /var/log/postgresql/postgresql-16-main.log

# Vacuum database
sudo -u postgres vacuumdb --analyze forgejo
```

### Disk Space Issues

```bash
make ssh

# Check space
df -h

# Find large directories
du -h /opt/forgejo | sort -rh | head -20

# Clean Docker
docker system prune -a

# Clean old backups
find /opt/forgejo/backups -type f -mtime +30 -delete

# Clean logs
sudo journalctl --vacuum-time=7d
```

## Security Operations

### Security Audit

```bash
make ssh

# Check for security updates
sudo apt update
sudo apt list --upgradable

# Review firewall rules
sudo ufw status verbose

# Check open ports
sudo netstat -tulpn

# Review SSH configuration
cat /etc/ssh/sshd_config

# Check failed login attempts
sudo grep "Failed password" /var/log/auth.log | tail -20
```

### Rotate Secrets

```bash
# Generate new secrets
openssl rand -base64 48

# Update vault
make ansible-vault-edit

# Redeploy with new secrets
make deploy
```

### SSL Certificate Renewal

```bash
# Auto-renewal is configured, but to force:
make ssh
sudo certbot renew --force-renewal
sudo systemctl reload nginx
```

### Security Updates

```bash
# Enable automatic security updates (already configured)
make ssh
sudo cat /etc/apt/apt.conf.d/50unattended-upgrades

# Check update history
sudo cat /var/log/unattended-upgrades/unattended-upgrades.log
```

## Scaling

### Vertical Scaling (More Resources)

```bash
# Update instance type
vim terraform/scaleway/compute/terraform.tfvars
# Change: instance_type = "DEV1-L"  # 8GB RAM

make terraform-apply PROVIDER=scaleway

# Service restarts automatically
```

### Database Optimization

```bash
make ssh
sudo -u postgres psql forgejo

# Optimize settings
ALTER SYSTEM SET shared_buffers = '512MB';
ALTER SYSTEM SET effective_cache_size = '2GB';
ALTER SYSTEM SET maintenance_work_mem = '256MB';

SELECT pg_reload_conf();
```

### Add Storage Volume

```bash
# In inventory
forgejo_use_external_volume: true
forgejo_volume_device: /dev/sdb

# Redeploy
make deploy-tags TAGS=volume
```

## Disaster Recovery

### Complete Failure Recovery

```bash
# 1. Create new infrastructure
make terraform-apply PROVIDER=scaleway

# 2. Update inventory with new IP
vim ansible/inventory/production/hosts.yml

# 3. Update DNS records
# Point domain to new IP

# 4. Deploy with restore
# First, copy backup timestamp
make ssh
ls /opt/forgejo/backups/  # Note timestamp

# Then restore
make restore
# Enter timestamp when prompted

# 5. Verify
make health
```

### Backup Storage Migration

```bash
# Copy backups to new location
make ssh
rsync -avz /opt/forgejo/backups/ /new/backup/location/

# Update backup configuration
vim ansible/playbooks/vars/main.yml
# forgejo_backup_path: /new/backup/location

make deploy-tags TAGS=backup
```

### Database Recovery from Corruption

```bash
make ssh

# Stop Forgejo
docker compose down

# Backup corrupted database
sudo -u postgres pg_dump forgejo > /tmp/forgejo-corrupted.sql

# Drop and recreate
sudo -u postgres dropdb forgejo
sudo -u postgres createdb forgejo

# Restore from latest backup
gunzip -c /opt/forgejo/backups/database-TIMESTAMP.sql.gz | \
  sudo -u postgres psql forgejo

# Restart Forgejo
docker compose up -d
```

## Best Practices

### Daily Checklist

- [ ] Check service health: `make health`
- [ ] Review logs for errors: `make logs | grep -i error`
- [ ] Verify backups ran: `ls -lt /opt/forgejo/backups/ | head -5`
- [ ] Check disk space: `df -h`

### Weekly Checklist

- [ ] Review security logs: `sudo grep "Failed" /var/log/auth.log`
- [ ] Check for updates: `sudo apt update && apt list --upgradable`
- [ ] Test backup restore (in dev environment)
- [ ] Review performance metrics
- [ ] Check SSL certificate expiry

### Monthly Checklist

- [ ] Full security audit
- [ ] Review and update firewall rules
- [ ] Rotate secrets if needed
- [ ] Review and optimize database
- [ ] Update documentation
- [ ] Test disaster recovery procedures

## Emergency Contacts

Keep these handy:

```bash
# Quick recovery commands
make health              # Check status
make restart             # Restart service
make logs                # View logs
make backup              # Create backup
make restore             # Restore from backup
make ssh                 # SSH access

# Emergency rollback
make update              # Includes backup
# If issues: make restore  # Roll back
```

## Useful Scripts

### Health Check Script

```bash
#!/bin/bash
# Save as: health-check.sh

echo "=== Forgejo Health Check ==="
echo "Date: $(date)"
echo ""

echo "1. Service Status:"
docker ps | grep forgejo

echo ""
echo "2. Disk Space:"
df -h | grep -E "Filesystem|/dev/vda1|/dev/sda1"

echo ""
echo "3. Memory Usage:"
free -h

echo ""
echo "4. Latest Backup:"
ls -lth /opt/forgejo/backups/*.tar.gz | head -1

echo ""
echo "5. HTTP Status:"
curl -s -o /dev/null -w "%{http_code}" http://localhost:3000
```

---

**Remember**: Always test procedures in a development environment first!