CopyBot/mediacms
1
0
Fork 0
mirror of https://github.com/mediacms-io/mediacms.git synced 2025-12-05 12:02:31 -05:00
Code Issues Packages Projects Releases Wiki Activity

Docker story refactoring

Browse Source
This commit is contained in:
Markos Gogoulos
2025-11-15 15:01:39 +02:00
parent 9b3d9fe1e7
commit 66e67c751e
49 changed files with 2969 additions and 513 deletions
Download Patch File Download Diff File Expand all files Collapse all files

367
docs/DOCKER_V7.3_MIGRATION.md Normal file
View File

@@ -0,0 +1,367 @@
# MediaCMS 7.3 Docker Architecture Migration Guide
## Overview
MediaCMS 7.3 introduces a modernized Docker architecture that removes supervisord and implements Docker best practices with one process per container.
## What Changed
### Old Architecture (pre-7.3)
- Single multi-purpose image with supervisord
- Environment variables (`ENABLE_UWSGI`, `ENABLE_NGINX`, etc.) to control services
- All services bundled in `deploy/docker/` folder
- File mounts required for all deployments
### New Architecture (7.3+)
- **Dedicated images** for each service:
- `mediacms/mediacms:7.3` - Django/uWSGI application
- `mediacms/mediacms-worker:7.3` - Celery workers
- `mediacms/mediacms-worker:7.3-full` - Celery workers with extra codecs
- `mediacms/mediacms-nginx:7.3` - Nginx web server
- **No supervisord** - Native Docker process management
- **Separated services**:
- `migrations` - Runs database migrations on every startup
- `nginx` - Serves static/media files and proxies to Django
- `web` - Django application (uWSGI)
- `celery_short` - Short-running tasks (thumbnails, etc.)
- `celery_long` - Long-running tasks (video encoding)
- `celery_beat` - Task scheduler
- **No ENABLE_* environment variables**
- **Config centralized** in `config/` directory
- **File mounts only for development** (`docker-compose-dev.yaml`)
## Directory Structure
```
config/
├── nginx/
│ ├── nginx.conf # Main nginx config
│ ├── site.conf # Virtual host config
│ └── uwsgi_params # uWSGI parameters
├── nginx-proxy/
│ └── client_max_body_size.conf # For production HTTPS proxy
├── uwsgi/
│ └── uwsgi.ini # uWSGI configuration
└── imagemagick/
└── policy.xml # ImageMagick policy
scripts/
├── entrypoint-web.sh # Web container entrypoint
├── entrypoint-worker.sh # Worker container entrypoint
└── run-migrations.sh # Migration script
Dockerfile.new # Main Dockerfile (base, web, worker, worker-full)
Dockerfile.nginx # Nginx Dockerfile
docker-compose.yaml # Production deployment
docker-compose-cert.yaml # Production with HTTPS
docker-compose-dev.yaml # Development with file mounts
```
## Migration Steps
### For Existing Production Systems
#### Step 1: Backup your data
```bash
# Backup database
docker exec mediacms_db_1 pg_dump -U mediacms mediacms > backup.sql
# Backup media files
cp -r media_files media_files.backup
```
#### Step 2: Update configuration location
```bash
# The client_max_body_size.conf has moved
# No action needed if you haven't customized it
```
#### Step 3: Pull latest images
```bash
docker pull mediacms/mediacms:7.3
docker pull mediacms/mediacms-worker:7.3
docker pull mediacms/mediacms-nginx:7.3
```
#### Step 4: Update docker-compose file
If using **docker-compose.yaml**:
- No changes needed, just use the new version
If using **docker-compose-cert.yaml** (HTTPS):
- Update `VIRTUAL_HOST`, `LETSENCRYPT_HOST`, and `LETSENCRYPT_EMAIL` in the nginx service
- Update the path to client_max_body_size.conf:
```yaml
- ./config/nginx-proxy/client_max_body_size.conf:/etc/nginx/conf.d/client_max_body_size.conf:ro
```
#### Step 5: Restart services
```bash
docker compose down
docker compose up -d
```
### For Development Systems
Development now requires the `-dev` compose file:
```bash
# Old way (no longer works)
docker compose up
# New way (development)
docker compose -f docker-compose-dev.yaml up
```
## Deployment Options
### Standard Deployment (HTTP)
**File**: `docker-compose.yaml`
**Command**:
```bash
docker compose up -d
```
**Features**:
- Self-contained images (no file mounts)
- Nginx serves on port 80
- Separate containers for each service
- Named volumes for persistence
**Architecture**:
```
Client → nginx:80 → web:9000 (uWSGI)
↓
static_files (volume)
media_files (volume)
```
### Production Deployment (HTTPS with Let's Encrypt)
**File**: `docker-compose-cert.yaml`
**Prerequisites**:
1. Domain name pointing to your server
2. Ports 80 and 443 open
**Setup**:
```bash
# 1. Edit docker-compose-cert.yaml
# Update these values in the nginx service:
# VIRTUAL_HOST: 'your-domain.com'
# LETSENCRYPT_HOST: 'your-domain.com'
# LETSENCRYPT_EMAIL: 'your-email@example.com'
# 2. Start services
docker compose -f docker-compose-cert.yaml up -d
# 3. Check logs
docker compose -f docker-compose-cert.yaml logs -f nginx-proxy acme-companion
```
**Features**:
- Automatic HTTPS via Let's Encrypt
- Certificate auto-renewal
- Reverse proxy handles SSL termination
**Architecture**:
```
Client → nginx-proxy:443 (HTTPS) → nginx:80 → web:9000 (uWSGI)
```
### Development Deployment
**File**: `docker-compose-dev.yaml`
**Command**:
```bash
docker compose -f docker-compose-dev.yaml up
```
**Features**:
- Source code mounted for live editing
- Django debug mode enabled
- Django's `runserver` instead of uWSGI
- Frontend hot-reload on port 8088
- No nginx (direct Django access on port 80)
**Ports**:
- `80` - Django API
- `8088` - Frontend dev server
## Configuration
### Environment Variables
All configuration is done via environment variables or `cms/local_settings.py`.
**Key Variables**:
- `FRONTEND_HOST` - Your domain (e.g., `https://mediacms.example.com`)
- `PORTAL_NAME` - Your portal name
- `SECRET_KEY` - Django secret key
- `POSTGRES_*` - Database credentials
- `REDIS_LOCATION` - Redis connection string
- `DEBUG` - Enable debug mode (development only)
**Setting variables**:
Option 1: In docker-compose file:
```yaml
environment:
FRONTEND_HOST: 'https://mediacms.example.com'
PORTAL_NAME: 'My MediaCMS'
```
Option 2: Using .env file (recommended):
```bash
# Create .env file
cat > .env << EOF
FRONTEND_HOST=https://mediacms.example.com
PORTAL_NAME=My MediaCMS
SECRET_KEY=your-secret-key-here
EOF
```
### Customizing Settings
For advanced customization, you can build a custom image:
```dockerfile
# Dockerfile.custom
FROM mediacms/mediacms:7.3
COPY my_local_settings.py /home/mediacms.io/mediacms/cms/local_settings.py
```
## Celery Workers
### Standard Workers
By default, `celery_long` uses the standard image:
```yaml
celery_long:
image: mediacms/mediacms-worker:7.3
```
### Full Workers (Extra Codecs)
To enable extra codecs for better transcoding (including Whisper for subtitles):
**Edit docker-compose file**:
```yaml
celery_long:
image: mediacms/mediacms-worker:7.3-full # Changed from :7.3
```
**Then restart**:
```bash
docker compose up -d celery_long
```
### Scaling Workers
You can scale workers independently:
```bash
# Scale short task workers
docker compose up -d --scale celery_short=3
# Scale long task workers
docker compose up -d --scale celery_long=2
```
## Troubleshooting
### Migrations not running
```bash
# Check migrations container logs
docker compose logs migrations
# Manually run migrations
docker compose run --rm migrations
```
### Static files not loading
```bash
# Ensure migrations completed (it runs collectstatic)
docker compose logs migrations
# Check nginx can access volumes
docker compose exec nginx ls -la /var/www/static
```
### Permission issues
```bash
# Check volume ownership
docker compose exec web ls -la /home/mediacms.io/mediacms/media_files
# If needed, rebuild images
docker compose build --no-cache
```
### Celery workers not processing tasks
```bash
# Check worker logs
docker compose logs celery_short celery_long
# Check Redis connection
docker compose exec redis redis-cli ping
# Restart workers
docker compose restart celery_short celery_long celery_beat
```
## Removed Components
The following are **no longer used** in 7.3:
- ❌ `deploy/docker/supervisord/` - Supervisord configs
- ❌ `deploy/docker/start.sh` - Start script
- ❌ `deploy/docker/entrypoint.sh` - Old entrypoint
- ❌ Environment variables: `ENABLE_UWSGI`, `ENABLE_NGINX`, `ENABLE_CELERY_BEAT`, `ENABLE_CELERY_SHORT`, `ENABLE_CELERY_LONG`, `ENABLE_MIGRATIONS`
**These are still available but moved**:
- ✅ `config/nginx/` - Nginx configs (moved from `deploy/docker/`)
- ✅ `config/uwsgi/` - uWSGI config (moved from `deploy/docker/`)
- ✅ `config/nginx-proxy/` - Reverse proxy config (moved from `deploy/docker/reverse_proxy/`)
## Persistent Volumes
MediaCMS 7.3 uses Docker named volumes for data persistence:
- **`media_files`** - All uploaded media (videos, images, thumbnails, HLS streams)
- Mounted on: migrations, web, nginx, celery_beat, celery_short, celery_long
- Persists across container restarts, updates, and image removals
- **`logs`** - Application and nginx logs
- Mounted on: migrations, web, nginx, celery_beat, celery_short, celery_long
- Nginx logs: `/var/log/mediacms/nginx.access.log`, `/var/log/mediacms/nginx.error.log`
- Django/Celery logs: `/home/mediacms.io/mediacms/logs/`
- Persists across container restarts, updates, and image removals
- **`static_files`** - Django static files (CSS, JS, images)
- Mounted on: migrations, web, nginx
- Regenerated during migrations via `collectstatic`
- **`postgres_data`** - PostgreSQL database
- Mounted on: db
- Persists across container restarts, updates, and image removals
**Important**: Use `docker compose down -v` to remove volumes (⚠️ causes data loss!)
## Benefits of New Architecture
1. **Better resource management** - Scale services independently
2. **Easier debugging** - Clear separation of concerns
3. **Faster restarts** - Restart only affected services
4. **Production-ready** - No file mounts, immutable images
5. **Standard Docker practices** - One process per container
6. **Clearer logs** - Each service has isolated logs, persistent storage
7. **Better health checks** - Per-service monitoring
8. **Data persistence** - media_files and logs survive all container operations
## Support
For issues or questions:
- GitHub Issues: https://github.com/mediacms-io/mediacms/issues
- Documentation: https://docs.mediacms.io

156
docs/admins_docs.md
View File

@@ -164,53 +164,123 @@ Database is stored on ../postgres_data/ and media_files on media_files/
## 4. Docker Deployment options
The mediacms image is built to use supervisord as the main process, which manages one or more services required to run mediacms. We can toggle which services are run in a given container by setting the environment variables below to `yes` or `no`:
**⚠️ IMPORTANT**: MediaCMS 7.3 introduces a new Docker architecture. If you're upgrading from an earlier version, please see the [Migration Guide](DOCKER_V7.3_MIGRATION.md).
* ENABLE_UWSGI
* ENABLE_NGINX
* ENABLE_CELERY_BEAT
* ENABLE_CELERY_SHORT
* ENABLE_CELERY_LONG
* ENABLE_MIGRATIONS
### Architecture Overview
By default, all these services are enabled, but in order to create a scaleable deployment, some of them can be disabled, splitting the service up into smaller services.
MediaCMS 7.3+ uses a modern microservices architecture with dedicated containers:
Also see the `Dockerfile` for other environment variables which you may wish to override. Application settings, eg. `FRONTEND_HOST` can also be overridden by updating the `deploy/docker/local_settings.py` file.
- **nginx** - Web server for static/media files and reverse proxy
- **web** - Django application (uWSGI)
- **celery_short** - Short-running background tasks
- **celery_long** - Long-running tasks (video encoding)
- **celery_beat** - Task scheduler
- **migrations** - Database migrations (runs on startup)
- **db** - PostgreSQL database
- **redis** - Cache and message broker
To run, update the configs above if necessary, build the image by running `docker compose build`, then run `docker compose run`
### Key Changes from Previous Versions
### Simple Deployment, accessed as http://localhost
- ✅ **No supervisord** - Native Docker process management
- ✅ **Dedicated images** per service
- ✅ **No ENABLE_* environment variables** - Services are separated into individual containers
- ✅ **Production images** don't mount source code (immutable)
- ✅ **config/** directory for centralized configuration
- ✅ **Separate celery workers** for short and long tasks
The main container runs migrations, mediacms_web, celery_beat, celery_workers (celery_short and celery_long services), exposed on port 80 supported by redis and postgres database.
### Configuration
The FRONTEND_HOST in `deploy/docker/local_settings.py` is configured as http://localhost, on the docker host machine.
Application settings can be overridden using environment variables in your docker-compose file or by building a custom image with a modified `cms/local_settings.py` file.
### Server with ssl certificate through letsencrypt service, accessed as https://my_domain.com
Before trying this out make sure the ip points to my_domain.com.
Key environment variables:
- `FRONTEND_HOST` - Your domain (e.g., `https://mediacms.example.com`)
- `PORTAL_NAME` - Portal name
- `SECRET_KEY` - Django secret key
- `DEBUG` - Enable debug mode (development only)
- Database and Redis connection settings
With this method [this deployment](../docker-compose-letsencrypt.yaml) is used.
See the [Migration Guide](DOCKER_V7.3_MIGRATION.md) for detailed configuration options
Edit this file and set `VIRTUAL_HOST` as my_domain.com, `LETSENCRYPT_HOST` as my_domain.com, and your email on `LETSENCRYPT_EMAIL`
### Simple Deployment (HTTP)
Edit `deploy/docker/local_settings.py` and set https://my_domain.com as `FRONTEND_HOST`
Use `docker-compose.yaml` for a standard HTTP deployment on port 80:
Now run `docker compose -f docker-compose-letsencrypt.yaml up`, when installation finishes you will be able to access https://my_domain.com using a valid Letsencrypt certificate!
```bash
docker compose up -d
```
### Advanced Deployment, accessed as http://localhost:8000
This starts all services (nginx, web, celery workers, database, redis) with the nginx container exposed on port 80. Access at http://localhost or http://your-server-ip.
Here we can run 1 mediacms_web instance, with the FRONTEND_HOST in `deploy/docker/local_settings.py` configured as http://localhost:8000. This is bootstrapped by a single migrations instance and supported by a single celery_beat instance and 1 or more celery_worker instances. Redis and postgres containers are also used for persistence. Clients can access the service on http://localhost:8000, on the docker host machine. This is similar to [this deployment](../docker-compose.yaml), with a `port` defined in FRONTEND_HOST.
**Features:**
- Production-ready with immutable images
- Named volumes for data persistence
- Separate containers for each service
### Advanced Deployment, with reverse proxy, accessed as http://mediacms.io
### Production Deployment with HTTPS (Let's Encrypt)
Here we can use `jwilder/nginx-proxy` to reverse proxy to 1 or more instances of mediacms_web supported by other services as mentioned in the previous deployment. The FRONTEND_HOST in `deploy/docker/local_settings.py` is configured as http://mediacms.io, nginx-proxy has port 80 exposed. Clients can access the service on http://mediacms.io (Assuming DNS or the hosts file is setup correctly to point to the IP of the nginx-proxy instance). This is similar to [this deployment](../docker-compose-http-proxy.yaml).
Use `docker-compose-cert.yaml` for automatic HTTPS with Let's Encrypt:
### Advanced Deployment, with reverse proxy, accessed as https://localhost
**Prerequisites:**
- Domain name pointing to your server
- Ports 80 and 443 open
The reverse proxy (`jwilder/nginx-proxy`) can be configured to provide SSL termination using self-signed certificates, letsencrypt or CA signed certificates (see: https://hub.docker.com/r/jwilder/nginx-proxy or [LetsEncrypt Example](https://www.singularaspect.com/use-nginx-proxy-and-letsencrypt-companion-to-host-multiple-websites/) ). In this case the FRONTEND_HOST should be set to https://mediacms.io. This is similar to [this deployment](../docker-compose-http-proxy.yaml).
**Setup:**
1. Edit `docker-compose-cert.yaml` and update:
- `VIRTUAL_HOST` - Your domain
- `LETSENCRYPT_HOST` - Your domain
- `LETSENCRYPT_EMAIL` - Your email
2. Run:
```bash
docker compose -f docker-compose-cert.yaml up -d
```
This uses `nginxproxy/nginx-proxy` with `acme-companion` for automatic HTTPS certificate management. Access at https://your-domain.com.
### Development Deployment
Use `docker-compose-dev.yaml` for development with live code reloading:
```bash
docker compose -f docker-compose-dev.yaml up
```
**Features:**
- Source code mounted for live editing
- Django debug mode enabled
- Frontend dev server on port 8088
- Direct Django access (no nginx) on port 80
### Scaling Workers
Scale celery workers independently based on load:
```bash
# Scale short task workers to 3 instances
docker compose up -d --scale celery_short=3
# Scale long task workers to 2 instances
docker compose up -d --scale celery_long=2
```
### Using Extra Codecs (Full Image)
For advanced transcoding features (including Whisper for automatic subtitles), use the full worker image:
Edit your docker-compose file:
```yaml
celery_long:
image: mediacms/mediacms-worker:7.3-full # Changed from :7.3
```
Then restart:
```bash
docker compose up -d celery_long
```
### A Scaleable Deployment Architecture (Docker, Swarm, Kubernetes)
The architecture below generalises all the deployment scenarios above, and provides a conceptual design for other deployments based on kubernetes and docker swarm. It allows for horizontal scaleability through the use of multiple mediacms_web instances and celery_workers. For large deployments, managed postgres, redis and storage may be adopted.
The architecture below provides a conceptual design for deployments based on kubernetes and docker swarm. It allows for horizontal scaleability through the use of multiple web instances and celery workers. For large deployments, managed postgres, redis and storage may be adopted.
![MediaCMS](images/architecture.png)
@@ -218,24 +288,36 @@ The architecture below generalises all the deployment scenarios above, and provi
## 5. Configuration
Several options are available on `cms/settings.py`, most of the things that are allowed or should be disallowed are described there.
It is advisable to override any of them by adding it to `local_settings.py` .
It is advisable to override any of them by adding it to `local_settings.py`.
In case of a the single server installation, add to `cms/local_settings.py` .
In case of a docker compose installation, add to `deploy/docker/local_settings.py` . This will automatically overwrite `cms/local_settings.py` .
Any change needs restart of MediaCMS in order to take effect.
Single server installation: edit `cms/local_settings.py`, make a change and restart MediaCMS
**Single server installation:** edit `cms/local_settings.py`, make changes and restart MediaCMS:
```bash
#systemctl restart mediacms
systemctl restart mediacms celery_beat celery_short celery_long
```
Docker Compose installation: edit `deploy/docker/local_settings.py`, make a change and restart MediaCMS containers
**Docker installation:** Configuration can be done in two ways:
1. **Environment variables** (recommended for simple changes):
Add to your docker-compose file:
```yaml
environment:
FRONTEND_HOST: 'https://mediacms.example.com'
PORTAL_NAME: 'My MediaCMS'
```
2. **Custom image with local_settings.py** (for complex changes):
- Create a custom Dockerfile:
```dockerfile
FROM mediacms/mediacms:7.3
COPY my_custom_settings.py /home/mediacms.io/mediacms/cms/local_settings.py
```
- Build and use your custom image
After changes, restart the affected containers:
```bash
#docker compose restart web celery_worker celery_beat
docker compose restart web celery_short celery_long celery_beat
```
### 5.1 Change portal logo

4
docs/saml_entraid_setup.md
View File

@@ -46,7 +46,7 @@ Before beginning, ensure the following:
## Step 1: Configure MediaCMS for SAML
The first step in enabling SAML authentication is to modify the `local_settings.py` (for Docker: `./deploy/docker/local_settings.py`) file of your MediaCMS deployment. Add the following configuration block to enable SAML support, role-based access control (RBAC), and enforce secure communication settings:
The first step in enabling SAML authentication is to modify the `local_settings.py` (for Docker: `./config/local_settings.py`) file of your MediaCMS deployment. Add the following configuration block to enable SAML support, role-based access control (RBAC), and enforce secure communication settings:
```python
USE_RBAC = True
@@ -292,7 +292,7 @@ Another issue you might encounter is an **infinite redirect loop**. This can hap
https://<MyDomainName>/accounts/saml/mediacms_entraid/login/
```
* Add the following line to `./deploy/docker/local_settings.py`:
* Add the following line to `./config/local_settings.py`:
```python
LOGIN_URL = "/accounts/saml/mediacms_entraid/login/"