IngestIQ
Getting StartedCore ConceptsData ConnectorsAI & ProcessingIntegrationsAPI ReferenceSelf-Hosting

Deployment

Deploy IngestIQ with Docker

Quick Start Deployment#

1. Clone Repository#

git clone https://github.com/avesta-hq/ingestiq-backend.git
cd ingestiq-backend

2. Configure Environment#

cp .env.example .env
# Edit .env with your API keys and secrets

3. Start Services#

docker compose up -d

4. Run Migrations#

npm install
npm run db:migrate
npm run db:seed

5. Start Server#

npm run dev  # Development
# or
npm start    # Production

IngestIQ is running at http://localhost:3000

Docker Compose Overview#

The docker-compose.yml includes:

services:
  vectordb:    # PostgreSQL 16 + pgvector
  redis:       # Redis 7
  nats:        # NATS with JetStream
  minio:       # S3-compatible storage
  gotenberg:   # Document conversion

Production Deployment#

1. Use Production Environment#

NODE_ENV=production

# Strong secrets
JWT_SECRET=[generate-32-char-secret]
JWT_REFRESH_SECRET=[generate-32-char-secret]

# Production database
MANAGEMENT_DB_URL=postgres://user:pass@prod-db:5432/ingestiq_mgmt
VECTOR_DB_URL=postgres://user:pass@prod-db:5432/ingestiq_vector

2. Build Production Image#

docker build -t ingestiq-backend:latest .

3. Run with Docker#

docker run -d \
  --name ingestiq \
  -p 3000:3000 \
  --env-file .env.production \
  ingestiq-backend:latest

Kubernetes Deployment#

Sample Deployment#

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingestiq-backend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ingestiq
  template:
    metadata:
      labels:
        app: ingestiq
    spec:
      containers:
      - name: ingestiq
        image: ingestiq-backend:latest
        ports:
        - containerPort: 3000
        envFrom:
        - secretRef:
            name: ingestiq-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "2Gi"
            cpu: "1000m"

Sample Service#

apiVersion: v1
kind: Service
metadata:
  name: ingestiq-backend
spec:
  selector:
    app: ingestiq
  ports:
  - port: 80
    targetPort: 3000
  type: LoadBalancer

Managed Services#

For production, consider managed services:

ComponentManaged Option
PostgreSQLAWS RDS, Google Cloud SQL, Supabase
RedisAWS ElastiCache, Redis Cloud
S3AWS S3, Google Cloud Storage
NATSNATS.io Cloud

Health Checks#

API Health#

curl http://localhost:3000/api/health

Expected response:

{
  "status": "ok",
  "timestamp": "2024-01-28T12:00:00.000Z"
}

Service Health#

# Check all containers
docker compose ps

# Check logs
docker compose logs -f

Monitoring#

Recommended Stack#

  • Prometheus - Metrics collection
  • Grafana - Visualization
  • Loki - Log aggregation

Key Metrics#

MetricWhat to Monitor
API latencyResponse times
Error rates4xx, 5xx responses
Queue depthBullMQ job counts
Vector DB sizeStorage growth
Memory usageContainer limits

Backup Strategy#

Database Backups#

# PostgreSQL backup
pg_dump -h localhost -U postgres ingestiq_vector > backup.sql

# Restore
psql -h localhost -U postgres ingestiq_vector < backup.sql

S3 Backup#

# MinIO/S3 sync to backup location
aws s3 sync s3://ingestiq-documents s3://backup-bucket/ingestiq/

Security Checklist#

  • Strong JWT secrets (32+ chars)
  • Rotate API keys periodically
  • Use secrets manager (AWS Secrets Manager, HashiCorp Vault)
  • Run behind reverse proxy (nginx, Traefik)
  • Enable HTTPS/TLS
  • Restrict database ports to internal network
  • Set NODE_ENV=production
  • Enable rate limiting
  • Regular dependency updates

Troubleshooting#

Common Issues#

# Check database is running
docker compose logs vectordb

# Verify connection string
psql $VECTOR_DB_URL -c "SELECT 1"

Increase container memory limits:

deploy:
  resources:
    limits:
      memory: 4G
  • Check NATS consumer lag
  • Monitor embedding API response times
  • Consider scaling processors

Related#

Requirements
System requirements
Configuration
Environment variables
Documentation