Kubernetes deployment

This guide assumes that all prerequisites have been met. Please visit the corresponding Prerequisites page for your infrastructure provider.

You may also use this guide for deployments to other cloud platforms (e.g. Oracle Kubernetes Engine), however it is up to you to satisfy any prerequisites for those platforms. Use at your own risk.

Deploy Tower

Create a namespace

Create a namespace to group the Tower resources within your K8s cluster.

1. Create the namespace (e.g. tower-nf):

   kubectl create namespace tower-nf

2. Switch to the namespace:

   kubectl config set-context --current --namespace=tower-nf

Configure container registry credentials

Nextflow Tower is distributed as a collection of Docker containers available through the Seqera Labs container registry cr.seqera.io. Contact support to get your container access credentials. Once you have received your credentials, grant your cluster access to the registry using these steps:

1. Retrieve the name and secret values from the JSON file you received from Seqera Labs support.

2. Create a Kubernetes Secret, using the name and secret retrieved in step 1, with this command:

   kubectl create secret docker-registry cr.seqera.io \
     --docker-server=cr.seqera.io \
     --docker-username='/\<YOUR NAME\>/' \
     --docker-password='<YOUR SECRET>'

   Note: The credential name contains a dollar $ character. To prevent the Linux shell from interpreting this value as an environment variable, wrap it in single quotes.

3. The following snippet configures the Tower cron service and the Tower frontend and backend to use the Secret created in step 2 (see tower-cron.yml and tower-svc.yml):

imagePullSecrets:
  - name: "cr.seqera.io"

This parameter is already included in the templates linked above — if you use a name other than cr.seqera.io for the Kubernetes Secret, update this value accordingly in the configuration files.

Tower ConfigMap

configmap.yml

kind: ConfigMap
apiVersion: v1
metadata:
  name: tower-backend-cfg
  labels:
    app: backend-cfg
data:
  TOWER_SERVER_URL: "https://<YOUR PUBLIC TOWER HOST NAME>"
  TOWER_CONTACT_EMAIL: "support@tower.nf"
  TOWER_JWT_SECRET: "ReplaceThisWithALongSecretString"
  TOWER_DB_URL: "jdbc:mysql://<YOUR DB HOST NAME AND PORT>/tower"
  TOWER_DB_DRIVER: "org.mariadb.jdbc.Driver"
  TOWER_DB_USER: "tower"
  TOWER_DB_PASSWORD: "<YOUR DB PASSWORD>"
  TOWER_DB_DIALECT: "io.seqera.util.MySQL55DialectCollateBin"
  TOWER_DB_MIN_POOL_SIZE: "2"
  TOWER_DB_MAX_POOL_SIZE: "10"
  TOWER_DB_MAX_LIFETIME: "180000"
  TOWER_SMTP_HOST: "<YOUR SMTP SERVER HOST NAME>"
  TOWER_SMTP_USER: "<YOUR SMTP USER NAME>"
  TOWER_SMTP_PASSWORD: "<YOUR SMTP USER PASSWORD>"
  TOWER_CRYPTO_SECRETKEY: "<YOUR CRYPTO SECRET>"
  TOWER_LICENSE: "<YOUR TOWER LICENSE KEY>"
  TOWER_ENABLE_PLATFORMS: "awsbatch-platform,gls-platform,azbatch-platform,slurm-platform"
  FLYWAY_LOCATIONS: "classpath:db-schema/mysql"
  TOWER_REDIS_URL: "redis://<YOUR REDIS IP>:6379"
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tower-yml
  labels:
    app: backend-cfg
data:
  tower.yml: |
    mail:
      smtp:
        auth: true
        # FIXME `starttls` should be enabled with a production SMTP host
        starttls:
          enable: false
          required: false

    # Uncomment to specify the duration of Tower sign-in email link validity
    auth:
      mail:
        duration: 30m

1. Download and configure configmap.yml as per the configuration page.

2. Deploy the configmap to your cluster:

   kubectl apply -f configmap.yml

The configmap.yml manifest includes both tower.env and tower.yml. These files are made available to the other containers through volume mounts.

Redis

redis.aks.yml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
  labels:
    app: redis
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/disk.csi.azure.com
parameters:
  kind: Managed
  storageaccounttype: Premium_LRS
allowVolumeExpansion: true
reclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-data
  labels:
    app: redis
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  labels:
    app: redis
spec:
  selector:
    matchLabels:
      app: redis
  serviceName: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox
          command:
            - /bin/sh
            - -c
            - |
              sysctl -w net.core.somaxconn=1024;
              echo never > /sys/kernel/mm/transparent_hugepage/enabled
          securityContext:
            privileged: true
          volumeMounts:
            - name: host-sys
              mountPath: /sys
      containers:
        - image: cr.seqera.io/public/redis:5.0.8
          name: redis
          args:
            - --appendonly yes
          ports:
            - containerPort: 6379
          volumeMounts:
            - mountPath: "/data"
              name: "vol-data"
      volumes:
        - name: vol-data
          persistentVolumeClaim:
            claimName: redis-data
        - name: host-sys
          hostPath:
            path: /sys
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  ports:
    - port: 6379
      targetPort: 6379
  selector:
    app: redis

redis.eks.yml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
  labels:
    app: redis
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  fsType: ext4
allowVolumeExpansion: true
reclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-data
  labels:
    app: redis
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  labels:
    app: redis
spec:
  selector:
    matchLabels:
      app: redis
  serviceName: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox
          command:
            - /bin/sh
            - -c
            - |
              sysctl -w net.core.somaxconn=1024;
              echo never > /sys/kernel/mm/transparent_hugepage/enabled
          securityContext:
            privileged: true
          volumeMounts:
            - name: host-sys
              mountPath: /sys
      containers:
        - image: cr.seqera.io/public/redis:5.0.8
          name: redis
          args:
            - --appendonly yes
          ports:
            - containerPort: 6379
          volumeMounts:
            - mountPath: "/data"
              name: "vol-data"
      volumes:
        - name: vol-data
          persistentVolumeClaim:
            claimName: redis-data
        - name: host-sys
          hostPath:
            path: /sys
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  ports:
    - port: 6379
      targetPort: 6379
  selector:
    app: redis

redis.gke.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-data
  labels:
    app: redis
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  labels:
    app: redis
spec:
  selector:
    matchLabels:
      app: redis
  serviceName: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      initContainers:
        - name: init-sysctl
          image: busybox
          command:
            - /bin/sh
            - -c
            - |
              sysctl -w net.core.somaxconn=1024;
              echo never > /sys/kernel/mm/transparent_hugepage/enabled
          securityContext:
            privileged: true
          volumeMounts:
            - name: host-sys
              mountPath: /sys
      containers:
        - image: cr.seqera.io/public/redis:5.0.8
          name: redis
          args:
            - --appendonly yes
          ports:
            - containerPort: 6379
          volumeMounts:
            - mountPath: "/data"
              name: "vol-data"
      volumes:
        - name: vol-data
          persistentVolumeClaim:
            claimName: redis-data
        - name: host-sys
          hostPath:
            path: /sys
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  ports:
    - port: 6379
      targetPort: 6379
  selector:
    app: redis

Download the appropriate manifest for your infrastructure:

• redis.aks.yml

• redis.eks.yml

• redis.gke.yml

Deploy to your cluster:

kubectl apply -f redis.*.yml

You may also be able to use a managed Redis service such as Amazon Elasticache or Google Memorystore, however we do not explicitly support these services, and Tower is not guaranteed to work with them. Use at your own risk.

If you do use an externally managed Redis service, make sure to update configmap.yml accordingly:

TOWER_REDIS_URL: redis://<redis private IP>:6379

Tower cron service

tower-cron.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cron
  labels:
    app: cron
spec:
  selector:
    matchLabels:
      app: cron
  template:
    metadata:
      labels:
        app: cron
    spec:
      imagePullSecrets:
        - name: "cr.seqera.io"
      volumes:
        - name: config-volume
          configMap:
            name: tower-yml
      initContainers:
        - name: migrate-db
          image: cr.seqera.io/private/nf-tower-enterprise/backend:v23.1.3
          command: ["sh", "-c", "/migrate-db.sh"]
          envFrom:
            - configMapRef:
                name: tower-backend-cfg
          volumeMounts:
            - name: config-volume
              mountPath: /tower.yml
              subPath: tower.yml
      containers:
        - name: backend
          image: cr.seqera.io/private/nf-tower-enterprise/backend:v23.1.3
          envFrom:
            - configMapRef:
                name: tower-backend-cfg
          volumeMounts:
            - name: config-volume
              mountPath: /tower.yml
              subPath: tower.yml
          env:
            - name: MICRONAUT_ENVIRONMENTS
              value: "prod,redis,cron"
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 3
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 3
            failureThreshold: 10

1. Download the manifest:

• tower-cron.yml

2. Deploy to your cluster:

kubectl apply -f tower-cron.yml

This container will create the required database schema the first time it is instantiated. This process can take a few minutes to complete and must be finished before you instantiate the Tower backend. Make sure this container is in the READY state before proceeding to the next step.

Tower frontend and backend

tower-svc.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  labels:
    app: backend
spec:
  selector:
    matchLabels:
      app: backend
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: backend
    spec:
      imagePullSecrets:
        - name: "cr.seqera.io"
      volumes:
        - name: config-volume
          configMap:
            name: tower-yml
      containers:
        - name: backend
          image: cr.seqera.io/private/nf-tower-enterprise/backend:v23.1.3
          envFrom:
            - configMapRef:
                name: tower-backend-cfg
          env:
            - name: MICRONAUT_ENVIRONMENTS
              value: "prod,redis,ha"
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: config-volume
              mountPath: /tower.yml
              subPath: tower.yml
          resources:
            requests:
              cpu: "1"
              memory: "1200Mi"
            limits:
              memory: "4200Mi"
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 3
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            timeoutSeconds: 3
            failureThreshold: 10
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  labels:
    app: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      imagePullSecrets:
        - name: "cr.seqera.io"
      containers:
        - name: frontend
          image: cr.seqera.io/private/nf-tower-enterprise/frontend:v23.1.3
          ports:
            - containerPort: 80
      restartPolicy: Always
---
# Services
apiVersion: v1
kind: Service
metadata:
  name: backend
  labels:
    app: backend
spec:
  ports:
    - name: http
      port: 8080
      targetPort: 8080
  selector:
    app: backend
---
apiVersion: v1
kind: Service
metadata:
  name: backend-api
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  type: NodePort
  selector:
    app: backend
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  type: LoadBalancer
  ports:
    - port: 80
  selector:
    app: "frontend"
---

Download the manifest:

• tower-svc.yml

Deploy to your cluster:

kubectl apply -f tower-svc.yml

Tower ingress

An ingress is used to make Tower publicly accessible, load balance traffic, terminate SSL/TLS, and offer name-based virtual hosting. The included ingress will create an external IP address and forward HTTP traffic to the Tower frontend.

ingress.aks.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: front-ingress
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
    - host: YOUR-TOWER-HOST-NAME
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: frontend
                port:
                  number: 80

ingress.eks.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: front-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: YOUR-CERTIFICATE-ARN
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06
    alb.ingress.kubernetes.io/load-balancer-attributes: >
      idle_timeout.timeout_seconds=301,
      routing.http2.enabled=false,
      access_logs.s3.enabled=true,
      access_logs.s3.bucket=YOUR-LOGS-S3-BUCKET,
      access_logs.s3.prefix=YOUR-LOGS-PREFIX
spec:
  rules:
    - host: <YOUR-TOWER-HOST-NAME>
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: ssl-redirect
                port:
                  name: use-annotation
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: frontend
                port:
                  number: 80

ingress.gke.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: front-ingress
  annotations:
    kubernetes.io/ingress.class: "gce"
spec:
  rules:
    - host: YOUR-TOWER-HOST-NAME
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: frontend
                port:
                  number: 80

Download the appropriate manifest and configure it according to your infrastructure:

• ingress.aks.yml

• ingress.eks.yml

• ingress.gke.yml

Deploy to your cluster:

kubectl apply -f ingress.*.yml

See the Kubernetes documentation on Ingress for more information. If you don't need to make Tower externally accessible, you can also use a NodePort or a LoadBalancer service to make it accessible within your intranet.

Additionally, see the relevant documentation for configuring an Ingress on each cloud provider:

• Amazon
• Azure
• Google Cloud

Check status

Finally, make sure that all services are up and running:

kubectl get pods

Test the application

To make sure that Tower is properly configured, follow these steps:

1. Log in to Tower.

2. Create an organization.

3. Create a workspace within that organization.

4. Create a new Compute Environment. Refer to Compute Environments for detailed instructions.

5. Select Quick Launch from the Launchpad tab in your workspace.

6. Enter the repository URL for the nf-core/rnaseq pipeline (https://github.com/nf-core/rnaseq).

7. In the Config profiles dropdown, select the test profile.

8. In the Pipeline parameters textarea, change the output directory to a sensible location based on your Compute Environment:

   # save to S3 bucket
   outdir: s3://<your-bucket>/results
   
   # save to scratch directory (Kubernetes)
   outdir: /scratch/results

9. Select Launch.

   You'll be transitioned to the Runs tab for the workflow. After a few minutes, you'll see the progress logs in the Execution log tab for that workflow.

Optional addons

Database console

dbconsole.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dbconsole
  labels:
    app: dbconsole
spec:
  selector:
    matchLabels:
      app: dbconsole
  template:
    metadata:
      labels:
        app: dbconsole
    spec:
      containers:
        - image: adminer:4.7.7
          name: dbconsole
          ports:
            - containerPort: 8080
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: dbconsole
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  type: NodePort
  selector:
    app: dbconsole

The included dbconsole.yml can be used to deploy a simple web frontend to the Tower database. It is not required but it can be useful for administrative purposes.

1. Deploy the database console:

   kubectl apply -f dbconsole.yml

2. Port-forward the database console to your local machine:

   kubectl port-forward deployment/dbconsole 8080:8080

   The database console will be available in your browser at http://localhost:8080.

High availability

When configuring Tower for high availability, it should be noted that:

• The cron service may only have a single instance

• The backend service can be run in multiple replicas

• The frontend service is replicable, however in most scenarios it is not necessary