Skip to content

Service Enhancement Proposal SEP-01 - Dispatcher on Swarm

Objective

Explore the feasibility of running our Dispatcher pool on a Docker Swarm, to include advantages/disadvantages, opportunities and challenges, how operations would change in that environment.

Pros

Docker Secrets for key to unlock SSL certs

The current (VM) dispatchers are deployed with the keys unencrypted, so this would be a security win.

Allows us to use latest version of Apache HTTPD

The RHEL 7 version of Apache is 2.4.6, released Jul 15, 2013. We expect the SSLCertificateFile to contain the entire certificate chain (minus the root), but support for this feature came with 2.4.8 Mar 11, 2014.

RHEL 8 has Apache HTTPD version 2.4.37, so this particular issue would not be a problem, but ModAuthCAS is not available pre-build for RHEL 8, and the SysOps group is not currently (April 2020) ready to support RHEL 8.

Can use the vt_httpd_base image!

We have been beta testing this with Grafana.

Scaling may become simpler

Adding a new node should deploy a new instance without further effort.

Healthchecks

Docker swarm would keep the instance running; restart if it were to die.ss

Cons

Reliance on stable names

We currently rely on stable names / addresses for

  • f5 load balancer
  • Even / Odd load balancing with Publishers
  • Defining flush queues on Publishers

Could the apparent host name be the same as the real host name?

  • I have tried in the past to set name as an environment variable, but have not found a clean way to do this.
  • May be able to pass in the hostname to the container. Will this work from a stack file?

Is there a way to dynamically register hosts with the appropriate publisher? YES - it looks like there is. AEM Documentation for uses of cURL has an example of creating a replication agent. This example is for an author replicating to a publisher, but I think we can adapt it to create a dispatcher flush agent.

Procedural Changes

Configuration Options

  • Build an image with the current config on top of vt_httpd_base
  • Certificates inside image? Technically they are part of the configuration.
    • Docker Secret to decrypt key

Routine Jobs (Cron)

Currently, cron jobs are used to managed logs. See logs below.

Logs

Currently cron jobs are used to manage logs.

  • If logs are sent to a volume mounted on the host, then all the same old processes will work.

    • Requires cron on the host system to run jobs that compress and prune logs.
    • Existing Splunk (system) processes will work the same as the VM dispatchers.

  • Could use Fluent Bit to send logs to Splunk, ElasticSearch or our existing Docker logging Pipeline (which sends to AWS CloudWatch).

Cluster Management

Good practice suggests 3 managers. At first, our cluster will be the 2 new nodes.

  • Option: Make 1 manager and 1 worker, then add 2 new managers as the old machines are updated.