Clustered NiFi with SSL

The default apache NiFi docker image works well when you want to run NiFi with SSL as a standalone instance, but when you want to run it clustered with SSL enabled, you will run into difficulties. Moreover, most of the documentation highlights how to set up NiFi with SSL using the NiFi provided tls toolkit. If you already have your own SSL certificate infrastructure, using tls toolkit is not an easy option. Nonetheless, NiFi can run perfectly fine in clustered mode with SSL enabled provided you configure it correctly.

Generate CA

Assume you have multiple services that you want to expose via HTTPS, with NiFi being one of them. You would like for them to all use the same Certificate Authority (CA). In another guide I describe how to set up such a CA and have it generate certificates for you.

Cluster Setup

In this set up we assume we have 7 node cluster using, for example, Apache Mesos. Three nodes are configured as masters, with the remaining nodes configured as agents:

+-------------------+---------+
|     hostname      | purpose |
+-------------------+---------+
| node1.example.com | master  |
| node2.example.com | master  |
| node3.example.com | master  |
| node4.example.com | agent   |
| node5.example.com | agent   |
| node6.example.com | agent   |
| node7.example.com | agent   |
+-------------------+---------+

Each of the masters has Zookeeper running, which NiFi will use as well.

The services on the cluster are made available via a proxy server (e.g. HAProxy) running on each master. The DNS name for each master will be service.example.com.

Make sure that each node can resolve the DNS names of the other nodes correctly. If necessary, set up dnsmasq on a separate machine and configure each node to use that as its resolver.

Generate Node Certificates

Once you have your CA, generate certificates for each of the cluster nodes that NiFi will run on. Unfortunately this set up cannot dynamically add certificates to NiFi so you will have to create certificates for all the nodes you *think* NiFi will run on!

First step is to create a private key for node4:

openssl genrsa -out node4.key 2048

Next create a Certificate Signing Request using our newly minted key:

openssl req -new -key node4.key -out node4.csr -subj '/CN=node4.example.com/O=Acme Corp' -addext 'subjectAltName = DNS:node4.example.com,DNS:node4,DNS:service.example.com'

Now that we have our CSR, we can go ahead and sign it with our CA:

openssl ca -batch -config ca.conf -notext -in node4.csr -out node4.crt

Next, package they key and certificate into a PKCS12 key store as follows:

openssl pkcs12 -export -chain -CAfile ca.crt -in node4.crt -inkey node4.key -passout pass:myadminpw > node4.p12

Now do the same for the three remaining nodes (generate a key, certificate, and PKCS12 keystore).

Generate Admin Certificate

Since NiFi uses two way SSL authentication, we also need to generate an admin certificate so you can access the NiFi UI.

First create the admin private key:

openssl genrsa -des3 -passout pass:myadminpw -out nifi.key 2048

Note that here we encrypt the private key as well (which we did not do for the nodes).

Now generate a CSR for our key:

openssl req -new -key nifi.key -passin pass:myadminpw -out nifi.csr -subj '/CN=NiFi Admin/O=Acme Corp'

Now sign the CSR with the same CA as we used for our nodes:

openssl ca -batch -config ca.conf -notext -in nifi.csr -out nifi.crt

Now that we have our NiFi Admin key and certificate, let’s bundle it together into a PKCS12 key store:

openssl pkcs12 -export -chain -CAfile ca.crt -in nifi.crt -inkey nifi.key -passin pass:myadminpw -passout pass:myadminpw > nifi.p12

Generate NiFi Trust Store

Since NiFi runs on the JVM, we need to provide it with a trust store that will contain all the certificates that can be trusted. This trust store will contain all the certificates of each node, as well as the admin certificate and CA certificate.

First create a trust store file containing our CA certificate:

keytool -importcert -v -trustcacerts -alias root -file ca.crt -keystore nifi-trust.jks -storepass myadminpw -noprompt

Next import our admin certificate into the trust store:

keytool -importcert -alias nifi-admin -file nifi.crt -keystore nifi-trust.jks -storepass myadminpw -noprompt

Now import the certificate of each agent node into the trust store. Here we do it for node4:

keytool -importcert -alias node4.example.com -file node4.crt -keystore nifi-trust.jks -storepass myadminpw -noprompt

Do the same for the certificates of node5, node6, and node7.

Custom NiFi Docker Image

As the standard docker image does not do clustered SSL well, we need to customise our docker image slightly. Here is the Dockerfile:

The above Dockerfile also shows how to add a custom processor package (nar) and associated flow files (for templates) and an additional set of JDBC drivers if required (e.g. PostgreSQL).

The associated docker-entrypoint.sh file looks as follows:

Build the custom docker image and push it to your repository.

Copy Key Stores to Cluster Share

Copy the node key stores to a shared location (e.g. an NFS share or CephFS) where each cluster node can access it:

cp node*.p12 /mnt/shared/certs

Also copy over the trust store we created:

cp nifi-trust.jks /mnt/shared/certs

Lauch NiFi on Agents

Now that our custom image is available we can launch it on each agent as follows:

docker run -d --network host --name nifi \
    -v /mnt/shared/certs:/etc/certs
    -e NIFI_CLUSTER_IS_NODE="true" \
    -e NIFI_WEB_HTTPS_PORT="8443" \
    -e NIFI_WEB_PROXY_HOST="service.example.com" \
    -e NIFI_AUTH="tls" \
    -e NIFI_CERTS_DIR="/etc/certs" \
    -e NIFI_TRUSTSTORE_FILE="nifi-trust.jks" \
    -e NIFI_TRUSTSTORE_TYPE="JKS" \
    -e NIFI_TRUSTSTORE_PASSWORD="myadminpw" \
    -e NIFI_KEYSTORE_FILE="node4.p12" \
    -e NIFI_KEYSTORE_TYPE="PKCS12" \
    -e NIFI_KEYSTORE_PASSWORD="myadminpw" \
    -e NIFI_ADMIN_IDENTITY="CN=NiFi Admin, O=Acme Corp" \
    -e NIFI_NODE_IDENTITIES="node4,node5,node6,node7" \
    -e NIFI_CLUSTER_NODE_PROTOCOL_PORT="8082" \
    -e NIFI_ZK_CONNECT_STRING="node1:2181,node2:2181,node3:2181" \
    -e NIFI_ELECTION_MAX_WAIT="20 sec" \
    acme/nifi-image:1.0.0

Note that you have to use the NIFI_KEYSTORE_FILE that corresponds to the node you are launching on! In the example above we are launching on node4 hence we use node4.p12.

The Mesos Marathon descriptor would look something like this:

{
  "id": "nifi",
  "cpus": 2,
  "mem": 2048,
  "disk": 0,
  "instances": 4,
  "constraints": [
    ["hostname", "UNIQUE"]
  ],
  "env": {
    "NIFI_CLUSTER_IS_NODE": "true",
    "NIFI_WEB_HTTPS_PORT": "20000",
    "NIFI_WEB_PROXY_HOST": "service.example.com",
    "NIFI_AUTH": "tls",
    "NIFI_CERTS_DIR": "/etc/certs",
    "NIFI_TRUSTSTORE_FILE": "nifi-trust.jks",
    "NIFI_TRUSTSTORE_TYPE": "JKS",
    "NIFI_TRUSTSTORE_PASSWORD": "myadminpw",
    "NIFI_ADMIN_IDENTITY": "CN=NiFi Admin, O=Acme Corp",
    "NIFI_NODE_IDENTITIES": "node4,node5,node6,node7",
    "NIFI_CLUSTER_NODE_PROTOCOL_PORT": "8082",
    "NIFI_ZK_CONNECT_STRING": "node1:2181,node2:2181,node3:2181",
    "NIFI_ELECTION_MAX_WAIT": "20 sec"
  },
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "acme/nifi-image:1.0.0",
      "network": "HOST",
      "parameters": [
        { "key": "rm", "value": "true"}
      ]
    },
    "volumes": [
      {
         "hostPath": "/mnt/shared/certs",
         "containerPath": "/etc/certs",
         "mode": "RO"
      }
    ]
  },
  "portDefinitions": [
    {
      "port": 20000,
      "protocol": "tcp",
      "name": "http"
    },
    {
      "port": 8082,
      "protocol": "tcp",
      "name": "cluster"
    }
  ],
  "requirePorts" : true,
  "healthChecks": [
    {
      "portIndex": 0,
      "protocol": "TCP",
      "gracePeriodSeconds": 120,
      "intervalSeconds": 60,
      "timeoutSeconds": 30,
      "maxConsecutiveFailures": 3
    }
  ],
  "labels": {
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_MODE": "tcp",
    "HAPROXY_0_STICKY":"true",
    "HAPROXY_1_ENABLED": "false"
  }
}