Replacing Internal Certificates
Internal Communication and Certificates​
NeuVector includes default self-signed certificates for encryption for the Manager (console/UI access), Controller (REST API, internal), Enforcer (internal), and Scanner (internal) communications.
These certificates can be replaced by your own to further harden communication. For replacing certificates used by external access to NeuVector (i.e, browser to the Manager, or REST API to the Controller), please see this section. See below for replacing the certificates used in internal communication between NeuVector containers.
Replacing certificates is recommended to be performed only during initial deployment of NeuVector. Replacing on a running cluster (even with rolling upgrade) may result in an unstable state where NeuVector pods are unable to communicate with each other due to a mismatch in certificates, and DATA LOSS may occur.
Replacing Certificates Used in Internal Communications of NeuVector​
Replace the internal encryption files ca.crt
, tls.key
, tls.crt
as follows:
- Create a new
ca.cfg
file with your favorite editor:
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = California
L = San Jose
O = NeuVector Inc.
OU = Neuvector
CN = Neuvector
[v3_req]
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = Neuvector
For additional information on ca.cfg
, see https://open-docs.neuvector.com/configuration/console/replacecert.
- Choose your scenario from the three options below:
- New certificate
- Update current certificate with SANs
- Regenarate certificate files and add SANs
If your certificate is about to expire and you need to generate a new one, follow the steps below:
- Delete the old
ca.crt
,tls.key
,tls.crt
, kubernetes secret, and generate new ones:
kubectl delete secret internal-cert -n neuvector
openssl genrsa -out ca.key 2048
openssl req -x509 -sha256 -new -nodes -key ca.key -days 3650 -out ca.crt
openssl genrsa -out tls.key 2048
openssl req -new -key tls.key -sha256 -out cert.csr -config ca.cfg
openssl req -in cert.csr -noout -text
openssl x509 -req -sha256 -in cert.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt -days 3650 -extensions 'v3_req' -extfile ca.cfg
openssl x509 -in tls.crt -text
kubectl create secret generic internal-cert -n neuvector --from-file=tls.key --from-file=tls.crt --from-file=ca.crt
Then edit the Controller, Enforcer, and Scanner deployment yamls, adding:
containers:
- name: neuvector-controller/enforcer/scanner-pod
volumeMounts:
- mountPath: /etc/neuvector/certs/internal/tls.key
name: internal-cert
readOnly: true
subPath: tls.key
- mountPath: /etc/neuvector/certs/internal/tls.crt
name: internal-cert
readOnly: true
subPath: tls.crt
- mountPath: /etc/neuvector/certs/internal/ca.crt
name: internal-cert
readOnly: true
subPath: ca.crt
volumes:
- name: internal-cert
secret:
defaultMode: 420
secretName: internal-cert
Then proceed to deploy NeuVector as before. You can also shell into the controller/enforcer/scanner pods to confirm that the ca.crt, tls.key, tls.crt files are the customized ones and that the NeuVector communications are working using the new certificates.
Sample patch commands for controller (change namespace to cattle-neuvector-system if needed, and modify for use on enforcer, scanner):
NAMESPACE=neuvector
kubectl patch deployment -n ${NAMESPACE} neuvector-controller-pod --type='json' -p='[{"op": "add", "path": "/spec/template/spec/volumes/-", "value": {"name": "internal-cert", "secret": {"defaultMode": 420, "secretName": "internal-cert"}} } ]'
kubectl patch deployment -n ${NAMESPACE} neuvector-controller-pod --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/volumeMounts", "value": [{"mountPath": "/etc/neuvector/certs/internal/cert.key", "name": "internal-cert", "readOnly": true, "subPath": "cert.key"}, {"mountPath": "/etc/neuvector/certs/internal/cert.pem", "name": "internal-cert", "readOnly": true, "subPath": "cert.pem"}, {"mountPath": "/etc/neuvector/certs/internal/ca.cert", "name": "internal-cert", "readOnly": true, "subPath": "ca.cert"} ] } ]'
If your certificate files were created before NeuVector version 5.3, you need to upate the certificate with at least one Subject Alternative Name, or SAN for short.
If you still have the files ca.key
and ca.crt
accessible, run the commands as follows:
kubectl delete secret internal-cert -n neuvector
openssl genrsa -out tls.key 2048
openssl req -new -key tls.key -sha256 -out cert.csr -config ca-new.cfg
openssl req -in cert.csr -noout -text
openssl x509 -req -sha256 -in cert.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt -days 3650 -extensions 'v3_req' -extfile ca-new.cfg
openssl x509 -in tls.crt -text
kubectl create secret generic internal-cert -n neuvector --from-file=tls.key --from-file=tls.crt --from-file=ca.crt
Once the certificate files have been updated, restart the deployments in order to use the updated certificate:
kubectl rollout restart deployment neuvector-controller-pod
kubectl rollout restart deployment neuvector-scanner-pod
kubectl rollout restart deployment neuvector-registry-adapter-pod
kubectl rollout restart ds neuvector-enforcer-pod
If your certificate files were created before NeuVector version 5.3, you need to upate the certificate with at least one Subject Alternative Name, or SAN for short.
If you don´t have the files ca.key
and ca.crt
anymore, follow the steps below:
- Backup your original certificate
kubectl get secret internal-cert -o yaml > internal-cert.yaml
- Export the existing internal-cert
kubectl get secret internal-cert -o json | jq -r '.data."ca.crt"' | base64 -d > old-ca.crt
kubectl get secret internal-cert -o json | jq -r '.data."tls.crt"' | base64 -d > old-tls.crt
kubectl get secret internal-cert -o json | jq -r '.data."tls.key"' | base64 -d > old-tls.key
- Create new certificate files and internal certificates
openssl genrsa -out ca.key 2048
openssl req -x509 -sha256 -new -nodes -key ca.key -days 3650 -out ca.crt
openssl genrsa -out tls.key 2048
openssl req -new -key tls.key -sha256 -out cert.csr -config ca.cfg
openssl req -in cert.csr -noout -text
openssl x509 -req -sha256 -in cert.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt -days 3650 -extensions 'v3_req' -extfile ca.cfg
openssl x509 -in tls.crt -text
- Merge the old and new
ca.crt
files
cat old-ca.crt > /tmp/ca.crt
cat ca.crt >> /tmp/ca.crt
- Update the Kubernetes secret with the merged
ca.crt
kubectl delete secret internal-cert -n neuvector
kubectl create secret generic internal-cert -n neuvector --from-file=tls.key=old-tls.key --from-file=tls.crt=old-tls.crt --from-file=ca.crt=/tmp/ca.crt
- Restart the deployments in order to use the updated certificate
kubectl rollout restart deployment neuvector-controller-pod
kubectl rollout restart deployment neuvector-scanner-pod
kubectl rollout restart deployment neuvector-registry-adapter-pod
kubectl rollout restart ds neuvector-enforcer-pod
- Wait for the restart to complete
kubectl rollout status deployment neuvector-controller-pod
kubectl rollout status deployment neuvector-scanner-pod
kubectl rollout status deployment neuvector-registry-adapter-pod
kubectl rollout status ds neuvector-enforcer-pod
-
Make sure the console can be accessed and controllers are all online.
-
Update the Kubernetes secret with the new
tls.key
kubectl delete secret internal-cert -n neuvector
kubectl create secret generic internal-cert -n neuvector --from-file=tls.key=tls.key --from-file=tls.crt=tls.crt --from-file=ca.crt=/tmp/ca.crt
- Restart the deployments in order to use the updated certificate
kubectl rollout restart deployment neuvector-controller-pod
kubectl rollout restart deployment neuvector-scanner-pod
kubectl rollout restart deployment neuvector-registry-adapter-pod
kubectl rollout restart ds neuvector-enforcer-pod
- Wait for the restart to complete
kubectl rollout status deployment neuvector-controller-pod
kubectl rollout status deployment neuvector-scanner-pod
kubectl rollout status deployment neuvector-registry-adapter-pod
kubectl rollout status ds neuvector-enforcer-pod
-
Make sure the console can be accessed and controllers are all online.
-
Update the Kubernetes secret with the new
ca.crt
kubectl delete secret internal-cert -n neuvector
kubectl create secret generic internal-cert -n neuvector --from-file=tls.key=tls.key --from-file=tls.crt=tls.crt --from-file=ca.crt=ca.crt
- Restart the deployments in order to use the updated certificate
kubectl rollout restart deployment neuvector-controller-pod
kubectl rollout restart deployment neuvector-scanner-pod
kubectl rollout restart deployment neuvector-registry-adapter-pod
kubectl rollout restart ds neuvector-enforcer-pod
- Wait for the restart to complete
kubectl rollout status deployment neuvector-controller-pod
kubectl rollout status deployment neuvector-scanner-pod
kubectl rollout status deployment neuvector-registry-adapter-pod
kubectl rollout status ds neuvector-enforcer-pod
- Make sure the console can be accessed and controllers are all online.
Updating/Deploying with Helm​
As of Helm chart 2.4.1
we can now manage the internal certificate install. The chart values.yaml should be reviewed for all the settings. The below example uses RKE2, standard Ingress and installer certificates.
# add chart
helm repo add neuvector https://neuvector.github.io/neuvector-helm/
# update chart
helm repo update
# add domain for ingress
export domain=awesome.sauce
# run the helm
helm upgrade -i neuvector -n neuvector neuvector/core --create-namespace --set imagePullSecrets=regsecret --set k3s.enabled=true --set k3s.runtimePath=/run/k3s/containerd/containerd.sock --set manager.ingress.enabled=true --set manager.ingress.host=neuvector.$domain --set manager.svc.type=ClusterIP --set controller.pvc.enabled=true --set controller.pvc.capacity=500Mi --set controller.internal.certificate.secret=internal-cert --set cve.scanner.internal.certificate.secret=internal-cert --set enforcer.internal.certificate.secret=internal-cert