Helm
Our Helm charts repository is the quickest way to get started using Flex with Kubernetes.
Prerequisites
The minimum information Kpow requires to operate is:
- Community
- Enterprise
- License Details: Request a community license.
- Flink REST URL
See the Flex Documentation for a full list of configuration options.
- License Details: Start a free 30-day trial.
- Flink REST URL
See the Flex Documentation for a full list of configuration options.
Configure Kubernetes/EKS
You need to connect to a Kubernetes environment before you can install Kpow.
The following examples demonstrate installing Kpow in Amazon EKS.
aws eks --region <your-aws-region> update-kubeconfig --name <your-eks-cluster-name>
Updated context arn:aws:eks:<your-aws-region>:123123123:cluster/<your-eks-cluster-name> in /your/.kube/config
You can verify that your Kubernetes cluster is available by running the following command:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-...-21.ec2.internal Ready <none> 2m15s v1.32.9-eks-113cf36
...
Run Flex in Kubernetes
Configure the Flex Helm Repository
Add the Factor House Helm Repository in order to use the Flex Helm Chart.
helm repo add factorhouse https://charts.factorhouse.io
Update Helm repositories to ensure you install the latest version of Flex.
helm repo update
Start a Flex Instance
Start Flex with config from '--set env.XYZ'
When using helm install
, you can pass configuration with the --set env.XYZ
flag. This requires careful formatting for certain values.
Some fields, particularly integers and strings containing quotation marks, require quoting. You may also need to escape special characters (like commas or nested quotes) with a backslash (\
). For more details, see Helm's documentation on The Format and Limitations of --set
.
The following example shows how to install Flex from the command line, highlighting how to handle escaped commas and quotes:
- Community
- Enterprise
helm install flex factorhouse/flex-ce \
--set env.LICENSE_ID="00000000-0000-0000-0000-000000000001" \
--set env.LICENSE_CODE="FLEX_COMMUNITY" \
--set env.LICENSEE="John\, doe" \ # <-- note the escaped comma
--set env.LICENSE_EXPIRY="2022-01-01" \
--set env.LICENSE_SIGNATURE="638......A51" \
--set env.FLINK_REST_URL="http://flink-dev.svc" \
--create-namespace --namespace factorhouse
NAME: flex
LAST DEPLOYED: Mon May 31 17:22:21 2021
NAMESPACE: factorhouse
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace factorhouse -l "app.kubernetes.io/name=flex,app.kubernetes.io/instance=flex" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:3000 to use your application"
kubectl --namespace factorhouse port-forward $POD_NAME 3000:3000
helm install flex factorhouse/flex \
--set env.LICENSE_ID="00000000-0000-0000-0000-000000000001" \
--set env.LICENSE_CODE="FLEX_CREDIT" \
--set env.LICENSEE="Factor House\, Inc." \ # <-- note the escaped comma
--set env.LICENSE_EXPIRY="2022-01-01" \
--set env.LICENSE_SIGNATURE="638......A51" \
--set env.FLINK_REST_URL="http://flink-dev.svc" \
--create-namespace --namespace factorhouse
NAME: flex
LAST DEPLOYED: Mon May 31 17:22:21 2021
NAMESPACE: factorhouse
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace factorhouse -l "app.kubernetes.io/name=flex,app.kubernetes.io/instance=flex" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:3000 to use your application"
kubectl --namespace factorhouse port-forward $POD_NAME 3000:3000
Start Flex with Environment Variables from a ConfigMap
You can configure Flex with a ConfigMap of environment variables as follows:
- Community
- Enterprise
helm install flex factorhouse/flex-ce \
--set envFromConfigMap=flex-config \
--create-namespace --namespace factorhouse
helm install flex factorhouse/flex \
--set envFromConfigMap=flex-config \
--create-namespace --namespace factorhouse
This approach requires a ConfigMap
named flex-config
to already exist in the factorhouse
namespace. To configure Flex with a local ConfigMap template, see Configuring with an Existing ConfigMap.
For general guidance, see the Kubernetes documentation on configuring all key-value pairs in a ConfigMap as environment variables.
Manage a Flex Instance
Set the $POD_NAME variable and test the Flex UI
Follow the notes instructions to set the $POD_NAME variable and configure port forwarding to the Flex UI.
export POD_NAME=$(kubectl get pods --namespace factorhouse -l "app.kubernetes.io/name=flex,app.kubernetes.io/instance=flex" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:3000 to use your application"
kubectl --namespace factorhouse port-forward $POD_NAME 3000:3000
Flex is now available on http://127.0.0.1:3000.
Check the Flex Pod
kubectl describe pods --namespace factorhouse
Name: flex-9988df6b6-vvf8z
Namespace: factorhouse
Priority: 0
Node: ip-172-31-33-42.ap-southeast-2.compute.internal/172.31.33.42
Start Time: Mon, 31 May 2021 17:22:22 +1000
Labels: app.kubernetes.io/instance=flex
app.kubernetes.io/name=flex
pod-template-hash=9988df6b6
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
View the Flex Pod Logs
kubectl logs --namespace factorhouse $POD_NAME
11:36:49.111 INFO [main] flex.system ? start Flex
...
Remove Flex
helm delete --namespace factorhouse flex
Start Flex with Local Changes
- Community
- Enterprise
You can run Flex with local edits to these charts and provide local configuration when running Flex.
Pull and Untar the Flex Charts
helm pull factorhouse/flex-ce --untar --untardir .
Make Local Edits
Make any edits required to flex-ce/Chart.yaml
or flex-ce/values.yaml
(adding volume mounts, etc).
Run Local Charts
The command to run local charts is slightly different, see ./flex-ce
rather than factorhouse/flex-ce
.
helm install flex ./flex-ce \
<.. --set configuration, etc ..> \
--create-namespace --namespace factorhouse
Configuring with an Existing ConfigMap
This is the recommended method for managing configuration separately from the Helm chart.
1. Prepare Your ConfigMap Manifest
Copy the example file (flex-config.yaml.example), then edit it to set your desired metadata.name
(e.g., flex-config
) and fill in your configuration under the data
section.
cp ./flex-ce/flex-config.yaml.example flex-config.yaml
# now edit flex-config.yaml
2. Create the ConfigMap in Kubernetes
Before installing, use kubectl
to create the ConfigMap
object in your cluster from the file you just prepared.
kubectl apply -f flex-config.yaml --namespace factorhouse
3. Install the Chart
Install the Helm chart, using --set
to reference the name of the ConfigMap
you just created. The --create-namespace
flag will ensure the target namespace exists.
helm install flex ./flex-ce \
--set envFromConfigMap=flex-config \
--create-namespace --namespace factorhouse
The Flex pod will now start using the environment variables from your externally managed ConfigMap
.
See flex-config.yaml.example for an example ConfigMap file.
See the Kubernetes documentation on configuring all key-value pairs in a config map as container environment variables for more information.
You can run Flex with local edits to these charts and provide local configuration when running Flex.
Pull and Untar the Flex Charts
helm pull factorhouse/flex --untar --untardir .
Make Local Edits
Make any edits required to flex/Chart.yaml
or flex/values.yaml
(adding volume mounts, etc).
Run Local Charts
The command to run local charts is slightly different, see ./flex
rather than factorhouse/flex
.
helm install flex ./flex \
<.. --set configuration, etc ..> \
--create-namespace --namespace factorhouse
Configuring with an Existing ConfigMap
This is the recommended method for managing configuration separately from the Helm chart.
1. Prepare Your ConfigMap Manifest
Copy the example file (flex-config.yaml.example), then edit it to set your desired metadata.name
(e.g., flex-config
) and fill in your configuration under the data
section.
cp ./flex/flex-config.yaml.example flex-config.yaml
# now edit flex-config.yaml
2. Create the ConfigMap in Kubernetes
Before installing, use kubectl
to create the ConfigMap
object in your cluster from the file you just prepared.
kubectl apply -f flex-config.yaml --namespace factorhouse
3. Install the Chart
Install the Helm chart, using --set
to reference the name of the ConfigMap
you just created. The --create-namespace
flag will ensure the target namespace exists.
helm install flex ./flex \
--set envFromConfigMap=flex-config \
--create-namespace --namespace factorhouse
The Flex pod will now start using the environment variables from your externally managed ConfigMap
.
See flex-config.yaml.example for an example ConfigMap file.
See the Kubernetes documentation on configuring all key-value pairs in a config map as container environment variables for more information.
Manage Sensitive Environment Variables
This helm chart accepts the name of a secret containing sensitive parameters, e.g.
apiVersion: v1
kind: Secret
metadata:
name: flex-secrets
data:
SASL_JAAS_CONFIG: a3JnLmFwYWNoXS5rYWZrYS5jb21tb24uc2VjdXJpdHkucGxhaW4uUGxhaW5Mb2dpbk2vZHVsZSByZXF1aXJiZCB1c2VybmFtZT0iTFQ1V0ZaV1BRWUpHNzRJQyIgcGFzc3dvcmQ9IjlYUFVYS3BLYUQxYzVJdXVNRjRPKzZ2NxJ0a1E4aS9yWUp6YlppdlgvZnNiTG51eGY4SnlFT1dUeXMvTnJ1bTAiBwo=
CONFLUENT_API_SECRET: NFJSejlReFNTTXlTcGhXdjNLMHNYY1F6UGNURmdadlNYT0ZXSXViWFJySmx2N3A2WStSenROQnVpYThvNG1NSRo=
kubectl apply -f ./flex-secrets.yaml --namespace factorhouse
Then run the helm chart (this can be used in conjunction with envFromConfigMap
)
See the Kubernetes documentation on configuring all key value pairs in a secret as environment variables for more information.
- Community
- Enterprise
helm install flex ./flex-ce \
--set envFromSecret=flex-secrets \
--set envFromConfigMap=flex-config \
--create-namespace --namespace factorhouse
helm install flex ./flex \
--set envFromSecret=flex-secrets \
--set envFromConfigMap=flex-config \
--create-namespace --namespace factorhouse
Provide Files to the Flex Pod
There are occasions where you must provide files to the Flex Pod in order for Flex to run correctly, such files include:
- RBAC configuration
- SSL Keystores
- SSL Truststores
How you provide these files is down to user preference, we are not able to provide any support or instruction in this regard.
You may find the Kubernetes documentation on injecting data into applications useful.
Flex Memory and CPU Requirements
The chart runs Flex with Guaranteed QoS, having resource request and limit set to these values by default:
resources:
limits:
cpu: 2
memory: 8Gi
requests:
cpu: 2
memory: 8Gi
These default resource settings are conservative, suited to a deployment of Flex that manages multiple Flink clusters and associated resources.
When running Flex with a single Flink cluster you can experiment with reducing those resources as far as our suggested minimum:
Minimum Resource Requirements
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 2Gi
Adjust these values from the command line like so:
- Community
- Enterprise
helm install flex factorhouse/flex-ce \
--set resources.limits.cpu=1 \
--set resources.limits.memory=2Gi \
--set resources.requests.cpu=1 \
--set resources.requests.memory=2Gi \
--create-namespace --namespace factorhouse
helm install flex factorhouse/flex \
--set resources.limits.cpu=1 \
--set resources.limits.memory=2Gi \
--set resources.requests.cpu=1 \
--set resources.requests.memory=2Gi \
--create-namespace --namespace factorhouse
We recommend always having limits and requests set to the same value, as this set Flex in Guaranteed QoS and provides a much more reliable operation.