Kubernetes Deployment

If you have configured and built the sample stream applications to run with one of the supported message brokers, You can run them as stand-alone applications on a Kubernetes cluster.

This section walks you through how to deploy the three Spring Cloud Stream applications on Kubernetes.

Setting up the Kubernetes Cluster

For this example, we need a running Kubernetes cluster. For this example, we deploy to minikube.

Verifying Minikube is Running

To verify that Minikube is running, run the following command (shown with typical output if Minikube is running):

$minikube status

host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

Building Docker Images

To build Docker images, we use the Spring Boot Maven plugin.

If you built the apps from scratch using the Boot Maven plugin, the default configuration should work.

Now you can run the Maven build to create the Docker images in the minikube Docker registry. To do so, run the following commands:

$ eval $(minikube docker-env)
$./mvnw spring-boot:build-image

If you downloaded the project source, the project includes a parent pom to let you build all the modules with a single command. Otherwise, run the build for the source, processor, and sink individually. You need only run eval $(minikube docker-env) once for each terminal session.

If you built from the project source using -Pkafka or -Prabbit, then the images will contain the corresponding binder dependency. If you want to build the images for a specific message broker, use:

./mvnw clean spring-boot:build-image -P<broker>

Kafka RabbitMQ

Installing Apache Kafka

Now we can install the Kafka message broker by using the default configuration from Spring Cloud Data Flow. To do so, run the following command:

kubectl apply -f https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.8.4/src/kubernetes/kafka/kafka-deployment.yaml \
-f https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.8.4/src/kubernetes/kafka/kafka-svc.yaml \
-f https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.8.4/src/kubernetes/kafka/kafka-zk-deployment.yaml \
-f https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.8.4/src/kubernetes/kafka/kafka-zk-svc.yaml

Deploying the Stream

To deploy the stream, you must first copy and paste the following YAML and save it to usage-cost-stream.yaml:

kind: Pod
apiVersion: v1
metadata:
  name: usage-detail-sender
  labels:
    app: usage-cost-stream
spec:
  containers:
    - name: usage-detail-sender
      image: usage-detail-sender:0.0.1-SNAPSHOT
      env:
        - name: SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS
          value: kafka
        - name: SPRING_CLOUD_STREAM_BINDINGS_OUTPUT_DESTINATION
          value: user-details
---
kind: Pod
apiVersion: v1
metadata:
  name: usage-cost-processor
  labels:
    app: usage-cost-stream
spec:
  containers:
    - name: usage-cost-processor
      image: usage-cost-processor:0.0.1-SNAPSHOT
      ports:
        - containerPort: 80
          protocol: TCP
      env:
        - name: SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS
          value: kafka
        - name: SPRING_CLOUD_STREAM_BINDINGS_INPUT_GROUP
          value: usage-cost-stream
        - name: SPRING_CLOUD_STREAM_BINDINGS_INPUT_DESTINATION
          value: user-details
        - name: SPRING_CLOUD_STREAM_BINDINGS_OUTPUT_DESTINATION
          value: user-cost
---
kind: Pod
apiVersion: v1
metadata:
  name: usage-cost-logger
  labels:
    app: usage-cost-stream
spec:
  containers:
    - name: usage-cost-logger
      image: usage-cost-logger:0.0.1-SNAPSHOT
      env:
        - name: SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS
          value: kafka
        - name: SPRING_CLOUD_STREAM_BINDINGS_INPUT_GROUP
          value: usage-cost-stream
        - name: SPRING_CLOUD_STREAM_BINDINGS_INPUT_DESTINATION
          value: user-cost

Then you need to deploy the apps, by running the following command:

kubectl apply -f usage-cost-stream.yaml

If all is well, you should see the following output:

pod/usage-detail-sender created
pod/usage-cost-processor created
pod/usage-cost-logger created

The preceding YAML specifies three pod resources, for the source, processor, and sink applications. Each pod has a single container that references the corresponding docker image.

We set the Kafka binding parameters as environment variables. The input and output destination names have to be correct to wire the stream. Specifically, the output of the source must be the same as the input of the processor, and the output of the processor must be the same as the input of the sink. We also set the logical hostname for the Kafka broker so that each application can connect to it. Here we use the Kafka service name — kafka-broker, in this case. We set the app: user-cost-stream label to logically group our apps.

We set the Spring Cloud Stream binding parameters by using environment variables. The input and output destination names have to be correct to wire the stream. Specifically, the output of the source must be the same as the input of the processor, and the output of the processor must be the same as the input of the sink. We set the inputs and outputs as follows:

Usage Detail Sender: SPRING_CLOUD_STREAM_BINDINGS_OUTPUT_DESTINATION=user-details
Usage Cost Processor: SPRING_CLOUD_STREAM_BINDINGS_INPUT_DESTINATION=user-details and SPRING_CLOUD_STREAM_BINDINGS_OUTPUT_DESTINATION=user-cost
Usage Cost Logger: SPRING_CLOUD_STREAM_BINDINGS_INPUT_DESTINATION=user-cost

Installing RabbitMQ

You can install the RabbitMQ message broker by using the default configuration from Spring Cloud Data Flow. To do so, run the following command:

kubectl apply -f https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.8.4/src/kubernetes/rabbitmq/rabbitmq-deployment.yaml \
-f https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.8.4/src/kubernetes/rabbitmq/rabbitmq-svc.yaml

Deploying the Stream

To deploy the stream, you must first copy and paste the following YAML content and save it to usage-cost-stream.yaml:

kind: Pod
apiVersion: v1
metadata:
  name: usage-detail-sender
  labels:
    app: usage-cost-stream
spec:
  containers:
    - name: usage-detail-sender
      image: usage-detail-sender:0.0.1-SNAPSHOT
      env:
        - name: SPRING_RABBITMQ_ADDRESSES
          value: rabbitmq

---
kind: Pod
apiVersion: v1
metadata:
  name: usage-cost-processor
  labels:
    app: usage-cost-stream
spec:
  containers:
    - name: usage-cost-processor
      image: usage-cost-processor:0.0.1-SNAPSHOT
      env:
        - name: SPRING_RABBITMQ_ADDRESSES
          value: rabbitmq
---
kind: Pod
apiVersion: v1
metadata:
  name: usage-cost-logger
  labels:
    app: usage-cost-stream
spec:
  containers:
    - name: usage-cost-logger
      image: usage-cost-logger:0.0.1-SNAPSHOT
      env:
        - name: SPRING_RABBITMQ_ADDRESSES
          value: rabbitmq

Then you can deploy the apps, as follows:

kubectl apply -f usage-cost-stream.yaml

If all is well, you should see the following output:

pod/usage-detail-sender created
pod/usage-cost-processor created
pod/usage-cost-logger created

The preceding YAML specifies three pod resources, for the source, processor, and sink applications. Each pod has a single container that references the respective docker image.

We set the logical hostname for the RabbitMQ broker for each app to connect to it. Here we use the RabbitMQ service name (rabbitmq in this case). We also set the label app: user-cost-stream to logically group our apps.

Verifying the Deployment

To verify the deployment, use the following command to tail the log for the usage-cost-logger sink:

kubectl logs -f usage-cost-logger

You should see messages similar to the following streaming:

2021-03-05 17:40:52.280  INFO 1 --- [fOjmHePqc5VWA-1] i.s.d.s.u.UsageCostLogger     : {"userId": "user5", "callCost": "28.1", "dataCost": "7.3500000000000005" }
2021-03-05 17:40:53.279  INFO 1 --- [fOjmHePqc5VWA-1] i.s.d.s.u.UsageCostLogger     : {"userId": "user2", "callCost": "21.400000000000002", "dataCost": "22.900000000000002" }
2021-03-05 17:40:54.279  INFO 1 --- [fOjmHePqc5VWA-1] i.s.d.s.u.UsageCostLogger     : {"userId": "user3", "callCost": "0.7000000000000001", "dataCost": "26.35" }
2021-03-05 17:40:55.281  INFO 1 --- [fOjmHePqc5VWA-1] i.s.d.s.u.UsageCostLogger     : {"userId": "user5", "callCost": "17.5", "dataCost": "25.8" }
2021-03-05 17:40:56.286  INFO 1 --- [fOjmHePqc5VWA-1] i.s.d.s.u.UsageCostLogger     : {"userId": "user3", "callCost": "25.3", "dataCost": "3.0" }
2021-03-05 17:40:57.289  INFO 1 --- [fOjmHePqc5VWA-1] i.s.d.s.u.UsageCostLogger     : {"userId": "user4", "callCost": "6.300000000000001", "dataCost": "19.700000000000003" }
2021-03-05 17:40:58.290  INFO 1 --- [fOjmHePqc5VWA-1] i.s.d.s.u.UsageCostLogger     : {"userId": "user4", "callCost": "1.4000000000000001", "dataCost": "19.6" }

Clean Up

To delete the stream, we can use the label we created earlier, as follows:

kubectl delete pod -l app=usage-cost-stream

kubectl delete -f usage-cost-stream.yaml

To uninstall RabbitMQ, run the following command:

kubectl delete all -l app=rabbitmq

To uninstall Kafka, run the following command:

kubectl delete all -l app=kafka