Installing by Using Docker Compose

Spring Cloud Data Flow provides a Docker Compose file to let you quickly bring up Spring Cloud Data Flow, Skipper, MySQL and Apache Kafka. The additional customization guides help to extend the basic configuration, showing how to switch the binder to RabbitMQ, use different database, enable monitoring more.

Also, when doing development of custom applications, you need to enable the Docker containers that run the Data Flow and the Skipper servers to see your local file system. The Accessing the Host File System chapter below shows how to do that.

For the impatient, here is a quick start, single-line command:

wget -O docker-compose.yml https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.6.3/spring-cloud-dataflow-server/docker-compose.yml; \
DATAFLOW_VERSION=2.6.3 SKIPPER_VERSION=2.5.2 \
docker-compose up

curl https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.6.3/spring-cloud-dataflow-server/docker-compose.yml -o docker-compose.yml & set DATAFLOW_VERSION=2.6.3& set SKIPPER_VERSION=2.5.2& docker-compose up

Detailed instructions of how to configure and start Spring Cloud Data FLow using Docker Compose are provided below.

Downloading the Docker Compose File

Click here to get the installation Docker Compose file or use the wget or curl tools to download it:

wget -O docker-compose.yml https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.6.3/spring-cloud-dataflow-server/docker-compose.yml

curl https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.6.3/spring-cloud-dataflow-server/docker-compose.yml -o docker-compose.yml

Starting Docker Compose

From within the docker-compose.yml directory run:

export DATAFLOW_VERSION=2.6.3
export SKIPPER_VERSION=2.5.2
docker-compose up

set DATAFLOW_VERSION=2.6.3
set SKIPPER_VERSION=2.5.2
docker-compose up

$Env:DATAFLOW_VERSION="2.6.3"
$Env:SKIPPER_VERSION="2.5.2"
docker-compose up

once the emitting of log messages on the command prompt stop, open the Spring Cloud Data Flow Dashboard at http://localhost:9393/dashboard or use the Shell as explained below.

The following environment variables can be used to configure the docker-compose.yml:

The docker-compose.yml configurations expose the following container ports to the host machine:

Stopping Spring Cloud Data Flow

1. Press Ctrl+C to shut down the docker compose process.
2. Run the following command to clean the used Docker containers:

docker-compose down

If errors occur due to old or hanging containers, clean all containers:

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

FOR /f "tokens=*" %i IN ('docker ps -aq') DO docker rm %i -f

Shell

For convenience and as an alternative to the Spring Cloud Data Flow Dashboard, you can use the Spring Cloud Data Flow Shell. The shell supports tab completion for commands and application configuration properties.

If you have started Spring Cloud Data Flow by using Docker Compose, the shell is also included in the springcloud/spring-cloud-dataflow-server Docker image. To use it, open another console window and type the following:

docker exec -it dataflow-server java -jar shell.jar

If you have started the Data Flow server with java -jar, you can download and start the shell. To download the Spring Cloud Data Flow Shell application, run the following command:

wget -O spring-cloud-dataflow-shell-2.6.3.jar https://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-shell/2.6.3/spring-cloud-dataflow-shell-2.6.3.jar

curl https://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-shell/2.6.3/spring-cloud-dataflow-shell-2.6.3.jar -o spring-cloud-dataflow-shell-2.6.3.jar

Accessing the Host File System

If you develop custom applications on your local machine, you need to register them with Spring Cloud Data Flow. Since Data Flow server run inside a Docker container, you need to configure this container to access to your local file system to resolve the applications registration references. In order to deploy those custom applications, the Skipper Server also needs to access them from within its own Docker container.

By default docker-compose.yml mounts the local host folder (e.g. folder where the docker-compose process is started) to a /root/scdf folder inside both the dataflow-server and the skipper containers.

The HOST_MOUNT_PATH and DOCKER_MOUNT_PATH environment variables (see the configuration table) allow you to customize the default host and container paths.

For example, if the my-app-1.0.0.RELEASE.jar is stored in the /tmp/myapps/ folder on the host machine (C:\Users\User\MyApps on Windows), you can make it accessible to the dataflow-server and skipper containers by setting the HOST_MOUNT_PATH like this:

export HOST_MOUNT_PATH=/tmp/myapps

set HOST_MOUNT_PATH=C:\Users\User\MyApps

$Env:HOST_MOUNT_PATH="C:\Users\User\MyApps"

and follow the starting docker-compose instructions to start the cluster.

See the compose-file reference for further configuration details.

Once the host folder is mounted, you can register the app starters (from /root/scdf), with the Data Flow Shell or Dashboard by using the file:// URI schema. The following example shows how to do so:

app register --type source --name my-app --uri file://root/scdf/my-app-1.0.0.RELEASE.jar

You can also pre-register the apps directly, by modifying the app-import configuration in the docker-compose.yml. For every pre-registered app starer, add an additional wget statement to the app-import block configuration, as the following example shows:

app-import:
  image: alpine:3.7
  command: >
    /bin/sh -c "
      ....
      wget -qO- 'https://dataflow-server:9393/apps/source/my-app' --post-data='uri=file:/root/apps/my-app.jar&metadata-uri=file:/root/apps/my-app-metadata.jar"

See the Data Flow REST API for further details.

Maven Local Repository Mounting

You can develop applications and install them in the local Maven repository (using mvn install) while the Data Flow server is running and have immediate access to the new built applications.

To do this you must mount the host’s local maven repository to the dataflow-server and skipper containers using a volume called /root/.m2/. The Maven Local repository location defaults to ~/.m2 for Linux and OSX and to C:\Users\{your-username}\.m2 for Windows.

We can leverage the HOST_MOUNT_PATH and DOCKER_MOUNT_PATH variables to configure mount volumes like this:

export HOST_MOUNT_PATH=~/.m2
export DOCKER_MOUNT_PATH=/root/.m2/

set HOST_MOUNT_PATH=%userprofile%\.m2
set DOCKER_MOUNT_PATH=/root/.m2/

$Env:HOST_MOUNT_PATH="~\.m2"
$Env:DOCKER_MOUNT_PATH="/root/.m2/"

and follow the starting docker-compose instructions to start the cluster.

Now you can use the maven:// URI schema and Maven coordinates to resolve jars installed in the host’s maven repository, as the following example shows:

app register --type processor --name pose-estimation --uri maven://org.springframework.cloud.stream.app:pose-estimation-processor-rabbit:2.0.2.BUILD-SNAPSHOT --metadata-uri maven://org.springframework.cloud.stream.app:pose-estimation-processor-rabbit:jar:metadata:2.0.2.BUILD-SNAPSHOT

This approach lets you use applications that are built and installed on the host machine (for example, by using mvn clean install) directly with the Spring Cloud Data Flow server.

Monitoring

The basic Data Flow docker-compose configuration does not enable the monitoring functionality for Stream and Task applications. Follow the Monitoring with Prometheus and Grafana or Monitoring with InfluxDB and Grafana customization guides to learn how to enable and configure the monitoring for Spring Cloud Data Flow.

To learn more about the monitoring experience in Spring Cloud Data Flow with Prometheus and InfluxDB, see the Stream Monitoring feature guide.

Debugging

The Debug Stream Applications guide, shows how to enables remote debugging for Stream Applications deployed by Data Flow.

The Debug Data Flow Server guide, shows how extend the docker compose configuration to enables remote Data Flow Server debugging with your IDE such as IntelliJ or Eclipse.

The Debug Skipper Server guide, shows how extend the docker compose configuration to enables remote Skipper Server debugging with your IDE such as IntelliJ or Eclipse.