Ylem

The open-source data streaming platform is a one-stop-shop solution for orchestrating data streams on top of Apache Kafka, Amazon SQS, Google Pub/Sub, RabbitMQ, various APIs, and data storages.

Installation

Install Docker 4

If you don't yet have Docker 4 installed, install it from their official website for your OS.

Install Ylem

Option 1. Install from pre-build containers

The best way to install Ylem is to clone the repository https://github.com/ylem-co/ylem-installer and follow the installation instructions from it. It will install Ylem from the latest version of pre-build containers stored on Docker Hub.

Ylem will be available at http://localhost:7331/

Option 2. Build and install from the source

If you want to compile Ylem from the source, run docker compose up or docker compose up -d from this repository. It will compile the code and run all the necessary containers.

Ylem is available at http://127.0.0.1:7330/

⚠️ Please pay attention. Compiling from the source might take some time and will keep the resources of your machine busy.

To rebuild a particular container

If you want to rebuild a particular container from a source locally, run the following:

docker compose build --no-cache %%CONTAINER_NAME%%

E.g.

docker compose build --no-cache ylem_users

Using your own Apache Kafka cluster

Ylem uses Apache Kafka to exchange messages for processing pipelines and tasks. By default Ylem already comes with the pre-configured Apache Kafka container, however, you might already have an Apache Kafka cluster in your infrastructure and might want to reuse it.

In this case, you need to take the steps described here in our documentation.

Configuring environment variables in .env files

Some particular integrations might require extra steps and using .env files. Configure them if you need to.

The list of such integrations and more information about them is in our documentation.

Folder structure in this repository

Ylem is a set of microservices. Each microservice is represented by one or more containers in the same network and communicates with each other via the API.

|-- api                  # api microservice
|-- backend
|--|-- integrations      # integrations with external APIs, databases and other software
|--|-- pipelines         # pipelines, tasks, connectors
|--|-- statistics        # statistics of pipeline and task runs
|--|-- users             # users and organizations
|-- database             # a container for storing databases for all the microservices
|-- processor
|--|-- python_processor  # processor of the Python code written in pipelines 
|--|-- taskrunner        # task runner and load balancer
|-- server               # Nginx container in front of all the microservice APIs allowing to avoid CORS issues on the UI side
|-- ui                   # user interface

Each microservice has its own README file containing more information about its usage and functionality.

Documentation

The user and developer documentation of Ylem is available at https://docs.ylem.co/.

The open-source section contains information about the task-processing architecture and configuration of integrations using .env files and parameters.

Explore our additional integration packages

Integration	Repository	Description
	https://github.com/ylem-co/ylem-kafka-trigger	Containerized Apache Kafka listener to stream data to Ylem
	https://github.com/ylem-co/ylem-rabbitmq-consumer	Containerized RabbitMQ consumer to stream data to Ylem
	https://github.com/ylem-co/s3-lambda-trigger	AWS Lambda function to stream data from AWS S3 to Ylem
	https://github.com/ylem-co/tableau-http-wrapper	Containerized HTTP wrapper to stream data from Ylem to Tableau

Key contributors

• olschaefer
• schneekatze
• lunoshot
• Ardem