Skip to content

kafka #253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 19, 2025
Merged

kafka #253

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion apps/docs/content/homepage.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ export const databases = [
{ name: "Meilisearch", link: "/meilisearch/overview", icon: <Icons.meilisearch/> },
{ name: "Qdrant", link: "/qdrant/overview", icon: <Icons.qdrant/> },
{ name: "NATS", link: "/nats/overview", icon: <Icons.nats/> },
{ name: "Kafka", link: "/kafka/overview", icon: <Icons.kafka/> },
{ name: "KeyDB", link: "/keydb/overview", icon: <Icons.keydb/> },
{ name: "Kafka", icon: <Icons.kafka/> },
]

export const storages = [
Expand Down
114 changes: 114 additions & 0 deletions apps/docs/content/kafka/overview.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
---
title: Apache Kafka
desc: Production-ready Apache Kafka on Zerops platform with automated scaling and enterprise reliability. Deploy, manage, and scale with zero infrastructure overhead.
---

import UnorderedList from '@site/src/components/UnorderedList';
import UnorderedCodeList from '@site/src/components/UnorderedCodeList';
import data from '@site/static/data.json';

Zerops provides a fully managed [Apache Kafka](https://kafka.apache.org/) messaging platform with automated scaling and zero infrastructure overhead, letting developers focus entirely on development.

## Supported Versions

Currently supported Kafka version:
<UnorderedList data={data.kafka.readable}/>

Import configuration version:
<UnorderedCodeList data={data.kafka.import}/>

## Service Configuration

Our Kafka implementation features optimized default settings designed for common use cases.

### Key Configuration

* **Client Connections:** Data brokers available on port `9092`
* **Authentication:** Secure SASL PLAIN with automatically generated credentials
* **Data Persistence:** Topic data stored indefinitely (no time or size limit)
* **Performance:** Optimized settings for reliability and throughput

### Resource Allocation

Zerops automatically allocates resources to your Kafka service based on demand:

* **Memory:** Up to 40GB RAM for high-performance message processing
* **Storage:** Up to 250GB for persistent storage of messages and logs
* **Auto-scaling:** Resources scale up and down automatically based on workload

## Deployment Modes

:::important
Deployment mode is selected during service creation and cannot be changed later.
:::

### High-Availability (HA) Setup

The recommended solution for production workloads and mission-critical data:

* Creates a multi-node Kafka cluster with 3 broker nodes
* Configures 6 partitions across the cluster
* Implements replication factor of 3 (each broker node has a copy of each partition)
* Default topic replication is also 3 (overridable by user application)
* Zerops automatically attempts to repair the cluster and data replication in case of a node failure

### Single Node Instance

Suitable for development and testing environments:

* Consists of 1 broker node
* Configures 3 partitions
* No data replication
* Lower resource requirements

:::note
Use for development or non-critical data only, as data loss may occur due to container volatility.
:::

## Authentication Management

Authentication credentials are automatically generated and managed by the platform using SASL PLAIN authentication.

**Access your credentials through:**
* The service access details in the Zerops GUI
* Environment variables in your service configuration:
* `user` - Username for authentication
* `password` - Generated secure password
* `port` - Kafka port (value: `9092`)

## Client Access

Client implementations differ, please refer to your chosen client's configuration manual for specific details.

### Access Methods

#### Seed Broker Connection
Connect to the Kafka cluster using the "seed" (or "bootstrap") broker server:
```
<hostname>:9092
```

#### Specific Broker Access
To access a single specific broker or a list of all/some brokers:
```
node-stable-1.db.<hostname>.zerops:9092,node-stable-2.db.<hostname>.zerops:9092,...
```

## Best Practices

### Production Workloads
* Use HA mode for all production deployments
* Configure proper retention policies for your topics based on your data requirements
* Monitor consumer lag to ensure messages are being processed efficiently
* Use consumer groups to distribute processing load

### Development Environments
* Single node instances are suitable for development and testing
* Be aware of potential data loss in non-HA deployments
* Consider using smaller message sizes during development to reduce resource usage

## Support

For advanced configurations or custom requirements:
* Join our [Discord community](https://discord.gg/zerops)
* Contact support via [email](mailto:support@zerops.io)
103 changes: 29 additions & 74 deletions apps/docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,15 @@ module.exports = {
},
className: "homepage-sidebar-item service-sidebar-item",
},
{
type: "ref",
id: "kafka/overview",
label: "Kafka",
customProps: {
sidebar_icon: "kafka",
},
className: "homepage-sidebar-item service-sidebar-item",
},
{
type: 'ref',
id: 'keydb/overview',
Expand All @@ -425,15 +434,6 @@ module.exports = {
},
className: 'homepage-sidebar-item service-sidebar-item',
},
// {
// type: "ref",
// id: "kafka/overview",
// label: "Kafka",
// customProps: {
// sidebar_icon: "kafka",
// },
// className: "homepage-sidebar-item service-sidebar-item",
// },
],
},
{
Expand Down Expand Up @@ -2294,71 +2294,26 @@ module.exports = {
},
},
],
// rabbitmq: [
// {
// type: "ref",
// id: "homepage",
// label: "Back to home",
// customProps: {
// sidebar_is_back_link: true,
// sidebar_icon: "back-arrow",
// },
// },
// {
// type: "doc",
// id: "rabbitmq/overview",
// label: "Zerops RabbitMQ Service",
// customProps: {
// sidebar_is_title: true,
// sidebar_icon: "rabbitmq",
// },
// },
// {
// type: "category",
// label: "How-to",
// collapsible: false,
// customProps: {
// sidebar_is_group_headline: true,
// },
// items: [
// {
// type: "doc",
// id: "rabbitmq/how-to/create",
// label: "Create RabbitMQ service",
// },
// {
// type: "doc",
// id: "rabbitmq/how-to/connect",
// label: "Connect to RabbitMQ",
// },
// {
// type: "doc",
// id: "rabbitmq/how-to/manage",
// label: "Manage users and databases",
// },
// {
// type: "doc",
// id: "rabbitmq/how-to/export-import-data",
// label: "Export and import data",
// },
// {
// type: "doc",
// id: "rabbitmq/how-to/scale",
// label: "Scale RabbitMQ service",
// },
// {
// type: "doc",
// id: "rabbitmq/how-to/control",
// label: "Stop and start RabbitMQ service",
// },
// {
// type: "doc",
// id: "rabbitmq/how-to/delete",
// label: "Delete RabbitMQ service",
// },
// ],
// },
// ],
kafka: [
{
type: 'ref',
id: 'homepage',
label: 'Back to home',
customProps: {
sidebar_is_back_link: true,
sidebar_icon: 'back-arrow',
},
},
{
type: 'doc',
id: 'kafka/overview',
label: 'Zerops Kafka Service',
customProps: {
sidebar_is_title: true,
sidebar_icon: 'kafka',
},
},
],
sharedstorage: [
{
type: 'ref',
Expand Down
78 changes: 77 additions & 1 deletion apps/docs/static/llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14185,8 +14185,8 @@ export const databases = [
{ name: "Meilisearch", link: "/meilisearch/overview", icon: },
{ name: "Qdrant", link: "/qdrant/overview", icon: },
{ name: "NATS", link: "/nats/overview", icon: },
{ name: "Kafka", link: "/kafka/overview", icon: },
{ name: "KeyDB", link: "/keydb/overview", icon: },
{ name: "Kafka", icon: },
]
export const storages = [
{ name: "Object storage", link: "/object-storage/overview", icon: },
Expand Down Expand Up @@ -15950,6 +15950,82 @@ Have you build something that others might find useful? Don't hesitate to share

----------------------------------------

# Kafka > Overview

Zerops provides a fully managed [Apache Kafka](https://kafka.apache.org/) messaging platform with automated scaling and zero infrastructure overhead, letting developers focus entirely on development.
## Supported Versions
Currently supported Kafka version:
Import configuration version:
## Service Configuration
Our Kafka implementation features optimized default settings designed for common use cases.
### Key Configuration
* **Client Connections:** Data brokers available on port `9092`
* **Authentication:** Secure SASL PLAIN with automatically generated credentials
* **Data Persistence:** Topic data stored indefinitely (no time or size limit)
* **Performance:** Optimized settings for reliability and throughput
### Resource Allocation
Zerops automatically allocates resources to your Kafka service based on demand:
* **Memory:** Up to 40GB RAM for high-performance message processing
* **Storage:** Up to 250GB for persistent storage of messages and logs
* **Auto-scaling:** Resources scale up and down automatically based on workload
## Deployment Modes
:::important
Deployment mode is selected during service creation and cannot be changed later.
:::
### High-Availability (HA) Setup
The recommended solution for production workloads and mission-critical data:
* Creates a multi-node Kafka cluster with 3 broker nodes
* Configures 6 partitions across the cluster
* Implements replication factor of 3 (each broker node has a copy of each partition)
* Default topic replication is also 3 (overridable by user application)
* Zerops automatically attempts to repair the cluster and data replication in case of a node failure
### Single Node Instance
Suitable for development and testing environments:
* Consists of 1 broker node
* Configures 3 partitions
* No data replication
* Lower resource requirements
:::note
Use for development or non-critical data only, as data loss may occur due to container volatility.
:::
## Authentication Management
Authentication credentials are automatically generated and managed by the platform using SASL PLAIN authentication.
**Access your credentials through:**
* The service access details in the Zerops GUI
* Environment variables in your service configuration:
* `user` - Username for authentication
* `password` - Generated secure password
* `port` - Kafka port (value: `9092`)
## Client Access
Client implementations differ, please refer to your chosen client's configuration manual for specific details.
### Access Methods
#### Seed Broker Connection
Connect to the Kafka cluster using the "seed" (or "bootstrap") broker server:
```
:9092
```
#### Specific Broker Access
To access a single specific broker or a list of all/some brokers:
```
node-stable-1.db..zerops:9092,node-stable-2.db..zerops:9092,...
```
## Best Practices
### Production Workloads
* Use HA mode for all production deployments
* Configure proper retention policies for your topics based on your data requirements
* Monitor consumer lag to ensure messages are being processed efficiently
* Use consumer groups to distribute processing load
### Development Environments
* Single node instances are suitable for development and testing
* Be aware of potential data loss in non-HA deployments
* Consider using smaller message sizes during development to reduce resource usage
## Support
For advanced configurations or custom requirements:
* Join our [Discord community](https://discord.gg/zerops)
* Contact support via [email](mailto:support@zerops.io)

----------------------------------------

# Keydb > Getting Started

This quick start allows you to get hands-on experience of Zerops by running a predefined application consisting of a KeyDB service and a runtime service that handles the SQL queries.
Expand Down
Loading