Skip to content

Commit

Permalink
Refer back to pipelines in docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Jeffail committed Apr 13, 2018
1 parent a8fa1cd commit beb721a
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 20 deletions.
13 changes: 2 additions & 11 deletions resources/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -429,17 +429,8 @@ use. If the number of inputs is less than or close to the number of processing
threads then it is also important to use a [buffer][buffers] in order to
decouple those inputs.

Here are some examples to illustrate:

``` yaml
TODO: config
```

Is a config that creates pipelines like this:

```
TODO: diagram
```
Please refer [to the documentation regarding pipelines][pipeline] for some
examples.

[default-conf]: ../../config/everything.yaml
[pipeline]: ./pipeline.md
Expand Down
18 changes: 9 additions & 9 deletions resources/docs/pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ output.
If you have processors that are heavy on CPU and aren't specific to a certain
input or output they are best suited for the pipeline section. It is
advantageous to use the pipeline section as it allows you to set an explicit
number of parallel threads of execution which, should ideally match the number
of available logical CPU cores.
number of parallel threads of execution which should ideally match the number of
available logical CPU cores.

If [a buffer is chosen][buffers] these processors are applied to messages read
from it. It is therefore possible to use buffers to distribute messages from a
single input across multiple parallel processing threads.
from it. It is therefore possible to use buffers as a way of distributing
messages from a single input across multiple parallel processing threads.

The following are some examples of how to get good performance out of your
processing pipelines.
Expand Down Expand Up @@ -59,7 +59,8 @@ output:
type: bar
```
With this config the block diagram of our Benthos instance might look like this:
With this config the pipeline within our Benthos instance would look something
like the following:
```
foo -> memory buffer ---> processor ---> bar
Expand Down Expand Up @@ -95,9 +96,7 @@ input:
inputs:
- type: baz
buffer:
type: memory
memory:
limit: 5000000
type: none
pipeline:
threads: 4
processors:
Expand All @@ -109,7 +108,8 @@ output:
type: bar
```
With this config the block diagram of our Benthos instance might look like this:
With this config the pipeline within our Benthos instance would look something
like the following:
```
baz -\
Expand Down

0 comments on commit beb721a

Please sign in to comment.