Update changelog

CHANGELOG.md

@@ -5,9 +5,11 @@ All notable changes to this project will be documented in this file.
 
 ## Unreleased
 
+## 0.14.5 - 2018-06-21
+
 ### Added
 
-- New `regexp_exact` and `regexp_partial` conditions.
+- New `regexp_exact` and `regexp_partial` content condition operators.
 
 ## 0.14.4 - 2018-06-19
 

README.md

@@ -169,7 +169,7 @@ go install -tags "ZMQ4" ./cmd/...
 [config-interp]: docs/config_interpolation.md
 [compose-examples]: resources/docker/compose_examples
 [streams-api]: docs/api/streams.md
-[streams-mode]: docs/streams_mode.md
+[streams-mode]: docs/streams/README.md
 [general-docs]: docs/README.md
 [cookbook-docs]: docs/cookbook/README.md
 [env-config]: config/env/default.yaml

docs/api/streams.md

@@ -8,6 +8,8 @@ stream.
 Each stream has its own input, buffer, pipeline and output sections which
 contains an isolated stream of data with its own lifetime.
 
+A walkthrough on using this API [can be found here][streams-api-walkthrough].
+
 ## API
 
 ### GET `/streams`
@@ -91,3 +93,5 @@ Attempt to shut down and remove a stream identified by `id`.
 #### Response 200
 
 The stream was found, shut down and removed successfully.
+
+[streams-api-walkthrough]: ../streams/using_REST_API.md

docs/streams/README.md

@@ -0,0 +1,23 @@
+Streams Mode
+============
+
+A Benthos stream consists of four components; an input, an optional buffer,
+processor pipelines and an output. Under normal use a Benthos instance is a
+single stream, and these components are configured within the service config
+file.
+
+Alternatively, Benthos can be run in `--streams` mode, where a single running
+Benthos instance is able to run multiple entirely isolated streams. Adding
+streams in this mode can be done in two ways:
+
+1. [Static configuration files][static-files] allows you to maintain a directory
+   of static stream configuration files that will be traversed by Benthos.
+
+2. An [HTTP REST API][rest-api] allows you to dynamically create, read the
+   status of, update, and delete streams at runtime.
+
+These two methods can be use interchangably, i.e. it's possible to update and
+delete streams that were created with static files.
+
+[static-files]: using_config_files.md
+[rest-api]: using_REST_API.md

docs/streams/using_REST_API.md

@@ -0,0 +1,241 @@
+Streams Via REST API
+====================
+
+By using the Benthos `--streams` mode REST API you can dynamically control which
+streams are active at runtime. The full spec for the Benthos streams mode REST
+API can be [found here][http-interface].
+
+Note that stream configs created and updated using this API do *not* benefit
+from [environment variable interpolation][interpolation] (function interpolation
+will still work).
+
+## Walkthrough
+
+Start by running Benthos in streams mode:
+
+``` bash
+$ benthos --streams
+```
+
+On a separate terminal we can add our first stream `foo` by `POST`ing a JSON or
+YAML config to the `/streams/foo` endpoint:
+
+``` bash
+$ curl http://localhost:4195/streams/foo -X POST --data-binary @- <<EOF
+input:
+  type: http_server
+buffer:
+  type: memory
+pipeline:
+  threads: 4
+  processors:
+  - type: jmespath
+    jmespath:
+      query: "{id: user.id, content: body.content}"
+output:
+  type: http_server
+EOF
+```
+
+Now we can check the full set of streams loaded by `GET`ing the `/streams`
+endpoint:
+
+``` bash
+$ curl http://localhost:4195/streams | jq '.'
+{
+  "foo": {
+    "active": true,
+    "uptime": 7.223545951,
+    "uptime_str": "7.223545951s"
+  }
+}
+```
+
+Good, now let's add another stream `bar` the same way:
+
+``` bash
+$ curl http://localhost:4195/streams/bar -X POST --data-binary @- <<EOF
+input:
+  type: kafka
+  kafka:
+    addresses:
+    - localhost:9092
+    topic: my_topic
+buffer:
+  type: none
+pipeline:
+  threads: 1
+  processors:
+  - type: sample
+    sample:
+      retain: 10
+output:
+  type: elasticsearch
+  elasticsearch:
+    urls:
+    - http://localhost:9200
+EOF
+```
+
+And check the set again:
+
+``` bash
+$ curl http://localhost:4195/streams | jq '.'
+{
+  "bar": {
+    "active": true,
+    "uptime": 10.121344484,
+    "uptime_str": "10.121344484s"
+  },
+  "foo": {
+    "active": true,
+    "uptime": 19.380582951,
+    "uptime_str": "19.380583306s"
+  }
+}
+```
+
+It's also possible to get the configuration of a loaded stream by `GET`ing the
+path `/streams/{id}`:
+
+``` bash
+$ curl http://localhost:4195/streams/foo | jq '.'
+{
+  "active": true,
+  "uptime": 30.123488951,
+  "uptime_str": "30.123488951s"
+  "config": {
+    "input": {
+      "type": "http_server",
+      "http_server": {
+        "address": "",
+        "cert_file": "",
+        "key_file": "",
+        "path": "/post",
+        "timeout_ms": 5000
+      }
+    },
+	"buffer": {
+      "type": "memory",
+      "memory": {
+        "limit": 10000000
+      }
+    },
+	... etc ...
+  }
+}
+```
+
+Next, we might want to update stream `foo` by `PUT`ing a new config to the path
+`/streams/foo`:
+
+``` bash
+$ curl http://localhost:4195/streams/foo -X PUT --data-binary @- <<EOF
+input:
+  type: http_server
+buffer:
+  type: none
+pipeline:
+  threads: 4
+  processors:
+  - type: jmespath
+    jmespath:
+      query: "{id: user.id, content: body.content}"
+output:
+  type: http_server
+EOF
+```
+
+We have removed the memory buffer with this change, let's check that the config
+has actually been updated:
+
+``` bash
+$ curl http://localhost:4195/streams/foo | jq '.'
+{
+  "active": true,
+  "uptime": 12.328482951,
+  "uptime_str": "12.328482951s"
+  "config": {
+    "input": {
+      "type": "http_server",
+      "http_server": {
+        "address": "",
+        "cert_file": "",
+        "key_file": "",
+        "path": "/post",
+        "timeout_ms": 5000
+      }
+    },
+	"buffer": {
+      "type": "none"
+    },
+	... etc ...
+  }
+}
+```
+
+Good, we are done with stream `bar` now, so let's delete it by `DELETE`ing the
+`/streams/bar` endpoint:
+
+``` bash
+$ curl http://localhost:4195/streams/bar -X DELETE
+```
+
+And let's `GET` the `/streams` endpoint to see the new set:
+
+``` bash
+$ curl http://localhost:4195/streams | jq '.'
+{
+  "foo": {
+    "active": true,
+    "uptime": 31.872448851,
+    "uptime_str": "31.872448851s"
+  }
+}
+```
+
+Great. Another useful feature is `POST`ing to `/streams`, this allows us to set
+the entire set of streams with a single request.
+
+The payload is a map of stream ids to configurations and this will become the
+exclusive set of active streams. If there are existing streams that are not on
+the list they will be removed.
+
+``` bash
+$ curl http://localhost:4195/streams -X POST --data-binary @- <<EOF
+bar:
+  input:
+    type: http_client
+	http_client:
+	  url: http://localhost:4195/baz/get
+  output:
+    type: stdout
+baz:
+  input:
+    type: http_server
+  output:
+    type: http_server
+EOF
+```
+
+Let's check our new set of streams:
+
+``` bash
+$ curl http://localhost:4195/streams | jq '.'
+{
+  "bar": {
+    "active": true,
+    "uptime": 3.183883444,
+    "uptime_str": "3.183883444s"
+  },
+  "baz": {
+    "active": true,
+    "uptime": 3.183883449,
+    "uptime_str": "3.183883449s"
+  }
+}
+```
+
+Done.
+
+[http-interface]: ../api/streams.md

docs/streams_mode.md → docs/streams/using_config_files.md

@@ -1,24 +1,15 @@
-Streams Mode
-============
+Streams Via Config Files
+========================
 
-A Benthos stream consists of four components; an input, an optional buffer,
-processor pipelines and an output. Under normal use a Benthos instance is a
-single stream, and the components listed are configured within the service
-config file.
+When running Benthos in `--streams` mode it's possible to create streams with
+their own static configurations by setting the `--streams-dir` flag to a
+directory containing a config file for each stream (`/benthos/streams` by
+default).
 
-Alternatively, Benthos can be run in `--streams` mode, where a single running
-Benthos instance is able to run multiple entirely isolated streams. Streams can
-be configured by setting the `--streams-dir` flag to a directory containing a
-config file for each stream (`/benthos/streams` by default).
+Note that stream configs loaded in this way can benefit from
+[interpolation][interpolation].
 
-During runtime streams can also be added, updated, removed and monitored using
-[a REST HTTP interface][http-interface].
-
-Note that stream configs loaded from `--streams-dir` can benefit from
-[interpolation][interpolation] but configs loaded through the HTTP REST
-endpoints cannot.
-
-## Using directory of stream configs
+## Walkthrough
 
 Make a directory of stream configs:
 
@@ -87,7 +78,7 @@ $ curl http://localhost:4195/streams | jq '.'
 }
 ```
 
-And you can query the loaded configuration of a stream:
+You can also query a specific stream to see the loaded configuration:
 
 ``` bash
 $ curl http://localhost:4195/streams/foo | jq '.'
@@ -104,23 +95,12 @@ $ curl http://localhost:4195/streams/foo | jq '.'
         "key_file": "",
         "path": "/post",
         "timeout_ms": 5000
-      },
-      "processors": [
-        {
-          "type": "bounds_check",
-          "bounds_check": {
-            "max_part_size": 1073741824,
-            "max_parts": 100,
-            "min_part_size": 1,
-            "min_parts": 1
-          }
-        }
-      ]
+      }
     },
     "buffer": {
       "type": "memory",
       "memory": {
-        "limit": 0
+        "limit": 10000000
       }
     },
     "pipeline": {
@@ -150,8 +130,8 @@ $ curl http://localhost:4195/streams/foo | jq '.'
 }
 ```
 
-There are other endpoints [in the REST API][http-interface] for creating,
-updating and deleting streams.
+There are other endpoints [in the REST API][rest-api] for creating, updating and
+deleting streams.
 
-[http-interface]: api/streams.md
-[interpolation]: config_interpolation.md
+[rest-api]: using_REST_API.md
+[interpolation]: ../config_interpolation.md

resources/docker/streams/README.md

@@ -24,5 +24,5 @@ Send data to Amazon S3:
 curl http://localhost:4195/s3/send -d '{"foo":"hello world"}'
 ```
 
-[streams]: ../../../docs/streams_mode.md
+[streams]: ../../../docs/streams/README.md
 [http-streams]: ../../../docs/api/streams.md