Skip to content

Commit 43b2144

Browse files
damemidamemi
committed
Add GCP collector config, readme, samples
1 parent 4c13e5a commit 43b2144

9 files changed

+237
-4
lines changed

‎.licenserc.json

+3-2
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@
4141
"src/featureflagservice/assets/vendor/",
4242
"src/featureflagservice/priv/",
4343
"src/productcatalogservice/genproto/",
44-
"internal/tools/"
44+
"internal/tools/",
45+
"gcp-config-values.yml"
4546
]
46-
}
47+
}

‎CONTRIBUTING.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Welcome to OpenTelemetry Demo Webstore repository Google Cloud integration!
44

55
This is the fork for the Google Cloud integration. To contribute to the demo
66
itself, see
7-
https://github.com/open-telemetry/opentelemetry-demo/blob/main/CONTRIBUTING.md.
7+
[the OpenTelemetry `CONTRIBUTING.md`](https://github.com/open-telemetry/opentelemetry-demo/blob/main/CONTRIBUTING.md).
88

99
If you would like to contribute to the Google Cloud integration, see
1010
instructions below.
@@ -184,3 +184,4 @@ on each other), the owner should try to get people aligned by:
184184
the owner should bring it to the OpenTelemetry Community Demo SIG
185185
[meeting](README.md#contributing).
186186

187+
[docs]: https://opentelemetry.io/docs/demo/

‎README.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@
99

1010
## Welcome to the OpenTelemetry Astronomy Shop Demo with Google Cloud
1111

12-
This repository containes a fork of https://github.com/open-telemetry/opentelemetry-demo,
12+
This repository contains a fork of [open-telemetry/opentelemetry-demo](https://github.com/open-telemetry/opentelemetry-demo),
1313
which integrates with Google Cloud Monitoring, Logging, and Trace.
1414

1515
**This is not an officially supported Google product.**
1616

17+
For details on how to use this with GCP, see [`README_GCP.md`](README_GCP.md).
18+
1719
This repository contains the OpenTelemetry Astronomy Shop, a microservice-based
1820
distributed system intended to illustrate the implementation of OpenTelemetry in
1921
a near real-world environment.

‎README_GCP.md

+87
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Running the demo on Google Cloud
2+
3+
The demo can send logs, traces, and metrics to Google Cloud. The easiest way to
4+
do this is with the [`gcp-config-values.yml`](gcp-config-values.yml) file (when
5+
deploying via Helm on GKE) or
6+
[`src/otelcollector/otelcol-config-extras.yml`](src/otelcollector/otelcol-config-extras.yml)
7+
when running with `docker-compose` on GCE.
8+
9+
## Running on GKE
10+
11+
The recommended way to run the demo on GKE is with the official Helm chart. If
12+
running on a GKE Autopilot cluster (or any cluster with Workload Identity), you
13+
must follow the prerequisite steps to set up a Workload Identity-enabled service
14+
account below. Otherwise, you can skip to the next section.
15+
16+
### Workload Identity prequisites
17+
18+
Follow the [Workload Identity
19+
docs](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to)
20+
to set up an IAM service account in your GCP project with permission to use
21+
Workload Identity and write logs, traces, and metrics:
22+
23+
```console
24+
export GCLOUD_PROJECT=<your project id>
25+
```
26+
27+
```console
28+
gcloud iam service-accounts create opentelemetry-demo \
29+
--project=${GCLOUD_PROJECT}
30+
gcloud projects add-iam-policy-binding ${GCLOUD_PROJECT} \
31+
--member "serviceAccount:opentelemetry-demo@${GCLOUD_PROJECT}.iam.gserviceaccount.com" \
32+
--role "roles/logging.logWriter"
33+
gcloud projects add-iam-policy-binding ${GCLOUD_PROJECT} \
34+
--member "serviceAccount:opentelemetry-demo@${GCLOUD_PROJECT}.iam.gserviceaccount.com" \
35+
--role "roles/monitoring.metricWriter"
36+
gcloud projects add-iam-policy-binding ${GCLOUD_PROJECT} \
37+
--member "serviceAccount:opentelemetry-demo@${GCLOUD_PROJECT}.iam.gserviceaccount.com" \
38+
--role "roles/cloudtrace.agent"
39+
gcloud iam service-accounts add-iam-policy-binding opentelemetry-demo@${GCLOUD_PROJECT}.iam.gserviceaccount.com \
40+
--role roles/iam.workloadIdentityUser \
41+
--member "serviceAccount:${GCLOUD_PROJECT}.svc.id.goog[default/opentelemetry-demo]"
42+
```
43+
44+
Update [`gcp-config-valus.yml`](gcp-config-values.yml) to annotate the
45+
Kubernetes service account with your project:
46+
47+
```console
48+
sed -i "s/%GCLOUD_PROJECT%/${GCLOUD_PROJECT}/g" gcp-config-values.yml
49+
```
50+
51+
Next, when you deploy the Helm chart, the
52+
[`gcp-config-values.yml`](gcp-config-values.yml) file will create the Kubernetes
53+
service account and annotate it to use Workload Identity.
54+
55+
### Deploying the Helm chart
56+
57+
Follow the [OpenTelemetry docs to run the Helm chart](https://opentelemetry.io/docs/demo/kubernetes-deployment):
58+
59+
```console
60+
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
61+
helm repo update
62+
helm install my-otel-demo open-telemetry/opentelemetry-demo --values gcp-config-values.yml
63+
```
64+
65+
## Running on GCE
66+
67+
Follow the [OpenTelemetry docs to run with Docker](https://opentelemetry.io/docs/demo/docker-deployment/):
68+
69+
```console
70+
make start
71+
```
72+
73+
## Seeing telemetry
74+
75+
With the demo running, you should see telemetry automatically created by the
76+
demo's load generator. You can see metrics under "Prometheus Target" in Cloud
77+
Monitoring:
78+
79+
![metrics](gcp_metrics.png)
80+
81+
Traces in the Trace explorer:
82+
83+
![traces](gcp_traces.png)
84+
85+
And logs in the Logs explorer organized by service:
86+
87+
![logs](gcp_logs.png)

‎gcp-config-values.yml

+83
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Copyright 2023 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# this file is used by Helm chart launch steps.
16+
# when merging YAML with the default values, objects are merged but lists are overwritten.
17+
# so, where a list is used, we have copied the defaults from the upstream values to preserve them.
18+
19+
opentelemetry-collector:
20+
# Service account for workload identity
21+
serviceAccount:
22+
create: true
23+
annotations: {"iam.gke.io/gcp-service-account":"opentelemetry-demo@%GCLOUD_PROJECT%.iam.gserviceaccount.com"}
24+
name: "opentelemetry-demo"
25+
26+
config:
27+
processors:
28+
memory_limiter:
29+
check_interval: 1s
30+
limit_percentage: 65
31+
spike_limit_percentage: 20
32+
33+
batch:
34+
send_batch_max_size: 200
35+
send_batch_size: 200
36+
timeout: 5s
37+
38+
resourcedetection:
39+
detectors: [gcp]
40+
timeout: 10s
41+
42+
transform/collision:
43+
metric_statements:
44+
- context: datapoint
45+
statements:
46+
- set(attributes["exported_location"], attributes["location"])
47+
- delete_key(attributes, "location")
48+
- set(attributes["exported_cluster"], attributes["cluster"])
49+
- delete_key(attributes, "cluster")
50+
- set(attributes["exported_namespace"], attributes["namespace"])
51+
- delete_key(attributes, "namespace")
52+
- set(attributes["exported_job"], attributes["job"])
53+
- delete_key(attributes, "job")
54+
- set(attributes["exported_instance"], attributes["instance"])
55+
- delete_key(attributes, "instance")
56+
- set(attributes["exported_project_id"], attributes["project_id"])
57+
- delete_key(attributes, "project_id")
58+
59+
# See https://github.com/open-telemetry/opentelemetry-demo/issues/1330
60+
filter/currency:
61+
metrics:
62+
metric:
63+
- 'IsMatch(name, "(.*)app_currency(.*)")'
64+
65+
exporters:
66+
googlecloud:
67+
log:
68+
default_log_name: opentelemetry.io/opentelemetry-demo
69+
70+
googlemanagedprometheus:
71+
72+
service:
73+
pipelines:
74+
logs:
75+
processors: [memory_limiter, resourcedetection, resource, batch]
76+
exporters: [googlecloud]
77+
traces:
78+
processors: [memory_limiter, resourcedetection, resource, batch]
79+
exporters: [googlecloud] #spanmetrics disabled
80+
metrics:
81+
receivers: [otlp] #spanmetrics disabled
82+
processors: [filter/ottl, filter/currency, memory_limiter, resourcedetection, transform, transform/collision, resource, batch]
83+
exporters: [googlemanagedprometheus]

‎gcp_logs.png

755 KB
Loading

‎gcp_metrics.png

344 KB
Loading

‎gcp_traces.png

481 KB
Loading

‎src/otelcollector/otelcol-config-extras.yml

+59
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@
44
# extra settings to be merged into OpenTelemetry Collector configuration
55
# do not delete this file
66

7+
# this file is used by docker-compose launch steps. it is merged with the default Collector
8+
# config in otelcol-config.yaml.
9+
710
## Example configuration for sending data to your own OTLP HTTP backend
811
## Note: the spanmetrics exporter must be included in the exporters array
912
## if overriding the traces pipeline.
@@ -16,3 +19,59 @@
1619
# pipelines:
1720
# traces:
1821
# exporters: [spanmetrics, otlphttp/example]
22+
23+
processors:
24+
memory_limiter:
25+
check_interval: 1s
26+
limit_percentage: 65
27+
spike_limit_percentage: 20
28+
29+
batch:
30+
send_batch_max_size: 200
31+
send_batch_size: 200
32+
timeout: 5s
33+
34+
resourcedetection:
35+
detectors: [gcp]
36+
timeout: 10s
37+
38+
transform/collision:
39+
metric_statements:
40+
- context: datapoint
41+
statements:
42+
- set(attributes["exported_location"], attributes["location"])
43+
- delete_key(attributes, "location")
44+
- set(attributes["exported_cluster"], attributes["cluster"])
45+
- delete_key(attributes, "cluster")
46+
- set(attributes["exported_namespace"], attributes["namespace"])
47+
- delete_key(attributes, "namespace")
48+
- set(attributes["exported_job"], attributes["job"])
49+
- delete_key(attributes, "job")
50+
- set(attributes["exported_instance"], attributes["instance"])
51+
- delete_key(attributes, "instance")
52+
- set(attributes["exported_project_id"], attributes["project_id"])
53+
- delete_key(attributes, "project_id")
54+
55+
# See https://github.com/open-telemetry/opentelemetry-demo/issues/1330
56+
filter/currency:
57+
metrics:
58+
metric:
59+
- 'IsMatch(name, "(.*)app_currency(.*)")'
60+
61+
exporters:
62+
googlecloud:
63+
64+
googlemanagedprometheus:
65+
66+
service:
67+
pipelines:
68+
logs:
69+
processors: [memory_limiter, resourcedetection, resource, batch]
70+
exporters: [googlecloud]
71+
traces:
72+
processors: [memory_limiter, resourcedetection, resource, batch]
73+
exporters: [googlecloud] #spanmetrics disabled
74+
metrics:
75+
receivers: [otlp] #spanmetrics disabled
76+
processors: [filter/ottl, filter/currency, memory_limiter, resourcedetection, transform, transform/collision, resource, batch]
77+
exporters: [googlemanagedprometheus]

0 commit comments

Comments
 (0)