Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanity check for Cygnus returns 502 Bad Gateway #359

Open
dkastl opened this issue May 21, 2024 · 7 comments
Open

Sanity check for Cygnus returns 502 Bad Gateway #359

dkastl opened this issue May 21, 2024 · 7 comments
Labels
bug Something isn't working

Comments

@dkastl
Copy link
Contributor

dkastl commented May 21, 2024

Checklist
Put an x in the boxes that apply

  • I have run the latest release.
  • I have inspeced the status of docker containers.
  • I have run the sanity check described in after-installation.md.
  • I have inspected all error messages.
  • I have inspected the log files in the /var/log/fiware directory.
  • I have run the make collect command.
  • I have provided a Github Star to this repository.

Not sure I inspected all errors and logs, but unfortunately I already deleted my previous setup.

Describe the bug

When I run the sanity check for Cygnus I get a "502 Bad Gateway" response like

sudo ngsi version --host cygnus.mydomain.com --pretty
cbVersion002 error 502 Bad Gateway <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

This happens, when only 1 sink is set to true. When 2 or more sinks are enabled, it seems to be working.

Eventually there is a problem when CYGNUS_MULTIAGENT is not set true, see:
https://github.com/lets-fiware/FIWARE-Big-Bang/blob/main/lets-fiware.sh#L2248-L2250

Your system environment
Run make collect in a directory where you ran the lets-fiware.sh script and paste the results here.

Date: Tue May 21 15:54:24 UTC 2024
Version: 0.37.0
OS: Ubuntu 2404
Docker: Docker version 26.1.3, build b72abbb
Docker compose: Docker Compose version v2.27.0
uid=0(root) gid=0(root) groups=0(root)
Hash: 53423c9f29f4242ffa8131ec9a722288259b01e3787850763a26cb89e4cebdb2  lets-fiware.sh
git-hash: 
a849b38e7f164b8556e21721e175ce928b06f0a3
b4cf3f3e124b0d64d6aebd0853f30575962d2905
04c575519e1f47aa1491403b1ca927883ab2aa4d
App list: KEYROCK ORION CYGNUS COMET NODE_RED GRAFANA QUANTUMLEAP IOTAGENT_UL IOTAGENT_JSON IOTAGENT_HTTP MOSQUITTO 
Install: completed
WILMA_AUTH_ENABLED: true
Docker containers: 
make[1]: Entering directory '/srv/fiware-big-bang'
sudo ./setup/ps.sh
WARN[0000] /srv/fiware-big-bang/docker-compose.yml: `version` is obsolete 
NAME                                    IMAGE                                   COMMAND                  SERVICE               CREATED          STATUS                      PORTS
fiware-big-bang-comet-1                 telefonicaiot/fiware-sth-comet:2.10.0   "docker/entrypoint.s…"   comet                 12 minutes ago   Up 12 minutes (healthy)     8666/tcp
fiware-big-bang-comet-wilma-1           letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   comet-wilma           12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-crate-1                 crate:4.6.6                             "/docker-entrypoint.…"   crate                 12 minutes ago   Up 12 minutes               4300/tcp, 127.0.0.1:4200->4200/tcp, 5432/tcp
fiware-big-bang-cygnus-1                telefonicaiot/fiware-cygnus:3.2.0       "/cygnus-entrypoint.…"   cygnus                12 minutes ago   Up 12 minutes (healthy)     5050/tcp, 5080/tcp
fiware-big-bang-cygnus-wilma-1          letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   cygnus-wilma          12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-grafana-1               grafana/grafana:6.1.6                   "/run.sh"                grafana               12 minutes ago   Up 12 minutes               3000/tcp
fiware-big-bang-iotagent-http-wilma-1   letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   iotagent-http-wilma   12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-iotagent-json-1         telefonicaiot/iotagent-json:2.4.2       "docker-entrypoint.s…"   iotagent-json         12 minutes ago   Up 12 minutes (healthy)     4041/tcp, 7896/tcp
fiware-big-bang-iotagent-json-wilma-1   letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   iotagent-json-wilma   12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-iotagent-ul-1           telefonicaiot/iotagent-ul:2.4.2         "docker-entrypoint.s…"   iotagent-ul           12 minutes ago   Up 12 minutes (healthy)     4061/tcp, 7896/tcp
fiware-big-bang-iotagent-ul-wilma-1     letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   iotagent-ul-wilma     12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-keyrock-1               letsfiware/fiware-idm:8.4.0             "docker-entrypoint.s…"   keyrock               12 minutes ago   Up 12 minutes (healthy)     3000/tcp
fiware-big-bang-mongo-1                 mongo:4.4                               "docker-entrypoint.s…"   mongo                 12 minutes ago   Up 12 minutes               27017/tcp
fiware-big-bang-mosquitto-1             eclipse-mosquitto:1.6                   "/docker-entrypoint.…"   mosquitto             12 minutes ago   Up 12 minutes               1883/tcp
fiware-big-bang-mysql-1                 mysql:8.1                               "docker-entrypoint.s…"   mysql                 12 minutes ago   Up 12 minutes               3306/tcp, 33060/tcp
fiware-big-bang-nginx-1                 nginx:1.25                              "/docker-entrypoint.…"   nginx                 12 minutes ago   Up 12 minutes               0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:8883->8883/tcp, :::8883->8883/tcp
fiware-big-bang-node-red001-1           letsfiware/node-red:0.37.0              "./entrypoint.sh"        node-red001           12 minutes ago   Up 12 minutes (healthy)     1880/tcp
fiware-big-bang-node-red002-1           letsfiware/node-red:0.37.0              "./entrypoint.sh"        node-red002           12 minutes ago   Up 12 minutes (healthy)     1880/tcp
fiware-big-bang-node-red003-1           letsfiware/node-red:0.37.0              "./entrypoint.sh"        node-red003           12 minutes ago   Up 12 minutes (healthy)     1880/tcp
fiware-big-bang-orion-1                 telefonicaiot/fiware-orion:3.10.1       "sh -c 'rm /tmp/cont…"   orion                 12 minutes ago   Up 12 minutes (healthy)     1026/tcp
fiware-big-bang-orion-wilma-1           letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   orion-wilma           12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-perseo-core-1           telefonicaiot/perseo-core:1.13.0        "/code/perseo_core-e…"   perseo-core           12 minutes ago   Up 12 minutes (healthy)     8080/tcp
fiware-big-bang-perseo-fe-1             telefonicaiot/perseo-fe:1.27.0          "docker-entrypoint.s…"   perseo-fe             12 minutes ago   Up 12 minutes (healthy)     9090/tcp
fiware-big-bang-perseo-fe-wilma-1       letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   perseo-fe-wilma       12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-postfix-1               letsfiware/postfix:0.37.0               "docker-entrypoint.sh"   postfix               12 minutes ago   Up 12 minutes               25/tcp
fiware-big-bang-postgres-1              postgres:15                             "docker-entrypoint.s…"   postgres              12 minutes ago   Up 12 minutes               5432/tcp
fiware-big-bang-quantumleap-1           orchestracities/quantumleap:0.8.3       "python app.py"          quantumleap           12 minutes ago   Up 12 minutes (healthy)     8668/tcp
fiware-big-bang-quantumleap-wilma-1     letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   quantumleap-wilma     12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-redis-1                 redis:6                                 "docker-entrypoint.s…"   redis                 12 minutes ago   Up 12 minutes               6379/tcp
fiware-big-bang-tokenproxy-1            letsfiware/tokenproxy:0.37.0            "docker-entrypoint.sh"   tokenproxy            12 minutes ago   Up 12 minutes               1029/tcp
make[1]: Leaving directory '/srv/fiware-big-bang'
Keyrock: 
{"keyrock":{"version":"8.4.0","release_date":"2023-06-09T13:19:25.000Z","uptime":"00:11:49.9","doc":"https://keyrock-fiware.github.io/","api":{"version":"v1","link":"https://keyrock.mydomain.com/v1"}}}

To Reproduce
Steps to reproduce the behavior:

  1. Configure Cygnus with only one sink set to true. -> 502 Bad Gateway
  2. Configure Cygnus with more than one sink set to true. -> it works

Expected behavior
A clear and concise description of what you expected to happen.

As descibed in the documentation it should return something like

{
  "success": "true",
  "version": "3.2.0.9b21eae9095e3250ba8799e61411905d1c756191"
}

Server (please complete the following information):

  • OS: Ubuntu 2404
  • Docker: Docker version 26.1.3, build b72abbb
  • Docker compose: Docker Compose version v2.27.0
@fisuda
Copy link
Contributor

fisuda commented May 25, 2024

I have not been able to reproduce this error. The hash value of the letsfiware.sh file is different from the original one of 0.37.0. Please make a FIWARE instance again using the official release file.

$ cd FIWARE-Big-Bang-0.37.0/

$ cp -a config.sh{,.bak}

$ vi config.sh

$ diff config.sh config.sh.bak
97c97
< CYGNUS=cygnus
---
> CYGNUS=
100c100
< CYGNUS_MONGO=true
---
> CYGNUS_MONGO=

$ ./lets-fiware.sh letsfiware.jp

$ ngsi version --host cygnus.letsfiware.jp
{"success":"true","version":"3.2.0.9b21eae9095e3250ba8799e61411905d1c756191"}

$ make collect
./config/script/collect.sh

Sat May 25 18:16:53 JST 2024
Version: 0.37.0
OS: Ubuntu 2204
Docker: Docker version 26.1.3, build b72abbb
Docker compose: Docker Compose version v2.27.0
uid=1000(letsfiware) gid=1000(letsfiware) groups=1000(letsfiware),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),122(lpadmin),135(lxd),136(sambashare),999(docker)
Hash: af89b60a9483d6a8ac4a0a709c6638d909cf3773f0b1bb035d1548e994b4ef7a  lets-fiware.sh
App list: KEYROCK ORION CYGNUS
Install: completed
WILMA_AUTH_ENABLED: false
Docker containers:
make[1]: Entering directory '/home/letsfiware/FIWARE-Big-Bang-0.37.0'
sudo ./setup/ps.sh
WARN[0000] /home/letsfiware/FIWARE-Big-Bang-0.37.0/docker-compose.yml: `version` is obsolete
NAME                                IMAGE                               COMMAND                  SERVICE      CREATED       STATUS                   PORTS
fiware-big-bang-0370-cygnus-1       telefonicaiot/fiware-cygnus:3.2.0   "/cygnus-entrypoint.…"   cygnus       2 hours ago   Up 2 hours (healthy)     5050/tcp, 5080/tcp
fiware-big-bang-0370-keyrock-1      letsfiware/fiware-idm:8.4.0         "docker-entrypoint.s…"   keyrock      2 hours ago   Up 2 hours (healthy)     3000/tcp
fiware-big-bang-0370-mongo-1        mongo:4.4                           "docker-entrypoint.s…"   mongo        2 hours ago   Up 2 hours               27017/tcp
fiware-big-bang-0370-mysql-1        mysql:8.1                           "docker-entrypoint.s…"   mysql        2 hours ago   Up 2 hours               3306/tcp, 33060/tcp
fiware-big-bang-0370-nginx-1        nginx:1.25                          "/docker-entrypoint.…"   nginx        2 hours ago   Up 2 hours               0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp
fiware-big-bang-0370-orion-1        telefonicaiot/fiware-orion:3.10.1   "sh -c 'rm /tmp/cont…"   orion        2 hours ago   Up 2 hours (healthy)     1026/tcp
fiware-big-bang-0370-tokenproxy-1   letsfiware/tokenproxy:0.37.0        "docker-entrypoint.sh"   tokenproxy   2 hours ago   Up 2 hours               1029/tcp
fiware-big-bang-0370-wilma-1        letsfiware/fiware-pep-proxy:8.4.0   "docker-entrypoint.s…"   wilma        2 hours ago   Up 2 hours (unhealthy)   1027/tcp
make[1]: Leaving directory '/home/letsfiware/FIWARE-Big-Bang-0.37.0'
Keyrock:
{"keyrock":{"version":"8.4.0","release_date":"2023-06-09T13:19:25.000Z","uptime":"01:29:53.6","doc":"https://keyrock-fiware.github.io/","api":{"version":"v1","link":"https://keyrock.letsfiware.jp/v1"}}}

@dkastl
Copy link
Contributor Author

dkastl commented May 27, 2024

Thank you for taking a look!

I can confirm that it works with only postgres enabled in a setup, where only "orion" and "cygnus" are configured with subdomain.

Before I tried with my original setup, that has many more services enabled, and the failure happend with main, v37.0 and an updated Cygnus as I saw in your pull request.

So the problem must occur with more services enabled, and I can try to enable one by one.

Which would be the best log file to look at?
Or, is there eventually a minimum requirement for server hardware, that prevents Cygnus to start correctly? Eventually 4GB RAM is not sufficient?

@dkastl
Copy link
Contributor Author

dkastl commented May 27, 2024

Just as a reference, the config.sh I used has the following modifications:

diff --git a/config.sh b/config.sh
index 67b7d60..67e74e4 100755
--- a/config.sh
+++ b/config.sh
@@ -94,12 +94,12 @@ IMAGE_TIMESCALE=timescale/timescaledb-postgis:1.7.5-pg12
 # Cygnus
 #
 # Set a sub-domain name of Cygnus
-CYGNUS=
+CYGNUS=cygnus
 
 # Use Cygnus sink (true or false) Default: false
 CYGNUS_MONGO=
 CYGNUS_MYSQL=
-CYGNUS_POSTGRES=
+CYGNUS_POSTGRES=true
 CYGNUS_ELASTICSEARCH=
 
 # Expose port (none, local, all) Default: none
@@ -129,7 +129,7 @@ IMAGE_ELASTICSEARCH_DB=elasticsearch:7.6.2
 # Comet
 #
 # Set a sub-domain name of Comet
-COMET=
+COMET=comet
 
 # Expose port (none, local, all) Default: none
 COMET_EXPOSE_PORT=
@@ -145,7 +145,7 @@ IMAGE_COMET=telefonicaiot/fiware-sth-comet:2.10.0
 #
 # Perseo
 #
-PERSEO=
+PERSEO=perseo
 
 # Perseo Max age (default: 6000)
 PERSEO_MAX_AGE=
@@ -189,7 +189,7 @@ IMAGE_DRACO=ging/fiware-draco:2.1.0
 # Quantumleap
 #
 # Set a sub-domain name of Quantumleap
-QUANTUMLEAP=
+QUANTUMLEAP=quantumleap
 
 # Expose port (none, local, all) Default: none
 QUANTUMLEAP_EXPOSE_PORT=
@@ -236,7 +236,7 @@ IMAGE_MEMCACHED=memcached:1
 # IoT Agent over Mosquitto
 #
 # Set a sub-domain name of Mosquitto
-MOSQUITTO=
+MOSQUITTO=mosquitto
 
 # Use MQTT (Port 1883). (true or false) Default: false
 MQTT_1883=
@@ -261,7 +261,7 @@ IMAGE_MOSQUITTO=eclipse-mosquitto:1.6
 # IoT Agent over HTTP
 #
 # Set a sub-domain name to use IoT Agent over HTTP.
-IOTAGENT_HTTP=
+IOTAGENT_HTTP=iotagent-http
 
 # Authorization for IoT Agent over HTTP. (none, basic or bearer) Default: bearer
 IOTA_HTTP_AUTH=
@@ -276,7 +276,7 @@ IOTA_HTTP_BASIC_PASS=
 # IoT Agent for UltraLight 2.0
 #
 # Set a sub-domain name of IoT Agent for UltraLight 2.0
-IOTAGENT_UL=
+IOTAGENT_UL=iotagent-ul
 
 IOTA_UL_DEFAULT_RESOURCE=/iot/ul
 IOTA_UL_TIMESTAMP=true
@@ -292,7 +292,7 @@ IMAGE_IOTAGENT_UL=telefonicaiot/iotagent-ul:2.4.2
 # IoT Agent for JSON
 #
 # Set a sub-domain name of IoT Agent for JSON
-IOTAGENT_JSON=
+IOTAGENT_JSON=iotagent-json
 
 IOTA_JSON_DEFAULT_RESOURCE=/iot/json
 IOTA_JSON_TIMESTAMP=true
@@ -308,11 +308,11 @@ IMAGE_IOTAGENT_JSON=telefonicaiot/iotagent-json:2.4.2
 # Node-RED
 #
 # Set a sub-domain name of Node-RED
-NODE_RED=
+NODE_RED=nodered
 
 # Node-RED multi instance
 # Number of Node-RED instance. default: 1
-NODE_RED_INSTANCE_NUMBER=
+NODE_RED_INSTANCE_NUMBER=3
 
 # username for Node-RED instance. default: node-red
 NODE_RED_INSTANCE_USERNAME=
@@ -338,7 +338,7 @@ IMAGE_NODE_RED=letsfiware/node-red:0.37.0
 # Grafana
 #
 # Set a sub-domain name of Grafana
-GRAFANA=
+GRAFANA=grafana
 
 # Logging level for Grafana
 #   https://grafana.com/docs/grafana/latest/administration/configuration/#configure-with-environment-variables

@dkastl
Copy link
Contributor Author

dkastl commented May 27, 2024

Eventually Comet makes a difference, beside enabling all database adapters.

The 502 Bad Gateway error happens for Cygnus, when the default config.sh is modified like

diff --git a/config.sh b/config.sh
index 67b7d60..7051045 100755
--- a/config.sh
+++ b/config.sh
@@ -94,12 +94,12 @@ IMAGE_TIMESCALE=timescale/timescaledb-postgis:1.7.5-pg12
 # Cygnus
 #
 # Set a sub-domain name of Cygnus
-CYGNUS=
+CYGNUS=cygnus
 
 # Use Cygnus sink (true or false) Default: false
 CYGNUS_MONGO=
 CYGNUS_MYSQL=
-CYGNUS_POSTGRES=
+CYGNUS_POSTGRES=true
 CYGNUS_ELASTICSEARCH=
 
 # Expose port (none, local, all) Default: none
@@ -129,7 +129,7 @@ IMAGE_ELASTICSEARCH_DB=elasticsearch:7.6.2
 # Comet
 #
 # Set a sub-domain name of Comet
-COMET=
+COMET=comet
 
 # Expose port (none, local, all) Default: none
 COMET_EXPOSE_PORT=

This returns

$ sudo ngsi version --host cygnus.example.com | jq
cbVersion002 error 502 Bad Gateway <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

Cygnus works without comet enabled:

diff --git a/config.sh b/config.sh
index 67b7d60..92fd770 100755
--- a/config.sh
+++ b/config.sh
@@ -94,12 +94,12 @@ IMAGE_TIMESCALE=timescale/timescaledb-postgis:1.7.5-pg12
 # Cygnus
 #
 # Set a sub-domain name of Cygnus
-CYGNUS=
+CYGNUS=cygnus
 
 # Use Cygnus sink (true or false) Default: false
 CYGNUS_MONGO=
 CYGNUS_MYSQL=
-CYGNUS_POSTGRES=
+CYGNUS_POSTGRES=true
 CYGNUS_ELASTICSEARCH=
 
 # Expose port (none, local, all) Default: none

And Cygnus works with Comet if all databases are set to true.

diff --git a/config.sh b/config.sh
index 67b7d60..5d16b4d 100755
--- a/config.sh
+++ b/config.sh
@@ -94,12 +94,12 @@ IMAGE_TIMESCALE=timescale/timescaledb-postgis:1.7.5-pg12
 # Cygnus
 #
 # Set a sub-domain name of Cygnus
-CYGNUS=
+CYGNUS=cygnus
 
 # Use Cygnus sink (true or false) Default: false
-CYGNUS_MONGO=
-CYGNUS_MYSQL=
-CYGNUS_POSTGRES=
+CYGNUS_MONGO=true
+CYGNUS_MYSQL=true
+CYGNUS_POSTGRES=true
 CYGNUS_ELASTICSEARCH=
 
 # Expose port (none, local, all) Default: none
@@ -129,7 +129,7 @@ IMAGE_ELASTICSEARCH_DB=elasticsearch:7.6.2
 # Comet
 #
 # Set a sub-domain name of Comet
-COMET=
+COMET=comet
 
 # Expose port (none, local, all) Default: none
 COMET_EXPOSE_PORT=

Then the ngsi command return correctly

$ sudo ngsi version --host cygnus.example.com | jq
{
  "success": "true",
  "version": "3.2.0.9b21eae9095e3250ba8799e61411905d1c756191"
}

@fisuda
Copy link
Contributor

fisuda commented May 27, 2024

This error seems to occur when Cygnus mongodb and postgresql sinks are enabled.

CYGNUS_MONGO=true
CYGNUS_POSTGRES=true
CYGNUS_MONGO= CYGNUS_MYSQL= CYGNUS_POSTGRES= OK/NG
true true NG
true true OK
true true OK
true true true OK

As a workaround, enable the cygnus mysql sink as well.

COMET=comet
CYGNUS=cygnus
CYGNUS_MONGO=
CYGNUS_MYSQL=true
CYGNUS_POSTGRES=true

Or disable STH-Comet.

COMET=
CYGNUS=cygnus
CYGNUS_MONGO=
CYGNUS_POSTGRES=true

Your configuration is bellow:

COMET=comet
CYGNUS=cygnus
CYGNUS_MONGO=
CYGNUS_POSTGRES=true

When enabled STH-Comet and Cygnus, the FIWARE Big Bang sets CYGNUS_MONGO to true automatically
and support the formal mode (Cygnus + STH-Comet).

Therefore, your configuration is equivalent to:

COMET=comet
CYGNUS=cygnus
CYGNUS_MONGO=true
CYGNUS_POSTGRES=true

@dkastl
Copy link
Contributor Author

dkastl commented May 28, 2024

I see. That makes sense. And thank you for the suggestions!

I usually prefer PostgreSQL, if I have the choice. That's why I initially disabled the other alternatives. But there is no problem to enable all.

That said, I thought it may be helpful to report the problem here.

@fisuda fisuda added the bug Something isn't working label May 28, 2024
@fisuda
Copy link
Contributor

fisuda commented Jun 2, 2024

I put a question about this issue on the Stack Overflow site as shown:
https://stackoverflow.com/questions/78565261/failed-to-connect-to-fiware-cygnus-3-8-0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants