Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout trying to upload qcamera files #34941

Open
robbederks opened this issue Mar 26, 2025 · 1 comment
Open

Timeout trying to upload qcamera files #34941

robbederks opened this issue Mar 26, 2025 · 1 comment
Labels

Comments

@robbederks
Copy link
Contributor

Describe the bug

When uploader tries to upload certain qcamera.ts files, it experiences a ReadTimeout from the Azure bucket. The file is actually uploaded successfully, since it does show up (and I can download it from) useradmin.

Uploader keeps retrying the same file until you remove it.
Never saw issues with qlogs (which go through the same process and go to the same bucket) or other uploads like bootlogs.

Exception:

ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='commadata2.blob.core.windows.net', port=443): Read timed out. (read timeout=10)")),
  Traceback (most recent call last):
    File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
      response = conn.getresponse()
                 ^^^^^^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connection.py", line 507, in getresponse
      httplib_response = super().getresponse()
                         ^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse
      response.begin()
    File "/usr/lib/python3.12/http/client.py", line 331, in begin
      version, status, reason = self._read_status()
                                ^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/http/client.py", line 292, in _read_status
      line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/socket.py", line 707, in readinto
      return self._sock.recv_into(b)
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/ssl.py", line 1252, in recv_into
      return self.read(nbytes, buffer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/ssl.py", line 1104, in read
      return self._sslobj.read(len, buffer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  TimeoutError: The read operation timed out

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):
    File "/usr/local/venv/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
      resp = conn.urlopen(
             ^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 843, in urlopen
      retries = retries.increment(
                ^^^^^^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment
      raise reraise(type(error), error, _stacktrace)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
      raise value
    File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
      response = self._make_request(
                 ^^^^^^^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 538, in _make_request
      self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
    File "/usr/local/venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 369, in _raise_timeout
      raise ReadTimeoutError(
  urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='commadata2.blob.core.windows.net', port=443): Read timed out. (read timeout=10)

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/data/openpilot/system/loggerd/uploader.py", line 186, in upload
      stat = self.do_upload(key, fn)
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/data/openpilot/system/loggerd/uploader.py", line 159, in do_upload
      response = requests.put(url, data=stream, headers=headers, timeout=10)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/requests/api.py", line 130, in put
      return request("put", url, data=data, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/requests/api.py", line 59, in request
      return session.request(method=method, url=url, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
      resp = self.send(prep, **send_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
      r = adapter.send(request, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/venv/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
      raise ReadTimeout(e, request=request)
  requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='commadata2.blob.core.windows.net', port=443): Read timed out. (read timeout=10)

Provide a route where the issue occurs

N/A

openpilot version

0.9.9

Additional info

No response

@robbederks robbederks added the bug label Mar 26, 2025
@robbederks
Copy link
Contributor Author

The retry might also explain #34922

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant