Change media upload to use the API v2 endpoints #38

GCorbel · 2025-01-23T17:27:00Z

This fix #37

X announced the creation of the API v2 endpoints for media upload : https://devcommunity.x.com/t/announcing-media-upload-endpoints-in-the-x-api-v2/234175/3

It is very similar, so there are not a lot of changes. The biggest change is that it requires to use the authentication with OAuth 2.0 but once you have a user token, the client can be done with X::Client.new(bearer_token: user_token). This client works the same for all other endpoints.

GCorbel · 2025-01-23T17:30:19Z

lib/x/response_parser.rb

+      begin
+        JSON.parse(response.body, array_class:, object_class:)
+      rescue JSON::ParserError
+      end


After the call of POST media/upload, the content-type is "application/octet-stream" despite the body is a valid json.

Here is an example of a response :

-> "HTTP/1.1 200 OK\r\n" -> "date: Thu, 23 Jan 2025 17:19:10 GMT\r\n" -> "perf: 7402827104\r\n" -> "vary: Origin\r\n" -> "pragma: no-cache\r\n" -> "server: tsa_b\r\n" -> "expires: Tue, 31 Mar 1981 05:00:00 GMT\r\n" -> "set-cookie: guest_id_marketing=v1%3A173765274968771643; Max-Age=63072000; Expires=Sat, 23 Jan 2027 17:19:10 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None\r\n" -> "set-cookie: guest_id_ads=v1%3A173765274968771643; Max-Age=63072000; Expires=Sat, 23 Jan 2027 17:19:10 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None\r\n" -> "set-cookie: personalization_id=\"v1_6xr6bdpzJyLG+/nW6b3LqQ==\"; Max-Age=63072000; Expires=Sat, 23 Jan 2027 17:19:10 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None\r\n" -> "set-cookie: lang=en; Path=/\r\n" -> "set-cookie: guest_id=v1%3A173765274968771643; Max-Age=63072000; Expires=Sat, 23 Jan 2027 17:19:10 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None\r\n" -> "api-version: 2.development\r\n" -> "content-type: application/octet-stream\r\n" -> "cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0\r\n" -> "last-modified: Thu, 23 Jan 2025 17:19:10 GMT\r\n" -> "x-transaction: 409457a465fdfcc5\r\n" -> "content-length: 160\r\n" -> "x-frame-options: SAMEORIGIN\r\n" -> "x-transaction-id: 409457a465fdfcc5\r\n" -> "x-xss-protection: 1; mode=block\r\n" -> "x-rate-limit-limit: 1000\r\n" -> "x-rate-limit-reset: 1737653013\r\n" -> "x-content-type-options: nosniff\r\n" -> "x-rate-limit-remaining: 992\r\n" -> "x-twitter-response-tags: BouncerCompliant\r\n" -> "strict-transport-security: max-age=631138519\r\n" -> "x-response-time: 1142\r\n" -> "x-connection-hash: 4d2a5453c0fcc53df1a7580c322112b153fb3731fd16b2367e9869e2472add55\r\n" -> "connection: close\r\n" -> "\r\n" reading 160 bytes... -> "{\"id\":\"1882478190238502912\",\"media_key\":\"3_1882478190238502912\",\"size\":753855,\"expires_after_secs\":86400,\"image\":{\"image_type\":\"image\\/jpeg\",\"w\":1980,\"h\":1114}}"

Rubocop complains about that because I remove an exception, but I have no better idea. Do you have one ?

GCorbel · 2025-01-23T17:31:05Z

x.gemspec

@@ -9,7 +9,7 @@ Gem::Specification.new do |spec|
  spec.summary = "A Ruby interface to the X API."
  spec.homepage = "https://sferik.github.io/x-ruby"
  spec.license = "MIT"
-  spec.required_ruby_version = ">= 3.2"
+  # spec.required_ruby_version = ">= 3.2"


It's temporary but I need to comment this to be able to test ATM.

Is there a reason to force the version of Ruby ?

Support for Ruby 3.1 is scheduled to end in two months, on March 31, 2025. After that point, running Ruby 3.1 in production will be insecure: any new vulnerabilities will be made public but won't be patched. Ruby 3.2 was released over two years ago, on December 25, 2022, and offers many features and performance improvements over Ruby 3.1. I'm interested to understand why you're waiting to upgrade.

We are still using Ruby 3.1 and want to upgrade. We are currently working on the upgrade.

The syntax changes between 3.1 and 3.2 are quite minor. I'm curious what's blocking your upgrade.

We need to upgrade our stack which is not all compatible with Ruby 3.2.

GCorbel · 2025-01-23T17:34:35Z

@sferik can you take a quick look? I will change specs and finish the implementation once you validated this draft.

sferik · 2025-01-24T03:15:52Z

Thank you for this patch. It looks like you're on the right track! I will happily merge this once all the tests are passing.

GCorbel · 2025-01-28T17:03:11Z

lib/x/media_uploader.rb

    end

    def chunked_upload(client:, file_path:, media_category:, media_type: infer_media_type(file_path,
-      media_category), boundary: SecureRandom.hex, chunk_size_mb: 8)


The size of chunks decreased to 1mb : https://docs.x.com/x-api/media/quickstart/media-upload-chunked#step-2-post-media-upload-append.

GCorbel · 2025-01-28T17:15:15Z

lib/x/media_uploader.rb

      chunk_size = chunk_size_mb * BYTES_PER_MB
-      append(upload_client:, file_paths: split(file_path, chunk_size), media:, media_type:, boundary:)
-      upload_client.post("media/upload.json?command=FINALIZE&media_id=#{media["media_id"]}")


Sometimes, the API return a payload with data and sometimes it doesn't.

A call to "media/upload?media_category=tweet_image" returns :

{"id":"1884301761822838785","media_key":"3_1884301761822838785","size":20259,"expires_after_secs":86400,"image":{"image_type":"image\/jpeg","w":932,"h":176}}

But with a "command" attribute it gives :

{"data":{"id":"1884301283227561984","media_key":"7_1884301283227561984","size":9840497,"expires_after_secs":86399,"video":{"video_type":"video\/mp4"},"processing_info":"state":"succeeded","progress_percent":100}}}

A request with "command=APPEND" returns an empty body with "NoContent" in headers.

GCorbel · 2025-01-28T18:18:53Z

lib/x/response_parser.rb

+      begin
+        JSON.parse(response.body, array_class:, object_class:)
+      rescue JSON::ParserError
+      end


Rubocop complains about that because I remove an exception, but I have no better idea. Do you have one ?

lib/x/response_parser.rb

sferik · 2025-01-28T19:10:13Z

lib/x/response_parser.rb

-      JSON.parse(response.body, array_class:, object_class:)
+      begin
+        JSON.parse(response.body, array_class:, object_class:)
+      rescue JSON::ParserError # rubocop:disable Lint/SuppressedException


What are the cases you're expecting to rescue here? Do you expect the X API to return invalid JSON? Or just an empty string?

This is related to the comment I did here : #38 (comment)

Previously, we relied on headers to parse JSON but now, there is a request made with "application/octet-stream" which contains a valid JSON. As before, nil should be returned when the body does not contain a JSON.

sferik

Looks good overall! Please address my two comments/questions above.

sferik · 2025-01-28T22:21:11Z

lib/x/response_parser.rb

-
-    def json?(response)
-      JSON_CONTENT_TYPE_REGEXP === response["content-type"]
-    end


If you're removing this method, you can remove the JSON_CONTENT_TYPE_REGEXP constant, since this is the only place it's used.

That said, the X::HTTPError class performs a similar check when parsing error JSON. We probably want to handle these cases in the same way.

I will give a try tomorrow.

I tried something as :

diff --git a/lib/x/errors/http_error.rb b/lib/x/errors/http_error.rb index 9f54aeb..ce0871d 100644 --- a/lib/x/errors/http_error.rb +++ b/lib/x/errors/http_error.rb @@ -3,8 +3,6 @@ require_relative "error" module X class HTTPError < Error - JSON_CONTENT_TYPE_REGEXP = %r{application/(problem\+|)json} - attr_reader :response, :code def initialize(response:) @@ -14,7 +12,7 @@ module X end def error_message(response) - if json?(response) + if X::ResponseParser.json?(response) message_from_json_response(response) else response.message @@ -33,9 +31,5 @@ module X response.message end end - - def json?(response) - JSON_CONTENT_TYPE_REGEXP === response["content-type"] - end end end diff --git a/lib/x/response_parser.rb b/lib/x/response_parser.rb index da5606d..fa55cb2 100644 --- a/lib/x/response_parser.rb +++ b/lib/x/response_parser.rb @@ -34,17 +34,20 @@ module X 503 => ServiceUnavailable, 504 => GatewayTimeout }.freeze - JSON_CONTENT_TYPE_REGEXP = %r{application/json} + + def self.json?(response) + JSON.parse(response.body) + true + rescue JSON::ParserError, TypeError + false + end def parse(response:, array_class: nil, object_class: nil) raise error(response) unless response.is_a?(Net::HTTPSuccess) - return if response.instance_of?(Net::HTTPNoContent) + return if !self.class.json?(response) || response.instance_of?(Net::HTTPNoContent) - begin - JSON.parse(response.body, array_class:, object_class:) - rescue JSON::ParserError # rubocop:disable Lint/SuppressedException - end + JSON.parse(response.body, array_class:, object_class:) end private

But it breaks 22 tests so there are too many changes for this PR.

I refactored a little and removed JSON_CONTENT_TYPE_REGEXP so it's simple.

sferik · 2025-01-30T21:16:40Z

lib/x/response_parser.rb

+    def json?(response)
+      return false if response.instance_of?(Net::HTTPNoContent)
+
+      JSON.parse(response.body)


This is better than before but it seems wasteful to parse the response body and throw away the result right before parsing JSON again in the parse method. I would move this check into the parse method, especially if it's a private method that you're not going to call from the X::HTTPError class.

I agree, I changed to have a similar version than 8d16719.

sferik · 2025-01-30T22:33:29Z

Looks good!

sferik · 2025-01-30T22:46:28Z

FYI, I just pushed these two cleanup commits:

Change media upload to use the API v2 endpoints

20086aa

GCorbel commented Jan 23, 2025

View reviewed changes

GCorbel marked this pull request as draft January 23, 2025 17:31

GCorbel added 5 commits January 28, 2025 11:49

fix when the payload root is data

d4bbbce

fix tests

d52d747

ignore rubocop rule

c61d464

readd required ruby version

07c822e

fix linter issues

ba51cae

GCorbel commented Jan 28, 2025

View reviewed changes

GCorbel marked this pull request as ready for review January 28, 2025 18:40

GCorbel requested a review from sferik January 28, 2025 18:41

sferik reviewed Jan 28, 2025

View reviewed changes

lib/x/response_parser.rb Show resolved Hide resolved

sferik reviewed Jan 28, 2025

View reviewed changes

sferik requested changes Jan 28, 2025

View reviewed changes

sferik reviewed Jan 28, 2025

View reviewed changes

refactor json check

8d16719

GCorbel requested a review from sferik January 29, 2025 15:35

fix mutant error

18322a4

sferik reviewed Jan 30, 2025

View reviewed changes

parse the body only once

a8a966e

sferik merged commit 001721f into sferik:main Jan 30, 2025
6 checks passed

yoshida-sigmoid mentioned this pull request Feb 1, 2025

Is a new gem release planned soon? #39

Closed

sferik mentioned this pull request Mar 13, 2025

X::BadRequest when using await_processing #42

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change media upload to use the API v2 endpoints #38

Change media upload to use the API v2 endpoints #38

GCorbel commented Jan 23, 2025

GCorbel Jan 23, 2025

GCorbel Jan 28, 2025

GCorbel Jan 23, 2025

sferik Jan 24, 2025 •

edited

Loading

GCorbel Jan 28, 2025

sferik Jan 28, 2025

GCorbel Jan 28, 2025 •

edited

Loading

GCorbel commented Jan 23, 2025

sferik commented Jan 24, 2025

GCorbel Jan 28, 2025

GCorbel Jan 28, 2025

GCorbel Jan 28, 2025

sferik Jan 28, 2025 •

edited

Loading

GCorbel Jan 28, 2025

sferik left a comment

sferik Jan 28, 2025

GCorbel Jan 28, 2025

GCorbel Jan 29, 2025

sferik Jan 30, 2025

GCorbel Jan 30, 2025

sferik commented Jan 30, 2025

sferik commented Jan 30, 2025

Change media upload to use the API v2 endpoints #38

Change media upload to use the API v2 endpoints #38

Conversation

GCorbel commented Jan 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sferik Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GCorbel Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

GCorbel commented Jan 23, 2025

sferik commented Jan 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sferik Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sferik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sferik commented Jan 30, 2025

sferik commented Jan 30, 2025

sferik Jan 24, 2025 •

edited

Loading

GCorbel Jan 28, 2025 •

edited

Loading

sferik Jan 28, 2025 •

edited

Loading