Multiline string bodies for multipart form data produce errors on some servers due to unix line ending between body parts

[GGoetzelmann]

What is the current bug behavior?

Hurl file with multiline string payload is causing errors due to newline

Depending on the OS hurl creates a curl command such as this:

curl --header 'Content-Type: multipart/form-data; boundary=boundary' --data $'--boundary\nContent-Disposition: form-data; name="record"; filename="blob"\nContent-Type: application/json\n\n{"id":"0"}\n--boundary\nContent-Disposition: form-data; name="document"; filename="blob"\nContent-Type: application/json\n\n{"some":"json"}\n--boundary--\n' 'http://localhost:8888'

This, however, seems to be rejected with an error (such as MalformedStreamException: Stream ended unexpectedly) by some servers (such as applications based on SpringBoot and Tomcat) due to unexpected line endings.

To me it looks like the produced curl command may not be compliant to RFC 9110:

8.3.3. Multipart Types
MIME provides for a number of "multipart" types -- encapsulations of one or more representations within a single message body. All multipart types share a common syntax, as defined in Section 5.1.1 of [RFC2046], and include a boundary parameter as part of the media type value.
The message body is itself a protocol element; a sender MUST generate only CRLF to represent line breaks between body parts.

Steps to reproduce

• use hurl on an OS with Unix line endings
• use a hurl file like this sample file
• use it against a SpringBoot/Tomcat based server such as this example (I have not ties to this project, it just popped up as an easy to use dockerized minimal example of a spring boot app. I know it is quite old, but the problem occurs for me on modern spring boot systems as well, I just don't have one handy for you.)

By the way the exact same thing can be reproduced with the documentation example about multiline string body (once you remove the "" around boundary from the sample, otherwise it will fail anyway on the mentioned systems).

What is the expected correct behavior?

Produce a curl with crlf line endings to make sure that libs such as those in tomcat core can split the parts correctly - I guess? Maybe I am misunderstanding something here completely, because this would be the first time for me that using CRLF would solve, not cause problems.

Execution context

The problem does NOT occur with hurl on windows (due to obvious reasons)

The problem does NOT occur if you convert the hurl file with "unix2dos" command first on unix based systems before executing it.

• Hurl Version (hurl --version): 6.0.0

Possible fixes

I was not able to manually adapt the hurl file so that I may include correct line endings in the string myself - pointers would be appreciated. Maybe it can simply be solved with correct input - if so, this may be a topic for documentation.

[jcamiel]

Hi @GGoetzelmann

Thanks for this very detailed description. There is actually no logic in multiline string, characters that are between ``` markers are taken "as it" from the Hurl file. If there is a \n, or a \r\n in the file:

POST 
```
line1    => ending with a \n
line2    => ending with a \r\n
line3    => ending with a \n
```

We're sending "line1\nline2\r\nline3\n"

Concerning multipart, I wasn't aware that message parts should use CRLF separators so, this snippet, from the docs, can be incorrect if the user has no written the right CRLF at the right place:

POST https://example.org/upload
Content-Type: multipart/form-data; boundary="boundary"
```
--boundary
Content-Disposition: form-data; name="key1"

value1
--boundary
Content-Disposition: form-data; name="upload1"; filename="data.txt"
Content-Type: text/plain

Hello World!
--boundary
Content-Disposition: form-data; name="upload2"; filename="data.html"
Content-Type: text/html

<div>Hello <b>World</b>!</div>
--boundary--
``` 

I think we don't want to put any logic with "classic" multiline stings, so the only way we can support writting a multipart request like this is:

• solution 1: if a request body is using a multiline strings ``` xxx ``` and Content-Type is of type multipart, we process the multiline string body and insure that there is the right CRLF delimiter at the right places.
• solution 2: use a language "hint" that indicates that the multiline string should be read as a multipart request:

POST https://example.org/upload
Content-Type: multipart/form-data; boundary="boundary"
```multipart
--boundary
Content-Disposition: form-data; name="key1"

value1
--boundary
Content-Disposition: form-data; name="upload1"; filename="data.txt"
Content-Type: text/plain

Hello World!
--boundary
Content-Disposition: form-data; name="upload2"; filename="data.html"
Content-Type: text/html

<div>Hello <b>World</b>!</div>
--boundary--
``` 

In this case, we process the request body so that we send a correct multipart body.

In both case, we need to have a Content-Type header to set the boundary value. I tend to prefer the second solution with an explicit ```multipart marker to hint that the body could be processed before sending.

@fabricereix @lepapareil any comment?

[lepapareil]

Hi @jcamiel, I tend to prefer the explicit 2nd solution :).

[fabricereix]

also in favor if the 2nd solution

[GGoetzelmann]

@jcamiel thank you for looking into it (and from the user perspective I wouldn't mind explicitness here, either. Multipart form data is strange enough without additional hidden treatment).

Just a side note: on very-verbose mode the current curious side effect is that on a working request the curl command in the verbose output is broken (I think it is split erroneously on a newline and only a part is provided in the output), while on a faulty string input, the curl command appears in the log correctly.

Wanted to mention it since on special treatment for the multipart string this problem may remain.