Feat/1117 auto content type

[zar3bski]

Landing a Pull Request (PR)

Long form explanations of most of the items below can be found in the CONTRIBUTING guide.

Branching checklist

• There is an issue associated with your PR (bug, feature, etc.. if not, create one)
• Your PR description references the associated issue (i.e. fixes #123456)
• Code is in its own branch
• Branch name is related to the PR contents
• PR targets main

Static analysis checks

• All rust files are formatted using cargo fmt
• All clippy checks pass when running cargo clippy --all-targets --all-features -- -D warnings -A clippy::mutex-atomic
• All existing tests pass

I do not understand the clipy erros yet, just noticing it while submitting this PR

Documentation

• New code is documented using doc comments
• Documentation about your PR is included in the docs, as needed. The docs live in a separate repository. Update the appropriate pages at the links below.
  • update example config file section
  • update help output section
  • add an example

Not yet

Additional Tests

• New code is unit tested
• New code is integration tested, as needed
• New tests pass

Not

[zar3bski]

I removed the post-form composite arg, added Unit tests, squashed the branch and I think that we are good to go. While syncing my fork, Github integrated origin/main in a weird fashion (contrib + Fish)

I handled x-www-form-urlencoded encoding without -Q, for it is a body to be displayed in Body banner section.

One test fails nlp::model::tests::model_generates_expected_tf_idf_scores but seems unrelated to this feature. I do not understand the problem pointed by the clippy tests.

Ready for review, @epi052 . No rush :-)

[zar3bski]

I'm puzzled by how tests behave differently locally and on the CI

Locally

Whether I run cargo test from main or from 289389d, I end up with this unrelated error:

failures:
    nlp::model::tests::model_generates_expected_tf_idf_scores

Same goes when I run cargo nextest run --all-features --all-targets --retries 4 --no-fail-fast

     Summary [   8.731s] 440 tests run: 439 passed, 1 failed, 1 skipped
  TRY 5 FAIL [   0.016s] feroxbuster nlp::model::tests::model_generates_expected_tf_idf_scores

Env:

• arch linux
• rustc 1.84.0 (9fc6b4312 2025-01-07)
• cargo 1.84.0 (66221abde 2024-11-19)

CI

cargo nextest run --all-features --all-targets --retries 4 --no-fail-fast
ends up with an error on

feroxbuster::test_main main_parallel_creates_output_directory

no sign of nlp::model::tests::model_generates_expected_tf_idf_scores.

This difference of behavior together with the fact that I'm quite new to rust and do not understand in details the architecture of feroxbuster makes tests debugging pretty hard for me. Are those false positives?

[epi052]

@zar3bski don't sweat those two tests. usually when i see a difference between ci/local tests that pass/fail differently, it's because my local toolchain is behind what runs in CI. If you do a rustup update, the test fails/passes may match afterward.

As for the tests themselves, it's not code that's related to your changes. I'll take a look before we merge your branch and get things sorted. Same goes for clippy (which are just new lints in the latest toolchain version).

[epi052]

ok, i did some thinking about this and looked at how some other cli tools handle this sort of thing (if they do at all).

Below is what I'm thinking at this point, please let me know what you think!

• --data
  • implies -m POST
  • if the user provides a -m that should override the default
• --post-data should become --data-urlencoded
  • implies -m POST
  • if the user provides a -m that should override the default
  • implies -H Content-Type: application/x-www-form-urlencoded
  • values from each key/value pair within the body are url
  • should support using @ symbol to use a file path (same as --data)
• --post-json should become --data-json
  • implies -m POST
  • if the user provides a -m that should override the default
  • implies -H Content-Type: application/json
  • should support using @ symbol to use a file path (same as --data)
• if the user provides both --data-* and -m POST, we can emit an informational log line stating that the combination is redundant