Add TandemRequestProvider for combined RequestList and RequestQueue usage

[CodeMan62]

Overview

This PR introduces two new components to improve request management in Crawlee:

• TandemRequestProvider: A provider that seamlessly combines RequestList and RequestQueue, allowing crawlers to use both sources efficiently.
• RequestListAdapter: An adapter that makes RequestList compatible with the IRequestProvider interface.
  Fixes #

Problem Solved

When using both RequestList and RequestQueue in a crawler, users often need to implement custom logic to ensure URLs aren't processed twice. This implementation standardizes and simplifies this pattern by:

1. First processing requests from the RequestList
2. Automatically transferring these requests to the RequestQueue in the background
3. Ensuring each URL is processed exactly once
4. Gracefully transitioning to RequestQueue after the list is exhausted

Implementation Details

• The RequestListAdapter wraps a RequestList instance and adapts its interface to match IRequestProvider.
• The TandemRequestProvider orchestrates the flow between the list and queue:
  • It implements the IRequestProvider interface
  • Handles request processing through the queue while ensuring list requests are enqueued first
  • Maintains proper request state tracking across both sources
  • Implements a background transfer mechanism to move requests efficiently

Integration

The implementation is already integrated into BasicCrawler._initializeRequestProviders(), making it immediately available to all crawler types without requiring additional configuration changes.

Backward Compatibility

This change is fully backward compatible:

• Existing code using either RequestList or RequestQueue alone will continue to work without changes
• This introduces an enhancement without modifying existing behavior

Testing

The implementation was tested to ensure it correctly:

• Transfers requests from list to queue
• Processes requests in the correct order
• Handles errors appropriately
• Maintains proper request state

This feature will simplify many crawler implementations and eliminate a common source of bugs when dealing with multiple request sources.

[barjin]

Please merge the latest master into your branch so we only get your changes in the diff. Cheers!

[CodeMan62]

@barjin Everything is done know you can review the PR thanks