Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Distiguish File loading from Document loading #3820

Open
JJK801 opened this issue Jan 7, 2025 · 0 comments
Open

[FEATURE] Distiguish File loading from Document loading #3820

JJK801 opened this issue Jan 7, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@JJK801
Copy link
Contributor

JJK801 commented Jan 7, 2025

Describe the feature you'd like

ATM, File loading (Direct Upload, S3 Source, ...) is deeply correlated to Document loading (Langchain loaders, Unstructured API, ...), but i think it should be separated concerns in order to be able to reuse the Document loading logic.

A good way would be to introducing FileLoader entity that can be an input for DocumentLoader entity.

Additional context

Example: I am unable to load files from S3 without using Unstructured API for chunking (even if it is possible to use any loader at code level)

Solution: I should have a S3File (FileLoader) node that can be linked as an input for UnstructuredFile (DocumentLoader) node or any other document loader accepting files.

@HenryHengZJ HenryHengZJ added the enhancement New feature or request label Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants