Address cosmetic changes

Author: amritghimire

docs/commands/cp.md

@@ -14,6 +14,12 @@ usage: datachain cp [-h] [-v] [-q] [-r] [--team TEAM]
 
 This command copies files and directories between local and/or remote storage. This uses the credentials in your system by default or can use the cloud authentication from Studio.
 
+The command supports two main modes of operation:
+
+- By default, the command operates directly with clouds using credentials in your system, supporting various copy scenarios between local and remote storage.
+- When using `-s` or `--studio-cloud-auth` flag, the command uses credentials from Studio for cloud operations. This mode provides enhanced authentication and access control for cloud storage operations.
+
+
 ## Arguments
 
 * `source_path` - Path to the source file or directory to copy
@@ -22,35 +28,20 @@ This command copies files and directories between local and/or remote storage. T
 ## Options
 
 * `-r`, `-R`, `--recursive` - Copy directories recursively
-* `--team TEAM` - Team name to use the credentials from.
+* `--team TEAM` - Team name to use the credentials from. (Default: from config)
 * `-s`, `--studio-cloud-auth` - Use credentials from Studio for cloud operations (Default: False)
 * `--update` - Update cached list of files for the source when downloading from cloud using local credentials.
 * `-h`, `--help` - Show the help message and exit
 * `-v`, `--verbose` - Be verbose
 * `-q`, `--quiet` - Be quiet
 
-## Copy Operations
-
-The command supports two main modes of operation:
 
-### Default Mode
-By default, the command operates directly with clouds using credentials in ypur system, supporting various copy scenarios between local and remote storage.
-
-### Studio Cloud Auth Mode
-When using `-s` or `--studio-cloud-auth` flag, the command uses credentials from Studio for cloud operations. This mode provides enhanced authentication and access control for cloud storage operations.
-
-## Supported Storage Protocols
+## Notes
+* When using Studio cloud auth mode, you must be authenticated with `datachain auth login` before using it
+* The default mode operates directly with storage providers
 
-The command supports the following storage protocols:
-- **Local file system**: Direct paths (e.g., `/path/to/directory` or `./relative/path`)
-- **AWS S3**: `s3://bucket-name/path`
-- **Google Cloud Storage**: `gs://bucket-name/path`
-- **Azure Blob Storage**: `az://container-name/path`
 
 ## Examples
-
-The command automatically determines the operation type based on the source and destination protocols:
-
 ### Local to Local
 
 **Operation**: Direct local file system copy
@@ -115,15 +106,3 @@ datachain cp gs://my-bucket/data/file.py gs://my-bucket/archive/file.py
 # Copy within same bucket with Studio cloud auth
 datachain cp gs://my-bucket/data/file.py gs://my-bucket/archive/file.py --studio-cloud-auth
 ```
-
-### Additional Examples
-
-```bash
-# Copy with specific team:
-datachain cp -s --team other-team /path/to/file.txt s3://my-bucket/data/file.txt
-```
-
-
-## Notes
-* When using Studio cloud auth mode, you must be authenticated with `datachain auth login` before using it
-* The default mode operates directly with storage providers

docs/commands/mv.md

@@ -1,6 +1,6 @@
 # mv
 
-Move storage files and directories through Studio.
+Move storage files and directories in clouds or local filesystem.
 
 ## Synopsis
 
@@ -11,7 +11,7 @@ usage: datachain mv [-h] [-v] [-q] [--recursive]
 
 ## Description
 
-This command moves files and directories within storage. The move operation is performed within the same bucket - you cannot move files between different buckets. The command supports both individual files and directories, with the `--recursive` flag required for moving directories.
+This command moves files and directories within storage. The command supports both individual files and directories, with the `--recursive` flag required for moving directories.
 
 ## Arguments
 
@@ -21,7 +21,7 @@ This command moves files and directories within storage. The move operation is p
 ## Options
 
 * `--recursive` - Move recursively
-* `--team TEAM` - Team name to move storage contents from
+* `--team TEAM` - Team name to use the credentials from. (Default: from config)
 * `-s`, `--studio-cloud-auth` - Use credentials from Studio for cloud operations (Default: False)
 * `-h`, `--help` - Show the help message and exit
 * `-v`, `--verbose` - Be verbose
@@ -31,6 +31,11 @@ This command moves files and directories within storage. The move operation is p
 
 The command supports moving files and directories within the same bucket:
 
+## Notes
+* When using Studio cloud auth mode, you must be authenticated with `datachain auth login` before using it
+* The default mode operates directly with storage providers
+* **Warning**: This is a destructive operation. Always double-check the path before executing the command
+
 ### Move Single File
 
 ```bash

docs/commands/rm.md

@@ -1,6 +1,6 @@
 # rm
 
-Delete storage files and directories through Studio.
+Delete storage files and directories from cloud or local system.
 
 ## Synopsis
 
@@ -19,12 +19,19 @@ This command deletes files and directories within storage. The command supports
 ## Options
 
 * `--recursive` - Delete recursively
-* `--team TEAM` - Team name to delete storage contents from
+* `--team TEAM` - Team name to use the credentials from. (Default: from config)
 * `-s`, `--studio-cloud-auth` - Use credentials from Studio for cloud operations (Default: False)
 * `-h`, `--help` - Show the help message and exit
 * `-v`, `--verbose` - Be verbose
 * `-q`, `--quiet` - Be quiet
 
+
+## Notes
+* When using Studio cloud auth mode, you must be authenticated with `datachain auth login` before using it
+* The default mode operates directly with storage providers
+* **Warning**: This is a destructive operation. Always double-check the path before executing the command
+
+
 ## Examples
 
 The command supports deleting files and directories:
@@ -48,9 +55,3 @@ datachain rm gs://my-bucket/data/directory --recursive
 # Delete directory with Studio cloud auth
 datachain rm gs://my-bucket/data/directory --recursive --studio-cloud-auth
 ```
-
-
-## Notes
-* When using Studio cloud auth mode, you must be authenticated with `datachain auth login` before using it
-* The default mode operates directly with storage providers
-* **Warning**: This is a destructive operation. Always double-check the path before executing the command

src/datachain/cli/__init__.py

@@ -115,36 +115,36 @@ def handle_command(args, catalog, client_config) -> int:
     return 1
 
 
-def _get_storage_implementation(args: "Namespace", catalog: "Catalog"):
+def _get_file_handler(args: "Namespace", catalog: "Catalog"):
     from datachain.cli.commands.storage import (
         LocalCredentialsBasedFileHandler,
-        StorageCredentialFileHandler,
+        StudioAuthenticatedFileHandler,
     )
     from datachain.config import Config
 
     config = Config().read().get("studio", {})
     token = config.get("token")
     studio = False if not token else args.studio_cloud_auth
     return (
-        StorageCredentialFileHandler(args, catalog)
+        StudioAuthenticatedFileHandler(args, catalog)
         if studio
         else LocalCredentialsBasedFileHandler(args, catalog)
     )
 
 
 def handle_cp_command(args, catalog):
-    storage_implementation = _get_storage_implementation(args, catalog)
-    return storage_implementation.cp()
+    file_handler = _get_file_handler(args, catalog)
+    return file_handler.cp()
 
 
 def handle_mv_command(args, catalog):
-    storage_implementation = _get_storage_implementation(args, catalog)
-    return storage_implementation.mv()
+    file_handler = _get_file_handler(args, catalog)
+    return file_handler.mv()
 
 
 def handle_rm_command(args, catalog):
-    storage_implementation = _get_storage_implementation(args, catalog)
-    return storage_implementation.rm()
+    file_handler = _get_file_handler(args, catalog)
+    return file_handler.rm()
 
 
 def handle_clone_command(args, catalog):

src/datachain/cli/commands/storage/__init__.py

@@ -1,10 +1,10 @@
 from .local import LocalCredentialsBasedFileHandler
-from .studio import StorageCredentialFileHandler
+from .studio import StudioAuthenticatedFileHandler
 from .utils import build_file_paths, validate_upload_args
 
 __all__ = [
     "LocalCredentialsBasedFileHandler",
-    "StorageCredentialFileHandler",
+    "StudioAuthenticatedFileHandler",
     "build_file_paths",
     "validate_upload_args",
 ]

src/datachain/cli/commands/storage/local.py

@@ -12,8 +12,10 @@ def cp(self):
         source_path = self.args.source_path
         destination_path = self.args.destination_path
         destination_cls = Client.get_implementation(destination_path)
+
         source_cls = Client.get_implementation(source_path)
         source_fs = source_cls.create_fs()
+        _, src_subpath = source_cls.split_url(source_path)
 
         update = self.args.update
         if source_cls.protocol == "file":
@@ -29,7 +31,9 @@ def cp(self):
         file_paths = {}
 
         if not is_file:
-            chain.to_storage(destination_path, placement="normpath")
+            chain.to_storage(
+                destination_path, placement="normpath", relative_to=src_subpath
+            )
 
         for (file,) in chain.to_iter("file"):
             if is_file:
@@ -40,7 +44,9 @@ def cp(self):
                 )
                 file.save(dst)
             else:
-                dst = file.get_destination_path(destination_path, "normpath")
+                dst = file.get_destination_path(
+                    destination_path, "normpath", relative_to=src_subpath
+                )
             _, dst_path = destination_cls.split_url(str(dst))
             file_paths[dst_path] = file.size
 

src/datachain/cli/commands/storage/studio.py

@@ -7,7 +7,7 @@
     from datachain.client.fsspec import Client
 
 
-class StorageCredentialFileHandler(CredentialBasedFileHandler):
+class StudioAuthenticatedFileHandler(CredentialBasedFileHandler):
     def cp(self):
         from datachain.client.fsspec import Client
 

src/datachain/cli/parser/studio.py

@@ -169,7 +169,7 @@ def add_storage_parser(subparsers, parent_parser) -> None:
     storage_cp_parser.add_argument(
         "--team",
         action="store",
-        help="Team name to copy storage contents to",
+        help="Team name to use the credentials from.",
     )
 
     storage_cp_parser.add_argument(
@@ -211,7 +211,7 @@ def add_storage_parser(subparsers, parent_parser) -> None:
     mv_parser.add_argument(
         "--team",
         action="store",
-        help="Team name to move storage contents from",
+        help="Team name to use the credentials from.",
     )
 
     mv_parser.add_argument(
@@ -242,7 +242,7 @@ def add_storage_parser(subparsers, parent_parser) -> None:
     rm_parser.add_argument(
         "--team",
         action="store",
-        help="Team name to delete storage contents from",
+        help="Team name to use the credentials from.",
     )
     rm_parser.add_argument(
         "-s",

src/datachain/lib/dc/datachain.py

@@ -2413,6 +2413,7 @@ def to_storage(
         num_threads: Optional[int] = EXPORT_FILES_MAX_THREADS,
         anon: Optional[bool] = None,
         client_config: Optional[dict] = None,
+        relative_to: Optional[str] = None,
     ) -> None:
         """Export files from a specified signal to a directory. Files can be
         exported to a local or cloud directory.
@@ -2468,6 +2469,7 @@ def to_storage(
             link_type,
             max_threads=num_threads or 1,
             client_config=client_config,
+            relative_to=relative_to,
         )
         file_exporter.run(
             (rows[0] for rows in chain.to_iter(signal)),

src/datachain/lib/file.py

@@ -58,13 +58,15 @@ def __init__(
         link_type: Literal["copy", "symlink"],
         max_threads: int = EXPORT_FILES_MAX_THREADS,
         client_config: Optional[dict] = None,
+        relative_to: Optional[str] = None,
     ):
         super().__init__(max_threads)
         self.output = output
         self.placement = placement
         self.use_cache = use_cache
         self.link_type = link_type
         self.client_config = client_config
+        self.relative_to = relative_to
 
     def done_task(self, done):
         for task in done:
@@ -77,6 +79,7 @@ def do_task(self, file: "File"):
             self.use_cache,
             link_type=self.link_type,
             client_config=self.client_config,
+            relative_to=self.relative_to,
         )
         self.increase_counter(1)
 
@@ -422,10 +425,11 @@ def export(
         use_cache: bool = True,
         link_type: Literal["copy", "symlink"] = "copy",
         client_config: Optional[dict] = None,
+        relative_to: Optional[str] = None,
     ) -> None:
         """Export file to new location."""
         self._caching_enabled = use_cache
-        dst = self.get_destination_path(output, placement)
+        dst = self.get_destination_path(output, placement, relative_to)
         dst_dir = os.path.dirname(dst)
         client: Client = self._catalog.get_client(dst_dir, **(client_config or {}))
         client.fs.makedirs(dst_dir, exist_ok=True)
@@ -549,7 +553,10 @@ def get_fs_path(self) -> str:
         return path
 
     def get_destination_path(
-        self, output: Union[str, os.PathLike[str]], placement: ExportPlacement
+        self,
+        output: Union[str, os.PathLike[str]],
+        placement: ExportPlacement,
+        relative_to: Optional[str] = None,
     ) -> str:
         """
         Returns full destination path of a file for exporting to some output
@@ -568,6 +575,8 @@ def get_destination_path(
             raise NotImplementedError("Checksum placement not implemented yet")
         elif placement == "normpath":
             path = unquote(self.get_path_normalized())
+            if relative_to:
+                path = posixpath.relpath(path, relative_to)
         else:
             raise ValueError(f"Unsupported file export placement: {placement}")
         return posixpath.join(output, path)  # type: ignore[union-attr]