Merge pull request #108 from Jonathan-Walsh-Amey/amey_main

Update CosmosDataSinkExtension.cs to enable support for shared throughput in Cosmos containers

Author: bowencode

Extensions/Cosmos/Cosmos.DataTransfer.CosmosExtension/CosmosDataSinkExtension.cs

@@ -59,7 +59,7 @@ public async Task WriteAsync(IAsyncEnumerable<IDataItem> dataItems, IConfigurati
                     containerProperties.PartitionKeyPath = settings.PartitionKeyPath;
                 }
 
-                ThroughputProperties? throughputProperties = settings.IsServerlessAccount
+                ThroughputProperties? throughputProperties = settings.IsServerlessAccount || settings.UseSharedThroughput
                     ? null
                     : settings.UseAutoscaleForCreatedContainer
                     ? ThroughputProperties.CreateAutoscaleThroughput(settings.CreatedContainerMaxThroughput ?? 4000)
@@ -211,4 +211,4 @@ public record ItemResult(string? Id, HttpStatusCode StatusCode)
             public int ItemCount => IsSuccess ? 1 : 0;
         }
     }
-}
+}

Extensions/Cosmos/Cosmos.DataTransfer.CosmosExtension/CosmosSinkSettings.cs

@@ -13,6 +13,7 @@ public class CosmosSinkSettings : CosmosSettingsBase, IDataExtensionSettings
         public int? CreatedContainerMaxThroughput { get; set; }
         public bool UseAutoscaleForCreatedContainer { get; set; } = true;
         public bool IsServerlessAccount { get; set; } = false;
+        public bool UseSharedThroughput { get; set; } = false;
         public DataWriteMode WriteMode { get; set; } = DataWriteMode.Insert;
         public List<string>? PartitionKeyPaths { get; set; }
 

Extensions/Cosmos/README.md

@@ -44,7 +44,7 @@ Or with RBAC:
 }
 ```
 
-Sink requires an additional `PartitionKeyPath` parameter which is used when creating the container if it does not exist. To use hierarchical partition keys, instead use the `PartitionKeyPaths` setting to supply an array of up to 3 paths. It also supports an optional `RecreateContainer` parameter (`false` by default) to delete and then recreate the container to ensure only newly imported data is present. The optional `BatchSize` parameter (100 by default) sets the number of items to accumulate before inserting. `ConnectionMode` can be set to either `Gateway` (default) or `Direct` to control how the client connects to the CosmosDB service. For situations where a container is created as part of the transfer operation `CreatedContainerMaxThroughput` (in RUs) and `UseAutoscaleForCreatedContainer` provide the initial throughput settings which will be in effect when executing the transfer. The optional `WriteMode` parameter specifies the type of data write to use: `InsertStream`, `Insert`, `UpsertStream`, or `Upsert`. The `IsServerlessAccount` parameter specifies whether the target account uses Serverless instead of Provisioned throughput, which affects the way containers are created. Additional parameters allow changing the behavior of the Cosmos client appropriate to your environment.
+Sink requires an additional `PartitionKeyPath` parameter which is used when creating the container if it does not exist. To use hierarchical partition keys, instead use the `PartitionKeyPaths` setting to supply an array of up to 3 paths. It also supports an optional `RecreateContainer` parameter (`false` by default) to delete and then recreate the container to ensure only newly imported data is present. The optional `BatchSize` parameter (100 by default) sets the number of items to accumulate before inserting. `ConnectionMode` can be set to either `Gateway` (default) or `Direct` to control how the client connects to the CosmosDB service. For situations where a container is created as part of the transfer operation `CreatedContainerMaxThroughput` (in RUs) and `UseAutoscaleForCreatedContainer` provide the initial throughput settings which will be in effect when executing the transfer. To instead use shared throughput that has been provisioned at the database level, set the `UseSharedThroughput` parameter to `true`. The optional `WriteMode` parameter specifies the type of data write to use: `InsertStream`, `Insert`, `UpsertStream`, or `Upsert`. The `IsServerlessAccount` parameter specifies whether the target account uses Serverless instead of Provisioned throughput, which affects the way containers are created. Additional parameters allow changing the behavior of the Cosmos client appropriate to your environment.
 
 ### Sink
 
@@ -62,6 +62,7 @@ Sink requires an additional `PartitionKeyPath` parameter which is used when crea
     "CreatedContainerMaxThroughput": 1000,
     "UseAutoscaleForCreatedContainer": true,
     "WriteMode": "InsertStream",
-    "IsServerlessAccount": false
+    "IsServerlessAccount": false,
+    "UseSharedThroughput": false
 }
 ```