Skip to content

Commit 9fc7f13

Browse files
committed
Added python script to extract workload from trace_func.py
Modified test file to check ZipHealth and Cleanup of local files, added E2E test for AzureFunctions Signed-off-by: Kavithran <104263022+cavinkavi@users.noreply.github.com>
1 parent eaa1d2d commit 9fc7f13

16 files changed

+829
-4
lines changed

.github/workflows/e2e_azure.yaml

+71
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
name: End-to-End Azure Functions Tests
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main # Trigger the workflow when the PR targets the main branch
7+
workflow_dispatch: # Allows manual triggering of the workflow
8+
9+
env:
10+
GOOS: linux
11+
GO111MODULE: on
12+
13+
jobs:
14+
test-azure:
15+
name: Test Azure Functions Cloud Deployment
16+
runs-on: ubuntu-20.04
17+
env:
18+
AZURE_APP_ID: ${{ secrets.AZURE_APP_ID }}
19+
AZURE_PASSWORD: ${{ secrets.AZURE_PASSWORD }}
20+
AZURE_TENANT: ${{ secrets.AZURE_TENANT }}
21+
22+
steps:
23+
- name: Check if environment variables are set # Validate secrets are passed
24+
run: |
25+
if [[ -z "$AZURE_APP_ID" ]]; then
26+
echo "AZURE_APP_ID is not set. Please check if secrets.AZURE_APP_ID is in the repository."
27+
exit 1
28+
fi
29+
if [[ -z "$AZURE_PASSWORD" ]]; then
30+
echo "AZURE_PASSWORD is not set. Please check if secrets.AZURE_PASSWORD is in the repository."
31+
exit 1
32+
fi
33+
if [[ -z "$AZURE_TENANT" ]]; then
34+
echo "AZURE_TENANT is not set. Please check if secrets.AZURE_TENANT is in the repository."
35+
exit 1
36+
fi
37+
38+
- name: Checkout GitHub Repository
39+
uses: actions/checkout@v4
40+
with:
41+
lfs: true
42+
43+
- name: Install Azure CLI
44+
run: |
45+
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
46+
az --version
47+
48+
- name: Install Golang
49+
uses: actions/setup-go@v5
50+
with:
51+
go-version: 1.21
52+
53+
- name: Set up Python 3.10
54+
uses: actions/setup-python@v4
55+
with:
56+
python-version: '3.10'
57+
58+
- name: Run Zip File Health Test
59+
run: go test -v ./pkg/driver/deployment/azure_functions_test.go -run TestZipHealth
60+
61+
- name: Run Cleanup Test
62+
run: go test -v ./pkg/driver/deployment/azure_functions_test.go -run TestCleanup
63+
64+
- name: Azure CLI Login Using Service Principal
65+
run: az login --service-principal --username $AZURE_APP_ID --password $AZURE_PASSWORD --tenant $AZURE_TENANT
66+
67+
- name: Build and Run Loader
68+
run: go run cmd/loader.go --config cmd/config_azure_trace.json
69+
70+
- name: Check the output
71+
run: test -f "data/out/experiment_duration_5.csv" && test $(grep true data/out/experiment_duration_5.csv | wc -l) -eq 0 # test the output file for errors (true means failure to invoke)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# azurefunctionsconfig.yaml
2+
azurefunctionsconfig:
3+
resource_group: ExperimentResourceGroup # Name of the resource group
4+
storage_account_name: invitrostorage # Name of the storage account
5+
function_app_name: invitrofunctionapp # Name of the function app
6+
location: EastUS # Region where resource created
+66
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
import re
2+
import os
3+
4+
#Script to extract execute_function from trace_func.py and inject it into azureworkload.py
5+
6+
# Paths to source and target workload files
7+
TRACE_FUNC_PATH = "server/trace-func-py/trace_func.py"
8+
AZURE_WORKLOAD_PATH = "azurefunctions_setup/shared_azure_workload/azurefunctionsworkload.py"
9+
10+
def extract_execute_function(src_path):
11+
# Extract the execute_function logic from trace_func.py.
12+
with open(src_path, "r") as f:
13+
content = f.read()
14+
15+
# Use regex to extract the execute_function definition and body
16+
match = re.search(r"def execute_function\(.*?\):.*?(?=def |\Z)", content, re.DOTALL)
17+
if not match:
18+
raise ValueError("execute_function() not found in trace_func.py")
19+
20+
return match.group(0)
21+
22+
def inject_function(func_code, workload_path):
23+
# Inject or replace execute_function in azureworkload.py.
24+
with open(workload_path, "r") as f:
25+
content = f.read()
26+
27+
# Check if execute_function already exists and replace it
28+
if "def execute_function" in content:
29+
updated_content = re.sub(
30+
r"def execute_function\(.*?\):.*?(?=def |\Z)", # Match existing function
31+
func_code, # Replace with new function code
32+
content,
33+
flags=re.DOTALL,
34+
)
35+
else:
36+
# Add execute_function at the end of the file
37+
updated_content = f"{content}\n\n{func_code}"
38+
39+
# Write updated content back to the file
40+
with open(workload_path, "w") as f:
41+
f.write(updated_content)
42+
43+
44+
def validate_injection(workload_path):
45+
# Validate that execute_function is present in azureworkload.py after injection.
46+
with open(workload_path, "r") as f:
47+
content = f.read()
48+
49+
if "def execute_function" not in content:
50+
raise RuntimeError("Injection failed: execute_function() not found in azureworkload.py")
51+
52+
53+
if __name__ == "__main__":
54+
try:
55+
# Extract execute_function from trace_func.py
56+
execute_function_code = extract_execute_function(TRACE_FUNC_PATH)
57+
58+
# Inject the extracted function into azureworkload.py
59+
inject_function(execute_function_code, AZURE_WORKLOAD_PATH)
60+
61+
# Validate the injection
62+
validate_injection(AZURE_WORKLOAD_PATH)
63+
64+
except Exception as e:
65+
print(f"Error during workload injection: {e}")
66+
exit(1)

azurefunctions_setup/host.json

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"version": "2.0",
3+
"logging": {
4+
"applicationInsights": {
5+
"samplingSettings": {
6+
"isEnabled": true,
7+
"excludedTypes": "Request"
8+
}
9+
}
10+
},
11+
"extensionBundle": {
12+
"id": "Microsoft.Azure.Functions.ExtensionBundle",
13+
"version": "[4.*, 5.0.0)"
14+
}
15+
16+
}
+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"IsEncrypted": false,
3+
"Values": {
4+
"FUNCTIONS_WORKER_RUNTIME": "python",
5+
"AzureWebJobsFeatureFlags": "EnableWorkerIndexing",
6+
"AzureWebJobsStorage": ""
7+
}
8+
9+
}

azurefunctions_setup/requirements.txt

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
azure-functions
2+
numpy>=1.21,<1.26
3+
psutil>=5.9,<6.0
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
import time
2+
import socket
3+
import json
4+
import azure.functions as func
5+
import logging
6+
from time import process_time_ns
7+
import math
8+
from psutil import virtual_memory
9+
from numpy import empty, float32
10+
11+
# Global variable for hostname
12+
hostname = socket.gethostname()
13+
14+
# Placeholder for `execute_function`, retrieve according to server/trace-func-py/trace_func.py
15+
def execute_function(input, runTime, totalMem):
16+
startTime = process_time_ns()
17+
18+
chunkSize = 2**10 # size of a kb or 1024
19+
totalMem = totalMem*(2**10) # convert Mb to kb
20+
memory = virtual_memory()
21+
used = (memory.total - memory.available) // chunkSize # convert to kb
22+
additional = max(1, (totalMem - used))
23+
array = empty(additional*chunkSize, dtype=float32) # make an uninitialized array of that size, uninitialized to keep it fast
24+
# convert to ns
25+
runTime = (runTime - 1)*(10**6) # -1 because it should be slighly bellow that runtime
26+
memoryIndex = 0
27+
while process_time_ns() - startTime < runTime:
28+
for i in range(0, chunkSize):
29+
sin_i = math.sin(i)
30+
cos_i = math.cos(i)
31+
sqrt_i = math.sqrt(i)
32+
array[memoryIndex + i] = sin_i
33+
memoryIndex = (memoryIndex + chunkSize) % additional*chunkSize
34+
return (process_time_ns() - startTime) // 1000
35+
36+
def main(req: func.HttpRequest) -> func.HttpResponse:
37+
logging.info("Processing request.")
38+
39+
start_time = time.time()
40+
41+
# Parse JSON request body
42+
try:
43+
req_body = req.get_json()
44+
logging.info(f"Request body: {req_body}")
45+
except ValueError:
46+
logging.error("Invalid JSON received.")
47+
return func.HttpResponse(
48+
json.dumps({"error": "Invalid JSON"}),
49+
status_code=400,
50+
mimetype="application/json"
51+
)
52+
53+
runtime_milliseconds = req_body.get('RuntimeInMilliSec', 1000)
54+
memory_mebibytes = req_body.get('MemoryInMebiBytes', 128)
55+
56+
logging.info(f"Runtime requested: {runtime_milliseconds} ms, Memory: {memory_mebibytes} MiB")
57+
58+
# Directly call the execute_function
59+
duration = execute_function("",runtime_milliseconds,memory_mebibytes)
60+
result_msg = f"Workload completed in {duration} microseconds"
61+
62+
# Prepare the response
63+
response = {
64+
"Status": "Success",
65+
"Function": req.url.split("/")[-1],
66+
"MachineName": hostname,
67+
"ExecutionTime": int((time.time() - start_time) * 1_000_000), # Total time (includes HTTP, workload, and response prep)
68+
"DurationInMicroSec": duration, # Time spent on the workload itself
69+
"MemoryUsageInKb": memory_mebibytes * 1024,
70+
"Message": result_msg
71+
}
72+
73+
logging.info(f"Response: {response}")
74+
75+
return func.HttpResponse(
76+
json.dumps(response),
77+
status_code=200,
78+
mimetype="application/json"
79+
)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"bindings": [
3+
{
4+
"authLevel": "anonymous",
5+
"type": "httpTrigger",
6+
"direction": "in",
7+
"name": "req",
8+
"methods": ["post"]
9+
},
10+
{
11+
"type": "http",
12+
"direction": "out",
13+
"name": "$return"
14+
}
15+
],
16+
"scriptFile": "azurefunctionsworkload.py"
17+
}
18+

cmd/config_azure_trace.json

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"Seed": 42,
3+
4+
"Platform": "AzureFunctions",
5+
"InvokeProtocol" : "http1",
6+
"EndpointPort": 80,
7+
8+
"BusyLoopOnSandboxStartup": false,
9+
10+
"TracePath": "data/traces/example",
11+
"Granularity": "minute",
12+
"OutputPathPrefix": "data/out/experiment",
13+
"IATDistribution": "exponential",
14+
"CPULimit": "1vCPU",
15+
"ExperimentDuration": 5,
16+
"WarmupDuration": 0,
17+
18+
"IsPartiallyPanic": false,
19+
"EnableZipkinTracing": false,
20+
"EnableMetricsScrapping": false,
21+
"MetricScrapingPeriodSeconds": 15,
22+
"AutoscalingMetric": "concurrency",
23+
24+
"GRPCConnectionTimeoutSeconds": 15,
25+
"GRPCFunctionTimeoutSeconds": 900,
26+
27+
"DAGMode": false
28+
}

cmd/loader.go

+2-1
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ func main() {
9696
"AWSLambda",
9797
"Dirigent",
9898
"Dirigent-Dandelion",
99+
"AzureFunctions",
99100
}
100101

101102
if !slices.Contains(supportedPlatforms, cfg.Platform) {
@@ -151,7 +152,7 @@ func parseYAMLSpecification(cfg *config.LoaderConfiguration) string {
151152
case "firecracker":
152153
return "workloads/firecracker/trace_func_go.yaml"
153154
default:
154-
if cfg.Platform != "Dirigent" && cfg.Platform != "Dirigent-Dandelion" {
155+
if cfg.Platform != "Dirigent" && cfg.Platform != "Dirigent-Dandelion" && cfg.Platform != "AzureFunctions" {
155156
log.Fatal("Invalid 'YAMLSelector' parameter.")
156157
}
157158
}

docs/loader.md

+40-1
Original file line numberDiff line numberDiff line change
@@ -262,4 +262,43 @@ Note:
262262
- Under `Manage Quota`, select `AWS Lambda` service and click `View quotas` (Alternatively, click [here](https://us-east-1.console.aws.amazon.com/servicequotas/home/services/lambda/quotas))
263263
- Under `Quota name`, select `Concurrent executions` and click `Request increase at account level` (Alternatively, click [here](https://us-east-1.console.aws.amazon.com/servicequotas/home/services/lambda/quotas/L-B99A9384))
264264
- Under `Increase quota value`, input `1000` and click `Request`
265-
- Await AWS Support Team to approve the request. The request may take several days or weeks to be approved.
265+
- Await AWS Support Team to approve the request. The request may take several days or weeks to be approved.
266+
267+
## Using Azure Functions
268+
269+
**Pre-requisites:**
270+
1. Microsoft Azure account with an active subscription ID
271+
2. Go installed
272+
3. Python3 installed
273+
274+
**Quick Setup for Azure Deployment:**
275+
1. Install the Azure CLI and verify installation:
276+
```bash
277+
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
278+
az --version
279+
```
280+
2. Log in as a user (Note: This will open a browser window to select Azure account):
281+
```bash
282+
az login
283+
```
284+
3. Create an Azure Service Principal:
285+
```bash
286+
az ad sp create-for-rbac --name "InVitro" --role Contributor --scopes /subscriptions/<your-subscription-id>
287+
```
288+
4. Set the following environment values:
289+
```bash
290+
export AZURE_APP_ID=<appId>
291+
export AZURE_PASSWORD=<password>
292+
export AZURE_TENANT=<tenant>
293+
```
294+
5. Use the Service Principal credentials in order to run automated scripts without manual login:
295+
```bash
296+
az login --service-principal --username $AZURE_APP_ID --password $AZURE_PASSWORD --tenant $AZURE_TENANT
297+
```
298+
6. Start the Azure Functions deployment experiment:
299+
```bash
300+
go run cmd/loader.go --config cmd/config_azure_trace.json
301+
```
302+
---
303+
Note:
304+
- Current deployment is via ZIP

go.mod

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,9 @@ require (
1717

1818
require (
1919
github.com/aws/aws-lambda-go v1.47.0
20-
github.com/stretchr/testify v1.10.0
2120
github.com/containerd/log v0.1.0
2221
github.com/google/uuid v1.6.0
22+
github.com/stretchr/testify v1.10.0
2323
github.com/vhive-serverless/vSwarm/utils/protobuf/helloworld v0.0.0-20240827121957-11be651eb39a
2424
github.com/vhive-serverless/vSwarm/utils/tracing/go v0.0.0-20240827121957-11be651eb39a
2525
go.mongodb.org/mongo-driver v1.17.1

0 commit comments

Comments
 (0)