GH action to generate report #199
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: disk-usage-report-job | ||
| namespace: jupyterhub | ||
| spec: | ||
| backoffLimit: 0 # No retry on failure | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: disk-usage-report | ||
| spec: | ||
| containers: | ||
| - name: disk-usage-report | ||
| image: dandiarchive/dandihub-report-generator:latest | ||
| args: | ||
| - "/home/" | ||
| volumeMounts: | ||
| - name: persistent-storage | ||
| mountPath: "/home" | ||
| subPath: "home" | ||
| restartPolicy: Never | ||
| nodeSelector: | ||
| NodeGroupType: default | ||
| NodePool: default | ||
| hub.jupyter.org/node-purpose: user | ||
| tolerations: | ||
| - key: "hub.jupyter.org/dedicated" | ||
| operator: "Equal" | ||
| value: "user" | ||
| effect: "NoSchedule" | ||
| volumes: | ||
| - name: persistent-storage | ||
| persistentVolumeClaim: | ||
| claimName: efs-persist |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # manifests/hello-world-pod.yaml | ||
| apiVersion: v1 | ||
| kind: Pod | ||
| metadata: | ||
| name: hello-world-pod | ||
| spec: | ||
| containers: | ||
| - name: hello | ||
| image: busybox | ||
| command: ['sh', '-c', 'echo Hello, World! && sleep 30'] | ||
| nodeSelector: | ||
| NodeGroupType: default | ||
| NodePool: default | ||
| hub.jupyter.org/node-purpose: user | ||
| tolerations: | ||
| - key: "hub.jupyter.org/dedicated" | ||
| operator: "Equal" | ||
| value: "user" | ||
| effect: "NoSchedule" | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,63 @@ | ||||||
| #!/usr/bin/env bash | ||||||
|
|
||||||
| set -e | ||||||
|
|
||||||
| # Load environment variables from the file if they are not already set | ||||||
| ENV_FILE=".ec2-session.env" | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| if [ -f "$ENV_FILE" ]; then | ||||||
| echo "Loading environment variables from $ENV_FILE..." | ||||||
| source "$ENV_FILE" | ||||||
| else | ||||||
| echo "Warning: Environment file $ENV_FILE not found." | ||||||
| fi | ||||||
|
|
||||||
| # Ensure required environment variables are set | ||||||
| if [ -z "$INSTANCE_ID" ]; then | ||||||
| echo "Error: INSTANCE_ID is not set. Cannot proceed with cleanup." | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| if [ -z "$ALLOC_ID" ]; then | ||||||
| echo "Error: ALLOC_ID is not set. Cannot proceed with cleanup." | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| # Check for AWS CLI and credentials | ||||||
| if ! command -v aws &>/dev/null; then | ||||||
| echo "Error: AWS CLI is not installed. Please install it and configure your credentials." | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| if ! aws sts get-caller-identity &>/dev/null; then | ||||||
| echo "Error: Unable to access AWS. Ensure your credentials are configured correctly." | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| # Terminate EC2 instance | ||||||
| echo "Terminating EC2 instance with ID: $INSTANCE_ID..." | ||||||
| if aws ec2 terminate-instances --instance-ids "$INSTANCE_ID" --no-cli-pager; then | ||||||
| echo "Instance termination initiated. Waiting for the instance to terminate..." | ||||||
| if aws ec2 wait instance-terminated --instance-ids "$INSTANCE_ID"; then | ||||||
| echo "Instance $INSTANCE_ID has been successfully terminated." | ||||||
| else | ||||||
| echo "Warning: Instance $INSTANCE_ID may not have terminated correctly." | ||||||
| fi | ||||||
| else | ||||||
| echo "Warning: Failed to terminate instance $INSTANCE_ID. It may already be terminated." | ||||||
| fi | ||||||
|
|
||||||
| # Release Elastic IP | ||||||
| echo "Releasing Elastic IP with Allocation ID: $ALLOC_ID..." | ||||||
| if aws ec2 release-address --allocation-id "$ALLOC_ID"; then | ||||||
| echo "Elastic IP with Allocation ID $ALLOC_ID has been successfully released." | ||||||
| else | ||||||
| echo "Warning: Failed to release Elastic IP with Allocation ID $ALLOC_ID. It may already be released." | ||||||
| fi | ||||||
|
|
||||||
| # Cleanup environment file | ||||||
| if [ -f "$ENV_FILE" ]; then | ||||||
| echo "Removing environment file $ENV_FILE..." | ||||||
| rm -f "$ENV_FILE" | ||||||
| fi | ||||||
|
|
||||||
| echo "Cleanup complete." | ||||||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| #!/usr/bin/env python3 | ||
|
|
||
| import os | ||
| import time | ||
| import json | ||
| import sys | ||
| import gzip | ||
| from datetime import datetime | ||
|
|
||
| def list_files_with_metadata(directory, output_file): | ||
|
There was a problem hiding this comment. write a simple test, could even probably be in this file, where you populate directory with nested folders and symlinks and you know the ground truth to aim for and compare against. |
||
| # Record the start time | ||
| start_time = time.time() | ||
|
|
||
| # Get the current date and time for indexing | ||
| index_timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S") | ||
|
|
||
| files_metadata = [] | ||
|
|
||
| for root, dirs, files in os.walk(directory): | ||
|
There was a problem hiding this comment. FTR. walk seems to be already doing desired (right) thing and does not follow symlinked folders. We get for note: we do not monitor empty folders below |
||
| for name in files: | ||
| filepath = os.path.join(root, name) | ||
| relative_path = os.path.relpath(filepath, directory) | ||
|
|
||
| try: | ||
| metadata = { | ||
| "path": relative_path, | ||
| "size": os.path.getsize(filepath), | ||
asmacdo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "modified": time.ctime(os.path.getmtime(filepath)), | ||
| "created": time.ctime(os.path.getctime(filepath)) | ||
| } | ||
| files_metadata.append(metadata) | ||
| except (FileNotFoundError, PermissionError) as e: | ||
| print(f"Skipping {filepath}: {e}") | ||
asmacdo marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # Record the end time and calculate the duration | ||
| end_time = time.time() | ||
| duration = end_time - start_time | ||
|
|
||
| # Prepare the output data with additional metadata | ||
| output_data = { | ||
| "index_timestamp": index_timestamp, | ||
| "duration_seconds": duration, | ||
| "files": files_metadata | ||
| } | ||
|
|
||
| # Compress and write the output data to a .json.gz file | ||
| with gzip.open(output_file, "wt", encoding="utf-8") as gz_file: | ||
| json.dump(output_data, gz_file, indent=4) | ||
|
|
||
| print(f"Indexing completed. Compressed results written to {output_file}") | ||
|
|
||
| # Ensure the script is called with the required arguments | ||
| if __name__ == "__main__": | ||
| if len(sys.argv) != 3: | ||
| print("Usage: python script.py <directory_to_index> <output_json_gz_file>") | ||
| sys.exit(1) | ||
|
|
||
| directory_to_index = sys.argv[1] | ||
| output_json_gz_file = sys.argv[2] | ||
|
|
||
| # Ensure the output filename ends with .json.gz for clarity | ||
| if not output_json_gz_file.endswith(".json.gz"): | ||
| output_json_gz_file += ".json.gz" | ||
|
|
||
| list_files_with_metadata(directory_to_index, output_json_gz_file) | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,133 @@ | ||||||
| #!/usr/bin/env bash | ||||||
|
|
||||||
| set -e | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| # Check for AWS CLI and credentials | ||||||
| if ! command -v aws &>/dev/null; then | ||||||
| echo "Error: AWS CLI is not installed. Please install it and configure your credentials." | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| if ! aws sts get-caller-identity &>/dev/null; then | ||||||
| echo "Error: Unable to access AWS. Ensure your credentials are configured correctly." | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| # Set variables | ||||||
| AWS_REGION="us-east-2" | ||||||
| # TODO document that this key needs to be created | ||||||
| KEY_NAME="dandihub-gh-actions" | ||||||
| # TODO create if DNE | ||||||
| # allow gh-actions to ssh into ec2 job instance from anywhere | ||||||
| SECURITY_GROUP_ID="sg-0bf2dc1c2ff9c122e" | ||||||
| # TODO retrieve subnet id (public, created by dandi-hub eks-dandihub-public-us-east-2a) | ||||||
| SUBNET_ID="subnet-0f544cca61ccd2804" | ||||||
| AMI_ID="ami-088d38b423bff245f" | ||||||
| EFS_ID="fs-02aac16c4c6c2dc27" | ||||||
| LOCAL_SCRIPTS_DIR=".github/scripts" | ||||||
| REMOTE_SCRIPTS_DIR="/home/ec2-user/scripts" | ||||||
| MOUNT_POINT="/mnt/efs" | ||||||
| ENV_FILE=".ec2-session.env" | ||||||
|
There was a problem hiding this comment. I dislike the fact that it would be just dumping into some hidden file in my current directory.
Suggested change
but then we might want to add logic to react if file already exist -- since that would mean likely that cleanup did not remove it and instance might still be running etc. |
||||||
|
|
||||||
| # Ensure the environment file is writable | ||||||
| echo "# Environment variables for EC2 session" > $ENV_FILE | ||||||
| echo "# Auto-generated by launch script on $(date)" >> $ENV_FILE | ||||||
|
|
||||||
| # Run EC2 instance | ||||||
| echo "Launching EC2 instance..." | ||||||
| export INSTANCE_ID=$(aws ec2 run-instances \ | ||||||
| --image-id $AMI_ID \ | ||||||
| --count 1 \ | ||||||
| --instance-type t3.micro \ | ||||||
| --key-name $KEY_NAME \ | ||||||
| --security-group-ids $SECURITY_GROUP_ID \ | ||||||
| --subnet-id $SUBNET_ID \ | ||||||
| --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=dandihub-gh-actions}]" \ | ||||||
| --query 'Instances[0].InstanceId' \ | ||||||
| --output text) | ||||||
|
|
||||||
| if [ -z "$INSTANCE_ID" ]; then | ||||||
| echo "Error: Failed to launch EC2 instance." | ||||||
| exit 1 | ||||||
| fi | ||||||
| echo "Instance ID: $INSTANCE_ID" | ||||||
| echo "export INSTANCE_ID=$INSTANCE_ID" >> $ENV_FILE | ||||||
|
|
||||||
| # Wait for instance to initialize | ||||||
| echo "Waiting for instance to reach status OK..." | ||||||
| aws ec2 wait instance-status-ok --instance-ids "$INSTANCE_ID" | ||||||
|
|
||||||
| # Allocate Elastic IP | ||||||
| echo "Allocating Elastic IP..." | ||||||
| export ALLOC_ID=$(aws ec2 allocate-address \ | ||||||
| --tag-specifications "ResourceType=elastic-ip,Tags=[{Key=Name,Value=dandihub-gh-actions-eip}]" \ | ||||||
| --query 'AllocationId' \ | ||||||
| --output text) | ||||||
|
|
||||||
| if [ -z "$ALLOC_ID" ]; then | ||||||
| echo "Error: Failed to allocate Elastic IP." | ||||||
| exit 1 | ||||||
| fi | ||||||
| echo "Elastic IP Allocation ID: $ALLOC_ID" | ||||||
| echo "export ALLOC_ID=$ALLOC_ID" >> $ENV_FILE | ||||||
|
|
||||||
| # Associate Elastic IP with instance | ||||||
| echo "Associating Elastic IP with instance..." | ||||||
| export EIP_ASSOC=$(aws ec2 associate-address \ | ||||||
| --instance-id "$INSTANCE_ID" \ | ||||||
| --allocation-id "$ALLOC_ID" \ | ||||||
| --query 'AssociationId' \ | ||||||
| --output text) | ||||||
|
|
||||||
| if [ -z "$EIP_ASSOC" ]; then | ||||||
| echo "Error: Failed to associate Elastic IP." | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| # Get Elastic IP address | ||||||
| export PUBLIC_IP=$(aws ec2 describe-addresses \ | ||||||
| --allocation-ids "$ALLOC_ID" \ | ||||||
| --query 'Addresses[0].PublicIp' \ | ||||||
| --output text) | ||||||
|
|
||||||
| echo "Elastic IP Address: $PUBLIC_IP" | ||||||
| echo "export PUBLIC_IP=$PUBLIC_IP" >> $ENV_FILE | ||||||
|
|
||||||
| # Upload scripts to EC2 instance | ||||||
| echo "Uploading scripts to EC2 instance..." | ||||||
| scp -i "$EC2_SSH_KEY" -o "StrictHostKeyChecking=no" \ | ||||||
| $LOCAL_SCRIPTS_DIR/produce-report.py $LOCAL_SCRIPTS_DIR/create-file-index.py \ | ||||||
| ec2-user@"$PUBLIC_IP":"$REMOTE_SCRIPTS_DIR/" | ||||||
|
|
||||||
| if [ $? -eq 0 ]; then | ||||||
| echo "Scripts uploaded successfully to $REMOTE_SCRIPTS_DIR on the instance." | ||||||
| else | ||||||
| echo "Error: Failed to upload scripts to the instance." | ||||||
| exit 1 | ||||||
| fi | ||||||
|
|
||||||
| # TODO automate | ||||||
| # eks-dandihub-efs sg is created by dandi-hub install | ||||||
| # this sg needs to accept incoming 2049 from the sg created for this ec2 | ||||||
| # sg-061d875722e569724 - eks-dandihub-efs | ||||||
| # aws ec2 authorize-security-group-ingress \ | ||||||
| # --group-id sg-061d875722e569724 \ | ||||||
| # --protocol tcp \ | ||||||
| # --port 2049 \ | ||||||
| # --source-group $SECURITY_GROUP_ID | ||||||
|
|
||||||
| # Mount EFS on the EC2 instance | ||||||
| echo "Mounting EFS on the EC2 instance..." | ||||||
| ssh -i "$EC2_SSH_KEY" -o "StrictHostKeyChecking=no" ec2-user@"$PUBLIC_IP" \ | ||||||
| "sudo yum install -y amazon-efs-utils && \ | ||||||
| sudo mkdir -p $MOUNT_POINT && \ | ||||||
| sudo mount -t efs $EFS_ID:/ $MOUNT_POINT && \ | ||||||
| echo '$EFS_ID:/ $MOUNT_POINT efs defaults,_netdev 0 0' | sudo tee -a /etc/fstab && \ | ||||||
| echo 'EFS mounted at $MOUNT_POINT'" | ||||||
|
|
||||||
| # Output SSH command for convenience | ||||||
| echo "To connect to your instance, use:" | ||||||
| echo "ssh -i \$EC2_SSH_KEY ec2-user@$PUBLIC_IP" | ||||||
|
|
||||||
| echo "Environment variables saved to $ENV_FILE." | ||||||
| echo "Run 'source $ENV_FILE' to restore the environment variables." | ||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.