Skip to content

Nsight Profiling to Phoenix Benchmark Cases #929

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Nsight Profiling to Phoenix Benchmark Cases #929

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
658f82e
added nsight profiling to phoenix benchmark cases
Malmahrouqi3 Jul 8, 2025
60266bb
Merge branch 'master' into nsys-profiling
Malmahrouqi3 Jul 9, 2025
122ce46
Merge branch 'master' into nsys-profiling
sbryngelson Jul 12, 2025
d5969b3
Merge branch 'master' into nsys-profiling
Malmahrouqi3 Jul 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/workflows/bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,31 @@ jobs:
(cd pr && . ./mfc.sh load -c ${{ matrix.flag }} -m g)
(cd pr && ./mfc.sh bench_diff ../master/bench-${{ matrix.device }}.yaml ../pr/bench-${{ matrix.device }}.yaml)

- name: Process Nsight Profiling Report
run: |
if [ -f "pr/report.nsys-rep" ]; then
echo "=== Nsight Profiling Summary ==="
echo "Master"
(cd master && nsys stats --report nvtx_sum report.nsys-rep)
echo "Pr"
(cd pr && nsys stats --report nvtx_sum report.nsys-rep)

echo "=== CUDA API CALLS ==="
echo "Master"
(cd master && nsys stats --report cuda_api_sum --format table report.nsys-rep | head -100)
echo "Pr"
(cd pr && nsys stats --report cuda_api_sum --format table report.nsys-rep | head -100)

echo "=== GPU KERNELS ==="
echo "Master"
(cd master && nsys stats --report cuda_gpu_kern_sum --format table report.nsys-rep | head -100)
echo "Pr"
(cd pr && nsys stats --report cuda_gpu_kern_sum --format table report.nsys-rep | head -100)

else
echo "No Nsight report found, skipping profiling analysis"
fi

- name: Print Logs
if: always()
run: |
Expand All @@ -107,5 +132,6 @@ jobs:
path: |
pr/bench-${{ matrix.device }}.*
pr/build/benchmarks/*
pr/report.nsys-rep
master/bench-${{ matrix.device }}.*
master/build/benchmarks/*
4 changes: 2 additions & 2 deletions .github/workflows/phoenix/bench.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ mkdir -p $currentdir
export TMPDIR=$currentdir

if [ "$device" = "gpu" ]; then
./mfc.sh bench --mem 12 -j $(nproc) -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks
nsys profile -o report ./mfc.sh bench --mem 12 -j $(nproc) -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks
else
./mfc.sh bench --mem 1 -j $(nproc) -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks
nsys profile -o report ./mfc.sh bench --mem 1 -j $(nproc) -o "$job_slug.yaml" -- -c phoenix-bench $device_opts -n $n_ranks
fi

sleep 10
Expand Down
Loading