Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre calculated Machine energy profiles #76

Merged
merged 77 commits into from
Jun 18, 2024
Merged

Pre calculated Machine energy profiles #76

merged 77 commits into from
Jun 18, 2024

Conversation

ArneTR
Copy link
Member

@ArneTR ArneTR commented Jun 4, 2024

No description provided.

Copy link

github-actions bot commented Jun 4, 2024

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run 28.5882 136.957 3.70154 70
Sleep 3s 2.9 3.53779 1.7689 24
ls -alhR /usr/lib 31.7416 128.113 4.00352 33
Sleep 3s 1.17 5.30669 1.7689 3

📈 Energy graph:

 
 4.40 ┤                ╭╮
 4.14 ┤ ╭╮   ╭╮        ││            ╭──╮
 3.87 ┤ │╰───╯╰────────╯╰────────────╯  │
 3.61 ┤ │                               │
 3.35 ┤ │                               │
 3.08 ┤ │                               │
 2.82 ┤ │                               │
 2.56 ┤ │                               │
 2.29 ┤ │                               │
 2.03 ┤ │                               │
 1.77 ┼─╯                               ╰──
                  Watts over time

🌳 CO2 Data:
City: Boydton, Lat: 36.677696, Lon: -78.37471
CO₂ from energy is: .05930238100000000000
CO₂ from manufacturing (embodied carbon) is: .01997196656202434362
Carbon Intensity for this location: 433 gCO₂eq/kWh
SCI: 0.079274 gCO₂eq / pipeline run emitted

ArneTR added 2 commits June 13, 2024 21:09
* main:
  Removing old GitHub runners
  Update README.md
  Changes the ip resolver from ip-api.com to ipapi.com because of https (#78)
@ArneTR
Copy link
Member Author

ArneTR commented Jun 14, 2024

The overhead is now down from 23s to 5s

It is unclear to me why the step for a measurement takes 5s. In local testing the time is less than 200ms

Even when I look at the detail logs the time seems to be less than 1s.

Screenshot 2024-06-14 at 8 30 26 AM

Copy link

github-actions bot commented Jun 14, 2024

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 5.51167 52.515 1.69 31
Sleep 3s 3.17667 14.1241 2.02 7
ls -alhR /usr/lib 12.1875 22.9073 2.86 8
Sleep 3s 1.32714 15.4836 1.94 8

🌳 CO2 Data:
City: Chicago, Lat: 41.8874, Lon: -87.6318
CO₂ from energy is: 0.022318875
CO₂ from manufacturing (embodied carbon) is: 0.008844728
Carbon Intensity for this location: 425 gCO₂eq/kWh
SCI: 0.031164 gCO₂eq / pipeline run emitted

Copy link

github-actions bot commented Jun 14, 2024

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 12.1103 85.3859 2.19 39
Sleep 3s 5.62143 19.762 2.82 7
ls -alhR /usr/lib 20.0214 49.3388 3.08 16
Sleep 3s 2.85 16.2851 2.04 8

🌳 CO2 Data:
City: Boydton, Lat: 36.6534, Lon: -78.375
CO₂ from energy is: 0.036288625
CO₂ from manufacturing (embodied carbon) is: 0.011127239
Carbon Intensity for this location: 425 gCO₂eq/kWh
SCI: 0.047416 gCO₂eq / pipeline run emitted

Copy link

github-actions bot commented Jun 14, 2024

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 12.469 82.6879 2.12 39
Sleep 3s 5.69 18.1724 2.60 7
ls -alhR /usr/lib 20.3871 48.8222 3.25 15
Sleep 3s 4.17571 15.6933 1.74 9

🌳 CO2 Data:
City: Chicago, Lat: 41.8874, Lon: -87.6318
CO₂ from energy is: 0.035141975
CO₂ from manufacturing (embodied carbon) is: 0.011127239
Carbon Intensity for this location: 425 gCO₂eq/kWh
SCI: 0.046269 gCO₂eq / pipeline run emitted

@ArneTR
Copy link
Member Author

ArneTR commented Jun 14, 2024

I found the culprit. Aparently sending a process to the background makes GitHub steps hang for a while of the input and output streams are not handled.

This code part fixes it: 4425e1b

Brought it now also to this branch

Copy link

github-actions bot commented Jun 14, 2024

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 9.12625 19.338 1.93 10
Sleep 3s 6.565 5.64411 2.82 2
ls -alhR /usr/lib 24.865 7.06576 3.53 2
Sleep 3s 1.32 6.62815 2.21 3

🌳 CO2 Data:
City: San Jose, Lat: 37.1835, Lon: -121.7714
CO₂ from energy is: 0.001295646
CO₂ from manufacturing (embodied carbon) is: 0.002853138
Carbon Intensity for this location: 67 gCO₂eq/kWh
SCI: 0.004149 gCO₂eq / pipeline run emitted

Copy link

github-actions bot commented Jun 14, 2024

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 21.3288 53.699 2.98 18
Sleep 3s 6.715 6.574 3.29 2
ls -alhR /usr/lib 33.7367 39.4299 3.94 10
Sleep 3s 2.37 7.69506 2.57 3

🌳 CO2 Data:
City: Boydton, Lat: 36.6534, Lon: -78.375
CO₂ from energy is: 0.023573861
CO₂ from manufacturing (embodied carbon) is: 0.005135649
Carbon Intensity for this location: 439 gCO₂eq/kWh
SCI: 0.028710 gCO₂eq / pipeline run emitted

Copy link

github-actions bot commented Jun 14, 2024

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 19.3417 52.2195 2.75 19
Sleep 3s 8.21 6.8245 3.41 2
ls -alhR /usr/lib 28.8633 37.7405 3.77 10
Sleep 3s 2.44333 7.65455 1.91 4

🌳 CO2 Data:
City: Phoenix, Lat: 33.4475, Lon: -112.0866
CO₂ from energy is: 0.015665700
CO₂ from manufacturing (embodied carbon) is: 0.005420962
Carbon Intensity for this location: 300 gCO₂eq/kWh
SCI: 0.021087 gCO₂eq / pipeline run emitted

Copy link

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 14.757 23.0745 2.10 11
Sleep 3s 6.445 5.65728 2.83 2
ls -alhR /usr/lib 26.5233 11.2577 2.81 4
Sleep 3s 0.856667 6.15952 2.05 3

🌳 CO2 Data:
City: Washington, Lat: 38.7095, Lon: -78.1539
CO₂ from energy is: 0.010129486
CO₂ from manufacturing (embodied carbon) is: 0.003138452
Carbon Intensity for this location: 439 gCO₂eq/kWh
SCI: 0.013268 gCO₂eq / pipeline run emitted

Copy link

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 23.6765 56.5316 2.98 19
Sleep 3s 10.005 7.25165 3.63 2
ls -alhR /usr/lib 36.5856 41.2396 4.12 10
Sleep 3s 4.68 8.04038 2.68 3

🌳 CO2 Data:
City: San Jose, Lat: 37.1835, Lon: -121.7714
CO₂ from energy is: 0.003787577
CO₂ from manufacturing (embodied carbon) is: 0.005420962
Carbon Intensity for this location: 67 gCO₂eq/kWh
SCI: 0.009209 gCO₂eq / pipeline run emitted

Copy link

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 23.9335 56.6945 3.15 18
Sleep 3s 13.445 7.49903 3.75 2
ls -alhR /usr/lib 35.6756 40.706 4.07 10
Sleep 3s 6.97667 8.48949 2.83 3

🌳 CO2 Data:
City: San Jose, Lat: 37.1835, Lon: -121.7714
CO₂ from energy is: 0.003798498
CO₂ from manufacturing (embodied carbon) is: 0.005135649
Carbon Intensity for this location: 67 gCO₂eq/kWh
SCI: 0.008934 gCO₂eq / pipeline run emitted

@ArneTR ArneTR merged commit 7ed0e7e into main Jun 18, 2024
4 checks passed
@ArneTR ArneTR deleted the pre-calculated branch June 18, 2024 10:06
ArneTR added a commit that referenced this pull request Jun 18, 2024
Copy link
Member

@ribalba ribalba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the new solution. Makes the code a lot clearer and easier to maintain next to all the other advantages.

An uncertainty is if Hyper-Threading / SMT is turned on or off, but we believe it is reasonable to assume that for Shared runners they will turn it on as it generally increases
throughput and performance in shared environments.

If you have trouble finding out the splitting factor for your system: Open an issue! We are happy to help!!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ArneTR "Open an issue" could be a link

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add

#!/usr/bin/env bash
set -eu
declare -A cloud_energy_hashmap
cloud_energy_hashmap[0.00]=93.3974838256836
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you choose this format? Wouldn't a format

0.00=93.3974838256836
...

be enough? And then use grep/awk to get the values?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file is source via bash and then accessible via a hashmap. This is way faster than using awk/grep for every utilization value. See my comment regarding the performance.

# but 100x faster in reading. The net gain is after ~ 5 measurements

That is why the file has this format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants