Statistics subsystem: new available statistics, plotting scripts, tests #74

Piccions · 2022-07-21T01:32:52Z

This PR aims at improving the usability and the capabilities of the statistics subsystem used to take internal measurements of the simulator behaviour. In particular:

The following statistics have been inserted in the simulator, the python interface and the default generated textual report:
- Anti-messages count
- Recovery costs, intended as the cost to restore a LP checkpoint and send the related anti-messages; the coasting forward operation costs are captured in another statistics value
- Size of LP states when checkpointed
- Costs of checkpoints
- Event execution costs
- Count of LPs
A comprehensive test for the statistics subsystem, including the related python scripts, has been implemented
Several bug related to the statistics subsystem have been fixed
- Serial runtime statistics issues
- Edge cases with divisions by zero in absence of rollbacks/checkpoints/processes messages

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Also updated the textual report with the correct logged average logged state size. Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

This allows to use high resolution timers and meaningfully compare them to normal timer values Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

In the previous commit we added a new node wide value which is now properly documented Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

TODO: put actual testing logic inside of it! Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

github-actions · 2022-07-21T01:33:53Z

Documentation coverage is 69.9% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  40.3% (27/67)
Enums      :  36.4% (4/11)
Files      :  95.7% (66/69)
Functions  :  71.3% (239/335)
Namespaces :   0.0% (0/2)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  61.7% (113/183)
-----------------------------------
Total      :  69.9% (591/846)

Piccions · 2022-07-21T01:40:05Z

This PR is a work in progress, things are missing and/or broken (we are still missing the full suite of python plotting scripts 😢 ).
I would appreciate some opinions on how to properly test the statistics python scripts. Right now the approach is kind of hacky: I invoke the python interpreter from C code using the system() function and the resulting textual report is checked with the help of another python script. Yes, also this latter script is invoked using system(). Any good ideas in this regard?

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Now we have two separate plotting scripts to plot single runs and multi runs statistics. Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

alessandropellegrini · 2022-07-23T17:30:22Z

Another possibility can be to have python tests, not necessarily using a python testing framework (if the number of python tests increases it can be introduced).

We can launch these python tests directly from ctest:

find_package(Python REQUIRED)
add_test(python_test_whateber
   ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test/tests/log/test.py
  )

From there, we could import the various functions from the scripts and test them.
The binary file could be an asset in the test directory.

This does not test the generation-parsing interaction. In that case we could call the simple model from python (using some $<TARGET_FILE:model> generator).

Piccions · 2022-07-23T22:01:38Z

Unfortunately FindPython is not available in CMake 3.10, it has been introduced in the 3.12 version. We can still write a Python test though, but we will have to hardcode the Python interpreter path.
Also, I feel like the binary file shouldn't be a static asset in the repository since the test wouldn't break when it should, e.g.: after heavy changes in the statistics module.
So my new proposal would be to split the current test in two parts: the one that tests the C code and the one that tests the Python parsing script. We can then impose a dependency relationship on those two tests (the Python parsing script consumes the output of the C test). CMake offers this feature to do this!

alessandropellegrini · 2022-07-23T22:04:54Z

I don't see any major problem in bumping the CMake version to 3.12 though, because we are anyhow planning to package everything on our side. In this case, having a "newer" dependency is less critical if we can directly provide binaries.

Regarding the test dependency, that's the solution I like the most. I couldn't check this feature's availability (again, luckily, I'm not the CMake expert! 😄).

Piccions · 2022-07-23T22:12:01Z

The only issue in bumping the version to 3.12 is that Ubuntu 18 LTS ships with CMake version 3.10. Yeah, not a big issue. If you agree, we will bump CMake version to 3.12!

codecov · 2022-07-28T00:57:05Z

Codecov Report

Merging #74 (362cccd) into develop (7bb0c8d) will increase coverage by 0.51%.
The diff coverage is 70.58%.

@@             Coverage Diff             @@
##           develop      #74      +/-   ##
===========================================
+ Coverage    84.93%   85.44%   +0.51%     
===========================================
  Files           46       46              
  Lines         2131     2158      +27     
  Branches        56       55       -1     
===========================================
+ Hits          1810     1844      +34     
+ Misses         298      293       -5     
+ Partials        23       21       -2

Impacted Files	Coverage Δ
src/datatypes/msg_queue.c	`93.93% <ø> (ø)`
src/distributed/mpi.c	`55.88% <ø> (ø)`
test/tests/mm/buddy_hard.c	`87.09% <ø> (ø)`
test/tests/mm/main.c	`100.00% <ø> (ø)`
test/tests/visibility/visibility_override.c	`100.00% <ø> (ø)`
test/tests/visibility/visibility_weak.c	`100.00% <ø> (ø)`
src/lib/random/random.c	`34.28% <7.14%> (ø)`
test/framework/test.c	`87.64% <50.00%> (-3.72%)`	⬇️
test/tests/datatypes/bitmap.c	`88.00% <80.00%> (-0.47%)`	⬇️
src/log/stats.c	`67.50% <88.88%> (-0.30%)`	⬇️
... and 14 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

github-actions · 2022-07-28T01:26:39Z

Documentation coverage is 69.0% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  40.3% (27/67)
Enums      :  36.4% (4/11)
Files      :  94.4% (67/71)
Functions  :  71.5% (254/355)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  58.2% (113/194)
-----------------------------------
Total      :  69.0% (607/880)

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

github-actions · 2022-07-29T20:16:37Z

Documentation coverage is 68.8% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  40.3% (27/67)
Enums      :  36.4% (4/11)
Files      :  94.4% (67/71)
Functions  :  71.5% (254/355)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  57.7% (113/196)
-----------------------------------
Total      :  68.8% (607/882)

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

github-actions · 2022-07-29T20:29:03Z

Documentation coverage is 68.8% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  40.3% (27/67)
Enums      :  36.4% (4/11)
Files      :  94.4% (67/71)
Functions  :  71.5% (254/355)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  57.7% (113/196)
-----------------------------------
Total      :  68.8% (607/882)

github-actions · 2022-07-29T22:24:24Z

Documentation coverage is 69.4% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  40.3% (27/67)
Enums      :  36.4% (4/11)
Files      :  94.4% (67/71)
Functions  :  71.8% (255/355)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  59.7% (117/196)
-----------------------------------
Total      :  69.4% (612/882)

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

github-actions · 2022-07-29T22:36:01Z

Documentation coverage is 69.4% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  40.3% (27/67)
Enums      :  36.4% (4/11)
Files      :  94.4% (67/71)
Functions  :  71.8% (255/355)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  59.7% (117/196)
-----------------------------------
Total      :  69.4% (612/882)

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

github-actions · 2022-07-30T08:21:54Z

Documentation coverage is 71.8% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  70.1% (47/67)
Enums      :  45.5% (5/11)
Files      :  94.4% (67/71)
Functions  :  71.8% (255/355)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  59.7% (117/196)
-----------------------------------
Total      :  71.8% (633/882)

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

github-actions · 2022-07-30T08:51:02Z

Documentation coverage is 71.9% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  70.1% (47/67)
Enums      :  45.5% (5/11)
Files      :  94.4% (67/71)
Functions  :  72.0% (255/354)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  59.7% (117/196)
-----------------------------------
Total      :  71.9% (633/881)

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

github-actions · 2022-07-30T08:56:10Z

Documentation coverage is 71.9% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  70.1% (47/67)
Enums      :  45.5% (5/11)
Files      :  94.4% (67/71)
Functions  :  72.0% (255/354)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  59.7% (117/196)
-----------------------------------
Total      :  71.9% (633/881)

Implemented a new ROOT-Sim API that allows to interrupt a currently running simulation Also fixed a bug in the logging module initialization Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

github-actions · 2022-07-30T10:41:06Z

Documentation coverage is 71.8% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  70.1% (47/67)
Enums      :  45.5% (5/11)
Files      :  94.4% (67/71)
Functions  :  71.8% (255/355)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  59.7% (117/196)
-----------------------------------
Total      :  71.8% (633/882)

alessandropellegrini

Thanks, looks good! Just one minor comment in the review.

src/gvt/termination.c

github-actions · 2022-09-11T01:16:34Z

Documentation coverage is 71.8% 👍

Classes    : 100.0% (1/1)
Defines    :  75.4% (107/142)
Enum Values:  70.1% (47/67)
Enums      :  45.5% (5/11)
Files      :  94.4% (67/71)
Functions  :  71.8% (255/355)
Namespaces :   0.0% (0/3)
Pages      : 100.0% (3/3)
Structs    :  95.2% (20/21)
Typedefs   :  91.7% (11/12)
Variables  :  59.7% (117/196)
-----------------------------------
Total      :  71.8% (633/882)

github-advanced-security

Found 11 potential problems in the proposed changes. Check the Files changed tab for more details.

github-advanced-security

Found 38 potential problems in the proposed changes. Check the Files changed tab for more details.

github-advanced-security

Found 41 potential problems in the proposed changes. Check the Files changed tab for more details.

github-advanced-security

Found 68 potential problems in the proposed changes. Check the Files changed tab for more details.

Piccions added 18 commits July 6, 2022 15:48

Removed unneeded stats from the final text report

0809bb0

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Added the size of the logged state as a simulator statistics

d4683d2

Also updated the textual report with the correct logged average logged state size. Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Added the total simulation time to the final statistics text report

d3ccb1d

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Added an high resolution measure of the total time

0f7b5fc

This allows to use high resolution timers and meaningfully compare them to normal timer values Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Fixed the documentation for the statistics binary format

3b0cffb

In the previous commit we added a new node wide value which is now properly documented Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

PRNG of tests is now "properly" seeded in multithreaded tests

0afa7fc

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Reformat of the bitmap test

ced245c

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Stub of the statistics module test

12399ba

TODO: put actual testing logic inside of it! Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Improved the statistic module tests, fixed a pair of bugs

ec6dd66

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Merge branch 'develop' into stats_examples

94a34bd

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Renamed the expected failure tests of the test framework

7c83993

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Removed a (now) useless timestamp from the statistics subsystem

e0fb82d

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Fixed a bug in the silent execution with SetState()

83e2952

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Fixed a bug in the statistics subsystem with large samples

c90a917

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Added the checkpoint average time in the statistics textual report

78096c1

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Fixed a bug in the thread count initialization in the serial runtime

b7b31f4

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Added LPs count to the statistic subsystem

3ff2522

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Added several stats to the statistics subsystem

139fc9d

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Piccions requested a review from alessandropellegrini as a code owner July 21, 2022 01:32

Piccions marked this pull request as draft July 21, 2022 01:33

Piccions added 2 commits July 21, 2022 13:16

Improved the statistics tests

409f207

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Example plotting script for matplotlib

c45be7b

Now we have two separate plotting scripts to plot single runs and multi runs statistics. Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Piccions mentioned this pull request Jul 22, 2022

Test threadpool race condition which negates concurrency #70

Closed

Small tweaks in the python plotting scripts

d86b59f

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Fix of the statistics subsystem under windows

c2923d5

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Piccions marked this pull request as ready for review July 28, 2022 01:32

Piccions linked an issue Jul 28, 2022 that may be closed by this pull request

Test the stats subsystem #50

Open

Statistics example scripts output tsv files with "--tsv"

788aff7

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Minor reformat of the tests

8b60b75

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Improved Doxygen documentation

0d8fcbe

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Piccions force-pushed the stats_examples branch from a048936 to 0d8fcbe Compare July 29, 2022 22:35

Improved the Doxygen documentation of stats.h

e56bcb1

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Fixed some Codacy issues

ac7f519

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

Fixed the statistics test

563a540

Criticism and suggestions are always welcome! Signed-off-by: Andrea Piccione <piccione@diag.uniroma1.it>

alessandropellegrini approved these changes Aug 17, 2022

View reviewed changes

src/gvt/termination.c Show resolved Hide resolved

Merge branch 'develop' into stats_examples

362cccd

github-advanced-security bot found potential problems Sep 11, 2022

View reviewed changes

Piccions merged commit b1fa574 into develop Sep 11, 2022

Piccions deleted the stats_examples branch October 1, 2022 23:00

Statistics subsystem: new available statistics, plotting scripts, tests #74

Statistics subsystem: new available statistics, plotting scripts, tests #74

Uh oh!

Conversation

Piccions commented Jul 21, 2022

Uh oh!

github-actions bot commented Jul 21, 2022

Uh oh!

Piccions commented Jul 21, 2022

Uh oh!

alessandropellegrini commented Jul 23, 2022

Uh oh!

Piccions commented Jul 23, 2022

Uh oh!

alessandropellegrini commented Jul 23, 2022

Uh oh!

Piccions commented Jul 23, 2022

Uh oh!

codecov bot commented Jul 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jul 28, 2022

Uh oh!

github-actions bot commented Jul 29, 2022

Uh oh!

github-actions bot commented Jul 29, 2022

Uh oh!

github-actions bot commented Jul 29, 2022

Uh oh!

github-actions bot commented Jul 29, 2022

Uh oh!

github-actions bot commented Jul 30, 2022

Uh oh!

github-actions bot commented Jul 30, 2022

Uh oh!

github-actions bot commented Jul 30, 2022

Uh oh!

github-actions bot commented Jul 30, 2022

Uh oh!

alessandropellegrini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Sep 11, 2022

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Jul 28, 2022 •

edited

Loading