Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'max_sub_interval' option to derivative sensor #125870

Open
wants to merge 7 commits into
base: dev
Choose a base branch
from

Conversation

karwosts
Copy link
Contributor

@karwosts karwosts commented Sep 13, 2024

Proposed change

Copying the implementation from #110685, add a max_sub_interval option to derivative sensor, which will force the derivative recalculation at the requested time interval if no updates come from the source sensor during that time.

Enabling this option will allow the derivative sensor to return to zero when the source stops changing. Presently the derivative will just forever keep the last non-zero value, which is not mathematically expected.

E.g.:
image

Type of change

  • Dependency upgrade
  • Bugfix (non-breaking change which fixes an issue)
  • New integration (thank you!)
  • New feature (which adds functionality to an existing integration)
  • Deprecation (breaking change to happen in the future)
  • Breaking change (fix/feature causing existing functionality to break)
  • Code quality improvements to existing code or addition of tests

Additional information

Checklist

  • The code change is tested and works locally.
  • Local tests pass. Your PR cannot be merged unless tests pass
  • There is no commented out code in this PR.
  • I have followed the development checklist
  • I have followed the perfect PR recommendations
  • The code has been formatted using Ruff (ruff format homeassistant tests)
  • Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

@home-assistant
Copy link

Hey there @afaucogney, mind taking a look at this pull request as it has been labeled with an integration (derivative) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of derivative can trigger bot actions by commenting:

  • @home-assistant close Closes the pull request.
  • @home-assistant rename Awesome new title Renames the pull request.
  • @home-assistant reopen Reopen the pull request.
  • @home-assistant unassign derivative Removes the current integration label and assignees on the pull request, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

@karwosts karwosts force-pushed the derivative_max_interval branch from ff8f0ed to 2fd5b97 Compare October 30, 2024 20:49
@mark007
Copy link
Contributor

mark007 commented Nov 23, 2024

Hoping this one can be reviewed / merged. It affects many of us, especially in the likes of our house where many sensors update very slowly/infrequently, eg where temperatures drop/rise very slowly. All I can ask is that those with the power to do so, review/merge it sooner rather than later, thanks!

@gjohansson-ST
Copy link
Member

Would this be solved by implementing also tracking reported and not only changed values?
I assume it's not expected that the input sensors stops reporting?
I guess it's not exactly the same as this PR but it would be less complicated perhaps to solve the issue.

@karwosts
Copy link
Contributor Author

karwosts commented Nov 23, 2024

Would this be solved by implementing also tracking reported and not only changed values? I assume it's not expected that the input sensors stops reporting? I guess it's not exactly the same as this PR but it would be less complicated perhaps to solve the issue.

That might help somewhat for some cases, but I'm not confident if that covers all the scenarios.

Like consider in the image in the description, there is a counter helper that just stops updating. I'm not convinced it would ever send more "reports"?

In general I'm not sure to what extent I can rely on an arbitrary sensor to keep sending reports of unchanged values, that seems very implementation dependent. I could imagine especially like battery powered devices would not send unnecessary reports to save battery.

@mekaneck
Copy link

I think using the state_reported event to update the derivative is a much needed change, but it should be a separate PR. Both changes are necessary to solve the problem correctly and completely. Both changes were implemented on the Riemann Sum integration (2 separate PR's) and I don't see a reason anything should be different for the derivative integration.

@Anton2079
Copy link

Sorry to ping when you're not listed, however @ThomDietrich are you able to review this, it is against @afaucogney but he hasn't been seen on Github since August.
This issue has been observed since December 2022.
#83496

@ThomDietrich
Copy link
Contributor

ThomDietrich commented Jan 8, 2025

Hey guys.
The discussed issue is very similar to a problem that we have over in the statistics component. See here and here. Maybe that's why @Anton2079 mentioned me here.

I welcome this PR here. As a user of the derivative component myself, I run into this problem too.

  1. I agree with @gjohansson-ST, the component should listen to the newly introduced async_track_state_report_event. I also agree with @karwosts, this should be a separate PR. However, I think the state report event PR would be the better and more significant improvement.
  2. I agree with the idea to add intermediary updates, also for the mentioned reasons.
    • @karwosts did you consider a service based approach instead? I am not sure if a fixed "sub interval" will solve this cleanly. A separate automation triggering an intermediary update would allow more control to the user, e.g. triggering an update just-in-time when another action needs the derivative value.
    • The feature of a fixed sub interval is still a good fallback to the said service.

@karwosts
Copy link
Contributor Author

karwosts commented Jan 8, 2025

did you consider a service based approach instead?

I didn't really consider anything else. I watched the development of this same feature in the integration (Riemann Sum) integration, it went through a long review process and many iterations to end up with this max_sub_interval option, so I just jumped straight to implementing the same option as there was a precedent for it (and it makes related integrations seem consistent).

@baudneo
Copy link

baudneo commented Feb 4, 2025

Patiently waiting for this to make its way to release. Really glad there's work being done to fix the derivative sensor.

@sophof
Copy link
Contributor

sophof commented Feb 10, 2025

@karwosts Hi, I've started on reviewing your PR after talking to you on discord, but I have some generic questions first before I go ahead and write up my suggested changes and such. To be honest, I think there will be few that will take some work..

I think it is clear that extra codeowners are needed for this integration based on the time this has spent in limbo currently. Besides that, according to the architecture guidelines, when adding a feature (such as here) one should also add themselves as codeowner. Are you willing to do that? You also implied that you didn't really know much anymore because of the time that had passed since submitting the PR. I interpreted from that, that you might not be that motivated.

As an alternative I could submit my PR and add myself as codeowner. It has similar ideas, but refactors the code even more (I might remove some of it to put it in a later PR to make things simpler). The only work that is left for me is to copy your unit tests and naming scheme, which I like better than mine. Then you could review my code.

So my question boils down to: are you willing to be code-owner? And if not, are you willing to review my PR to help it along instead?

@gr0mit1
Copy link

gr0mit1 commented Feb 10, 2025

This would be really good to see working. I've got an mqtt gas meter that reports my gas usage in kWh. I'm trying to plot 'power' and it never drops to zero, even when no gas is being consumed for hours.
Screenshot 2025-02-10 at 13 18 49
I'd be happy to test the fix once you've made the change. Many thanks

@frenck frenck requested a review from emontnemery March 1, 2025 22:45
@elupus
Copy link
Contributor

elupus commented Mar 8, 2025

Looks like reported event will be seen as opt in by integrations, so we must support something like this and probably enabled by default.

@karwosts
Copy link
Contributor Author

I'm thinking this has reached a dead-end. There was perhaps too much refactoring included in the initial PR, makes it probably too hard to review, and worse it is too hard to keep up to date with other changes.

The recent merging of state_reported now will require a significant amount of work to rewrite this again, and I'm not sure if it would do any good to attempt it.

Could try to split the necessary refactoring out into a separate PR before attempting any functional changes, but given that this integration has been abandoned by the codeowner I have no reason to think that would ever be reviewed either.

@sophof - if you have a version of this ready to go that has incorporated the state_reported change, I'm happy to close this and let you take it over if you would like to try to do so.

If not, I think this needs some direction from a maintainer on where to go from here, before I'm willing to invest any more time on it.

@sophof
Copy link
Contributor

sophof commented Mar 12, 2025

#139230 should already fix this now AFAIK.

@karwosts
Copy link
Contributor Author

I think the thinking is that it fixes it for some cases, but not all. Getting state_reported events from a constant sensor can't be guaranteed.

@sophof
Copy link
Contributor

sophof commented Mar 12, 2025

Yeah, I was just checking the code and I'm also unsure if it will always work. Depends on the implementation of the sensor I guess, but maybe that is the right approach? all sensors I own have the possibility to set both a min value to force reporting AND a fixed time. This approach keeps the code 'clean' (although I think that maybe I will still do some refactoring).

},
"data_description": {
"round": "Controls the number of decimal digits in the output.",
"time_window": "If set, the sensor's value is a time weighted moving average of derivatives within this window.",
"unit_prefix": "The output will be scaled according to the selected metric prefix and time unit of the derivative."
"unit_prefix": "The output will be scaled according to the selected metric prefix and time unit of the derivative.",
"max_sub_interval": "Recalculates derivative if the source did not change for this duration. Use 0 for no time based updates."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a hyphen to "time-based" should improve readability and (machine) translations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Derivative integration gives incorrect values when source does not change