[FEATURE] Implement CUPAC

[tikhomirovd]

🚀 Feature Proposal

Implement CUPAC in AB Tests

Motivation

Currently, AB tests in HypEx do not utilize advanced variance reduction techniques like CUPAC (Controlled Using Post-Assignment Covariates). CUPAC allows for additional variance reduction by incorporating post-assignment covariates, leading to more efficient and statistically powerful experiments. Adding CUPAC will enhance the robustness of AB tests in HypEx and provide users with another tool for improving experiment accuracy.

Feature Description

• Implement CUPAC variance reduction in the AB testing framework.
• Ensure CUPAC is an optional feature that can be toggled by users.
• Integrate the method into the existing experimental pipeline while maintaining backward compatibility.
• Provide clear documentation on how and when to use CUPAC.
• Add example notebooks demonstrating CUPAC’s impact on variance reduction.

Potential Impacts

• Improved statistical power in AB tests.
• Reduced sample size requirements for detecting significant effects.
• Increased reliability of experiment results by incorporating post-assignment covariates.

Alternatives

• Keep the current AB testing approach without post-assignment covariate adjustments (less efficient).
• Implement alternative post-experiment variance reduction techniques, such as doubly robust estimation.

Additional Context

• Follow best practices in implementing CUPAC (leveraging post-assignment covariates effectively).
• Ensure integration is compatible with existing experiment structures.
• Add unit tests to verify the correctness of CUPAC’s implementation.

Checklist

• Implement CUPAC in AB testing framework.
• Ensure seamless integration with existing experiments.
• Write unit tests covering CUPAC logic and correctness.
• Provide clear documentation and usage examples.
• Create a tutorial notebook demonstrating CUPAC’s impact.
• Review and refine before merging.