# Notes on using the `ErrorBreakdown` functionality


`Scatter` objects in the `YODA` format can carry information about bin-to-bin correlations thanks to the `ErrorBreakdown` annotation, which is set of nested `yaml` dictionaries.
The first dictionary level takes point indices as keys, the second level is a dictionary with uncertainty source names as keys, and finally, the third level provides the signed up (`up`) and down (`dn`) value of the relevant uncertainty in the relevant bin. The absolute uncertainties (rather than relative) are expected.


There are a number of important design decisions which the user should bear in mind when using this functionality, which are briefly listed below:
 - Only `Scatter` objects handle `ErrorBreakdown` objects natively
 - When a `Scatter` object is read in from a `YODA` file, the `ErrorBreakdown` `yaml` entry is not parsed automatically, as this can be a costly operation. The `yaml` is parsed as soon as the user asks the `Scatter` object for its list of variations (usign `Scatter::variations()`) or if the user explicitly calls `Scatter::parseVariations`: in other words, the `ErrorBreakdown` is parsed as soon as it is needed, but not before. When it is parsed, the `ErrorBreakdown` information is propagated to the `Points` objects associated with the scatter.
 - `Points` objects have an `errMap()` method which allows the user to retrieve the dictionary of `{syst Name: syst value}` for that point. The entry corresponding to the empty string `""` is the total uncertainty as specified in the `yerr-` or `yerr+` columns of the `Scatter2D` object (and similar for `Scatter1D` and `Scatetr3D` objects). 
 - Additional sources of uncertainty can be added to a point using the `Point::setErr()` methods and similar, by passing an additional `string` argument specifying the name of the uncertainty. Similar `Point::yErr()` getter methods also take a string argument if a particular source is to be retrieved.
 - The value of the `""` uncertainty is *not* affected by the user adding new entries to the map, unless `updateTotalUncertainty()` is called from either the `Point` or the parent `Scatter` object, in which case the sum of quadrature of all other sources in the `errMap()` is set as the new total.
 - `Point` objects need to know which `Scatter` object they belong to in order to retrieve their `errMap()`. When using the `addPoint()` method passing the point coordiates by value, the parentage of the created points is automatically set. The user should not normally have to set the parentage manually.
 - `Scatter` objects come with a `hasValidErrorBreakdown()` method which checks if the `ErrorBreakdown` is present and has all non-zero errors, so that it can be used to create a non-singular covariance matrix.
 - A covariance or correlation matrix constructed from the error breakdown can be called from a `Scatter2D` object using `Scatter2D::covarianceMatrix()` or `Scatter2D::correlationMatrix()`.


An example of a `Scatter` object with an error breakdown can be seed below:
```
BEGIN YODA_SCATTER2D_V2 /REF/ATLAS_2017_I1615206/d01-x01-y01
ErrorBreakdown: {0: {stat: {dn: -0.022, up: 0.026}, sys: {dn: -0.005, up: 0.007}}, 1: {stat: {dn: -0.02, up: 0.027}, sys: {dn: -0.003, up: 0.004}}, 2: {stat: {dn: -0.029, up: 0.036}, sys: {dn: -0.005, up: 0.008}}, 3: {stat: {dn: -0.017, up: 0.021}, sys: {dn: -0.003, up: 0.004}}, 4: {stat: {dn: -0.012, up: 0.014}, sys: {dn: -0.002, up: 0.003}}, 5: {stat: {dn: -0.0076, up: 0.01}, sys: {dn: -0.0008, up: 0.0013}}, 6: {stat: {dn: -0.0055, up: 0.0073}, sys: {dn: -0.0009, up: 0.0012}}, 7: {stat: {dn: -0.004, up: 0.0049}, sys: {dn: -0.0006, up: 0.0009}}, 8: {stat: {dn: -0.0015, up: 0.0019}, sys: {dn: -0.0001, up: 0.0003}}, 9: {stat: {dn: -0.00051, up: 0.00071}, sys: {dn: -3.0e-05, up: 0.0001}}}
IsRef: 1
Path: /REF/ATLAS_2017_I1615206/d01-x01-y01
Title: doi:10.17182/hepdata.79497.v1/t1
Type: Scatter2D
---
# xval   xerr-   xerr+   yval  yerr-   yerr+  
5.000000e+00  5.000000e+00  5.000000e+00  7.000000e-02  2.256103e-02  2.692582e-02
1.250000e+01  2.500000e+00  2.500000e+00  1.800000e-02  2.022375e-02  2.729469e-02
1.750000e+01  2.500000e+00  2.500000e+00  7.200000e-02  2.942788e-02  3.687818e-02
2.500000e+01  5.000000e+00  5.000000e+00  4.200000e-02  1.726268e-02  2.137756e-02
3.750000e+01  7.500000e+00  7.500000e+00  3.400000e-02  1.216553e-02  1.431782e-02
5.250000e+01  7.500000e+00  7.500000e+00  1.690000e-02  7.641989e-03  1.008415e-02
7.000000e+01  1.000000e+01  1.000000e+01  1.080000e-02  5.573150e-03  7.397973e-03
1.000000e+02  2.000000e+01  2.000000e+01  1.370000e-02  4.044750e-03  4.981967e-03
1.600000e+02  4.000000e+01  4.000000e+01  4.300000e-03  1.503330e-03  1.923538e-03
2.750000e+02  7.500000e+01  7.500000e+01  1.180000e-03  5.108816e-04  7.170077e-04
END YODA_SCATTER2D_V
```

An example of how to manipulate `ErrorBreakdown` information is shown below:
```
import yoda

s = yoda.Scatter2D("/foo")
s.addPoint(1.0, 5.5, 0.5, 1.0) 
print(s)
s.point(0).setYErrs(0.1, "syst1")
s.point(0).setYErrs(0.2, "syst2")
s.point(0).setYErrs(0.3, "syst3")
print(s)
print(s.variations())
s.writeVariationsToAnnotations()
yoda.write(s, "save.yoda")

s2 = yoda.read("save.yoda")["/foo"]
print(s2)
print(s2.variations())
```