Configuring AutoQA for subsystems

This document is intended for subsystem expert who configure the AutoQA reference histogram analysis in Offline QA for their subsystem. Please note that there are three parts to this process:

1. Configuring histogram descriptions
2. Configuring the data to use as a histogram reference
3. Configuring the analysis parameters.

Please follow the steps outlined below, beginning with what is necessary for all three of the above configuration tasks:

All configurations

Every configuration begins by using the QA browser. Please enter your RCF username and then select "2.1 shift work". The next step you take depends on whether you wish to view data and reference, or just the reference alone. When using the AutoQA system, please take note of the "Help" buttons which are situated at the top right corner of any panel for which contextual help exists and feel free to click these at any time.

I. Configuring histogram descriptions

To aid the people who review the QA histograms, each histogram's title and an informative description are to be maintained. Note that descriptions span any trigger types (i.e. a separate description is not needed for minbias histograms vs. high tower histograms). 

1. If this is all you want to do, you may simply view the references alone for this: use for "View reference histograms" from page 22 of the QA Browser (page 22 is titled "Fast offline, select cuts"), then click "OK on the next page to continue.
   • It is also fine to opt for a comparison with real data as it does not affect this step; i.e. this can be done concerrently with configuring reference histograms and/or configuring analysis parameters.
2. Use the menus to select the latest references (you may need to click the little arrows on some browsers), and then click on the "Analyze" button which appears.
3. After a few moments spent generating graphics of the histograms, the full list of histograms will appear. Scroll to the histograms for your subsystem, and click "Examine" next to any one of them. It is also worth being aware that double clicking on any histogram name will reveal its title, and clicking on "(more)" will also reveal its description (without having to click "Examine").
4. After a few moments spent retrieving the individual histogram, a page will display showing details of the histogram. Click on the "Edit (experts only!)" button. Once in Edit mode, you will stay there until you explicitly leave it through a similar action, and you will see a red-dashed border while in this mode.
5. The histogram title and description will appear in the middle of the page. Edit these as appropriate, and click the "Update description" button.
6. Use the left panel to navigate to the next histogram on which you wish to work (you can use "Prev"/"Next", the table of pages, or select "All" to return to the full list). Then repeat steps 4-5 above as appropriate.

II. Configuring reference histograms

Each histogram's analysis consists of a comparison against a reference histogram.

1. Locate a run, or set of runs, where the subsystem is believed to be in proper working order to define a reference.
   • If using a single run, opt for "Select auto-combined jobs and compare to a reference histogram."
     • Another way to get there is to arrive via the "Check Offline QA" link for any given run from the RunLog, then checking the box so that analysis options are shown, and then selecting plots for the file stream of choice
   • If using a set of runs, opt for "Combine several jobs and compare to a reference histogram."
   • If unsure of a good run, you will need to hunt through available runs for one which you believe looks good.
2. Find the run(s) of interest among those listed and click the OK button.
3. Use the provided menu to select the appropriate reference. By default, what the system believes is the appropriate reference (same Run Year, Trigger Setup, and lated Version) is automatically selected. The "Analyze" button will appear when a reference is selected, and you should then click it.
4. Repeat steps 3-4 from "Configuring histogram descriptions" listed above. The data histogram and any previous reference will appear near the top of the panel.
5. It is important to remember that reference histograms may be general, or specific to a trigger type. If a reference histogram does not exist for a specific trigger type, the analysis will be made comparing to the general one (if it exists). Please consider which purpose you want this histogram to serve and select the trigger type for which it applies appropriately.
6. Click the "Mark for Updating Reference" button. 
7. Repeat the selection of histograms you wish to choose as reference and steps 5-6 until you are ready to submit the entire set of new histograms for reference.
8. Click on the "Marked for update" button near the upper left.
9. The list of marked histograms will appear, along with a options for the new reference set. By default, the same Run Year and Trigger Setup will be selected, and any other Trigger Setups from the current operations will be listed as alternatives. Whichever is chosen, a new version number will be given automatically. Once the selections are made, a "Submit new reference" button will appear.
10. Before submitting, please input comments, such as "BSMDe histograms updated for new dead regions".
11. Click the "Submit new reference" button.

III. Configuring analysis parameters

Each analysis involves a comparison method ("mode"), possible options for that mode, and a value uses as a cut on whether the analysis passes or fails.

1. Repeat steps 1-4 from "Configuring histogram descriptions" listed above. The histogram analysis parameters will appear near the bottom of the panel.
2. It is important to remember that analysis parameters may apply to a histogram of any trigger type ("general"), or to a specific trigger type. If analysis parameters do not exist for a specific trigger type, the general analysis parameters for this histogram will be used (if they exist). Please consider which purpose you want these parameters to serve and select the trigger type for which they apply appropriately.
3. Select an analysis mode and enter any options. The default of Kolmogorov Maximum Distance and no options works pretty well. More details about the possible modes are listed below these instructions.
4. Select a numerical value between 0 and 1 to use as a pass/fail cut.
   • For guidance, something near 0.8 will likely catch most discrepancies, but it may be wise to choose a tighter value (e.g. 0.9 or 0.95) initially if no trend records for this analysis are available. The "View Trends" button exists to help in making this selection, presenting the records from any previously set analysis used when the QA shift examined data.
5. Click the "Update Cut" button to store the new parameters for use the next time an analysis is run (it is important to be aware that the currently shown analysis and parameter that were used will not update to reflect the new parameters - it must be run again to do so).
6. If an analysis is no longer desired to be used, clicking the "Delete Cut" button will exclude it from future analyses. This may be desirable in a case such as where a specific trigger typed analysis may be supplanted by a general analysis for the histogram.
7. Use the left panel to navigate to the next histogram on which you wish to work. Then repeat steps 2-6 above as appropriate.
8. To assess the performance of the new analysis parameters, click on "Go back ... Back to data selections" near the very top and compare data and reference to see the scores.

___________

Analysis Modes

Currently, three modes are available:

0. χ² Probability: using ROOT's TH1::Chi2Test(). This test can be very sensitive, giving results near 0 for anything which is not a very good match.
1. Kolmogorov Probability: using ROOT's TH1::KolmogorovTest(). This test is also quite sensitive, but a bit more forgiving that the χ².
2. Kolmogorov Maximum Distance: using ROOT's TH1::KolmogorovTest() with the "M" option, and taking 1.0 minus the maximum distance result (so that a small maximum distance gives a good score). This test tends to give much better scores to histograms which are less perfect matches, and seems to provide reasonable room for discrepancies, so it is the most commonly used test.

If you are interested in adding another analysis mode, please bring the topic to the starqa-hn hypernews forum. Such a mode can be a custom function written as a plug-in, or an existing root function.

___________

Last updated Feb. 24, 2016 (G. Van Buren)