Introduction

This is a follow-up to some discussions in 2023, motivated by my observation in attempting to calibrate TPC SpaceCharge for Run 23 AuAu200 that I was unable to get agreement between TPC and VPD vertex z positions for data in early Run 23.

The issue was that a different (internal THUB) 40 MHz clock was used for 1/4^th of TOF (including VPD west, but not east) than the rest of the system. Usually all are on one common external clock, and this was realized and fixed after run 24162048 shortly after 9pm on June 11, 2023. Geary described the issue with some additional details in an email sent to the MTDTOF mailing list. Zhangbu suggested that the drift in timing could be measured using the difference between TPC and VPD vertex z positions. And I suggested a method of correcting for this in a blog post: Correct timing error for wrong TOF THUB clock.

---

Measurement

I realized that it is simple to attempt to generate the expected sawtooth plot from the MuDsts. Here are some example plots from an st_physics MuDst file (st_physics_24161026_raw_0500029.MuDst.root), and an st_physics_adc MuDst file (st_physics_adc_24140017_raw_0000007.MuDst.root) of the timing error T0err [ns] vs. real time [s].
    T0err = 2*(VtxZ_TPC - VtxZ_VPD) / c
where c is the speed of light in [cm/ns].

MuDst->SetAlias("T0err","2*(MuEvent.mEventSummary.mPrimaryVertexPos.mX3-BTofHeader.mVpdVz[][0])/29.98");
MuDst->SetAlias("RealTime","(MuEvent.mEventInfo.mBunchCrossingNumber[][0]
+MuEvent.mEventInfo.mBunchCrossingNumber[][1]*pow(2,32))/9383180.0")
TCut VpdFoundVtx = "abs(BTofHeader.mVpdVz[][0]+999)>0.1";
MuDst->SetMarkerStyle(7);
MuDst->Draw("T0err:RealTime",VpdFoundVtx);

Both plots show the drifting (rising) timing error as real time elapses. The events in the st_physics_adc file are more sparse, making the bands less distinct, but it spans a larger duration of real time. I suspect the two outliers in the left plot are the result of events with low VPD multiplicity such that a random VPD hit alters its time calculation, or the use of a pile-up vertex from the TPC, and should be excluded.

Here are some quick observations...

1. The bunch crossing numbers appear to reset for each run, not each fill. This implies that the phase for the correction must be determined run-by-run, not fill-by-fill.
2. The maximum VPD time shows a rather clean cut at 51200 ns (51.2 μs), which is a small surprise as we were expecting it to be twice that, at 102.4 μs. This can be seen graphically rather easily from the MuDst as well, though I confirmed the values with print statements inside StVpdCalibMaker.

MuDst->Draw("BTofHeader.mVpdTime[][0][2]","BTofHeader.mVpdTime[][0][2]>50000")

Another point to discuss from the above plots is that there are clearly two bands at any given real time. I believe this can be understood to be due to the arrival of particles at the VPD detectors before or after rollovers of the time counters. As an illustrative example, one can imagine a moment in time when the arrival of particles at the east VPD is somewhere around halfway through the count up to 51.2 μs, perhaps at about 25.6 μs. And at that moment, if the counter for the west VPD is close to 180° out of phase, close to its moment of rollover back to zero, then the west VPD hits may have a time either very high (just below 51.2 μs) or very low (just above 0). So the VPD east-west time difference can be ±25.6 μs (near one or the other, not between somewhere).

Additional evidence to support this explanation comes from looking at the first plot above as a lego plot, which provides the approximate probability of being in one band vs. the other at any given real time. We see that the probability to be in any one band peaks when the time error is near zero, corresponding to the two counters being in phase. Being in phase is similar to normal operation of both VPDs using a common counter, and is a desirable situation because the probability of seeing only a single band is nearly 1.0 (i.e. the chance that some VPD hits arrive before a counter rollover, and some after a counter rollover is very close to zero, though not exactly zero). The probability to be in either of two bands equalizes (i.e. both bands are on their peaks' respective shoulders) half way between these peaks, when the counters are 180° out of phase.

But to be clear, there is no single "correct" band: both are equally legitimate, and all of it must be corrected to recover the data.

---

Correction Attempt

Next, I attempted a "by eye" correction. Ignoring the phase, I can simply try to make the T0err constant at its intercept value by subtracting a time-varying offset and adding a "band-wise" correction:
    CorrectedT0err = T0err - (102400 ns) * (fmod(RealTime,X)/X) + (51200 ns) * Y
where X is the periodicity in seconds, Y is an integer for the band, and effectively the value of (102400 ns / X) is the timing drift rate in [ns/s]. I manually tuned the value of X to within approximately ±0.01 seconds to make the resulting plot of CorrectedT0err vs. real time as flat as I could for the two MuDsts noted earlier, and here is what I observe:

double X = 73.48;
MuDst->Draw(Form("T0err-102400*fmod(RealTime,%5.2f)/%5.2f
+51200*(3-int((T0err-102400*fmod(RealTime,%5.2f)/%5.2f)/51200+3))
:RealTime",X,X,X,X),VpdFoundVtx);

Run 24161026: X = 73.48 seconds, drift rate = 1.39 ns/s




Run 24140017: X = 75.98 seconds, drift rate = 1.35 ns/s


(note: when making these plots, I found that if I had the 102400 ns value incorrect, e.g 104200 ns, it showed up clearly as discontinuities at the clock rollover points in real time, so the smoothness we see here is also further confirmation of the correctness of that 102400 ns value)

There are a couple striking things to observe here:

1. The two runs have clearly different drift rates (and consequently periods). It isn't a question whether they may be the same within errors: using one's X value for the other leaves a significant drift.
2. Even after the correction, there are remaining variations at the O(100 ns) level in the first plot, and O(1500 ns) level in the second plot. These are compatible with each other in the sense that the second plot spans a much larger real time interval, and in some real time intervals similar to the first plot's, it also shows only O(100 ns) variations. The variations are smoothly flowing, as if a clock speed slowly varies.

The implication seems to be that the difference between the two counters' oscillators is not constant. That will be bad news for correcting this timing error if we are unable to find some other explanation for this.

A quick order-of-magnitude check to consider is whether a poorly calibrated TPC (e.g. preliminary drift velocities) could contribute. But here we are speaking about a 100 ns shift in the VPD's time, not the TPC's. The TPC vertex position would need be shifted by 15 meters in the formula for T0err for the VPD to result in a 100 ns shift, so this is not TPC-induced.

I'll also add one more observation: in the corrected T0err plot for run 24161026, there is a small set of data points that appear to be shifted upwards by roughly 40 ns. It would be good to understand the cause...perhaps something like a single outlier VPD tube being included?

-Gene