User login

Navigation

Notes from Operations Meeting

Updated on Sun, 2024-10-06 09:31 by ptribedy. Originally created by leejh on 2021-11-16 10:51. Under:

council

October 6th, 2024

Recap of the Past 24 Hours:

MCR provided 56x56 Au+Au collisions around 5 AM (with crossing angle but no rebucketing yet). Collisions were stable with low background. We focused on setting up global timing, made significant progress, and took one physics run. The beam was dumped at 9:25 AM. sPHENIX had a 20-30 minute access, followed by APEX and RHIC rebucketing.
ZDC AND rate was 3 KHz. We started with tune_AuAu_2024. Global timing looked good, with TAC numbers for BBC (blue background was high, 2x collision rate). VPDs and TOF were turned on, starting with zdc-tac, then switching to zdc-mb.
Ran tune_AuAu_2024 with TPC, TOF, BTOW, and L4 (no L4 events).
Ran setup_AuAu_2024 with TPC, TOF, BTOW, and L4 (L4 events recorded, but no plots). Reached out to Jeff to resolve L4 issues, as plots were needed for global timing. Continued running without L4 until Jeff fixed the issue.
The evp /a disk was 60% full; Jeff cleared files. l4Cali and l4Disp are still down. Diyu was contacted and will work on the calibration server within 2-3 hours.
After receiving the L4 plot, global timing looked good, with only a 0.5 ns offset.
Vz was off by ~20 cm. MCR wasn't seeing the real-time vertex, but this was fixed and improved.
BBC setup plan: TAC is a bit off, but BBC looks good. Shift crew was instructed to take 5 runs per BBC HV setting, adjust config, collect 50k events, and log runs as BBC HV scan in ESL.
Oleg initiated a cal_scan, timing within 1 ns. He will review the data later.
Akio loaded BBC HV data.
EPD experts took initial scans for TAC adjustment, with a follow-up planned.
Run 25280025 was the first Au+Au 200 production setup; fastoffline was requested and is in progress (no forward tracker, eTOF, or GMT yet).
Forward trackers (FST/sTGC) will be turned on after rebucketing (post-10 AM).

Issues Encountered:

No major issues. BCW issue was solved after Tim accessed and fixed the create.
l4Cali and l4Disp are still down, Diyu is working on it.

Plan for the Day (RHIC & STAR):

Access now; RHIC will handle rebucketing.
Forward trackers will be turned on mid-next fill (when background is low).
EPD work is ongoing; experts will provide updates on EPD and calorimeters.

Looking Ahead:

First Au+Au collisions at STAR (56x65) occurred this morning; one physics run was completed, and the next fill is expected in a few hours (delays likely).
This fill wasn’t useful for sPHENIX; they’re taking access. They’ll request 6x6 for background studies when stable. RHIC is planning to install a 56 MHz RF on Monday for a narrow vertex. Stable beams are needed by Wednesday.
Commissioning plan: compile a list of detector experts (EPD: Mike needs a call when the next fill starts, FST & sTGC experts need a call mid-fill, FST: time bin, sTGC performance will be checked and followed up, VPD will use run23 files).
Update the detector readiness checklist (shift crew to continue with setup_AuAu_2024: TRG+DAQ+iTPC+TPX+TOF+BTOW+ETOW+L4+FCS).
For 6x6, we’ll use minbias for regular production; fastoffline will be requested.
Shift leader for the next owl found; the day shift will run with one DO.

Saturday, October 5th, 2024

Urgent Issues:

BCW VME crate repair: Tim was informed, but repairs are on hold due to RHIC's beam development
Shift leader missing: We have no shift leader for the next owl shift.

Recap of the Past 24 Hours:

Collected cosmic data throughout the day as no beam development work was done (despite efforts).
MCR called for APEX at 1:30 am but reported issues at 3 am.
Encountered L0 trigger not starting and BCW components not working, which were removed.
Yellow Abort Kicker issue continues.
Blue & Yellow injection and ramp setup continues, updates expected in a few hours.

Encountered Issues:

BCW configuration error and FPGA failure:
- BCW and BCW_DSM2 nodes are missing from the component tree.
- The BCW VME crate requires repair (need to discuss access and coordination).
- Tim will replace the board and power cycle it locally, which should take 30 minutes (we have spare boards).
- Follow-up with Tim; we can run min-bias without the fix and should take opportunistic access if available.
L0/L1 not responding:
- Power cycling crate 62 failed initially.
- Prashanth went in manually, and David helped resolve it.
2000+ timeouts:
- Stopped the run.
- Shift leader consulted Akio, Jeff, and Hank.
- Jeff tried multiple power cycles of L0/L1, and it eventually came back.
EPD hot tiles:
- Restarted the run, and the issue was gone.
- Maria will investigate further.
EVB23 issue:
- EVB23 is still out of the run, with EVB22 and EVB24 expected to return soon.
- Awaiting further updates from Jeff.

Plan for the Day (RHIC & STAR):

Continue Yellow Abort Kicker work.
Blue & Yellow injection and ramp scheduled for the evening.
Ramp development overnight.
Continue taking cosmic runs when no beam is available; switch to APEX otherwise.

Looking Ahead:

First Au+Au collisions at STAR expected Sunday evening (possible delay). First fill could be 56x56.
October 4-8: sPHENIX requests 111x111 initially, then 6x6 for background study when stable.
- RHIC plans to install 56 MHz RF on Monday for narrow vertex; stable beam required before Wednesday.
STAR needs/plans for bunches/luminosity:
- TPC group requests the same crossing angle for 6x6 (one or two fills based on sPHENIX needs).
Detector commissioning plan:
- Compile a list of experts to call.
- VPD will use Run23 files.
- EPD requires calibration runs (contact experts).
- FST (time bin), sTGC, and others need timing adjustments (trigger and global timing with JH/Akio), followed by cal-scan.
- Follow up with experts.
Update the detector readiness checklist:
- Use tune_AuAu_2024 (zdc_mb), setup_AuAu_2024, and production_AuAu_2024.
6x6 production: We can use minbias and regular production settings.
Shift sign-up issues: Dan is resolving these with various STAR institutions; we may run with one DO. Follow-up with Daniel/Pavel recommended.

Friday, October 4, 2024

Recap of the Past 24 Hours:

Cosmic data: Collected cosmic data throughout the day due to no beam development work.
Minor issues: Solved with help from experts.
Yellow Abort Kicker: Still awaiting new tools to address the issue.
RHIC power supply: Power supply work was completed; RF conditioning is ongoing.

Encountered Issues:

Run control GUI crash:
- The GUI crashed and disappeared at the start of the run.
- Re-established connection using XLaunch, which helped restart the GUI.
ETOW configuration failure:
- Crate 1 and 2 experienced a failure.
- After multiple unsuccessful reloads, ETOW was removed.
- Expert rebooted the crates, but DAQ mon still showed errors (Crate 1). The shift crew followed suggestions from experts, but the issue remains unresolved.
TRG L0 issue:
- Run failed to start due to a TRG L0 error.
- Power cycling the VME trigger crate resolved the issue.
FCS dead:
- The run was stopped multiple times (more than 60 seconds, three times consecutively) due to FCS dead errors.
- DOs performed a power cycle on the VME trigger crate, which fixed the issue.
EVB23 issue:
- EVB23 was dead in the component tree, preventing the run from starting.
- Awaiting further action from Jeff (work in progress).

Plan for the Day (RHIC & STAR):

Yellow Abort Kicker work and g9-blw-ps work.
Ramp development overnight.
Blue & Yellow injection setup overnight.
Continue collecting cosmic runs when no beam is available.
Perform the usual pedestal runs (FCS LED and others).
Possibility of controlled access for 4–6 hours (open for interest).

Looking Ahead:

First Au+Au collisions at STAR expected Saturday overnight (possible delay); first fill may be 56x56.
October 4–8: sPHENIX requests 111x111 initially, followed by 6x6 for background study once stable.
- RHIC plans to install 56 MHz RF on Monday for narrow vertex. Stable beam is needed before Wednesday.
STAR needs/plans for bunches/luminosity:
- TPC group requests the same crossing angle for 6x6 (one or two fills driven by sPHENIX needs).
Detector commissioning plan:
- Compile a list of experts to call.
- VPD will use Run23 files.
- EPD requires calibration runs (call experts).
- FST (time bin), sTGC, and others (timing adjustments, trigger, global timing with JH/Akio), then cal-scan. Follow up with experts.
Update detector readiness checklist:
- Use tune_AuAu_2024 (zdc_mb), setup_AuAu_2024, and production_AuAu_2024.
6x6 run can use minbias and regular production settings.
Shift sign-up issues: Being resolved; Dan is working with various STAR institutions. May need to run with one DO.

Thursday, October 3, 2024

Recap of the Past 24 Hours:

We ran cosmic data all day as no beam development work was done.
Ongoing Yellow Abort Kicker work.
RHIC power supply work completed, but RF conditioning will require more time.

Encountered Issues:

iTPC/TPC issues overnight: Power-cycling RDOs 1-4 did not resolve the problem. Currently, iTPC sector 1 RDOs 1, 2, 3 are masked, and iTPC sector 1 RDO 4 is not masked. (this is not correct according to Tonko) ZhengXi will address this.
EVB23 in the component tree prevented the run from starting yesterday. The shift crew removed it. Jeff will look into it, and I will follow up.
A hot tile on the EPD was found yesterday. Maria is investigating it, and I will follow up.

Plan for the Day (RHIC & STAR):

Continue work on DX training.
Yellow Abort Kicker work.
RF conditioning of storage cavities.
Blue injection setup scheduled for tonight.
Yellow injection setup planned for tomorrow.
Continue taking cosmic runs.
Take usual pedestal runs (FCS LED and others).

Looking Ahead:

First Au+Au collisions at STAR: Scheduled for Saturday overnight*, with the first fill possibly 56x56. (There may be a delay.)
Oct 4-8: sPHENIX requests 111x111 initially, then 6x6 for background study once stable. RHIC plans 56 MHz RF installation on Monday for narrow vertex; stable beams needed before Wednesday.
STAR needs/plans for bunches/luminosity? Remove crossing angle for 6x6 (to be determined).
Commissioning plans for detectors: VPD & EPD need calibration runs, FST (timebin), sTGC, and others (all timing adjustments, trigger, and global timing). Follow up with experts.
Plan for eTOF during this Au+Au run should mirror p+p configuration.
Update the detector readiness checklist (setup_auau200_2024_minbias, tune_auau200_2024_minbias, auau200_2024_minbias). Jeff will manage this, and the shift leader will review.
Urgently need a shift leader and detector operator for next week’s owl shift. Dan Cebra is unreachable; Frank will follow up.

Wednesday, October 2, 2024

Recap of the Past 24 Hours:

We ran cosmic data all day as no beam development work was done.
Akio implemented the initial file for the BBC high-voltage run for Au+Au 2024.
David Tlusty updated the TOF control for TOF West sector 5, resolving the previous issue.
ESMD was turned off as per Jacobs' request and will be excluded from future runs.
MCR postponed the Yellow Abort Kicker repair to today.

Encountered Issues:

EVB23 in the component tree prevented the run from starting. The shift crew removed it, and Jeff will look into it.
One hot tile on the EPD was identified. Maria is investigating.

Plan for the Day (RHIC & STAR):

No significant progress on DX training yet; it may start this evening or tomorrow morning.
Yellow Abort Kicker work is still ongoing.
RF conditioning: Most parts are running smoothly, and most components are conditioned.
Blue injection ramp setup is scheduled for tonight, with Yellow injection tomorrow.
We continue to run cosmics but will prioritize expert needs.

Looking Ahead:

First Au+Au collisions at STAR are expected Friday overnight, with the first fill potentially 56x56.
Oct 4-8: sPHENIX ramp-up/stochastic cooling, aiming for 111x111 (or possibly fewer), followed by 6x6 for background studies (planned for Tuesday) once silicon and TPC are stable. RHIC plans to install 56 MHz RF, and we'll explore the use of 6x6 (Gene will advise).
STAR needs/plans for bunches/luminosity: Remove the crossing angle for 6x6 (to be determined).
RHIC DX training and Yellow Abort Kicker work access will be completed.
eTOF plan for Au+Au should remain the same as for p+p runs.
Update the detector readiness checklist (setup_auau200_2024_minbias, tune_auau200_2024_minbias, auau200_2024_minbias). Jeff will follow up.

Tuesday, October 1, 2024
Recap of Past 24 Hours:

Yesterday we had access till from 8:00 till ~16:00
Took cosmics for rest of the day, went to apex around 4 am and then cosmics again ~6 am.
CAS brought the magnet down at the beginning of the access for some cleaning work. Ramped up at about noon.
TOF: problem with W5 POS HV is fixed, so we are back to 100% ToF acceptance
eToF: replaced TCD fanout box, works now (?)
FST status checked, no further refilling needed
After the access, STAR detector was put in the APEX/beam development mode. Again took cosmic when no beam related work was performed

Encountered issues

BSMD: CAEN HV system failed; no spare parts (controller boar) are available. BSMD is off now and not included in the run configuration. Most probably BSMD will be unavailable till the end of the Run24.
MXQ crate went off without apparent reason, fixed by power-cycling the crate

Plan for the Day (orders may be swapped)

Opportunity for restricted access from 8:00 – 18:00 (DX training for collider), some DX training for tomorrow (access opportunity tomorrow), please let us know if you want to use this
We continue running cosmic but prioritize expert’s needs
Separate work for Pulsed Power group for yellow abort kicker (need 6 hours of work)
RF conditioning of storage cavities overnight
Injection and ramp possible (APEX mode tonight !!)

Looking Ahead

First Au+Au collisions at STAR expected Wednesday (Oct. 2nd) night or Thursday (Oct. 3rd)
RHIC power supply not ready, DX training, access for yellow board kicker, crossing angles setup
What is the plan for eTOF during this AuAu run ?
Update the detector readiness checklist
Geary is asking about David’s schedule, IOC related work for TOF, Alex will contact him and cc Geary

Monday, September 30, 2024

Recap of Past 24 Hours:

Overall: Physics and data taking, the last p+p store ended at 08:00,

Encountered Issues:

BSMD HV did not turn ON. Shift team contacted Oleg, and then removed BSMD from the run configuration (starting at about midnight).A few common issues with TPC RDOs, Trigger/RunControl, and sTGC

Plan for the Day:

access opportunity till ~16:00
CAS brought the magnet down for some cleaning work. It needs coordination with Prashanth.
Plans for the access:
- Inspection if FST cooling refill is needed
- TOF: W5 POS HV cable to be moved from 7.5 to 3.0
- Time permitting: eTOF: check TCD connection to the rack
After the access:
- we plan to bring the magnet up
- STAR detector stays in the APEX/beam commissioning mode

Looking Ahead

Tuesday, Oct. 1st: We expect a few hours of access opportunity
The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

Sunday, September 29, 2024

Recap of Past 24 Hours:

Overall: Physics and data taking

Encountered Issues:

DAQ: EVB24 started causing problems at about 8 am today, removed from the run configuration after consulting Tonko. Jeff will work on solving this problem later today.
ESMD: MAPMT FEE 4P1 turned red for a while, the problem fixed itself automatically
A few common issues with TPC RDOs and Trigger/RunControl
B1U polarization measurements not available for the current fill (#35153) due to broken target. The issue showed up at the end of fill #35152.

Plan for the Day:

Physics

Looking Ahead

Monday, Sept. 30: Maintenance day

the last p+p store ends at 08:00,
then ~8 - 9 hour of maintenance
Plans for the access:
- We plan to keep the magnet on
- Inspection if FST cooling refill is needed

Tuesday, Oct. 1st: We expect a few hours of access opportunity
The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

Saturday, September 28, 2024

Recap of Past 24 Hours:

Most of the time: Physics and data taking
Problems with FCS ECAL. Akio and Tim made access at 19:40, replaced one of MPOD modules for FCS ECAL power supply, reconfigured and turned back on.
We are running with ToF without West Sector 5, so with 90% of ToF acceptance (and 90% of expected ToF multiplicity). Jeff modified IDs for triggers that include ToF Multiplicity to keep track of runs with this state.

Encountered Issues:

Evening shift noticed 2 new cold tiles in the EPD West ADC plot. Performed "Reboot All" according to the EPD cold tile response manual, but that did not solve the issue. It should be resolved after running pedestal_rhicclock_clean. Run pedestal_rhicclock_clean was postponed to the end of the fill, and done in the morning.
- Note to shift teams: please follow the EPD instruction carefully: After “Reboot All”, one needs to run pedestal_rhicclock_clean, even if we have beams.
A few common issues with TPC RDOs, Trigger/RunControl

Plan for the Day:

Physics

Looking Ahead

Physics for the rest of the week
Monday, Sept. 30: Maintenance day

the last p+p store ends at 08:00,
then ~8 - 9 hour of maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

Friday, September 27, 2024

Recap of Past 24 Hours:

One hour access for sPHENIX at 10 am, then some problems with injection.
back to Physics at 13:49, then data taking
EEMC HV monitoring is in "paused" mode to reduce the number of GUI crashes

Encountered Issues:

TOF: "over current" alarms in TOF HV West Sector 5, followed by the "Positive - HV West Sector 5" internal trip.
- Shift team contacted Rongrong, no success with resolving the problem during the night. Geary was trying to fix this issue today in the morning, but no success so far. We can run with the current ToF state (without TOF West Sector 5),
  which gives 90% of ToF acceptance, so 90% of ToF Multiplicity. We need to change the IDs for triggers that include ToF Multiplicity to keep track of this situation. To fix the problem, we need a short access.
TOF+MTD gas system lost communication with Slow Control.
- Alexei solved the problem by restarting the program that provides the connection to the Slow Control database.
Run Control disappeared, solved after contacting Jeff (RTS02 machine died)
- Jeff will work on preparing a spare machine in the case RTS02 dies for good.
FCS trigger was running very high, which caused FST 100% dead in DAQ (two cases). Resolving the 1st one required “reboot all”, the 2nd - fixed after restarting the run.
EEMC GUI communication problem, solved by the shift team.
- Note to shift teams: please read carefully the email from Will Jacobs on Sept. 26 about EEMC monitoring
A few common issues with TPC RDOs, Trigger/RunControl

Plan for the Day:

Physics

Looking Ahead

Physics for the rest of the week
Monday, Sept. 30: Maintenance day

the last p+p store ends at 08:00,
then ~8 - 9 hour of maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

Thursday, September 26, 2024

Recap of Past 24 Hours:

APEX (10:00 – 16:00), then machine development (16:00 - 18:00)
back to Physics at 19:16, then smooth data taking

Encountered Issues:

A few common issues with TPC RDOs, Trigger/RunControl, and sTGC
EEMC communication problem, resolved by the shift team
Shift team noticed single hot channel in Sector-22 from the TPC Sec. 22 charge per pad plot, but no corresponding peak in the RDO-bytes plot. The were unable to identify the RDO, and not sure if it's a known issue.
- It is safe to continue data taking in such a case, but pay attention to TPC dead time.
FCS trigger was running very high, which caused FST 100% dead in DAQ. Resolved after restarting the run.
- Note to shift leaders: Please pay attention to trigger rates. If the trigger rates are red, restart the run.

Plan for the Day:

Physics

Looking Ahead

Monday, Sept. 30: Maintenance day

the last p+p store ends at 08:00,
then ~8 - 9 hour of maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.
First Au+Au collisions at STAR expected on October 3rd.

Wednesday, September 25, 2024

Recap of Past 24 Hours:

Overall: Physics and data taking
TPC Grid Leak: David reset the HV power supply, TPC Grid Leak is back to normal conditions

Encountered Issues:

A few common issues with TPC RDOs, Trigger/RunControl, and sTGC
EEMC communication problem, solved after calling the expert

Note to shift teams: this EEMC communication problem, and how to resolve it, is discussed in the EEMC manual. Please check it before
calling the expert.

Plan for the Day:

APEX from 10:00 to 16:00,
then 2 hours of machine development (16:00-18:00) (test of a new
polarimeter target)
return to Physics at about 19:00

Looking Ahead

Monday, Sept. 30: Maintenance day

the last p+p store ends at 08:00,
then ~8 - 9 hour on maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Tuesday, September 24, 2024

Recap of Past 24 Hours:

Overall: Physics and data taking
1 hour access for sPHENIX at 2 pm, shift team took Cosmics during the access
We are running with very stable beam condition, so there is no need to take bgRemedyTest_2024 runs. They were removed from Detector Readiness Checklist

Encountered Issues:

TPC Grid Leak: sectors 20 and 24 lost their voltages. Alexei tried to fix the problem, but without success. Also intermittent alarms for
sector 23.
- We will request ~0.5 hour access today, after the current fill, to fix the problem (change the board)
FCS trigger rates were very high, which caused FST going 100% dead in DAQ. Restarting a run resolved the problem (cleared the bad FCS conditions)
TPC gas PI-10 yellow alarm, shift team contacted Alexei, the alarm
cleared by itself.
A few common issues with TPC RDOs, Trigger/RunControl
sTGC ROB #12 power cycled.

Note to shift leaders: please read carefully the "TPC reference plots and issue problem solving" manual. If there is a single hot channel,
there is no need to stop the run.

Plan for the Day: Physics

Looking Ahead

Wednesday, Sept. 25.:

APEX from 10:00 to 16:00,
then 2 hours of machine development (16:00-18:00) (test of a new
polarimeter target)
return to Physics at about 19:00

Monday, Sept. 30: Maintenance day

the last p+p store ends at 08:00,
then ~8 - 9 hour on maintenance

The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Monday, September 23, 2024

Recap of Past 24 Hours:

Overall: Physics and smooth data taking
Prashanth worked on the sTGC gas system. Shift team needs to monitor sTGC PT-1 pressure through database since its alarm is disabled, and call Prashanth if its pressure drops below 19 psi.

Encountered Issues:

A few common issues with TPC RDOs
Yellow alarms for sTGC PT-2 and PT-3 pressure. Shift team consulted Prashanth. The pressure should be monitored. Call Prashanth if the pressure drops down to 14-15 psi.

Plan for the Day: Physics

Looking Ahead
- plan for tomorrow: Physics
- APEX on 9/25
- No maintenance on 9/25
- Maintenance moved to 9/30 (start of Au run)
- The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Sunday, September 22, 2024

Recap of Past 24 Hours:

Overall: Physics and smooth data taking
Alexei refilled GMT gas bottle

Encountered Issues:

A few common issues with TPC RDOs and Trigger/RunControl
sTGC gas interlock alarm went off (PT1 gauge showing a high pressure). Shift team contacted Prashanth and turned down the pressure regulator on the gas tank outside the TPC gas room.
sTGC ROB #8: was power cycled and after that its current was lower than expected (0.5A vs 0.7A). Shift team consulted Prashanth. After starting new run, the issue resolved itself.
FCS: DEP05:1 failed. The shift team called Tonko, restarting the run solved the issue.
Note to shift leaders: Please read error messages carefully. In this case, one should try restarting the run one more time before calling the expert, as the error message describes.

Plan for the Day: Physics

Looking Ahead
- No maintenance on 9/25
- Maintenance moved to 9/30 (start of Au run)
- The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Saturday, September 21, 2024

Recap of Past 24 Hours:

Most of the time Physics and smooth data taking
Beam dumped at 19:00 for access for sPHENIX
Shift team took Cosmics during the access
Back to Physics and data taking at 0:30

Encountered Issues:

A few common issues with TPC RDOs and Trigger/RunControl
BSMD: HV GUI lost connection, resolved after consulting the expert (Oleg)
Important: If there is an issues with BSMD: do not wait for BSMD, remove BSMD from the run configuring and start the run. Then try to resolve the problem.

Plan for the Day: Physics

Looking Ahead
- Physics during the weekend
- No maintenance on 9/25
- Maintenance moved to 9/30 (start of Au run)
- The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Friday, September 20, 2024

Most of the time Physics and smooth data taking

Encountered Issues:

Shift team was confused what should be the state of the VME #55 GP-Pulser. It must be off!
Trigger: Critical error message for run #25264020 and #25264018: "l1 CRITICAL l1Task l1Task.C:#281 Got an invalid token: 0"
- Shift team consulted Akio and continued data taking. Jeff will investigate this issue. If this error happens again, it is safe to ignore, but please make a note in the Shift log.
Inconsistent temperature alarms on VME #51 and #55
A few minor issues with TPC RDOs and configuring MXQ_QTD
sTGC PT-2 and PT-3 gas alarms went off, shift team consulted Prashanth
Shift team stopped the laser run when noticed peaks in the TPC ADC vs time plots.
- During laser runs, one should expect spikes in the TPC ADC distribution, so no reason to stop the run. Please consult the "TPC reference plots and issue problem solving" manual.

Plan for the Day: Physics

Thursday, September 19, 2024

Recap of Past 24 Hours:

APEX till 23:30, then ~1 hour access for sPHENIX
Unexpected beam loss during the first fill.
Physics started at 3 am.
eTOF: Do not include eTOF in a run until further notice from experts.

Encountered Issues:

RunControl:
- Could not stop run from the run control. GUI froze. Resolved after calling Jeff.
- Configurations waiting for more than three minutes, runs were notstopping. Resolved by "reboot all" multiple times after consultingJeff.
Hot channels in TPC
- If there is a single hot channel, there is no need to stop the run. Flemming has sent to STAR operations email list his detailed suggestions for dealing with the hot channels in TPC:
- Spike in RDO bytes
  - Identify the RDO
  - Is it a single hot channel?
    - Yes: continue the run and reboot at end
    - No: (for example: Noisy full RDO, Noisy FEE or parts)
      - Stop run and reboot, unless it’s a laser run stop when there are >3000 events recorded.
  - Example from last few days
    - 261017 sector tpx 12-6 single hot pad
    - 263013 (last night) ~16 channels in TPX 18-6

Plan for the Day: Physics

Looking Ahead

No maintenance on 9/25
Maintenance moved to 9/30 (start of Au run)
The p+p run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Wednesday, September 18, 2024

Recap of Past 24 Hours:

Access for sPHENIX at 10:30 am yesterday, then some problems with cryo
Shift team took cosmics during the accesses,
Detector Readiness Checklist was updated (updated eTOF procedure, a note about GMT)
Physics started at 5:16 pm, data taking till 7:30 am this morning
Akio uploaded new FCS Ecal HV files

Encountered Issues:

Run control GUI froze completely. Fixed by Jeff by terminating the Run Control GUI remotely on his end, then the shift leader started the GUI.
A few minor issues with TPC RDOs.
Error configuring MXQ_QTD Node -> Shift team powercycled VME crate MXQ, which resolved the issue.

Plan for the Day

APEX from 8:00 till 23:00.
Dmitry will do maintenance on databases.
Tonko will work on RDOs and TCD.
Access for sPHENIX at 23:00 to work on cooling of MVTX (estimated duration: 30 minutes)
Back to physics at about midnight

Looking Ahead

No maintenance on 9/25
Maintenance moved to 9/30 (start of Au run)
Pp run ends on Sept. 30. The last 3 weeks of the Run24 will be Au+Au collisions.

Tuesday, July 30, 2024

TPC sector 11 channel 8 anode tripped, clear the trip manually
spike for the TPX sector 5 in this run, power-cycled
sTGC ROB10 channel 1 fluctuating

1st power dip (~11:00):

Lost the control for all the detectors. We got the global and sTGC interlock alarm, lost the power to the platform, lost water, network, MTD gas, air blowers.
PMD Power was off in the interlock page
powercycle the VME EQ4 crate
All back on ~ 14:33
TPC LASER controls were reset, we see pico drivers alive now.

2st power dip (~15:10):

MCW is running, but magnet tripped
reset the FST cooling
Turn on the BBC and ZDC. VPD is responding, so turned them off
BCE in red in the component tree, then fixed
Will recovered EEMC
BTOW, BSMD, ETOW, ESMD, FCS have been tested and ready to go. (18:04)
Magenet tripped (18:41)
restored control of TOF/MTD/eTOF HV and LV.
pedAsPhys run with TOF+MTD+TRG+DAQ, now only TOF tray 117 error remains, Rongrong masked out this tray.
Rebooted crate #63 (MXQ), rebooted trg/daq/l2. Now this run finished properly without any errors.
Magenet tripped again (21:41)
unable to turn on the VPD

Current issue:

“Bermuda” computer has a problem, Wayne had an access but couldn’t fix it. Copy the disk to a new one now, it is running ~30% at ~9:30. Wayne is also preparing a new desktop for this in the meantime.
MCW was lost due to blown fuses on the 80T chiller (for the MCW). Water is back online. Only MCW was lost, everything else is fine. (~6:20),
Lost the communication of TPC air blower (didn’t trigger the global interlock). - David & Tim
VME processor in Crate80 initiallizing correctly, but not communication. But right now is BTOW is back
GLW lost communication, need to be checked during access/ or David can re-establish com. - recovered - Tim
Can't start run due to mxq_qtd: qt32d-52 VP003 QT32D is dead - 63 crate - Hank will call control room
mix, mix_dsm2 - 69 crate - Need a physical power-cycle - Tim
Laser can be turn on but can’t be tuned

To shifters:

Shiftleaders please pass all the informations to the next shift, walkthrough all the problems happened during the shift, and the remaining problems
check the weather before processing the recovery, just in case there will be another thunder storm/power-dip happens soon
clean the control room

Monday, July 29, 2024

Status & Issues:

TPC:
- #25210022, a spike in the TPX RDO_bytes plot for sector 4. Power-cycled.
- #25211009, ITPC RDO S04:1, power-cycled
- #25211016, iTPC RDO iS17:2, TPX S13:4, power-cycled
- TPC Anode Trip, Sector 11 channel 8, 5 times - apply 45V down, will also remind the SC expert
Laser:
- The laser can turn on but is not able to tune. Prashanth will try to fix it during the next access (Monday afternoon/Wednesday).
- Now the procedure for laser run is: 1) Warm up the laser in advance for 5 minutes and do not try to tune the laser. 2) After 5 minutes, start the laser run. Do not tune the laser during the laser run.
Trigger:
- #25210037 couldn’t start the run, rebooted TRG+DAQ
- Carl did a test for the new trigger configuration. Need to do a quick check at the end of this fill
sTGC:
- Red alarm from sTGC Air blower AC failure, the problem cannot be fixed during the run, need to have access. It triggered sTGC interlock after about 20 minutes. DOs powered down the HV & LV. Shifters switch the bypass key from the left side to the right side following the instruction from David.
- David had short access ~ 18:30, then the sTGC blower AC was restored. (~18:50)
- sTGC ROB 10 channel 1 (sTGC::LV2::114::1::imon) keeps making yellow alarms repeatedly and quickly disappears. (~01:12).
TOF:
- Prashanth & Jim restarted TOF/MTD archiver from the TOF machine in the gas room. Changed SF6 cylinder and Freon cylinder.
FCS:
- Error in “FEE count 44, expected 55; FEE count 33, expected 55 -- restart run. If the problem persists contact expert”. Then got a “configuration error”. DOs power-cycled the FEEs and reboot the FCS in the run control. But still have the same issue. Called Oleg.
- a problem with FCS ECal North. One of the MPOD PS boards shows 'outputfailure Maxs' all V and currents are at 0. It is not clear if it is a failure of MPOD itself, or if it is caused by one of the ECal FEE.
- Gerard found that FCS power channel u4 configuration readback values were wrong, looked like all defaults. Likely, this channel got a radiation upset. Reconfiguring the MPOD with setup script 'setup_FCS_LV.sh' restored correct operation
- FCS: DEP08:3 failed, restart the run fixed the problem
Network:
- MQ01 server: Disconnected the MQ01 server, unplugged all 4 disks from the MQ01 server, installed in the backup server (labeled in STARGW1), and then connected the backup server online with Wayne’s help. After rebooting the server, things seem to be working fine. DB monitoring is also back online.
- TOF/MTD Gas monitoring: went to the gas room, and started the EpicsToF program. The PVs start to update online. Alarms cleared.
- EPD: Tim forced a reboot of TUFF1 and 2. Now the EPD GUI reports "connected". Working fine now.

Schedule & Plans:

cosmic 13:00-19:00 request by sPHENIX, access: AC. FCS S 10, VME 62, BBC East 1, the fan of TOF(east vpd); reboot scserv (Wayne), TPC Laser (Prashanth)
Physics for the rest of the time
Low luminosity tomorrow or Thursday (6x6)

Sunday, July 28, 2024

Status & Issues:

TPC:
- #25209041, iTPC S13:1, DOs power-cycled it
- #25209057, TPX S02:6, DOs power-cycled it
- #25209065, 100% TPX/ITPC deadtime for over 1 mintue
- #25210015, iTPC S09:3, DOs power-cycled it, but still get the same error, masked it out
- #25210020 - TPX S22:04, higher value in the TPX Totall bytes per RDO, power-cycled it after the run
MTD:
- #25209043, some hot strips in the MTD strips vs BL (CirticalShiftPlots->MTD-<StripsvsBL) plot
Network;
- 19:15, EPD, EPD: TUFF[2] dead - check TUFF if RUNNING!; 19:25, lost the connection; QA plots look okay
- 00:00, TOF/MTD Gas; lost the connection; The computers in the gas room running ok, it is just the online database stop updating
- DOs visit the gas room once an hour, check the gas values in-person, Alexei provided some inputs on which value we can look for
- Lost the control of laser for camera 1 and 3
- call from Wayne. He said the online monitoring network issue is caused by MQ01 computer. He let us to reboot the MQ01 and check the net work connection of dashboard1 computer in the DAQ room. The MQ01 is dead, will try to replace the power supply.
Others:
- DAQ rate is a little bit high
- TPC pulser crate #55 is in an unknown state! Please make sure it is OFF! - it is off

Schedule & Plans:

A short access after this fill (request by sPHENIX), physics for the rest of the day
Tomorrow afternoon - 6 hours cosmic request by sPHENIX

Saturday, July 27, 2024

Status & Issues:

TPC:
- ITPC S11:2, masked out
- TPX S19:3, power-cycled; Shift Crew should look for spikes in rdoNobytes, and if spikes look for appropriate sector adc plots—details in the TPC reference plots and issue problem-solving slides.
- TPX S01:6 (#25208024), power-cycled
- iTPC S21:2 (#25208045, #25208046), power-cycled
- ITPC S16:4 (#25308048), power-cycled
- (#25208050 - #25208053) ITPC S17:1, S04:1, S16:4, power-cycled
- (#25208057) a spike in RDO_bytes plot TPX S11:4, power-cycled
- (#25209003) ITPC S16:4, DOs power-cycled it
- (#25209005) ITPC S07:1, DOs power-cycled it
- (#25209007) ITPC S17:4, DOs power-cycled it
- (#25209016) ITPC S04:1, DOs power-cycled it
- (#25209019) ITPC S16:6, S16:3, DOs power-cycled them
Environment alarm:
- Had a temperature alarm again (13:30), followed by a series of similar alarms for different subsystems on July 22. Called MCR and Jameela. The CAS watch and AC people came and fixed the problem (~15:14). Jameela scheduled an AC maintenance on the next maintenance day.

Schedule & Plans:

physics for all-day

Friday, July 26, 2024

Status & Issues:

TPC:
- TPX: RDO S21:6, power-cycled
- iTPC S02:1 power-cycled, still create problem, masked out
- TPX[28] [0xBA1C] died/rebooted -- restart a new run and it looks good
- 25207049-25207052: ITPC: RDO S18:4 , many auto-recoverys, again in the late night (25207059), power-cycled the it
- (22:48) TPC Anode sector-1 channel-5 tripped, shifters tried to clear the trip it didn't worked. So, individually cleared the trip following the manual.
- ITPC: RDO S11:2 -- auto-recovery failed. Powercyle this RDO manually & restart run. (25208018, 25208019 )
FCS:
- fcs10 issues: It gets stuck in fcs10 HCAL South FEE scan, Tonko increased the logging level to capture it in the log for the next occurence

New guide for FCS If the blinking issue happen again, try follows:

1) Powercycle FCS HCAL South FEEs in the FCS slow control.

2) "Reboot" FCS in the run control

3) Start a new run tun

4) If that failed, mask out the FCS[10] and record that in the shift log

TOF:
- (#25208020) Several TOF trays in error and do the auto-recovery. got the a red alarm from TOF LV THUBNE at same time. After the auto-recovery done the red alarm disappeared.

The list of TOF tray in error:

TOF: tray # in error: 66 68 69 70 71 72 73 74 75 76 77 79 80 81 82 83 84 85 86 87 89 90 91 92 93 94 95 122 -- auto-recovery, wait 6 seconds…

(#25208022)TOF THUB NE auto-recovery and triggered the red alarm. Alarm disappeared after the auto-recovery finished

Schedule & Plans:

physics for all day and weekends
Cosmic in next Monday (likely) requested by sPHENIX, Carl & Xiaoxuan & JH will work on the triggers during that time
Works plans on the list: AC. FCS S 10, VME 62, BBC East 1 & bwoo6, the fan of TOF(east vpd); reboot scserv (Wayne)

Thursday, July 25, 2024

Beam until around 15:30 (extended since 7:00); We had a short access to fix bTOW problem after beam dump; APEX until midnight; running physics until this morning.

Status & Issues:

TPC:
- (25206021 & 022) iS02:1, masked out; tpc.C:#621 RDO4: automatically masking FEE #7 error
Laser:
- Jim showed shifters about how to operate Laser
- Checked the magic crystals for the TPC lasers. The quantity of crystals is good and should last several more days.
- Alexei and Jim decided to increase the amount of methane flowing to the TPC (slightly) to try to increase the drift velocity. (It has been falling in recent days). So I turned FM2 clockwise by 3mm at the end of the index needle.
TOF gas: DOs switched from TOF Freon Line B to Line A
BTOW: Oleg and Yu made an access, replaced blown fuses for crate 0x0b it is configuring OK. Powercycled PMT box 39 (on separate power supply) and restore communications with boxes 41,42 and 39. BTOW sysetm restored and ready to go.
FCS: DEP10:6 in unmasked at 22:30 during fcs_led_tcd_only; but create problem when try to start the emc-check at the beginning of the fill (1:04). Tried try reboot trg and fcs, doesnt’t work; tried to only mask the 10:6, doesn’t work; masked the 10; - Tonko will look at it
Run control: Run control was frozen this morning right before the beam dump, couldn't close the windows at the beginning. Force it to close with the windows task manager, but couldn't bring it back after several try. Called Jeff, found vcx-server was not running in the background. Run control is back after rebooted the vcx-server (xlaunch). Since it happened in the end of the fill when the beam is about to dump, the problem didn't affect any physics run. - shifters can use the old shitcrew PC (in front of shift leader desk, RTS02) to start the run control if this happens and stop us to start/stop a physics run in the future
Network:
- Any new host attempting (e.g. yesterday rebooted sc3) to connect to scserv initially fails in the same way. Wayne want to reboot scserv to see if it changes anything, but want to hold off until a maintenance period.
- Temporatory solusion: if this issue is encountered again, please wait two minutes and try connecting again.
Others:
- #25207018:
  - 06:03:03 1 tcd CRITICAL tExcTask mvmeIrqLib.c:#477 UNKNOWN EXCEPTION: Task 0x01DFE148 suspended, exception 0x00000400.
  - 06:03:03 1 tcd CRITICAL tNetTask mvmeIrqLib.c:#477 UNKNOWN EXCEPTION: Task 0x01DEDA70 suspended, exception 0x00000700.
- #25207019: EPD West hit count shows two (relatively) not-very-efficient areas. Issues disappeared in the next urn;

Schedule & Plans:

Machine development is cancelled, so physics for all day
sPHENIX is addressing the suggestions got from the safety walkthrough for the isobutane, no clear schedule yet; Carl and JH will try to test the low luminosity trigger configurations on Friday morning (Carl & JH), Carl will send a guide to summarize trigger configuration exam did last time
Works plans on the list: AC. FCS S 10, VME 62, BBC East 1 & bwoo6, the fan of TOF(east vpd); reboot scserv (Wayne)

Wednesday, July 24, 2024

Status & Issues:

SC3:
- lost control on VPD,BBC,ZDC and VME crates due to sc3 CPU crash. David bought control of VPD/BBC/ZDC back at SC5; Wayne came and rebooted SC3
BTOW:
- configuration failed error around 20:50; Tried restarting the run, but the caution persists. Then realized this might due to the crash of sc3
- Oleg T. found three BEMC PMT boxes (39, 41, 42) are dead, and they are masked out for now.
- Error at 05:21:09: 1 btow CAUTION btowMain emc.C:#467 BTOW: Errors in Crate IDs: 11;BTOW: configuration failed
- At similar time, VME-9 emcvme9_i4val made a red temperature alarm. (5:43), Oleg suspects that the issue with the BTOW is due to the blown fuse.
- Also have a problem on connecting to VME processes on the platform, for BTOW data collector and BTOW canbus,
- An access is requested after this fill for Oleg, and Wayne (if needed)
- now running without BTOW+BSMD
GMT: trip at u3 channel. DOs performed a reset trip operation.
Trigger: Hank points out the document about how to fix the trigger related problem for shifters (https://www.star.bnl.gov/public/trg/trouble)
FCS: DEP10:6 failed again, masked from the component tree. To the shifters:
- If it is DEP 10:6 problem, masked 10:6 and run (already masked)
- If it is entire DEP 10 problem, take FCS out from the run, contact Tonko
Others:
- STAR control room door handle is fixed
- An “umbrella” is installed to temporary fix the ceiling leaks
- J.H. opened a BERT window for the beam-beam parameter. Now we can check the beam-beam parameter by it.

Schedule & Plans:

APEX for today (July 24) 8:00 - 00:00, - problem on AGS RF cooling water, the beam extended
Machine development assigned for tomorrow (July 25) 11:00-15:00
Still no clear timeline about when sPHENIX will flow the isobutane / have access / low luminosity runs - exam the trigger configurations on Friday morning (Carl & JH), Cail will send an guide to summarize trigger configuration exam did last time
Works plans on the list: AC. FCS S 10, VME 62, BBC East 1 & bwoo6, the fan of TOF(east vpd)

Tuesday, July 23, 2024

unexpected beam abort (~ 20:06)
MCR had a fake ODH alarm, but based on the safety procedure, still dump the beam earlier (~06:20)

Status & Issues:

TPC:
- TPX: RDO S09:5, recovered after start a new run
- #25204040, The TPC went 99% dead, this indicates it is external to TPC.( By doing a reply of daq you will see that at 12:27 the JP1 SCA triger rate goes to 3 mHz)
- #25204053, many ITPC RDO auto-recoveries and 100% TPX/iTPC dead time
- RDO4: automatically masking FEE #7
- power-cycled TPX: RDO S15:6
EPD: Mariia Stefaniak tried to fix the EPD problem, reboot TRG and DAQ and take some pedestal_rhicclock_clean runs
sTGC: Before #25205016, shifters restarted the sTGC LV and found some empty lines in the sTGC hits/FOB and empty space in hits/Qudrant. Power cycled after this run, and thing back to normal in the next run.
EEMC:
- (Day shift) red+blue indicator for sector 1, P1 (most inner circle) at the EEMC MAPMT FEE GUI. DOs followed the manual and solved the problem
- a new noisy PMT in ESMD starting from Run# 25204041
Tigger:
- (at 9:45am): 1) STP reset is failing. Runs will not work, please power cycle L0/L1 crate #62 2) STP reset finally worked. Do not power cycle L0/L1 crate
- L0 and L1 got stuck on TRG + FCS, shifters rebooted all component; still fail to start the run, FCS keep blinking; called Jeff; take the fcs[10] out; run could start. But today’s morning it is working again
- #25204066: There was a warning in daq monitor for L2. Event timed out by 97 ms, Token 861, Total Timeouts = 11, Timed Out Nodes = MIX_DSM2::dsm2-23. - Will be discussed in the trigger meeting
- #25204068: BQ_QTD[trg] [0x801E] died/rebooted -- try restarting the Run. Shifters tried rebooting trigger, didn't work. Then rebooted all, run could be started.
Others:
- Takafumi brings it up that the reference QA plot is out of date (https://drupal.star.bnl.gov/STAR/content/reference-plots-and-instructions-shift-crew-current-official-version), will add a list of recent good run as an example of additional reference
- The control room AC is still leaking, Jamella came and said will try to fix it ASAP
- The door handle (white door) to enter STAR control room is loose - call MCR and maintenence team

Schedule & Plans:

Physics for the rest of the day with 6 hours of fills
Possible chance to access after the sPHENIX isobutane safety walk-through (start at 11:00) in the afternoon. Works planed last time: AC. FCS S 10, VME 62, scaler board 5 (BBC E) & bwoo6 (Chris Perkins), the TOF east vpd - we decide to wait for the next access

Monday, July 22, 2024

Status & Issues:

TPC: power cycled TPX: RDO S02:3; RDO iS19:1 bad (#25203050, and a few runs after 25203051), powercycled this RDO, but did not work, masked it out
#25203031 & 25203044 - The shift crew noticed in the QA plots that RDO_bytes have a spike around 75 (TPX Total bytes per RDO), - may related to the dead time
FCS: Tim came and had access around 2 pm; Tonko with Tim checked the fPRE DAQ link for sector 10:6, the DEP board#13(from 1) in crate #3 (from 0). The issue remains after replacing the patch cable and SFP module...but in any case, from further evaluation, the issue seems to be w/ the DEP board. Time constraints for the access did not allow for enough time to replace + configure + test a new board in the system. DOs unmasked sector 10 RDO 6. Again not working at ~ 23:22, shifters took it out. But showing ok during the midnight shift.
EEMC: Will reconfigured MAPMT box 1P1 (a.k.a ESMD crate @71) at ~ 10:05. OTOH it very simply responded to a reconfigure so appears ok. Then it again tripped many times during the day. Still shows the errors in Crate IDs: 71. Follow the manual can clear the trip, I will notice all the shifters about clear this trip manually.
VME: VME62 got stuck. DOs reset it (14:56)
Environment alarm:
- TOF LV -> (East vpd) Terminal voltage triggering the yellow alarm from time to time over from ~16:00
- Wide Angle Hall temperature is 30.7 degrees at 17:36 (yellow alarm); raise to 31.1 at 18:29 (red alarm). VME Crate 55 (no in-used) temperature yellow alarm at ~ 19:00; sTGC LV ROB# 10 current alarm at ~17:16 (yellow); VME Crate 51 PS Air temperature transient yellow alarm at ~19:26. Called MCR, and they sent CAS Watch to STAR to have a look. Looks like the AC in the IR is not running, so the 2nd platform shows a high-temperature alarm, but the original diagnosis is they need to have access to fix it. Since the temperature is still ok to run, we scheduled access for the CAS watch and AC guys to come, investigate, and fix it at the end of the fill (midnight). Then they found both ACs for WAH were down. They have successfully turned on one AC, and the temperature has started to decrease. Since the temperature is gradually back to normal, and running ok now, we will keep running until the next access
Trigger:
- #25203026, “The run got stopped due to: L2[trg] [0x8201] died/rebooted -- try restarting the Run”, could not start a new run for 2 times, reboot the trigger and everything was running again
- Jeff updated the low rate prescale setting for fcsDiJPAsy*EPDveto - Good so far
- Hank power-cycled scaler board 5. Tim checked the patch cable for the BBC E PMT signal. The cable is connected and visually seems fine. But still no response. We will need to check further at scheduled maintenance access.
Others:
- water leak at STAR control room, which seems to be from a bad sealing of the AC, AC team got informed and ordered new parts to fix the problem;
- Waters outside STAR assembly hall, maintenance team got informed. They shut the water down.
- If similar thing happened, called MCR first (and/or Jameela), and then maybe water group x4668
BERT: the system freeze time to time so the notice doesn’t pop up, keep an eye on the BERT system, restart it if needed

Schedule & Plans:

Physics for the rest of the day with 6 hours of fills
We are now running with one AC on in the IR, looks fine so far, will try to schedule a work once there is a chance to have a long time access. So for the next access: AC. FCS S 10, VME 62, scaler board 5 (BBC E), the TOF east vpd

Sunday, July 21, 2024

It was quite a smooth day for our data-taking.

Status & Issues:

TPC: #25202047 stopped due to TPC dead time (TPX: RDO S18:3 -- auto-recovery)
Laser: DO and shift crew should check both drift velocity and charge distribution vs phi plot. The latter should show spikes at about the sector centers. Two examples are printed and left near the laser PC and shift leader's desk.
ETOF usually stocks about 3-5 minutes after the beginning of the run with the errors: ETOF has 1136>1000 EVB errors. It keeps happening. We are currently running without ETOF. Do we want to include it?
FCS: FCS10 is ready to go after Tonko power-cycled the DEP crate. The DEP10:6 remains masked. - request an access for 30 mins
ESMD warning: "ESMD: Errors in Crate IDs: 71 -- refer to Endcap Operations Guide, Data Transfer Issues", run with this warning error for the rest of the shift
EPD: Run 25202062 - The shift crew observed a new cold tile in the EPD West <ADC> plot.
Trigger: Hank noticed the BBC East Scaler board 5 has problem
Others: Ceiling leaks at STAR control room (at the top of the coffee maker table), called site maintenance, they are sending people here; another leaking is found in the assembling hall (in front of the gas room), called site maintenance

Schedule & Plans:

Physics for the rest of the day with 6 hours of fills (Significant more down time now, need to discuss if longer fill is ok in tomorrow’s meeting)

Saturday, July 20, 2024

Status & Issues:

TPC:

TPX S10:6 was masked out for #25201034, power-cycled, problem was fixed.

iTPC S13:3 was bad, restarting the run to fix the problem.

TPX S09:3; S23:4 bad, power-cycled it manually

iTPC S05:3 is masked out

BSMD: RDO 2 -- too many auto-recoveries stopped the run, Oleg looked at it, and it’s back now.

GMT: single-tripped HV module (u3). DOs manual to clear the trip by resetting and restoring the channel (section 2).

FCS: Yesterday morning DEP10:6 failed frequently in the early morning. Tonko looked at it and found many possible reasons (fiber optics interface is glitching, low voltage at the PS (unlikely), the fiber has been slightly dislodged, or some other board failure), but all need to have access. Tim found the location of the board (South: crate:#3, DEP board #13 (count from 0)), but we are not able to have access. Then FCS stopped the run to start a new run at around midnight, called Jeff, and tried to mask out 10:1, 10:6, or 10:8 but still couldn’t start the run. [fcs10 00:36:01 202] (fcsMain): ERROR: fcs_fee_c.C [line 1548]: S10:1 -- FEE scan failed [2]. Masked the whole sector (10) out. FCS->Pres->PresSouth is empty. Tonko looked at it this morning and fixed the problem in sector 10. We take a fcs_led_tcd_only run, looks ok so far. DEP10:6 still could have problem, mask it if it happened.

Trigger: Jeff: Changed prescales for some FCS triggers to increase rates of low threshold triggers when the luminosity is low, according to Carl's triggerboard suggestions.

The Windows machine to monitor the magnet is back online now.

#25201048: run stopped by: 3514|esbTask.C|Recovery failed for RDO(s): 1 -- stopping run. Try restarting. Fixed after restart. Not sure what’s the problem

Schedule & Plans:

Physics for the rest of the day with 6 hours fills

Friday, July 19, 2024

Beam quality improved after machine development

Fill 34826: Physics for sPHENIX started at 20:23; Physics for STAR started at 21:19. Production run started at 21:25 with ZDC rate ~ 20k

Fill 34829: Physics for sPHENIX started at 00:30; Physics for STAR started at 1:17. Production run started at 01:24, with ZDC rate at 22.4k

Status & Issues:

TPC:

Unmasked iTPC RDOs: iS08-1; iS09-4; iS10-3; iS11-3; iS13-1

Have problems again after replacement. Masked: TPX S11-3; S11-6; S20-4; S20-5

TPX[30] [0xBA1E] died/rebooted (#25201011 ) - reboot seems not working, but then come back by itself

TPX and ITPC are 100% dead due to ITPC S02:4; S18:4; S04:1 (#25200043); then ITPC S02:4 S02:2 (#25200044); ITPC RDO S10:3 (#25200051-cosmic); ITPC RDO S10:3 (failed multiple times, masket out) iTPC RDO S08-1 (#25201006, failed multiple times, masked out)

TPC Anode Trip (sector-23 channel 5)

TOF: TOF LV alarm (yellow) - power cycled TOF LV - cleared.

FCS: DEP10:6 failed - 4 times - Looks like the fiber optics interface is glitching. Tonko: Could be due to low voltage at the PS (unlikely) or the fiber has been slightly dislodged. Or some other board failure. - Need access?

Crate #84 on the 1st floor is yellow. Hear no alarm. The temperature of PS is about 46 and red status. The Fan Speed is 1200 and yellow status (evening shift)

BBC: Tim and Akio made access to fix the BBC scaler. It was a BBC-west discriminator which had offset from 0 in output. This was moved to working channel one below in the same module, and the output width was adjusted to 10nsec to match the old one. Now it's coming at a reasonable rate for noise & pocket pulser.

Windows:

Shift leader computer crashed at 00:53 and 1:30, rebooted. TPC caen-anode HV alarming during the second crash (25201005). DOs brought them back following the instructions by click "wake me up". Not be able to stop the run. After the Run control was back, this run already run over 15mins. The QA plots looks okay, so still mark this run as good. - the run control can run on any of the linux machine

The machine to monitor the magnet is not recovered yet

Schedule & Plans:

Physics for the rest of the day with 6 hours fill

Thursday, July 18, 2024

Completed the scheduled access work during yesterday’s access: Network switch power supply (UPS), BSMD, ESMD crate 71, Magnet water for magnet, TPX, FST coolant refill, Powercycle main canbus

One fill so far since yesterday’s maintenance, 40 mins after sPHENIX declared physics, we start with the STAR ZDC coincidence rate ~ 22kHz

Status & Issues:

TPC:

RDO: power cycled RDO S02:2 and S02:4, also power cycled iS08:1 for 3 times, still frequently stopped the run, masked out

Anode trip once in the morning (sector-23, channel 5).

FST: FST -> HV -> ROD 3 and 4 in red, shifters brought them back manually.

The total daq rate > 5K and scalar rate were high in red (9M) for JP and BHT triggers (25200008-25200013, 25200020-25200025). DO originally thought it is a trigger problem, so called Jeff. Jeff mentioned it could be a problem due to the triggered detector. Shifters do not see any problem from QA plots. Tonko and Oleg called in, pointed out it is DSM crate problem (L0-L1). Shifters power-cycle the BC1, BCE, and BCW (VME 72, 73, 76). The rate looked reasonable now.

FCS triggers scalar rate is high > 9M (25200029), recovered in the next run

TOF gas is alarming for PT-2, changed the bottle

To shifters:

New expert call list is updated, contact Prashanth or/and Jim Tomas if there is any TPC related problems

Record to the log if the run is stopped due to the "TPC 100% dead" issues

If experts hang out the phone when you call in the mid-night, leave a message to experts, no need to call multiple times. Experts are getting to solve the problem as soon as possible after they received the messages

Schedule & Plans:

30 mins access requested by sPHENIX, possible to request another longer access after machine development - we used this time in the morning to access and try to fix the BBC problem, power-cycled the crate but it seems not working (Jamie & Akio). We will need to have a longer access if possible after machine development time

Machine development: 1000-1400 (put detector to APEX mode) - Toko will work on TPC during the APEX; request access after this if possible for BBC (Akio & Tim);

Physics: 1400+

Wednesday, July 17, 2024

Status & Issues:

General: Beam dump around 7:30, magnet is down, having access now

TPC: S17:3 tripped; RDO iS17:4 bad; iS09:4 bad error, power cycled S09-4 and S17-4; masked out iS09:4 in the end

TOF: PT-1 gas alarm, switched from B to A

ETOF: eTOF DAQ reconfiguration procedure is not working, "ETOF configuring front end, be patient!" for hours after restarting the eTOF DAQ. Geary called in and fixed the problem for the next run. Then it has >1000 EVB error again

FCS: Akio uploaded new FCS Ecal HV file

STGC: a yellow gas alarm for the Pentane Counter at 12:39; bottles refiled by 14:51

L4: L4 live events display has been updated to include the global tracks back. The space charge parameters for L4 have also been updated. Now it support the users to select global tracks or primary tracks themselves in UI..

Trigger:

Run 25199011 - By the end of this run, the rate increased to 4K, JP1 is 2.5K.

Cannot start run as trg-L0 get stuck, reboot trigger once

Others:

unexpected beam loss ~ 2:54 yesterday and then 16:30 yesterday - request extra polarization measurement in the middle? - get statistics about unexpected beam loss

40 mins delay after turn sPHENIX physics for the last fill, miscommunications MCR. We will keep 0 min or 40 min. Will be discussed during the spokersperson meeting

PC with BERT got frozen for about 5 minutes (day shift)

AC in the control room is back - don’t touch the thermostat, contact Jameela if needed

To shifters: write the shift log on time, and write the summary log with more details on the problems

Access plan for today (to 16:00):

Network switch power supply (UPS) - Wayne

BSMD (with magnet off) - Oleg

ESMD crate 71

Magnet water for magnet - Prashanth

TPX - Tonko & Tim

Laser tuning - Alexei

FST coolant refill - Prithwish & Yu

Powercycle main canbus - David

Tour to students- 11:30 & 13:20 by Jeff & Prashanth & Yu

Schedule & Plans:

sPHENIX will request a few hours cosmic and some fill with less bunch for low luminosity after changing to a new mixed gas: use this time for STAR to tune our trigger? (configuration changes should be discussed/finalized with more advance time due to EPIC collaboration meeting next week)

During nominal daytime hours (0800-2000) CAD will operate with 4-hour stores after STAR is brought into collisions. Polarization measurements will be taken at 0 and 4 hours (skipping the 2/3 hours measurement). Outside of daytime hours, resume the nominal 6-hour store length after bringing STAR into collisions and follow the existing store recipes (i.e. polarization measurements every 3 hours and dump) - will be revisit after get the statistics on how often is the unexpected beam loss

Tuesday, July 16, 2024

Status & Issues

• EQ1_QTD died/rebooted in run 2597030

• FCS power-cycled between fills (Oleg T.)

• Jeff updated some triggers after the trigger board meeting (FCS DiJP/DiJPAsy and EM0/1 with EPD veto); starting with run 25197047

• ETOW configuration error (crate 1 fixed by DO, crate 2 later required intervention by W. Jacobs)

• GMT u3 tripped and recovered by DO

• Beam dumped for SPHENIX access (EMCal); next fill lost due to QLI

• L0 stuck, rebooted (x2)

• iTPC/TPX 100% dead in three runs

• Filled dumped just after 9 am for another SPHENIX access to fix EMCal problems

• Issues with l2ped web page persist; the plots are all available but the archive is not updated properly which causes index.html to stop on July 3; (l2btowCal has similar problem, but stops on July 7)

Schedule & Plans

• Maintenance day, Wednesday 0800-1600

o Network switch power supply (UPS)

o BSMD (magnet off)

o ESMD crate 71

o Magnet water for magnet (Prashanth will check if water group is ready for valve replacement)

o iTPC/TPX recovery (Tonko, Tim)

o Laser tuning (Alexei)

• Then back to physics

• SPHENIX will request 56 bunch fill for low luminosity in a few days; possibilities to use this for STAR? (configuration changes should be discussed/finalized with more advance time due to EPIC collaboration meeting next week)

Monday, July 15, 2024

Status & Issues

• Connection to VME was lost at start of fill in the morning; DAQ warning about crate #55 (pulser); resolved in consultation with David; power-cycled following the slow control manual; VME 50 was still yellow; power-cycled between runs, lost connection to gating grid recovered by David and cathode interlock

• Beam abort with anode trip about an hour before scheduled dump time

• Cosmics for a few hours; observed higher rates than before

• iTPC deadtime spikes in run 24196057; L1 invalid token at start of run

• iTPC RDO iS13-1 masked after unsuccessful attempts at power-cycling

• Other RDOs which required manual power-cycle: iS13-2 iS13-4

• iTPC/TPX 100% dead (in three runs)

• high rates in forward triggers in run 25197028; stopped quickly and started new run

• level 2 monitoring plots have not been updated on the web page; the analysis is producing output, but it is not updated on online.star.bnl.gov/l2algo/l2ped

Schedule & Plans

• SPHENIX is asking for a short access after the current fill

• SPHENIX rates at the start of fill are currently below the 24 kHZ which are the threshold to bring STAR on; detectors should be brought up when physics is On for SPHENIX

• Continue with physics until Wednesday morning (maintenance day)

Sunday, July 14, 2024

Status & Issues

• 30 minute access turned into closer to 2 hours; new fill after 3 hours

• BTOW configuration errors while trying to take pedestals; rebooted trigger

• Then L0 hangs; reboot trigger; power-cycled VME-62 (twice)

• ESMD errors in crate #71 at start of every run; Will was informed and we can ignore this for now (EEMC MAPMT boxes 1S3 and 1P1)

• DAQ message “requesting reconfigure from Run Control” in combination with power-cycling RDO S20-5 and “critical: RECONFIG ERR tpx-34”; masked out S20-5; eventually able to start run after trigger rebooted

• Mostly smooth data taking through late afternoon and night; bgRemedyTest with 10k at start and end of each fill

• BTOW configuration failed in two more runs (not consecutive)

• EPD timing scan in runs 25195082 – 086

• sTGC hits/timebin has low counts early in fill 34799 (has happened before in some runs last week)

• L0 hangs one more time

• One run ends with 100% deadtime TPX & iTPC

Schedule & Plans

• Continue running physics until maintenance day (Wednesday)

• Include bgRemedyTest in fills as before (10k events)

• Discussion of beta* tomorrow

• Discussion of EPD timing cuts in trigger board meeting on Monday

Saturday, July 13, 2024

Status & Issues

• Akio power-cycled scaler crate; BBC And is back

• TPX RDO S11-3 and S11-6 are masked out due to power problems; Tim needs to take a look during maintenance day

• iTPC RDO iS09-3 investigation is continuing (added error messages for Tonko); mask again when it fails

• EPD veto on early hits is now in the production files (starting from run 25194034); shift crews have observed differences in EPD <TAC> (EPD expert suggested to reboot trigger and take pedestal_rhicclock_clean afterwards, this should have been added to the shiftlog)

• Stuck bit caused the high rates in EHT0; power-cycled TP-2 crate (Will J.)

• Rongrong tried to recover MTD BL 28; unsuccessful, still masked out

• Trigger group tested tier 1 file; everything back to default (?)

• Took some cosmics due to extended access/downtime

• Collisions at 1820

• Shift crew encountered: L0 died/rebooted, TPX[8] died/rebooted, iTP RDO iS10-3 power-cycle (repeatedly, then masked), iTPC[10] had to be power-cycled manually (Jeff)

• Power dip between fills at start of night shift with magnet trip, global interlock alarm, TPC FEE and RICH scalers white

• Magnet back up at 2:35 am

• Oleg T. recovered BEMC after MCW loss; HT TP 163 and 291 are masked out; BSMD is 50% dead and was turned off (until maintenance day)

• FST failure code 2 before first production run

• Combination of high rates in JP triggers and TPX/iTPC deadtime; rebooted trigger; power-cycled all RDOs; then again RDO S20-4/5 (again in the next run)

• Will J. recovered all MAPMT for ETOW; issue remaining MAPMT 1P1 (Will says it’s overheating, experts are aware)

• Run control was very slow in the morning, it seems to be running fine now

Schedule & Plans

• Continue physics data taking: pp200_production_radial

• bgRemedyTest_2024 at start and end fill

• EPD delay scan in next fill (non-intrusive during regular production run, see Hank’s email for details)

Friday, July 12, 2024

Status & Issues

• MTD HV trip in BL 15, 16 & 17 (early in fill 34785); power-cycled and back for next run

• Magnet trip at 10:30 am; strainers were cleaned during our downtime, but it is not completely clear where the problem is; valve replacement is ordered and should be replaced during maintenance; David Chan and team looked through temperature logs from different locations; magnet ramped up after 5 pm, temperatures looked fine and stabilized well under the trip threshold

• Network power switch died (splat-s60-2); Wayne was able to diagnose remotely; Jeff and Tim prepared access work; UPS was in “overheat error”; Tim plugged the network switch into the rack power

• MCR did a vernier scan for themselves while the magnet was down (and optimized our rate…?)

• Some problems coming back; Jeff, Rongrong, Gavin on zoom; one fill lost during ramp; everything was back for collisions at next fill

• MTD BL 28 is masked out

• FST problems with RDO 1-5 and 2-6; no problems when detector was at full HV

• BBC And is 0 in scaler GUI (Akio is looking at it)

• BTOW configuration failed in one run

• sTGC yellow/red gas alarm again this morning (Prashanth has been informed)

• elevated temperatures on VME-84 and 98 (EQ4, BDB)

• bgRemedyTest_2024; runs 25193…, 25194009, 017, 030

• ETOW HT trigger patch #81 is hot; EHT0 rate too high (prescaled at 50 now)

Schedule & Plans

• 1.5 hour access after this fill; dump time moved up to 10 am (condensation in tunnel, SPHENIX)

• STAR to get collisions at 24 kHz (SPHENIX)

• Carl’s bgRemedy studies confirm efficiency of background rejection for forward triggers; will send summary with configuration changes

• EPD delay scan (5 production runs) waiting for confirmation from Hank

Thursday, July 11, 2024

Status & Issues

• APEX study of spin direction at STAR was not successful and postponed

• Magnet trips at 12:17 pm and again at 1:26 pm; magnet at half field until 6pm, then back up to full field

• Collisions at 6:45 pm (75 minutes after SPHENIX)

• Several problems when starting run; BTOW configuration; TOF LV THUB NW tray 45, west 4 (power-cycled); iTPC RDO iS09-3 masked out

• Beam lost at start of second physics run

• East and west trim currents were not ramped up to full field; NMR showed 0.4965 T instead of 0.4988 T; ramped at 8:20 pm (mark the two runs as bad)

• Overnight fill with horrible yellow lifetime (tune changes during the ramp); STAR only 20 minutes behind SPHENIX but rates low from the start

• bgRemedyTest_2024 (runs 25192042, 25193016, did not include FCS)

• sTGC gas alarm (fluctuating, Prashanth was made aware)

• Tonko already looked at problematic RDOs from last night; iS01-1 reburned PROM; iS09-3 not clear what is wrong, unmasked again; iS09-4 disabled 4 FEEs

• STAR a little more than an hour behind SPHENIX

Schedule & Plans

• Physics until maintenance day (Wednesday, July 17)

• Vernier scan at the end of current fill (early afternoon)

• bgRemedyTest_2024 at start and end of two fills (Hank will double check tier 1 parameters and file/dates)

• Timing scan in regular runs on hold until after bgRemedyTest

Wednesday, July 10, 2024

Status & Issues

• TOF freon changed to bottle A

• epdTest-radial in new fill (run 25191030); cuts on early hits look good; in the process of being implemented -> bgRemedyTest_2024 is ready

• TPX RDO S01-5

• GMT u3 HV tripped (DO recovered, no further issues)

• L2 died/rebooted during configuration of one run; started new run without problem

• MTD low voltage THUBN alarm (run 25191041)

• iTPC RDO iS10-3 was masked out after repeated failure in pedAsPhys (waiting for collisions while SPHENIX was up already); masked out; Tonko reburned PROMs on iS06-2 and iS10-3 and unmasked them before APEX this morning

• Took cosmics data until APEX

• iTPC cluster occupancy in QA histogram is out of range early in the fill (e.g. compare runs 25191031 & 46)

Schedule & Plans

• Study of polarization vector during APEX today; take zdcPolarimetry runs when MCR does scan of different parameters (15-20 minutes x 2)

• Back to physics at 1600

• bgRemedyTest_2024 at start and end of fill

• trigger group requests five regular runs with modified settings (non-invasive to physics, details in Hank’s email)

Tuesday, July 9, 2024

Status & Issues

• iTPC RDO iS17-3 fixed and unmasked (Tonko)

• 3+ hours of cosmics data; first fill dumped after SPHENIX request for access (about one hour of collisions for SPHENIX)

• epdTest-radial with new TAC stop registers (run 25190055)

• ITPC RDO iS06-2 masked out after unsuccessful power-cycle

• Timeouts in l2ana01; low data rate (not sure if this is related, happened about 2 minutes apart)

• TPC anode trip S20-9

• TPX[24] died/rebooted

• iTPC RDO S02-4

• Fill extended due to problems with injection / BtA

Schedule & Plans

• bgRemedyTest_2024 updated after discussion in trigger board; ready for use once tier1 file is updated (Hank); take short run at start and end of each fill (TRG+DAQ+BEMC+EEMC+TOF+FCS)

• Wednesday APEX 0800-1600; continue physics until then

• Schedule a vernier scan in the near future (at the end of a fill)

Monday, July 8, 2024

Status & Issues

• GMT gas bottle replaced (reminder: even after switching to new bottle, the alarm keeps going until the empty bottle is replaced)

• TPX /iTPC RDOs: S11-6 (now masked out); iS02-4; power-cycled all after three failed attempts at run start

• TPX[31] died/rebooted during pedestal run

• Peak in TPC drift velocity is sometimes wide (run 25190013, improved in run 019)

• Magnet trip in fill 34764; restored without beam dump; polarization also looked ok in the next measurement

• Lost beam twice during injection / ramp

• Beam abort this morning; lead flows in sector 10 (problematic all week, being investigated now)

• EPD timing test looks good; background removed effectively (Hank)

Schedule & Plans

• Time between sPHENIX and STAR physics: over 100 minutes!

• Uptime 14 hours on Saturday; less than 9 hours on Sunday

• Short epdTest-radial in next fill

• Continue physics: pp200_production_radial until Wednesday morning (APEX)

Sunday, July 7, 2024

Status & Issues

• bgUpcTest with all detectors (25188041, 61, 68, 25189008)

• Lost laser view; no laser runs in fill 34758; Alexei got a short access between fills and restored the connection to laser platform

• STGC: ROB #03 bad FEB required power-cycle

• TPX[30] [0xBA1E] died/rebooted (running fine in the next run)

• Magnet trip at 6 pm; CAS were unable to clear the fault; clogged strainer for the supply; cleared by 6:55 pm when RHIC had just started injecting beam; ramped magnet and restarted RHIC fill

• TOF pt-2 alarm procedure updated (Alexei)

• GMT U3 HV tripped once

• Two peaks in “TPX Total bytes per RDO" (sectors 6 & 21); power-cycle cleared this

• sPHENIX had problems bringing down one of their detectors; unfortunately, MCR called us first while we were waiting for “ready to dump” from sPHENIX; ended up with 30 minute zdcPolarimetry

• Some issues with to many TOF recoveries; power-cycled LV; eventually had to go through CANbus restart procedure which solved the problem

• TPX RDO 17-5, 11-6, 11-3; iTPC RDO 02-4

Schedule & Plans

• Time between sPHENIX and STAR physics: 13, 8, 16, 32, 32, 59 minutes

• bgUpcTest is finished -> decision from trigger board (Monday)

• Continue physics: pp200_production_radial until Wednesday morning (APEX)

• Wayne is not available for next week (call Jeff for immediate help)

Saturday, July 6, 2024

Status & Issues

• Machine development; first attempted fill around 4 pm; lost when STAR going into collisions

• TPX/iTPC maintenance (Tonko)

o TPX S11-6 reburned PROM (still problematic)

o iTPC iS7-2 this is an older bad one but I think I fixed it; unmasked.

o iTPC S9-3: masked FEE #6; unmasked RDO

• NMR is now recorded in database; still logged overnight but not necessary anymore

• Jeff updated expected rates for DAQ monitoring

• Took EPD test runs for trigger group (epdTest-radial, runs 25187037, 38, 39, 40)

• Test configuration for bgUpcTest; 2-3 minutes at start and end of fill; didn’t use all required detectors in recent fill; redo in next fill

• EEMC HV yellow alarm; had to reload HV values several times; some sectors went from red to black; ran without ETOW in the meantime

• Trigger 100% dead, couldn’t properly end the run; force stop and reboot trigger

• A few runs going into TPX/iTPC 100% dead after a few minutes this morning

• TPX RDO S11-6, S15-6; iTPC RDO iS12-3, iS17-3 (recurring, had to be masked)

Schedule & Plans

• Time from Physics:On to STAR collisions: 13 min, 8 min, 16 min

• Beams look stable after sPHENIX is in collision; we can start bringing up detectors after MCR call; rates should be monitored carefully until STAR gets to Physics:On

• bgUpcTest with all detectors (only need one good fill)

• Continue physics: pp200_production_radial until Wednesday morning (APEX)

Friday, July 5, 2024

Status & Issues

• At the start of fill 34747, problems starting with TPX[30] and failing STP resets. Run couldn’t be stopped properly and VME #62 power cycling wasn’t successful. Akio looked remotely, but also couldn’t help. Jeff eventually separated problems with trigger from TPX. pedAsPhys was successful on second try. Then hard reset of TPX[30] in the DAQ room. (Error in dsm2-3 in STP monitoring is not critical for data taking.)

This happened again when fill 34748 was lost. Shift crew tried to power-cycle VME #62; no success from control room or Jeff remotely. David got a short access, couldn’t power-cycle on the crate itself. Tim was not available, so we decided to hard reset (unplug). Fortunately, this solved the problem and VME #62 came back just as RHIC was about to reinject.

• TPX, iTPC & FST deadtime issues in a few runs throughout the day. Clarified how to mark the runs and recovery with shift crew. (many auto-recoveries in early runs in new fill)

• sTGC pressure PT-1 yellow warning (fluctuating around threshold, may reappear during the daytime)

• FCS DEP 04:5 failed once (DAQ message has instructions for shift crew, no further issue)

• David changed the sTGC gas bottle

• Manual power-cycling of TPX RDOs 11:6 (many times), 22:6, 03:4, 14:6

Schedule & Plans

• Machine development today (~5 hours)

• Test run for trigger modifications; details will follow (Carl, Jeff)

• Continue physics: pp200_production_radial through the weekend

• Suggested to try to reproduce the VME #62 problems during next maintenance day for better diagnosis

Thursday, July 4, 2024

Status & Issues

• Lost beam before 10 am; then machine development

• BSMD shows high current at start of fill; Oleg T. said to run as is and power cycle later (~90 minutes)

• New cold channel in EPD (run 25185031)

• TPC 100% dead at start of one run (three other runs where it happened later; run 25185031 should not be marked as bad)

• Pentane refilled (David)

• NMR can be read from the control room now (David)

• Hank asked for repeat of epdTest_radial (run 25184043)

• “PCI Bus Error: status: 0x01” in emc-check and next run; reboot TRG + DAQ

• Few runs with TPX 100% deadtime after a few minutes; then L2 timeouts -> reboot all fixed it

• TPX RDO S11:3, S11:6; iTPC RDO S09:3 (many times this morning, now masked out)

• Beam abort around 5:20 am; beam permit dropped and couldn’t be cleared remotely; ran cosmics for a few hours; new fill coming up now

Schedule & Plans

• Machine development on Friday (about 5 hours)

• Continue physics: pp200_production_radial

Wednesday, July 3, 2024

Status & Issues

• BTOW crate 0x10 was recovered; trigger patches for this crate were un-masked and tested (Oleg T, Tim)

• MAPMT sectors 2&3 HVSys module replaced (Tim)

• TPX & iTPC maintenance done (S11-6 seemed ok, failed once during cosmics)

• Cosmics data throughout the afternoon

• TPC anode sector 23 channel 5 tripped; “clear trip” didn’t work; manual recovery

• Maintenance extended until 8 pm (request from sPHENIX); new fill up by 9 pm

• Jeff added log info for STP failure -> power cycle L0/L1 crate #62

• GMT HV gui wasn’t responding; DO power-cycled the crate following the manual

• Intermittent yellow alarm on sTGC PT2 & 3

• BEMC CANBus needed to be rebooted (white alarm on CANBus, VME-1, 12, 16, 20, 24, 27)

• epdTest runs (25184076, 077, 078 - all EPD detectors see the early hits now, detailed analysis is on-going)

• FST random noise (non-ZS) plots are empty (run 25185002)

• EEMC gui turned white after beam loss; two yellow warnings remain (VME-90, 97); expert was informed; ok to run for now

• ETOF was taken out of run control (Geary’s email); Norbert called around 2 am and said it should be ready again

• Power-cycle TPX RDO 20:6

Schedule & Plans

• Machine development from 10 am until 1 pm (or earlier); sPHENIX asked for 10 min. access

• Continue physics: pp200_production_radial

• APEX Wednesday. July 10 (maybe later)

Tuesday, July 2. 2024

Status & Issues

• Trigger configurations updated with TofMult0 after discussion in trigger board meeting; everything handled in the existing configuration files, no need to change the procedure for the shift crew (in effect from run 25183041, bgRemedyTest_2024 not needed for the time being)

• iTPC RDO 02:4 manual recovery

• FST deadtime 100% (Fleming suggested correlation with trigger rates at beginning of run, check run 25184013, mark runs as bad)

• BSMD HV not ready for first run in fill 34733

• TOF gas switched to line B (11:51 pm)

• TPX/iTPC 90% dead for three attempted runs, eventually masked TPX RDO S9:4 (Tonko, done)

• Maintenance day: beam dump at 8:10 am, magnet down

Schedule & Plans

• Sweep at 5:30 pm

• Wednesday: machine development (2nd storage ramp, 10 am, 3 hours)

• Next APEX: July 10 (possibly postponed / combined with next session)

Monday, July 1. 2024

Status & Issues

• BSMD trips at beginning of fill

• TPC gating grid error and anode trips (first run in fill)

• Investigation of “non-super-critical pedestal problem in EQ4” (Maria, Mike); slightly shifted ADC spectra, does not affect the trigger at the moment, will communicate with trigger group if this changes

• evbx2 connection error? (run 25182073), L2 died/rebooted in the next run; all ok in 076

• evb01 | sfClient | Can't stat "/d/mergedFile/SMALLFILE_st_upc_25182078_raw_2400007.daq" [no such file or directory] (run 25178079)

• New alarm (buzz) for critical alarms in DAQ log (DAQ_announcer.sh, David)

• Beam loss at 1:25 am; regulator card on yo1.tq6 replaced at 5:25 am

• VPD alarm on slot 15-13; DO couldn’t recover; Akio looked remotely and said to ignore for now; slow control should take a look and maybe change limits (3 V out of 2 kV

• Took cosmics data for the rest of the night

Schedule & Plans

• Background discussion

o Vertex study (special production with no vertex constraint, Ting’s fast offline analysis)

o bgRemedyTest_2024: 25182039, 047, 061, 069, 25183021

o Current bunch intensities are close to the loss limit at ramp (recent losses during rotator ramp)

• Test of separated collision setup for sPHENIX and STAR

• Continue physics: pp200_production_radial

• Maintenance day tomorrow (Tuesday, 9 am – 4:30 pm)

o Magnet ramp down after beam dump

o EEMC PS (Tim, Oleg)

o TPC electronics

o Laser (Alexei)

o Windows update shift leader desktop

Sunday, June 30, 2024

Status & Issues

• TPC RDO S11:6 remains masked

• ETOW configuration failed in one run

• L2 died in one run

• Isobutane fraction ratio was higher than expected, followed the procedure for restoring the ratio (after 30 min. wait)

• TPC, iTPC, FST hung a few runs on 100% deadtime; shift crew takes action within 2-3 minutes (when it doesn’t self-recover)

• “FCS: powercycling DEP02:4” turns into “K?[0m” in DAQ monitor

• TOF LV needed power cycling after too many errors (detector operators, tray 54 west 5 needed manual intervention)

• BSMD had some trips early in fill, excluded for one run

• Manual power-cycle on iTPC RDO iS13:3, TPX RDO S06:4

• sTGC: ROB #04 bad FEB (followed procedure to start new run, power-cycled eventually)

Schedule & Plans

• bgRemedyTest_2024: 25181040, 045, 059, 067, 25182019, 025

• Special fast offline production for background studies is running (and progressing nicely) [fills 34714, 16 after the most recent modifications to the beam on Thursday]

• Continue physics: pp200_production_radial

• Good turnaround times for RHIC with current bunch intensities; sampled luminosity still a little below 50% of 2015

• Windows update on shift leader desktop (maintenance day)

Saturday, June 29, 2024

Status & Issues

• “sTGC hits / timebin” high early in fill (25180030)

• TOF gas changed (PT-2), methane last night, isobutane this morning

• bgRemedyTest_2024 run 25180057

• FST 99% dead; start new run

• TPX & iTPC 100% dead repeatedly and not recovering; power cycled RDO S05:6, S11:3 and S11:6 (twice); S11:6 failed again twice the new fill and masked out for now

• BSMD had difficulty to ramp in new store; excluded for first few runs

Schedule & Plans

• Jamie updated the ZDC coincidence cross section: 0.23 mb (down from 0.264 mb previous years)

• Need gas bottle delivery; will run out in about 18 days (Alexei, Prashanth)

• sPHENIX is slow in ramping down (polarimetry & beam dump); we may gain 5 minutes before ZdcPolarimetry at beam dump

• Modifications to trigger configuration: get more data with bgRemedyTest_2024 with TRG+DAQ at beginning and end of fills throughout the weekend (takes about two minutes each with TOF, other detectors can ramp HV; bgRemedyTest before ZdcPolarimetry)

Friday, June 28, 2024

Status & Issues

• EEMC sectors 2&3 (maintenance day…)

• Follow-up on DSM board (Hank, may need an access)

• L2/L0 problems between fills (Akio/Hank, power cycled VME-72 & VME-100)

• Configuration errors in ESMD (emc-check and several runs after); sys-reset after call from Will

• BBQ, EQ2, EQ3 failed during pedestal run(s), success on 4th attempt

• sTGC ROB#3 power cycle

• Masked RDO iS01:2 after it couldn’t be recovered

• TPX RDO S11:6 power cycled manually

• One call to Jeff when TPX & iTPC went 100% dead repeatedly; power cycled all FEE’s

• Quality of laser events is often low, Alexei is following up with DO

Schedule & Plans

• Physics collisions until Tuesday morning

• Blue beam background studies (Ting looked at vertex distributions for abort gaps)

• Ask Gene to have FastOffline without vertex cuts for a few runs from fill 34714

• Modifications to TAC start to reduce early hits from background events (Hank)

• Modified trigger configuration for early runs has been prepared (bgRemedyTest_2024); will try to run test at next fill (needs TOF HV up, can run while others are still ramping)

• Updates to power-dip recovery work sheet (input from some subsystems still needed)

Thursday, June 27

Status & Issues

• EPD trigger test runs done (25178033, 25178040)

• 7-9 minutes from Physics On to data taking

• Cosmic data during Linac RF recovery in afternoon

• Severe thunderstorm warning in the evening; thunderstorm eventually came through at 2:30 am; then power dip with magnet trip

• MCW had a blown fuse; all VME crates were turned off (CAS watch & Prashanth, fixed at 7 am)

• tpcAirHygroF alarm; Prashanth reset the TPC air blower

• EEMC sectors 2&3 still tripping (looking for access opportunity)

• LeCroy communication lost (DO->David), Akio reset it remotely

• DSM board still causing trouble (Oleg -> Hank)

Schedule & Plans

• No APEX today

• Physics collisions: pp200_production_radial

• FastOffline data for abort gap studies of beam background (Ting?); trigger proposal postponed (Carl & Akio)

• Recovery procedure from power dip -> update detector check list

• Next maintenance on Tuesday, July 2

Wednesday, June 26

• Day of “Reflection on Safety” (Prashanth)

Status & Issues

• Eleanor requested changes to trigger registers in epdTest-emc-check and epdTest-radial (now using default values)

• Cogging adjustment: TPC vertex is -4 cm (BBC now at -10 cm ???)

• ETOF fails twice in first few hours of fill, not included afterwards as per updated instructions

• Glitch with BSMD HV GUI before pedestal run, restarted GUI (instructions updated)

• New BERT feature: “Prepare / Ready for Pol. Meas.”

• EEMC sectors 2&3 trip every few hours (Tim)

• TPX RDO 11:6 has to be power cycled manually about once per shift

• EVB11 is dead, taken out (3 am, Jeff)

• FST running 99% dead during laser run (stop and restart)

• Laser events are low, although bright spot on camera (Alexei)

Schedule & Plans

• Physics: pp200_production_radial

• Request for trigger test: epdTest-emc-check and epdTest-radial at the end of current fill (5 minutes each)

• Slow increase in bunch intensity (now 1.5e11), yellow polarization lifetime

• APEX still tentatively on Thursday (decision tonight, possibility for access in the morning 8 am, EEMC: Tim/Oleg)

• Next maintenance on Tuesday, July 2

From 06/05/2024 to 06/24/2024, Period Coordinator: Jaroslav Adam, notes:

06/25/24

I. RHIC schedule

a) Machine development today 10am till 1pm, then collisions also on Wednesday

b) APEX on Thursday Jun 27

c) Collisions on Friday Jun 28 and over the Weekend

d) Next maintenance on Tuesday July 2

II. STAR status and issues

a) Hot tower in BEMC eta-phi plot, trigger rates are normal, ignore it

b) Intermittent alarms from sTGC ROB#10, current fluctuating at threshold,
threshold to be moved

c) Gas alarms (intermittent) on boxes on window to DAQ room to be reported as log entry

d) TOF LV yellow alarms to be only reported as log entry (email by Geary
"Log entry at 14:55 yesterday")

e) Shift for z-vertex is seen for TPC L4, not for BBC (2 cm off), but similar shift
seen at sPHENIX, fast offline to be checked

f) Possibility for 3 hour access during APEX on Thursday Jun 27, during work hours
if it happens, BEMC is ready

g) No collision data yesterday, VPD west TAC looks same as for collisions,
result of blue beam background

h) Safety program tomorrow, 5 mins for safety during 10am meeting

i) Next period coordinator is Oleg Eyser

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/24/24

I. RHIC schedule

a) Store with 0 mrad crossing angle for sPHENIX at noon for 3+ hours,
no collisions for STAR, consider this store as beam development

b) Physics today again at 4+pm

II. STAR status and issues

a) Smooth running yesterday

b) z-vertex is shifted by about -11 cm, seen only for TPC vertex finder (space charge),
+/-5 cm shift is ok, position to be checked with VPD

c) sPHENIX will be asking for 4 hours without beam (TPC distortions) soon when RHIC is off
for some other reason, opportunity for BTOW crate 0x10 and EEMC HVSys A controller

d) Question on including eTOF later in the fill, crews observed more BUSY problems
at the beginning of the fill, suggestion to try twice in each store to include eTOF back

III. Plans

a) No data to be taken for store at noon today, the store is aimed for 0 mrad at sPHENIX

b) Radially polarized data taking at high luminosity, pp200_production_radial, for store at 4+pm

06/23/24

I. RHIC schedule

a) Physics today, adjustments for yellow polarization

b) Test for zero crossing angle at sPHENIX tomorrow Jun 24

c) Machine development on Tuesday Jun 25

II. STAR status and issues

a) Smooth running yesterday

b) Pentane bottle changed for sTGC, DOs rebooted EEMC controls

c) Shifts in vertex z position are being corrected from RHIC side

d) eTOF BUSY, procedure in eTOF manual from May 27 (in production run,
no need to stop the run, take in out for the next run)

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/22/24

I. RHIC schedule

a) Physics over the weekend

II. STAR status and issues

a) Zero field, low luminosity store took place yesterday 10pm till 3:30 am

b) VME crates were off from 1pm to 4pm, potential issue with MCW, reached 79F,
several issues when turning on (BCW turned on after several tries, multimeters
for field cage had to be power cycled during 5min access)

c) Inform David before turning off VMEs due to temperature

d) EEMC HV was restored with help of Will Jacobs and DOs configuring part
of it manually

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/21/24

I. RHIC schedule

a) Machine development today 11am till 3pm, physics after

b) Physics over the weekend

II. STAR status and issues

a) Opportunity for low luminosity, 56 bunches, zero field store after the development
at 3+pm, 30kHz BBC rate was requested for the store, call Akio when we get
the store; the store will be polarized

b) BTOW crate 0x10 is still masked and disconnected, Tim dealing with one board
from that crate in lab, then an access for several hours with magnet off
would be needed

c) EEMC problematic HVSYs A controller was replaced by a spare (Tim), spare
did not work, original controller is in place now

d) EPD crate 4 early hits, two new configurations (Eleanor+Jeff) to be tested
with timing setup, to be run after emc_check during normal polarization run,
email by Hank with details to be sent

e) Lecroy1445 for BBC/VPD/ZDC, procedure to restore communication for DOs now works

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial,
after the potential low luminosity zero field store

06/20/24

I. RHIC schedule

a) Maintenance now 8am till 6pm, physics after

b) Friday Jun 21 machine development 9am till 1pm, then physics also over weekend

II. STAR status and issues

a) Maintenance now, restricted access, work on west trim magnet (had multiple
trips past days), TPC RDOs (Tonko), BTOW (Oleg + Prashanth + Tim)

b) For NMR readout, wait for magnet ramp to finish before reporting to shift log
(and green indicator 'NMR LOCK' to the left of field value should be lit for field
to be valid), now hold readouts till 6pm

c) Procedure for magnet ramp to update to instruct MCR to wait with ramping magnet
back until STAR informs them is ready to do so

d) Visit to STAR today afternoon

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/19/24

I. RHIC schedule

a) Physics today, maintenance tomorrow, Thursday Jun 20

b) Friday Jun 21 machine development 9am till 1pm, then physics also over weekend

II. STAR status and issues

a) Magnet trip on all magnets after a power dip yesterday 6pm, CAS watch replaced
regulator card for west trim (current was ~10A lower than set value)

b) BCW crate #76 turned on only after several tries (was turning off itself after
several seconds), Jeff tested trigger, ok now

c) Lecroy1445 for BBC/VPD/ZDC lost communication, DOs could not recover because procedure
involves ssh login to one of SC machines on platform which did not work - now crews
should call David or me when it happens

d) +/-5V oscillation on power line, CAD investigating its cause

e) Current state of online QA plots to be checked by crews at shift change - TPC occupancy
may change over time depending on RDO availability, similar holds for BTOW

f) Maintenance tomorrow, Jun 20, work on west trim magnet (had multiple trips past days),
TPC RDOs (Tonko), BTOW (Oleg + Prashanth + Tim)

g) Visit to STAR tomorrow afternoon

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/18/24

I. RHIC schedule

a) Machine development today 11am till 2pm, then physics

b) Physics Wednesday, maintenance on Thursday Jun 20

c) Friday Jun 21 machine development 9am till 1pm, then physics also over weekend

II. STAR status and issues

a) Access yesterday for BTOW radstoneBoards and DSM1 board in BCW crate finished ok
(DSM1 board was replaced in BCW crate and controller for BTOW crate #80 was replaced
- radstone were ok)

b) Maintenance on Thursday Jun 20, work on west trim magnet (had multiple trips past
days), TPC RDOs (Tonko), BTOW (Oleg + Prashanth + Tim)

c) Alignment data with field off when machine is in stable condition, by end of June

d) FCS ECAL voltage file changed by Akio to compensate for radiation damage

e) Online plots seen to fill slowly in the morning, Jeff working on automatic
restarts for Jevp plots

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/17/24

I. RHIC schedule

a) Physics today, intensity increase 0.1e11/store

b) Next maintenance Thursday Jun 20 (not Wednesday because of holidays)

II. STAR status and issues

a) BTOW/BSMD out of the runs, radstoneBoards in crate #80 can't initialize,
access needed, end of current fill at 2:30

b) JP2 triggers firing hot, taken out, access needed for BW003 DSM board
(stuck bit)

c) Jevp plots crashed two times, recovered by Jeff and Wayne, new instruction
for shift crews to be provided

d) Multiple magnet trips for west trim, instruction for shift crews to first
     put detectors to magnet ramp and then call CAS watch (they're very quick
     in ramping the trim back), item for maintenance on Thursday from CAS side;
     update instruction to call Prashanth in case of magnet trip

e) Alignment data with field off, tbd at coordination meeting Tuesday

f) NMR field inconsistent with readings on magnet current - variations in read
current values

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/16/24

I. RHIC schedule

a) PS issues at RHIC, attempt for polarized beams ended in unexpected abort
at flattop at 4am

b) Next maintenance Thursday Jun 20 (not Wednesday because of holidays)

II. STAR status and issues

a) Magnet trips in west trim, 5 times

b) Jevp plots and run control window crashed, recovered by Jeff,
log at 17:52 yesterday

c) Cosmics since 4am

d) Alignment data with field off, tbd at coordination meeting Tuesday

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial,
after cold snake and PS issues are recovered

06/15/24

I. RHIC schedule

a) Unpolarized stores, polarization after cold snake is recovered,
expected later today

b) Physics over the weekend

c) Next maintenance Thursday Jun 20 (not Wednesday because of holidays)

II. STAR status and issues

a) Multiple trips after unexpected beam abort around 1:30am today (MTD, BSMD,
     sTGC, EEMC, TPC), updating Detector Readiness Checklist to wait 5 minutes
     after 'physics' is declared for a store to start bringing detectors to physics,
     also no Flat Top state in Detector States

b) BERT screen on SL desk not allowing to select STAR status in pull-down menu,
still remains

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial,
after cold snake is recovered

06/14/24

I. RHIC schedule

a) Unpolarized stores today, cold snake to be recovered by 8pm, polarization
after

b) Physics over the weekend

c) Next maintenance Thursday Jun 20 (not Wednesday because of holidays)

II. STAR status and issues

a) Access yesterday for DSM1 boards (noisy JP triggers) and MTD HV was ok,
issues seem fixed

b) New protected password, please login to drupal link and scroll to the bottom
of the page

c) BERT screen on SL desk not allowing to select STAR status in pull-down menu,
also beam dump window does not appear

d) Shift crews, please pay attention to AC water drain, was full now, and keep
doors closed when the AC is running, DAQ room doors also to be closed
at all times

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial,
after cold snake is recovered

06/13/24

I. RHIC schedule

a) Access now at 10am for two hours, then machine development at noon till 4pm

b) Polarized physics at 4pm (cold snake will be recovered early afternoon)

c) Next maintenance Thursday Jun 20 (not Wednesday because of holidays)

II. STAR status and issues

a) Noisy JP triggers, access now to replace 2 possible DSM1 boards

b) MTD, power failure in CAEN PS crate, same access now to replace power module

c) Trigger thresholds for B/EMC are changed (to account for lower gain in PMTs),
email on star-ops, subject 'Changes to B/EMC threshold settings'

d) Configuration 'pp200_production_radial' to be used for physics at 4pm again

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/12/24

I. RHIC schedule

a) APEX today starting 10am, polarization measurement today at 9pm (when cold
snake is restored), back to physics at midnight

b) Thursday 6/13 till Sunday 6/16: physics

c) Next maintenance Thursday Jun 20 (not Wednesday because of holidays)

II. STAR status and issues

a) Noisy JP triggers, 2+ hour access to replace 2 possible DSM1 boards, might
get such access tomorrow Thursday, after machine development ~2pm - to be updated

b) MTD, power failure in CAEN PS crate, power module to be replaced, 1 hour access

c) 7bit bunch Id, incorrect reset for counter, not happened since Monday morning

d) Trigger thresholds for barrel, test run 25163054 done last night (to compensate
for lower gains in PMTs), tba over star-ops by Carl

e) Online plots crashing from time to time, Jeff investigating

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/11/24

I. RHIC schedule

a) Physics today, last store will start at 10pm to last till APEX tomorrow at 10am,
then polarization measurement tomorrow at 9pm (when cold snake is restored)

b) Thursday 6/13 till Sunday 6/16: physics

II. STAR status and issues

a) EPD missing sectors were caused by eq3_qtd and eq4_qtd nodes masked in run control,
no clear reason why, eq4 lost first, eq3 in run after

b) Noisy JP triggers, BC102, DSM#1, tbd at trigger meeting

c) 7bit bunch Id, incorrect reset for counter (Akio), tbd at trigger meeting

d) Thresholds for barrel triggers to be readjusted to compensate for aging effects,
Carl will instruct SL on zoom for control room

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/10/24

I. RHIC schedule

a) Physics today and tomorrow, APEX on Wednesday, Jun 12

II. STAR status and issues

a) EPD has missing sectors, EQ3 and EQ4 not reading out, potential access
at noon or after (sPHENIX asked for 2 hours)

b) Noisy JP triggers, BC102, DSM#1, Hank looking into it

c) 7bit bunch Id, incorrect reset for counter (Akio)

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/09/24

I. RHIC schedule

a) Physics now till Tuesday, Jun 11

II. STAR status and issues

a) eTOF not in the runs, repeated 'scDeamon.C:#1904 ETOF has 1018>1000 EVB' message

b) GUI for VME 70 (EEMC canbus) shows incorrect voltages, crate itself works ok

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/08/24

I. RHIC schedule

a) Physics over the Weekend + Monday and Tuesday,
Jun 8 till 11

II. STAR status and issues

a) Recurrent trips for west trim magnet,
CAS worked on it yesterday

b) BSMD sector 2 and TPX[34] gave errors in pedestal run 25160019,
due to oncoming injection crews couldn't run the pedestal again

c) Transient TOF or MTD LV alarms can be ignored (not temperature),
log entry for persistent alarms, in email to star-ops by Geary
yesterday, subject 'TOF LV yellow alarms'

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/07/24

I. RHIC schedule

a) today: store at 3am, machine development now till 11am (spin tune study for blue snake),
then physics

b) Weekend Jun 8,9: physics

II. STAR status and issues

a) Wrong production configuration (pp200_production_High_Luminosity) was in Detector Readiness Checklist,
(typo introduced yesterday when the checklist was updated), correct configuration is pp200_production_radial

b) BSMD is included in production runs

c) Shift crews should subscribe to star-ops mailing list, star-ops-l@lists.bnl.gov

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/06/24

I. RHIC schedule

a) today: APEX was scheduled till 9pm, however no beam from AGS (Siemens exciter power supply),
some APEX sessions will be rescheduled, back to physics at 9pm

b) Friday Jun 7: spin tune study for blue snake (~ 2 hours) between stores

c) Weekend Jun 8,9: collisions

II. STAR status and issues

a) Maintenance completed yesterday

b) Cosmics overnight because of no beam

c) BSMD to be included, Oleg will give instruction

d) For crews: very humid these days, please keep control room and DAQ room doors closed
for AC to work properly. PS: also flush coffee water tray from time to time otherwise
it spills over the table

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

06/05/24

I. RHIC schedule

a) today: maintenance 8am till 4pm, then collisions

b) Thursday Jun 6: apex from 8am till 9pm, investigation for longitudinal
component, STAR will take ZDC polarimetry runs, then collisions by 10pm

c) Friday Jun 7: spin tune study for blue snake (~ 2 hours) between stores

d) Weekend Jun 8,9: collisions

II. STAR status and issues

a) Smooth running

b) Maintenance day today:

c) Magnet to be turned off after morning beam dump for 200T chiller for magnet turn-on,
then turn magnet on to test the chiller

d) Turn off TPC FEEs + VMEs + TOF HV, LV + MTD LV HV due to work on condenser fan
for the 80T chiller (cools MCW)

e) EEMC MAPMT FEE box cooling (Bill S. and Prashanth), when magnet is off,
barriers down and access to the (south) poletip

f) TPX/iTPC RDOs masked out (3 of them), Tonko will work on it when FEEs are back on

g) eTOF, colors on HV GUI -> only sector 3 is at full, all others are zero / re-open GUI
may clear colors

h) Crews should lookup reference plots, SL are passing information to those who asked

i) EPD lower gain at 3 tiles (outer) Maria Stefaniak

j) BSMD to be included tomorrow

III. Plans

a) Radially polarized data taking at high luminosity, pp200_production_radial

From 05/15/2024 to 06/04/2024, Period Coordinator: Zilong Chang, notes

STAR daily operation meeting 05/14/2024

(Period Coordinator: Zhangbu Xu)

Incoming Period Coordinator: Zilong Chang

RHIC Schedule

Plan for this week,

Maintenance day (Wednesday), APEX (Thursday), switch to Spin run

Emittance is large, achieving 2.1x10^11 per bunch at injection and 1.9x10^11 at top (ultimate goal is 3x10^11);

Yesterday: 28x28 and vernier scan for STAR Monday 6 hours (3-4PM); high-luminosity run 2 hours before hand;

56x56 sPHENIX not yet ready

STAR leveling at 20KHz

STAR status

Physics running at 7KHz. pp200-production-LowLuminosity
MB-EPD+forward: 1.4B evts, prescale (7-10); MB-EPD-TOF4: 1.55B evts, prescale 1
switch to leveling at 40KHz Monday and MB-EPD+forward all the way

Check prescale=1 for MB if we can have even higher DAQ rate (7—7.4KHz)

sTGC:
Wednesday access change one of the bad FEES;
Zero-field Alignment dataset (56x56 third priority Monday)?

Shift operation. Any issues?

Shift vacancy issue? Working progress (Cebra)

BBC QT in wrong bunch (prepost=-1) in some runs with some boards; production runs

High-Lumi configuration:
Configuration run taken (BBCA 500KHz);
zdcpolarimetry (front/end thresholds run15 setting), analysis shows consistency with previous runs; new gain file produced;

28x28 fill 8 hours

Vernier Scans at beginning and end; forward cross section data; Smooth runs;

Plans

a full Maintenance Day on May 15^th (Linda Horton’s visit at BNL); May 16, APEX;

STAR daily operation meeting 05/13/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

Emittance is large, doing shorter fills (2 hours) to effectively scrub;
achieving 2.1x10^11 per bunch at injection and 1.9x10^11 at top (ultimate goal is 3x10^11);

28x28 and vernier scan for STAR Monday 6 hours (3-4PM); high-luminosity run 2 hours before hand; 56x56 sPHENIX not yet ready

Two abort gaps blue and yellow background asymmetry

STAR leveling at 40KHz

STAR status

Physics running at 6.2KHz. pp200-production-LowLuminosity
MB-EPD+forward: 1.2B evts, prescale (7-10); MB-EPD-TOF4: 1.55B evts, prescale 1
switch to leveling at 20KHz Monday and MB-EPD+forward all the way?

L4 died a couple of times, is this indicative of a hardware issue?

sTGC:
1mV/HV 2900V/Bit ON, updated at 11:46AM yesterday after beam dump;
Wednesday access change one of the bad FEES;
Zero-field Alignment dataset (56x56 third priority Monday)?

Shift operation. Any issues?
shift leader desktop freezes from time to time (for 5 minutes), firefox issue (Wayne)?
Should be stand-down during thunderstorm request information from MCR; APEX mode;
AC unit above control roof. Permanent unit arrived, now only temporary unit (start to work). wiring done, work permit?

Shift vacancy issue? Working progress (Cebra)

Scaler timing for Polarization monitoring. DONE;
3 scaler boards left, #3 ZDC, #5,6 BBC; no backup

BBC QT in wrong bunch (prepost=-1) in some runs with some boards; production runs;

High-Lumi configuration:
When we are done, inform CAD. They will switch to 28x28;
BHT2 remove BBC TAC requirement; dimuon VPD TAC2 to VPD TAC (100cm);
UPC-J/Psi BHT bit bug fixed;
Jeff will take a couple of runs so that Carl and others can check the settings;

Plans

Spin Trigger and polarimetry commissioning
production trigger configuration, optimize trigger rates
polarimetry scalers; Monday instead, removing cross angle and leveling at 20KHz, with high luminosity configuration file; scaler working; need to check ZDC polarimetry data and thresholds; ZDCSMD gain matching, >30 minutes zdcpolarimetry (high priority on Monday, 1 million evts beginning and end); afternoon-early evening Monday 28x28 bunches;  vernier scan configuration file does have scaler readout every second and in the run control (second priority Monday); 6 hours

a full Maintenance Day on May 15^th (Linda Horton’s visit at BNL); May 16, APEX;

STAR daily operation meeting 05/12/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

Emittance is large, doing shorter fills (2 hours) to effectively scrub;
achieving 1.5x10^11 per bunch (ultimate goal is 3x10^11);

Beam since last evening; 2 hour store, high background, not the best for MB-EPD

Continue short store over weekend, 28x28 and vernier scan for STAR likely Monday

STAR leveling at 40KHz

No 9AM meeting on the weekend;

STAR status

Physics running at 5.8KHz. pp200-production-LowLuminosity
MB-EPD+forward: 1.1B evts, prescale (7-10); MB-EPD-TOF4: 1.3B evts, prescale 1
switch to leveling at 20KHz Monday and MB-EPD+forward all the way?

Need to make sure efficient running; data-taking right after PHYSICS ON;
do not wait for leveling; After luminosity leveling at BBCAnd~=40KHz is stable, stop and restart run.

sTGC:
gain 3mV/HV 2850V/Bit ON, mask two FEEs;
1mV/HV 2900V/Bit ON, NO on run control? Wednesday access change one of the bad FEES; 11:46AM beam dump;
Zero-field Alignment dataset Monday (56x56 third priority Monday)?

Shift operation. Any issues?
shift leader desktop freezes from time to time (for 5 minutes), firefox issue (Wayne)?
Should be stand-down during thunderstorm request information from MCR; APEX mode;
AC unit above control roof. Permanent unit arrived, now only temporary unit (start to work?). wiring done, work permit?

Shift vacancy issue? Working progress (Cebra)

Scaler timing for Polarization monitoring. DONE;
3 scaler boards left, #3 ZDC, #5,6 BBC

BBC QT in wrong bunch (prepost=-1) in some runs with some boards; production runs;

Plans

Spin Trigger and polarimetry commissioning
production trigger configuration, optimize trigger rates
polarimetry scalers; Monday instead, removing cross angle and leveling, with high luminosity configuration file; scaler working; need to check ZDC polarimetry data and thresholds; ZDCSMD gain matching, >30 minutes zdcpolarimetry (high priority on Monday); afternoon-early evening Monday 28x28 bunches;  vernier scan configuration file need to have scaler readout every second (second priority Monday);

a full Maintenance Day on May 15^th (Linda Horton’s visit at BNL); May 16, APEX;

STAR daily operation meeting 05/11/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

Emittance is large, doing shorter fills (2 hours) to effectively scrub;
achieving 1.5x10^11 per bunch (ultimate goal is 3x10^11);

Beam since last evening; 2 hour store, high background, not the best for MB-EPD

Continue short store over weekend, 28x28 and vernier scan for STAR likely Monday

STAR leveling at 40KHz

No 9AM meeting on the weekend;

STAR status

Physics running at 5.8KHz. pp200-production-LowLuminosity
MB-EPD+forward: 1.05B evts, prescale (7-10); MB-EPD-TOF4: 1.1B evts, prescale 1

Potential rate 8KHz, TPC hits the speed limit at 6KHz;
try TPC readout at 6KHz?

Overnight L0 power cycle; L4 down, Wayne/Diyu/Mike fixed it this morning;

Need to make sure efficient running; data-taking right after PHYSICS ON;
do not wait for leveling; If DAQ rate <5KHz after luminosity leveling or BBCAnd~=40KHz, stop and restart run.

clean up coffee station (dump trays);

VPD: DONE; VPD TAC2 windows; HT-VPD100;

MTD timing window cuts;

sTGC:
gain 3mV/HV 2850V/Bit ON, mask two FEEs;

Zero-field Alignment dataset later on

Shift operation. Any issues?
cosmic ray without TPC yesterday;
Should be stand-down during thunderstorm request information from MCR; APEX mode;
AC unit above control roof. Permanent unit arrived, now only temporary unit (start to work?). wiring done, work permit?

Shift vacancy issue? Working progress (Cebra)

Scaler timing for Polarization monitoring. DONE;
3 scaler boards left, #3 ZDC, #5,6 BBC

BBC QT in wrong bunch (prepost=-1) in some runs with some boards; production runs;

Plans

Spin Trigger and polarimetry commissioning
production trigger configuration, optimize trigger rates
polarimetry scalers; Monday instead, removing cross angle and leveling, with high luminosity configuration file; scaler working; need to check ZDC polarimetry data and thresholds; ZDCSMD gain matching; afternoon-early evening Monday 28x28 bunches;

a full Maintenance Day on May 15^th (Linda Horton’s visit at BNL)

STAR daily operation meeting 05/10/2024



RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

Emittance is large, doing shorter fills (2 hours) to effectively scrub;
achieving 1.5x10^11 per bunch (ultimate goal is 3x10^11);

Beam since last evening; 2 hour store, high background, not the best for MB-EPD

Continue short store over weekend, 28x28 and vernier scan for STAR likely Monday

STAR leveling at 40KHz

No 9AM meeting on the weekend;

CAD PS access, no STAR access after this fill for 2 hours; after that, longer store 6 hours for sPHENIX TPC conditioning; after that, short fills again

STAR status

Physics running at 5.8KHz. pp200-production-LowLuminosity
MB-EPD+forward: 980M evts, prescale (7-10); MB-EPD-TOF4: 950M evts, prescale 1

Potential rate 8KHz, TPC hits the speed limit at 6KHz;
try TPC readout at 6KHz?

VPD: production ID; active splitter checks, no issue discovered but need to find the source of issue (Tim); VPD west ADC (6 MXQ even chs) progressing;
raise thresholds to 80 (one channel masked out), east 50; noise persists with HV OFF; commissioning on tune configuration file;

sTGC:
gain 3mV/HV 2850V/Bit ON, mask two FEEs? Later today

Zero-field Alignment dataset later on

Shift operation. Any issues?
Should be stand-down during thunderstorm request information from MCR; APEX mode;
AC unit above control roof. Permanent unit arrived, now only temporary unit (start to work?). wiring done, work permit?

Shift vacancy issue? Working progress (Cebra)

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.
3 scaler boards left, #3 ZDC, #5,6 BBC

BBC QT in wrong bunch (prepost=-1) in some runs with some boards; production runs;

Plans

Spin Trigger and polarimetry commissioning
production trigger configuration, optimize trigger rates
polarimetry scalers; Monday instead, removing cross angle and leveling, with high luminosity configuration file; scaler working; need to check ZDC polarimetry data and thresholds;

a full Maintenance Day on May 15^th (Linda Horton’s visit at BNL)

STAR daily operation meeting 05/09/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

Emittance is large, doing shorter fills (2 hours) to effectively scrub
during the day and overnight long fills;

Blue injection kicker (arcing and oil blockage), AGS field unstable (Siemens PS)
no beam last 24 hours

Thursday Machine Development (until 16:00); 4-8PM scrubbing; 8PM physics

STAR leveling at 40KHz

Possible 58x58 (28x28) fill Monday

STAR status

Physics running at 5.8KHz. pp200-production-LowLuminosity
MB-EPD+forward: 910M evts, prescale (7-10); MB-EPD-TOF4: 850M evts, prescale 1

VPD: production ID; active splitter checks, no issue discovered but need to find the source of issue (Tim); VPD west ADC (6 MXQ even chs) progressing;
raise thresholds to 64 (one channel masked out), east 50; noise persists with HV OFF; commissioning on tune configuration file;

sTGC:
gain 3mV/HV 2850V/Bit ON, mask two FEEs? Later today

Zero-field Alignment dataset later on

Shift operation. Any issues?
Should be stand-down during thunderstorm request information from MCR; APEX mode;
AC unit above control roof. Permanent unit arrived, now only temporary unit (doesnot work well). wiring done, work permit?

Shift vacancy issue? Working progress (Cebra)

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.
3 scaler boards left, #3 ZDC, #5,6 BBC

BBC QT in wrong bunch (prepost=-1) in some runs with some boards; production runs;

Remove VPDA from cosmic ray trigger configuration file;

Plans

Spin Trigger and polarimetry commissioning
production trigger configuration, optimize trigger rates
polarimetry scalers; Monday instead, removing cross angle and leveling, with high luminosity configuration file; scaler working; need to check ZDC polarimetry data and thresholds;

a full Maintenance Day on May 15^th (Linda Horton’s visit at BNL)
Tim is not available for that day?

STAR daily operation meeting 05/08/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

Emittance is large, beam scrubbing for 4 hours;
limited success, maybe doing shorter fills (2 hours) to effectively scrub
during the day and overnight long fills;

Blue injection kicker (arcing and oil blockage) no beam last 24 hours

APEX Wednesday 8AM-11PM using yellow beam for now; Thursday Machine Development (10AM-noon)

STAR leveling at 40KHz

STAR status

Physics running at 5.8KHz. pp200-production-LowLuminosity
MB-EPD: 800M evts, prescale (7-10); MB-EPD-TOF4: 850M evts, prescale 1

VPD: production ID; active splitter checks, no issue discovered but need to find the source of issue (Tim); VPD west ADC (6 MXQ even chs) progressing;
raise thresholds; noise persists with HV OFF; commissioning on tune configuration file;
DSM VPD to MTD (MT101) =>QT board in the wrong slot, FIXED; Tim/Chris

sTGC:
gain 3mV/HV 2850V/Bit ON, mask two FEEs? Later today
increase forward components in MB-EPD and EPD-TOF4

MB-EPD-forward included MB-EPD evts?
Zero-field Alignment dataset later on

Shift operation. Any issues?
Cosmic run #17 this morning, 40-60 auto-configuring during thunderstorm; not in elog;

Should be stand-down during thunderstorm request information from MCR; APEX mode;
AC unit above control roof. Permanent unit arrived, now only temporary unit (doesnot work well). wiring done, work permit?

Shift vacancy issue? Working progress (Cebra)
14 worked on, 21 filled,

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.
3 scaler boards left, #3 ZDC, #5,6 BBC

BBC QT in wrong bunch (prepost=-1) in some runs with some boards; production runs;

ePIC HCAL HV settings and threshold (restricted access);

Plans

Spin Trigger and polarimetry commissioning
production trigger configuration, optimize trigger rates
polarimetry scalers; (Thursday/Friday 1-2 hour beam time) Monday instead, removing cross angle and leveling, with high luminosity configuration file; scaler working; need to check ZDC polarimetry data and thresholds;

a full Maintenance Day on May 15^th (Linda Horton’s visit at BNL)
Tim is not available for that day?

STAR daily operation meeting 05/07/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).
1.2x10¹¹ per bunch from about 0.9 earlier

Emittance is large, beam scrubbing for 4 hours;
limited success, maybe doing shorter fills (2-4 hours) to effectively scrub
during the day and overnight long fills;

6-hour access requested by sPHENIX/CeC/CEPS

APEX Wednesday 8AM-11PM; Thursday Machine Development (10AM-noon)

STAR leveling at 40KHz

STAR status

Physics running at 5.8KHz. pp200-production-LowLuminosity
MB-EPD: 800M evts, prescale (7-10); MB-EPD-TOF4: 850M evts, prescale 1

Readiness checklist update for physics available. Shift crew started to follow normal operation procedure.

VPD: production ID; active splitter checks today (Tim);
VPD west ADC (6 MXQ even chs) progressing; if noise persists, raise thresholds;
should be done today/tomorrow; commissioning on tune configuration file;
DSM VPD to MTD (MT101) wrong slots=>QT board in the wrong slot; Tim/Chris

sTGC:
Access to gain change Monday also change prod ID?
HV scan, another iteration yesterday and today; (Tonko/Jeff/Daniel)
24 hours afterward final change.
gain 3mV/HV 2850V/Bit ON; increase forward components in MB-EPD and EPD-TOF4

MB-EPD-forward included MB-EPD evts? Zero-field Alignment dataset later on

Shift operation. Any issues?
AC unit above control roof. Permanent unit arrived, now only temporary unit (doesnot work well). wiring done, should be done?

Shift vacancy issue? Working progress (Cebra)
14 worked on, 21 filled,

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.
3 scaler boards left, #3 ZDC, #5,6 BBC

BBC QT in wrong bunch (prepost=-1) in some runs with some boards; production runs;

FST reference plots updated;

ePIC HCAL HV settings and threshold (restricted access);

Plans

Spin Trigger and polarimetry commissioning
production trigger configuration, optimize trigger rates
polarimetry scalers; Thursday/Friday 1-2 hour beam time, removing cross angle and leveling, with high luminosity configuration file; scaler working; need to check ZDC polarimetry data and thresholds;

a full Maintenance Day on May 15^th (Linda Horton’s visit at BNL)
Tim is not available for that day?

STAR daily operation meeting 05/06/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).
1.2x10¹¹ per bunch from about 0.9 earlier

Emittance is large, beam scrubbing for 4 hours

Potential 2-hour access requested by sPHENIX

STAR leveling at 40KHz



STAR status

Physics running at 5.6KHz. pp200-production-LowLuminosity
MB-EPD: 700M evts, prescale (5-7); MB-EPD-TOF4: 700M evts, prescale 1.1—1.5
possible options to be discussed at triggerboard meeting: leveling at 30KHz, change prescale, short run at 10KHz? BHT3 and BHT3-L2gamma triggers

Readiness checklist update for physics available. Shift crew started to follow normal operation procedure.

VPD: tac offset correct (ONLINE), slewing upload and DONE (BBQ); production ID;
VPD west ADC (5 MXQ chs) progressing; power-cycle the active WEST splitters

eTOF included in the run (Geary)

sTGC:
Access to gain change Monday also change prod ID?
HV scan, another iteration yesterday and today; (Tonko/Jeff/Daniel)
24 hours afterward final change.
Zero-field Alignment dataset later on

Shift operation. Any issues?
elog: Water on the floor near restroom near STAR control room. Maybe a new roof leak? Dry it. (leak from the roof)
MTD/sTGC gas alarms overnight; gas bottles changed;
TPC sector 12 RDO 1 masked out permanently
setRHICclock after using local clock;

AC unit above control roof. Permanent unit arrived, now only temporary unit, no cooling, been worked on.
wiring done, waiting for the permit?

Shift vacancy issue? Working progress (Cebra)
5 vacancies coming weeks; no issue this week;

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.
3 scaler boards left, #3 ZDC, #5,6 BBC

Plans

a full Maintenance Day on May 15^th (Linda Horton’s visit at BNL)
Tim is not available for that day?

Now to May 14^th, a short run to commission trigger rate/background and zdcpolarimetry with high luminosity and zero crossing angle?

STAR daily operation meeting 05/05/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).
1.2x10¹¹ per bunch from about 0.9 earlier

STAR leveling at 40KHz



STAR status

Physics running at 5.6KHz. pp200-production-LowLuminosity
MB-EPD: 510M evts, prescale (5-7); MB-EPD-TOF4: 500M evts, prescale 1.1—1.5
Offline update on pileup and vertexing (Shengli)

Readiness checklist update for physics available. Shift crew started to follow normal operation procedure. removing the “prepare for physics” for BEMC.

VPD: tac offset correct (ONLINE), slewing upload today (BBQ); production ID;
VPD west ADC (5 MXQ chs) progressing;

eTOF included in the run (Geary)

sTGC:
Access to gain change Monday also change prod ID?
HV scan, another iteration yesterday and today; (Tonko/Jeff/Daniel)
24 hours afterward final change.
Zero-field Alignment dataset later on

Shift operation. Any issues?
TPC sector 12 RDO 1 masked out permanently
run38 last night took 20 minutes;
After the beam dump, turn on the VPD and take:
pedestal_rhicclock_clean with TRG+DAQ+FCS (1k events).
AC unit above control roof. Permanent unit arrived, now only temporary unit.
wiring done, should be done yesterday?

Shift vacancy issue? Working progress (Cebra)

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.
3 scaler boards left, #3 ZDC, #5,6 BBC

Plans

a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)
Tim is not available for that day.

STAR daily operation meeting 05/04/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

STAR leveling at 40KHz

No CNI Polarization measurement in the middle of a store for now



STAR status

Physics running at 5.6KHz. pp200-production-LowLuminosity
MB-EPD: 320M evts; MB-EPD-TOF4: 310M evts

Readiness checklist update for physics available. Shift crew started to follow normal operation procedure.

EPD trigger timing – EPD timing scan DONE.
EQ gates need to be updated in the tier1 file (DONE)

VPD: tac offset correct (ONLINE), slewing upload today (BBQ); production ID;
VPD west ADC (5 MXQ chs) progressing

sTGC:
Access to gain change Monday also change prod ID?
HV scan last night, another iteration today;
24 hours afterward final change.
Zero-field Alignment dataset later on

Shift operation. Any issues?
TPC sector 12 RDO 1 masked out permanently
AC unit above control roof. Permanent unit arrived, now only temporary unit.
wiring done, should be done yesterday?

Shift vacancy issue? Working progress (Cebra)

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.
3 scaler boards left, #3 ZDC, #5,6 BBC

Test run of gating grid at different rates (1 minute each)
went well, processing data, likely do another one with high luminosity

Plans

a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)
Tim is not available for that day.

After Flattop is achieved and the  beams have been steered to achieve collisions, take:

         emc-check with  TRG + DAQ + BTOW + ETOW + ESMD + FCS   (50k events)

Once beams reach PHYSICS ON status, turn on detectors according to Detector States Diagram. When detectors are ready, start running [pp200_production_lowlumisoty] with (all triggers included):

        TRG+DAQ+TPX+ITPC+BTOW+ETOW+ESMD+TOF+eTOF+MTD+GMT+FST+sTGC+FCS+L4

For now, BSMD is not included in the production data-taking.

STAR daily operation meeting 05/03/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

Access 13:30—14:30 today

Machine Development 14:30—17:00

STAR leveling at 40KHz; last fill this morning very high background first 1/2hour

No CNI Polarization measurement in the middle of a store for now



STAR status

Physics running at 5.6KHz. pp200-production-LowLuminosity
MB-EPD: 170M evts; MB-EPD-TOF4: 160M evts

Readiness checklist update for physics available. Shift crew started to follow normal operation procedure.

EPD trigger timing – EPD timing scan DONE.
EQ gates need to be updated in the tier1 file?

ZDCSMD issue, swap out daughter card#1. DONE!

Previous Issues:

low rate with QT crate issue (done).
trigger deadtime at high rate but not an issue at the moment

high lumi at 3KHz, need low deadtime, not an issue

VPD: tac zero out. Be ready today for data-taking
Done? VPD west ADC

sTGC:
Access to gain change? Daniel?
Zero-field Alignment dataset later on

Shift operation. Any issues?
AC unit above control roof. Permanent unit arrived, now only temporary unit.

Shift vacancy issue? Working progress (Cebra)

FST will switch back from 9 time bin to 3 time bin (DONE)

Plans

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.
3 scaler boards left, #3 ZDC, #5,6 BBC

Test run of gating grid at different rates (1 minute each) 11:30AM

a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)
Tim is not available for that day.

·       Once Beams reach FLAT TOP,
run EMC_check with TRG+DAQ+ BTOW+ETOW+ESMD+FCS

·       When MCR issues “prepare for dump”, start bringing detectors to the "Preparing beam dump" state and the SL clicks “Prepare to dump”.
run zdcpolarimetry_2024 with DAQ+TRG
After all detectors are in the safe mode, the SL clicks “Ready to dump”. After beams dumped, stop run.

STAR daily operation meeting 05/02/2024

RHIC Schedule

Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

10Hz Machine development with 12x12

Access 10:30—11:30AM today (CEC)

STAR leveling at 40KHz



STAR status

Physics running at 5.6KHz. pp200-production-LowLuminosity

EPD trigger timing – EPD timing scan again. Mike analyzing the data,
EQ gates need to be updated in the tier1 file (need to be done)

use EPD coincidence replacing BBCA as the main trigger component. EPD has wider gate which can see both signals and background. It works great!

ZDCSMD issue, have to swap out daughter card#1, access right now.

Previous Issues:

low rate with QT crate issue (investigation ongoing).
Chris worked on it, needs elog entries.
QT rate seems to be up (4KHz, 40% dead) => 5.6KHz normal deadtime

VPD: tac zero out. Be ready today for data-taking
still zero out, and this is not good for the production run
NEXT FILL will be fixed.

EPD parameters were left in a different state from earlier in the production configuration last night.

sTGC: time bin cut? Access to gain change? Zero-field Alignment dataset

Readiness checklist update for physics available. Shift crew started to follow normal operation procedure. Prom_Check DAQ+TRG+iTPC once a day (no event, stop run after it starts).

Shift operation. Any issues? AC unit above control roof. Permanent unit arrived, now only temporary unit. Detector operator (fill in two weeks, visa revoked at airport)

Plans

·       Before run official production pp200_production_LowLuminosity, whenever possible, run zdcpolarimetry_2024 with DAQ+TRG

FST will switch back from 9 time bin to 3 time bin (today)

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.

Test run of gating grid at different rates (1 minute each)

a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)

STAR daily operation meeting 05/01/2024

RHIC Schedule
Plan for this week,

111x111 bunch collisions for experiments remaining of the week (mostly).

10Hz Machine development with 12x12

STAR leveling at 40KHz, two short fills this morning



STAR status

Physics running started since Yesterday. pp200-production-LowLuminosity

EPD trigger timing - one clock late issue resolved. Eleanor requested 4 EPD runs
last night. Status? Readout Crossing should be 8 instead of the default 7?
EQ gates need to be updated in the tier1 file

Once EPD finalized, proposed to use EPD coincidence replacing BBCA as the main trigger component. EPD has wider gate which can see both signals and background.

ZDCSMD issue, Aihong took some pedestal runs and coordinated with Hank. The channel 4 has a high pedestal. ZDCSMD looks good in the fill since last night. There is a single run (25122011) from a short-lived fill this morning in which the same offending channel look a little suspicious, but it is not crazy. Aihong will continue monitoring it. May have to swap out daughter card#1

Previous Issues:

L0L1 turn-off issue (Tim changed fan tray for L0L1 crate 62)
fixed, no issue so far

low rate with QT crate issue (investigation ongoing).
Chris worked on it, needs elog entries.
QT rate seems to be up (4KHz, 40% dead) => 5KHz normal deadtime

VPD: tac zero out. Be ready today for data-taking

Gene analyzed data from Monday/Tuesday for studying the background effect.
see effects and need to keep BBC blue+1.7Yellow background <200KHz

sTGC: time bin cut? Access to gain change?

Readiness checklist update for physics available. Shift crew started to follow normal operation procedure. Prom_Check DAQ+TRG+iTPC once a day (no event, stop run after it starts).

Shift operation. Any issues?

Jeff changed the logic how to include forward detectors in the trigger.

Production configuration is needed. Jeff will clean up the file. Default configuration:pp200_production_LowLuminosity: done

Determine the gain for the polarimeter at the beginning the fill. Jeff has a configuration for this: file is available, will put instruction in the checklist

Plans

FST will switch back from 9 time bin to 3 time bin (today)

Readiness checklist update for physics available.

Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.

Test run of gating grid at different rates (1 minute each)

a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)

STAR daily operation meeting 04/30/2024
(Period Coordinator change: Kong Tu => Zhangbu Xu)

§ RHIC Schedule

4K cool down.

Plan for this week,

·      111x111 bunch collisions for experimental setup overnight.

·      Machine development today (10:00 – 14:00).

·      Crossing angle 1mrad at STAR; leveling at STAR made signal/background ratio very small. Reverted back to without leveling.



§ STAR status

·      Physics running started around 4am this morning. We created a temporary configuration and promoted MB-BBC, MB-TOFmult4, BBC, ZDC, EPD.

·      Global timing moved back 2ns. CAL SCAN was redone.

·      EPD trigger timing - one clock late issue resolved. Eleanor fixed it! A few minor changes should be done. All detectors calibration done. VPD E and W max tac value changed from 1950 to 2100.

·      ZDCSMD issue, Aihong took some pedestal runs and coordinated with Hank. The channel 4 has a high pedestal. Issue is associated with the QT broad. Will discuss in the trigger meeting.

·      Previous Issues:

o   L0L1 turn-off issue (Tim changed fan tray for L0L1 crate 62)

o   low rate with QT crate in issue (investigation ongoing).

·      Gene showed the space charge calibration plot and expressed concern about the space charge calibration.

·      Drilling finished yesterday.

·      Readiness checklist update for physics available. Shift crew started to follow normal operation procedure. (observation: a lot of not experienced shift crew and new trainee this year.)

·      Shift operation. One of the DO failed the training exam (Oxygen Deficiency training) multiple times. The DO had to contact training coordinator and just showed up with training finished.

§ Plans

·      Gene will analyze the first run this morning for studying the background.

·      VPD: tac alignment will be next when we have collisions.

·      FST will switch back from 9 time bin to 3 time bin.

·      Jeff will change the logic how to include forward detectors in the trigger.

·      Production configuration is needed. Jeff will clean up the file. Default configuration: pp200_production_LowLuminosity

·      Determine the gain for the polarimeter at the beginning the fill. Jeff has a configuration for this.

·      Readiness checklist update for physics available.

·      Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.

·      a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)

STAR daily operation meeting 04/29/2024

§ RHIC Schedule

4K cool down.

Plan for this week,

·      111x111 bunch collisions for experimental commission overnight.

·      Maintenance today 08:00 to 17:00; machine development Tuesday.



§ STAR status

·      The VPD earlistTAC was chopped off at 1950 (max). Since Run 25119110, global timing was moved by 2.5 ns early (Find delay 117 to 112). We might need another CAL SCAN to see with collisions. Endcap needs to scan anyway.

·      EPD trigger timing - one clock late, status: Eleanor find a blank VT201. ZDC, BBC, VPD need to revert back to original parameters.

·      ZDCSMD issue (west horizontal channel 4 was hot) and power cycle MXQ crate didn’t work. ZDCSMD gate scanning done, and default values are not changed. Hank: take another Ped run before evaluating this.

·      Running since last evening, pp_200_commissioning. Details about elevating to physics triggers will be discussed at the Trigger Board meeting.

·      L4 calibration. Diyu has received the calibration file from Gene from Run 15. Will investigate.

·      Drilling seems to be in the IR only. All evaluation were done. Lijuan: we should have this discussion early for next time.

·      VPD: tac alignment will be next when we have collisions. Will redo the voltage scan too. Call Daniel and Frank.

·      Previous Issues:

o   L0L1 turn-off issue.

o   low rate with QT crate in issue (investigation ongoing).

§ Plans

·      When we have beams tonight, call Oleg, VPD (Daniel. B), EPD (Maria, Mike), Prashanth, Akio.

·      Put in sTGC later today.

·      FST will switch back from 9 time bin to 3 time bin.

·      Determine the gain for the polarimeter at the beginning the fill. Jeff has a configuration for this.

·      Maintenance (access) today: 1) FCS moving in; 2) EEMC 5S2 box check and burp (Will J provided instruction and Prashanth received it); 3) Possible EPD air intake diverter; 4) L0L1 Crate work (Tim is planning to change the fan tray and change the voltage setting.) 5) Concrete drilling for ePIC (after 9 am).

·      Crossing angle of 1mrad to be added after all calibrations or close to physics.

·      Noise run should be taken.

·      Readiness checklist update for physics today.

·      Scaler timing for Polarization monitoring. It’s on the to-do list of Chris.

·      a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)

STAR daily operation meeting 04/28/2024

§ RHIC Schedule

4K cool down.

Plan for this week,

·      56x56 bunch collisions for experimental commission continue.

·      111x111 bunch collisions later tonight.

·      Maintenance day 08:00 to 17:00 Monday; machine development Tuesday.



§ STAR status

·      BBC, EPD (timing and bias scan), VPD, EMC are all commissioned. EPD still needs this trigger work - one clock late (trigger group will look into it).

·      Running since last evening, pp_200_commissioning, with MB trigger (BBC+TOF0) and high multiplicity trigger (with TOFMult4 > 8 for QA purposes for now). Fast Offline data has been requested and running for Prithwish, Shengli, et al. Shengli already produced QA plot which looks reasonable. Discussion tomorrow at the Trigger Board meeting.

·      L4 issue seems to be improved by Diyu with space charge calibration update [1]! Flemming suggested Diyu to consult with Gene about the pp 200 parameter for space charge calibration. Currently DCAz still looks strange.

·      Previous Issues: 1) L0L1 turn-off issue, 3) low rate with QT crate in issue (not solved yet). Update from Jeff, Tim, Hank (after yesterday 11:30am discussion)?

·      Aihong ZDCSMD work finished and currently still analyzing the data.

§ Plans

·      Crossing angle of 1mrad to be added after all calibrations or close to physics.

·      Maintenance (access) tomorrow: 1) FCS moving in; 2) EEMC 5S2 box check and burp (Will J provided instruction and Prashanth received it); 3) Possible EPD air intake diverter; 4) L0L1 Crate work (Tim is planning to change the fan tray and change the voltage setting.) 5) Concrete drilling for ePIC (after 9 am).

·      Triggers promoted to physics discussion at Trigger Board meeting tomorrow.

·      Readiness checklist update for physics next week.

·      Polarization monitoring. It’s on the to-do list of Chris.

·      a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)

STAR daily operation meeting 04/27/2024

§ RHIC Schedule

4K cool down.

Plan for this week,

·      56x56 bunch collisions for experimental commission started around 12:30am.

·      New store started at 7am.

·      Detector commission overnight and continue this weekend.



§ STAR status

·      No access.

·      ZDC, BBC, VPD DSM timing are calibrated (was one tick late), while EPD still needs this timing calibration (Chris will work on it). VPD-tac offset was restored to last year’s value instead of zero.

·      We observed the strange vertex z distribution on L4 but not L3 [1]. Diyu: calibration of the TPC? pp 500 parameters are used. Going to look at correlation between multiplicity and vertex distribution.

·      Previous Issues: 1) L0L1 turn-off issue, 2) L2 crashing with prepost issue (fixed), 3) low rate with QT crate in issue (not solved yet). Tim and Jeff left instructions to the shift crew for L0L1 issue. Issue 1): Under voltage turned off the crate. David will communicate with Tim, Jeff, etc. Hank will ask Jack about the voltage setting.

·      Forward detectors, not running yet. Will include them soon.

·      Finished: Cal Scan (Oleg) within 1ns w.r.t last year, BBC (Akio), EPD ongoing (Maria). Global time can be set.

§ Plans

·      Aihong should look at the ZDCSMD.

·      Continue trigger commission: EPD (Maria, Mike),VPD (Geary, Daniel Brandenburg, Frank).

·      VPD HV scan.

·      To shift crew: ETOF and MTD HV should OFF instead of STANDBY.

·      Polarization monitoring. It’s on the to-do list of Chris.

·      Plan after the trigger detector commission later today: 1) BBC-AND + TOF > 0 as MB and/or maybe 2) BBC trigger + mult>20 (to start with); all configurations should have the crossing angle.

·      Readiness checklist update for physics next week.

·      ½ day (9-1PM? Prashanth will find out and keep us posted on staropsa) of Maintenance on April 29^th (Monday) and a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)

§ When we have access.

·      (access needed) L0L1 crate shut off and check PS.

·      (access needed) EPD cooling: fan blowing to the fee box needs to be improved. Prashanth is working on it.

·      (access needed) ESMD issue, crate 85? Will: the issue is on the electronics or could be water flowing, but not understood yet. Someone can try to clear the bubbles by removing the quick release, etc…Prashanth/Will will email Bill to follow this up and try when we have access again.

STAR daily operation meeting 04/26/2024

§ RHIC Schedule

4K cool down.

Plan for this week,

·      Beam setup last night and 12x12 store for experimental setup early this morning. Collisions!

·      Global timing looked good, but the beam condition is not good with large background (see vertex z distribution run 25117023)

·      Continue beam setup in the AM, and more experimental commission in the PM and over the weekend.



§ STAR status

·      No access.

·      There are three issues: 1) L0L1 turn-off issue, 2) L2 crashing with prepost issue, 3) low rate with QT crate in issue. Chris worked on Issue 2) and it seems to be fixed. Status: stable for 1.5 days. Jeff: 1) happened once this morning due to “under voltage error 43”. David will look into the alarm system. Tim: could be PS. (will need access)

§ Plans

·      Trigger commission: Prepost (Chris), EPD (Maria, Mike), TOF, BBC (Akio), VPD (Daniel Brandenburg, Frank, Geary). We will have a call list when we have collisions, e.g., JH, Akio.

·      VPD HV scan.

·      Shift crew should pay attention to the events coming instead of deadtime.

·      Shift crew should look at the issue from the VME crate.

·      APEX mode for running single beams with the tune file.

·      Readiness checklist update for physics next week.

·      ½ day of Maintenance on May 1^st (Wednesday) and a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)

§ When we have access.

·      (access needed) L0L1 crate shut off and check PS.

·      (access needed) EPD cooling: fan blowing to the fee box needs to be improved. Prashanth is working on it.

·      (access needed) ESMD issue, crate 85? Will: the issue is on the electronics or could be water flowing, but not understood yet. Someone can try to clear the bubbles by removing the quick release, etc…Prashanth/Will will email Bill to follow this up and try when we have access again.

STAR daily operation meeting 04/25/2024

§ RHIC Schedule

4K cool down.

Plan for this week,

·      Beam setup last night.

·      First collision is expected to be this evening (maybe 6x6 bunches).

·      ½ day of Maintenance on May 1^st (Wednesday) and a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)

§ STAR status

·      No access.

·      There are three issues: 1) L0L1 turn-off issue, 2) L2 crashing with prepost issue, 3) low rate with QT crate in issue. Experts will investigate them.

·      When we have access.

o   (access needed) EPD cooling: fan blowing to the fee box needs to be improved. Prashanth is working on it.

o   (access needed) ESMD issue, crate 85? Will: the issue is on the electronics or could be water flowing, but not understood yet. Someone can try to clear the bubbles by removing the quick release, etc…Prashanth/Will will email Bill to follow this up and try when we have access again.

§ Plans

·      Trigger configuration for low lumi pp will be provided by Jeff.

·      Trigger commission: EPD (Maria, Mike), TOF, BBC (Akio), VPD (Daniel Brandenburg). We will have a call list when we have collisions, e.g., JH, Akio.

·      Will check the duration of the run.

·      Shift crew should look at the issue from the VME crate.

·      APEX mode for running single beams with the tune file.

·      Readiness checklist update for physics next week.

·      May 14^th (Tuesday), RHIC will open for Linda H (DOE office of science)

STAR daily operation meeting 04/24/2024

§ RHIC Schedule

4K cool down.

Plan for this week,

·      Blue injection kicker and PS work resolved/finished! Beam setup last night.

·      First collision is expected to be tomorrow evening or Friday.

·      ½ day of Maintenance on May 1^st (Wednesday) and a full Maintenance Day on May 14^th (Linda Horton’s visit at BNL)

·      Emergency power test. Prashanth: 10:30am, Wednesday.

§ STAR status

·      No access.

·      Jeff and Chris worked on the L0L1 and L2 issue and confirmed the cosmic configuration with prepost enabled also crashed the L2 and L0L1. Update? Also, Run-25114053 around 19:30, L2 and L0 crashed (tune_2024_prepost) and shift crew brought it back by following the expert’s instruction. Related? Experts baffled.

·      When we have access.

o   (access needed) EPD cooling: fan blowing to the fee box needs to be improved. Prashanth is working on it.

o   (access needed) ESMD issue, crate 85? Will: the issue is on the electronics or could be water flowing, but not understood yet. Someone can try to clear the bubbles by removing the quick release, etc…Prashanth/Will will email Bill to follow this up and try when we have access again.

§ Plans

·      Trigger configuration for low lumi pp will be provided by Jeff. Trigger commission: EPD (Maria, Mike), TOF, BBC (Akio), VPD (Daniel Brandenburg). We will have a call list when we have collisions.

·      APEX mode for running single beams with the tune file.

·      Readiness checklist update for physics next week.

·      May 14^th (Tuesday), RHIC will open for Linda H (DOE office of science)

STAR daily operation meeting 04/23/2024

§ RHIC Schedule

4K cool down.

Plan for this week,

·      Blue injection kicker and PS work continue. Beam setup early afternoon.

·      First collision is expected to be delayed due to the ongoing works and checks.

·      Maintenance on May 1^st (Wednesday).

·      Emergency power test next week. Prashanth: 10:30am, Wednesday.

§ STAR status

·      No access.

·      L2 seems to be running fine with prepost in tune configuration. (Hank and his team will investigate, as previous interpretation didn’t seem to explain). Jeff will do it when we have beams.

·      L0L1 VME crate crashed when running the tune_2024_prepost. We will keep an eye on it.

·      Same as yesterday.

o   (access needed) EPD cooling: fan blowing to the fee box needs to be improved. Prashanth is working on it.

o   (access needed) ESMD issue, crate 85? Will: the issue is on the electronics or could be water flowing, but not understood yet. Someone can try to clear the bubbles by removing the quick release, etc…Prashanth/Will will email Bill to follow this up and try when we have access again.

§ Plans

·      Jeff will investigate the system with L0/L1 and L2 when there’s beam activity.

·      APEX mode for running single beams with the tune file.

·      Readiness checklist update for physics next week.

·      May 14^th (Tuesday), RHIC will open for Linda H (DOE office of science)

STAR daily operation meeting 04/22/2024

RHIC Schedule

4K cool down.

Plan for this week,

Blue injection setup last night and saw some beam activities!

Blue main quad PS diagnostics continue and with some other issues to investigate. Some checks will be done with Yellow but most or less ready for injection.

First collision expected April 23-25 (?)

Maintenance on May 1^st (Wednesday).

Emergency power test next week, but not sure what day yet.

STAR status

No access, but we may go to restricted access later today.

BTOW hot spots. No action needed from shift crew.

L2 seems to be running fine with prepost in tune configuration. (Hank and his team will investigate, as previous interpretation didn’t seem to explain).

L0L1 VME crate crashed when running the tune_2024_prepost. Experts may look into it.

EPD is on now. EPD reduced the QT threshold to 3, and can bring it back to 35 after the scan.

Same as yesterday but we may have access later.

(access needed) EPD status: Mike: there are yellow and right lights on the TUFF box, need to look into what they mean. (Everything is fine.)

(access needed) EPD cooling: fan blowing to the fee box needs to be improved. Prashanth is working on it.

(access needed) ESMD issue, crate 85? Will: the issue is on the electronics or could be water flowing, but not understood yet. Someone can try to clear the bubbles by removing the quick release, etc…Prashanth/Will will email Bill to follow this up and try when we have access again.

FST time bin shift. Ziyue will do it (Monday) and keep the shift crew posted. Jeff will adjust some setting and will document it on shiftlog.

Plans

Jeff will investigate the system with L0/L1 and L2.

Whether VPD should be on or off during single beam, experts will monitor and advise.

APEX mode for running single beams with the tune file.

Readiness checklist update for physics next week.

May 14^th (Tuesday), RHIC will open for Linda H (DOE office of science)

STAR daily Operation meeting 04/21/2024

RHIC Schedule

4K cool down.

Plan for this week,

Blue injection failed yesterday again and continue today.

Yellow will depend on the blue beam schedule. First collision expected April 23-25.

Maintenance on May 1^st (Wednesday).

Emergency power test next week, but not sure what day yet (see Prashanth’s email)

STAR status

Back to controlled access.

Same as yesterday.

(access needed) EPD status: Mike: there are yellow and right lights on the TUFF box, need to look into what they mean.

(access needed) EPD cooling: fan blowing to the fee box needs to be improved. Prashanth is working on it.

(access needed) ESMD issue, crate 85? Will: the issue is on the electronics or could be water flowing, but not understood yet. Someone can try to clear the bubbles by removing the quick release, etc…Prashanth/Will will email Bill to follow this up and try when we have access again.

FST time bin shift. Ziyue will do it (Monday) and keep the shift crew posted.

L2 died couple of times and experts (Hank, Akio, Eleanor) helped fixed it (eliminate the prepost seemed to fix the issue). Christ will look into it. New instruction here, https://www.star.bnl.gov/public/trg/trouble/L2_stop_run_recovery.txt).

Plans

Jeff will make a different tune file without prepost for now. Run with prepost as the default tune file, but without it will be the plan B.

Cosmic data taking with Reverse Full field.

Can use APEX mode for running single beams.

Readiness checklist update for physics next week.

May 14^th (Tuesday), RHIC will open for Linda H (DOE office of science)

STAR daily Operation meeting 04/20/2024

§ RHIC Schedule

4K cool down.

Plan for this week,

·      Blue injection failed yesterday and continue today.

·      Yellow PS checkout over the weekend (controlled access) and injection on April 22 (Monday); first collision expected April 23-25.

·      Maintenance on May 1^st (Wednesday).

·      Emergency power test next week, but not sure what day yet (see Prashanth’s email)

§ STAR status

·      BEMC HV fixed.

·      Back to restricted access.

·      (access needed) EPD status: Mike said it’s still a mystery and will have someone look at the lights on the EPD rack in the Hall (DO just did). Tonko made a comment on starops and mystery seems to be resolved. Mike: there are yellow and right lights on the TUFF box, need to look into what they mean.

·      (access needed) EPD cooling: fan blowing to the fee box needs to be improved. Prashanth is working on it.

·      (access needed) ESMD issue, crate 85? Will: the issue is on the electronics or could be water flowing, but not understood yet. Someone can try to clear the bubbles by removing the quick release, etc…Prashanth/Will will email Bill to follow this up and try when we have access again.

·      (access needed) sTGC air blower alarm seems to have issues. Tim fixed it!

·      L2 died and experts instructed the correct way of bringing L2 back (MXQ message suggests a link, and experts are looking to see if it is updated. https://www.star.bnl.gov/public/trg/trouble/L2_stop_run_recovery.txt ). Hank will update the instruction.

·      FST time bin shift. Ziyue will do it (Monday) and keep the shift crew posted.

§ Plans

·      Shift crew  Check online plots timely.

·      Cosmic data taking with Reverse Full field.

·      Can use APEX mode for running single beams.

·      Readiness checklist update for physics next week.

·      May 14^th (Tuesday), RHIC will open for Linda H (DOE office of science)

STAR daily Operation meeting 04/19/2024

RHIC Schedule

4K cool down.

Plan for this week,

Blue and Yellow PS checkout and injection into blue? on April 19 (today).

Yellow PS checkout over the weekend (controlled access) and injection on April 22 (Monday); first collision expected April 23-25.

Maintenance on May 1^st (Wednesday).

STAR status

BEMC HV. Oleg: today, may need Tim to make cables to finish the work, depending on EPD.

sTGC air blower alarm seems to have issues. Tim will look into it when possible.

EPD status: Tim removed East EPD TUFF box from hall. Tim: reinstalled this morning. Tonko can take a look.

EPD cooling: fan blowing to the fee box needs to be improved. Prashanth is working on it.

ESMD issue, crate 85? Shift crew had asked, and Will. J said he will work on it today. Will: the issue is on the electronics or could be water flowing, but not understood yet. Someone can try to clear the bubbles by removing the quick release, etc…Prashanth/Will will email Bill to follow this up and try when we have access again.

L2 had died and brought back, but L4 is still dead (the machine died). L4 issue we should call Diyu. L4 should be back now. Will give an updated instruction to deal with this for shift crew.

TPC hot spots. Experts, please investigate. Flemming: tpc, there are 2 hotspots always there. itpc, there are hotspots come and go. Will update the reference plots Monday.

Jeff already made a tune file. Jeff will put one prepost in BBC, EPD, ZDC etc.

BCW has been running fine since yesterday.

Plans

Will clarify the schedule for emergency power tests with RHIC.

Cosmic data taking with Reverse Full field.

Can use APEX mode for running single beams.

Readiness checklist update for physics next week.

May 14^th (Tuesday), RHIC will open for Linda H (DOE office of science)

STAR daily operation meeting 04/18/2024

RHIC Schedule

4K cool down.

Plan for this week,

26 GeV PS test tonight (STAR can stay open during this test)

Blue beam (100 GeV) injection on April 19 evening (tomorrow evening).

Yellow beam injection on April 21 (Sunday); first collision expected April 23-25.

STAR status

Maintenance: only TPC water work (temperature sensor) was done, but other tests were postponed to next Wednesday (April 24).

sTGC air blower alarm seems to have issues, but the temperature is stable and has been included in the run. Tim will check this.

BTOW issue with crate id 26 and VME 24. Oleg fixed it and he will put back the BSMD after this meeting.

EPD status: Tim needs to replace TUFF box (aim for tomorrow.) Cooling: fan blowing to the fee box needs to be improved.

ESMD issue, crate 85? Status?

Will check to include ETOW in the run.

Communications:

Call MCR for communications to CAS.

Plans

Cosmic data taking with Reverse Full field.

Readiness checklist update for physics next week.

Akio wants to be on call list for beam tomorrow evening.

Jeff will make a tune file today.

BCW – need issue reports. Shift crews need to document all actions.

May 14^th (Tuesday), RHIC will open for Linda H (DOE office of science)

STAR daily Operation meeting 04/17/2024

§ RHIC Schedule

4K cool down.

Plan for this week,

·      Blue beam injection on April 19 (Friday).

·      Yellow beam injection on April 21 (Sunday)

·      First collision expected April 23-25.

§ STAR status

·      Power dip last night. Subdetectors were brought back on except a few issues:

o   sTGC air blower.

o   TPC air blower, Alexei will look into it with help from Tim.

o   BEMC is back, and EEMC CANBUS are down, no control?

o   Some works need to be done for the gas system of MTD. MTD can still be operated safely.

·      Mike Lisa: EPD seemed to have issue with TUFF box and bad voltages. Shift crew turned EPD off during the evening shift. Mike turned them on this morning, and Tim needs to take a look. Cooling will be added to the FEE box.

·      Geary: ETOF instruction was reminded on starops. Will remind shift crew and include ETOF for noise run later today after maintenance.

·      Eleanor: fixed the BCW and gave instructions to the shift crew.

·      Will J: EEMC chiller status and how to turn things off during the power tests. This is already noted.

·      ESMD issue, crate 85? Tim will try to fix it after the power tests.

·      Flemming: requested special run for TPC and was taken during the evening shift, Run 25107059

·      RHIC status computer on shift leader desk (Jim Thomas sent an email to Angelika for username and password)

·      CAS will come taking down the magnet.

§ Plans

·      Downtime (10:30-17:00) today. Emergency power test, magnet power test, MCW maintenance (part change, postponed to next week!), TPC water maintenance (temperature sensor)

·      Cosmic data taking with Reverse Full field.

·      Detector status update.

·      Readiness checklist update for physics next week.

·      Power dip recovery instruction needs to be reprinted.

RHIC/STAR Schedule [calendar]

Printer-friendly version
Login or register to post comments

The STAR experiment

General

User login

Navigation

Notes from Operations Meeting

Saturday, October 5th, 2024

Urgent Issues:

Recap of the Past 24 Hours:

Encountered Issues:

Plan for the Day (RHIC & STAR):

Looking Ahead:

Friday, October 4, 2024

Recap of the Past 24 Hours:

Encountered Issues:

Plan for the Day (RHIC & STAR):

Looking Ahead:

Thursday, October 3, 2024 Recap of the Past 24 Hours:

Encountered Issues:

Plan for the Day (RHIC & STAR):

Looking Ahead:

Recap of the Past 24 Hours:

Encountered Issues:

Plan for the Day (RHIC & STAR):

Looking Ahead:

Monday, September 30, 2024

Sunday, September 29, 2024

Saturday, September 28, 2024

Friday, September 27, 2024

Thursday, September 26, 2024

Wednesday, September 25, 2024

Tuesday, September 24, 2024

Monday, September 23, 2024

Sunday, September 22, 2024

Saturday, September 21, 2024

Friday, September 20, 2024

Thursday, September 19, 2024

Wednesday, September 18, 2024

Tuesday, July 16, 2024

Status & Issues

Schedule & Plans

Monday, July 15, 2024

Status & Issues

Schedule & Plans

Sunday, July 14, 2024

Status & Issues

Schedule & Plans

Saturday, July 13, 2024

Status & Issues

Schedule & Plans

Friday, July 12, 2024

Status & Issues

Schedule & Plans

Thursday, July 11, 2024

Status & Issues

Schedule & Plans

Wednesday, July 10, 2024

Status & Issues

Schedule & Plans

Tuesday, July 9, 2024

Status & Issues

Schedule & Plans

Monday, July 8, 2024

Status & Issues

Schedule & Plans

Sunday, July 7, 2024

Status & Issues

Schedule & Plans

Saturday, July 6, 2024

Status & Issues

Schedule & Plans

Friday, July 5, 2024

Status & Issues

Schedule & Plans

Thursday, July 4, 2024

Status & Issues

Schedule & Plans

Wednesday, July 3, 2024

Status & Issues

Schedule & Plans

Tuesday, July 2. 2024

Thursday, October 3, 2024

Recap of the Past 24 Hours: