Slow Controls Operations Manual
STAR
Slow Controls Operations Manual
Creighton University
version 7.0
Dec 02, 2007
This manual is supplemented by the figures in the
STAR Slow Controls Training presentation.
Slow Controls Operations Manual
Telephone Numbers *
Line Commands *
Slow Controls Architecture *
Hardware Controls Workstations 6
Color Coding of Data 6
The Alarm Handler 7
Starting and Running the Alarm Handler 9
Viewing Archived Data 12
Accessing Front End Processors 13
Special VME, DAQ, Trigger and Magnet Commands 13
Telephone Numbers (Nov 12, 2007)
Slow Controls at STAR (631) 344-8242
BNL apartment 22B (631) 344-1043
at Creighton (402) 280-2208
Michael Cherney
cell phone (will ring at BNL) (402) 305-0238
at Creighton (402) 280-3039
Line Commands
Open a terminal on sc3.starp and get
[sc3.starp.bnl.gov]:~>
From sc3.starp.bnl.gov there are many alias commands for bringing
up medm screens (guis) and launching tools.
The most important one to be able to do as a shift crew member is to start the
Alarm handler (sometimes called alh or ALH)
startalh & -- starts the Alarm Handler
Note: using the P button you can access almost every subsystem medm screen from the Alarm Handler
Other usefull aliases that can be used from sc3.starp. All give the same screens as those available from
the Alarm Handler.
vme_plat1 --- vme power supplies for the first floor of platform
vme_plat2 --- vme power supplies for the 2nd floor of platform
eemc_canbus --- vme power supplies for the EEMC
emc_canbus --- vme power supplies for the EMC
zdc --- ZDC HV controls
upVPD --- upVPD HV controls
bbchv --- BBC HV controls
richscal --- rich scalars
tpctop --- gets the tpc top level user interface
trighv --- starts the trigger high voltage control program
and many others see .user_aliases on sc3.starp and/or sc.starp
The second most important command you can do from the terminal is
to open a serial connection to a IOC processor by opening a telnet session.
Note : This can be done from any terminal session on a starp network computer.
There can only be one session open.
If a session is open the command will fail!
telnet scserv xxxx
connects to the serial port of the front end processor connected to
port xxxx of the terminal server on the platform.
A list of processors and port numbers is posted to the left of sc3.starp
Archived data may be viewed at the web site: http://sc3.starp.bnl.gov/archive
Front End Processors (located on the platform or in the DAQ room) carry out all of the control and monitoring functions. Sun Workstations display the data (processes on the workstations listen for broadcasts of parameter values and broadcast user requests to the front end processors over Ethernet).
The STAR Hardware Controls system sets, monitors and controls all subsystems. In addition, Hardware Controls can generate and display alarms and warnings for each subsystem. A real-time VxWorks environment is utilized for this purpose and a controls software package, EPICS, is used to provide a common interface to all subsystems. The baseline STAR detector consists of time projection chamber subsystems (anode high voltage, cathode field cage, gating grid, front-end electronics power supply, HDLC link, gas, laser, interlocks, and VME-crate control), mechanisms for the exchange of information with the STAR trigger and online, as well as external magnet and accelerator systems. Approximately 16,000 parameters governing experiment operation are currently controlled and monitored. Magnet status and accelerator information is obtained from the accelerator controls database using a Control DEVice (CDEV) client/server interface. The set of databases for detector configuration is maintained by Hardware Controls. STAR Hardware Controls maintains the system-wide control of the STAR detector with its EPICS databases at the subsystem level. Databases are constructed on the subsystem level, such that they can be used for subsystem testing and remain easily included in a larger detector configuration. An HDLC link provides the field bus used by the experiment for controls and an alternate data path.
STAR hardware is required to be designed failsafe. All required personnel and property protection is mandated to be independent of STAR Hardware Controls. This does not mean that you cannot cause damage to the detector using the slow controls interface. If you are in doubt about an action contact an expert.
EPICS was selected as the foundation for the STAR control software environment because it incorporates a standard means of sharing information and services, graphical display and control interfaces, and a robust, real-time operating system. At STAR, EPICS is run on Sun workstations connected to VME processors running the VxWorks operating system. The components of EPICS used at STAR are the Motif Editor and Display Manager (MEDM), the Graphics Database Configuration Tool (GDCT), the sequencer, the alarm handler, and the data archiver.
EPICS is a set of software tools and applications jointly developed by Argonne National Laboratory and Los Alamos National Laboratory for the purpose of controlling Particle Accelerators and Large Experiments. Present and future development is being done cooperatively by Argonne (ANL), Los Alamos National Laboratory (LANL), Lawrence Berkeley Laboratory (LBL), the Jefferson National Laboratory (CEBAF), the Spallation Neutron Source Collaboration , BESSY (Berliner Elektronenspeicherring-Gesellschaft für Synchrotronstrahlung m.b.H.) and DESY (Deutsches Elektronen-Synchrotron).
EPICS provides: interfaces to instrumentation from data acquisition, supervisory control and steady-state control through a table entry database, an operator interface to all control system parameters through interactive displays, data logging through a table entry archiving file, alarm management through a table entry alarm file, sequential control through a state definition language with convenient database interface routines, channel access routines for interfacing the control system data to data analysis, and other functions not provided in the control system. The basic components needed are the Operator Interface (OPI), Input Output Controller (IOC), and a Local Area Network, which allows the OPI and IOC to communicate.
Hardware Controls Workstations
Slow Controls workstations are located to the left of the Level 3 Workstation. The main controls workstation (sc3.starp.bnl.gov) and a second workstation (sc4.starp.bnl.gov) are positioned next to each other with sc4 to the right of sc3. The system should be running on sc3. Access from other computers is controlled. (Telnet is possible from sc4 and certain online computers.) The current username and password for shift operators are posted on the sc3.star.bnl.gov monitor. These may be updated. Please check each time you are on shift and do not use old login names and passwords.
Shutting down the workstation or restarting the workstation should have no effect on the front end processors. The user interface is independent of the front end controllers. Instructions for accessing these processors are included later in this document.
Green entries identify data which is believed to be correct and within bounds
Yellow indicates entries that are slightly outside of bounds. Corrective action should be taken or logbook entries should be made.
Red tags data that is beyond normal operating limits. Corrective action should be taken or an expert should be contacted. A logbook entry should be made.
White entries indicate channels where data is missing or where the validity is suspect. An expert should be contacted. A logbook entry should be made.
The Alarm Handler is used to identify alarms, study the symptoms, and contact an expert. It is an interactive graphical application used primarily to display and monitor EPICS database alarm states. It serves as an interface between an operator and the EPICS database and it communicates with the database using the same mechanism (channel access) as the controls screens. The user interface for the Alarm Handler contains a hierarchical display of an alarm configuration structure allowing both high level and detailed views of the alarm configuration structure in one window.
Purpose of the Alarm Handler
The Alarm Handler monitors alarm conditions of EPICS database records. The primary responsibilities of the Alarm Handler are to: bring alarms to the operator's attention, provide the operator guidance for handling of specific alarms, allow the operator to globally acknowledge alarms, provide a graphical view of current database alarms, and provide for logging alarms and display of the logged alarm history.
Alarm Handler Terminology
An alarm is an unexpected change of state for an EPICS database parameter. Examples of alarm conditions are: deviations from tolerance band, software or hardware errors, and loss of communication to hardware or linked parameters
When a change in a parameter is detected, a message is sent to each process monitoring the parameter. The Alarm Handler may be one of these processes.
There are two parts to an alarm: the alarm status and the severity of that alarm status. Alarm status and severity are set and checked whenever a parameter is processed. The alarm severity can take one of the following four values: no alarm, minor alarm, major alarm, or invalid data alarm. A correspondence between these alarm severities and the alarm state are preprogrammed in the EPICS database. The relationship is loaded when the front end processor is booted. The EPICS database specifies the alarm state of the each parameter. The state is defined on the basis of the value of the parameter (being within certain bounds) and on the proper completion of data collection and processing cycles.
The Alarm Handler displays alarms for an arbitrary set of channels. Each Alarm Channel refers to a specific EPICS database parameter. The Alarm Handler provides a grouping mechanism so that related channels can be grouped together. An Alarm Group consists of a named set of lower-level groups and/or parameters. An Alarm Channel will be considered in an error state if there is a loss of communication or communication error between the Alarm Handler and front end processor.
Alarm Handler Windows
The Alarm Handler display consists of two types of windows, a Runtime Window and a Main Window. While the Alarm Handler is executing, the Runtime Window is always displayed.
The Runtime Window is a small icon-like window that contains a single button containing the name of the alarm configuration main Alarm Group. The color of this button is used to show the highest alarm severity of any outstanding alarms.
White: Data Invalid Alarm
Red: Major Alarm
Yellow: Minor Alarm
Background color: No Alarm
Beeping and blinking of the button is used to show the presence of unacknowledged alarms. Pressing the Runtime Window button will open the Alarm Handler Main Window or, if already open, bring the MainWindow to the top of the window stack. The Close or Quit item on the window manager menu allows the user to exit the Alarm Handler.
The Alarm Handler Main Window is divided into three parts: a menu bar, an alarm configuration display area, and a message area. On the menu bar, there are selections for pull-down menu items that perform all the functions of the Alarm Handler. The alarm configuration display area is divided into two major parts: an alarm configuration tree structure display and an Alarm Group content display. The current alarm configuration tree structure appears in the first area, and a list of the contents of the currently selected Alarm Group from the alarm configuration tree structure appears in the second area. Color is used to show alarm severity. A single character severity code is also provided to assist operators who may have difficulty distinguishing colors. The message area can show the current execution mode, the current execution state, and the name of the current configuration file. It also contains buttons to silence alarms, and explanatory descriptions of the alarm configuration summary abbreviation codes and status data which appear in the group summary and channel status lines.
Starting and Running the Alarm Handler
Invoke the Alarm Handler by executing the command: startalh
The alarm handler should be operated at the left controls workstation (sc3). The Runtime Window appears. There will be beeping and the button will blink to indicate the presence of unacknowledged alarms.
Some users prefer using tpctop, a generic user interface. Normally the monitoring functions found in tpctop are carried out at the subsystem workstations and you will not need to make any changes from this screen. The same control screens available in tpctop are in general accessible from the Alarm Handler.
To get a top level control screen, at the UNIX shell prompt type: tpctop
Click on the small square next to the name of the subsystem that you want to examine.
From this screen you can make changes to the subsystems even if there is another similar interface open at the subsystem.
Click on the Runtime Window to bring up the Main Window.
You can then relocate the Runtime Window.
Click in a box in the lower right corner of the Main Window to silence beeping.
Silence Forever will stop the beeping forever. Silence Current will stop the beeping until you get another alarm. Silence Current is probably the sensible choice if you are watching the screen. Alarms will continue to be updated on the display regardless of your selection. Be sure Silence Forever is not clicked when you stop watching the screen.
To acknowledge an alarm, click on the left-most box.
To get the name of the expert responsible for this subsystem,
Click on "G" for Guidance.
To see the control screen for this subsystem, click on "P" for Process.
There are two ways to see which channels are involved in a Group.
Clicking on the Group name lists the channels in the frame on the right.
Clicking on the right-pointing triangle lists the channels in the left frame in a tree.
To collapse the tree back to the original view, click the triangle again.
You may want to leave this window open at all times when the experiment is running.
To Exit, click on "Close" on either the file menu on the menu bar or on the Runtime Window. This will bring up the Alarm Handler Exit dialog window.
Make sure you kill any icons that are no longer in use when you exit.
Enabling and Disabling Alarm Systems
Individual subsystem may be partially or total disabled using the control screen obtained by clicking on the process button of the DISABLE-ALARMS subsystem.
Alarm Handler Tree Structure
The alarm handler configures the display of parameters into a tree structure. The detector is divided into groups (subsystems) which are divided into subgroups that are made up of individual parameters. Branches of the tree (alarm groups) are displayed as separate Group Summary Lines. Buttons on the tree structure display allow the user to expand or deflate the alarm display.
This display line summarizes the status of all alarms for the group (normally a detector subsystem or part of a subsystem) named in this line and all lower level groups (smaller divisions of the subsystem or individual channels). The group summary display consists of the following items:
Acknowledgment Button
The button is activated only if an unacknowledged alarm is present for the group and the alarm handler is active. This button is color-coded and labeled with a corresponding letter representing the highest severity of unacknowledged alarm as follows:
White: labeled with V indicates an invalid data alarm
Red: labeled with R indicates a major alarm
Yellow: labeled with Y indicates a minor alarm
Background color indicates no alarm present
This button describes the highest severity of unacknowledged alarms for this group and all associated subgroups. Clicking on this button while alarm handler is active will send alarm acknowledgments to all alarm channels associated with the group. It has the same effect as individually acknowledging each subgroup channel in an unacknowledged alarm state.
Single Character Severity Code
This code shows the highest level alarm whether or not the alarm has been acknowledged. (The acknowledge button only identifies the highest level unacknowledged alarm.) A one-character severity display code is present only if at least one channel associated with the group is in alarm or in an error state. It is color-coded and represents the highest severity outstanding alarm following the same coding system as the acknowledge button. The code shows the highest severity current alarm for this group and all its subgroups.
Group Name Selection Button
This button is used to look at the next level down the tree. The alarm handler allows the operator to choose any alarm group or channel which is lower on the tree by clicking on this button. The result is to display one level of the contents associated with the selected subgroup. This subgroup is also becomes the currently selected group (your location in the tree). If this name is associated with a collection of channels, the button has an associated right-pointing triangle so that an operator can distinguish paths to channels (no triangle - end of tree) from paths to subgroups (triangle - additional channels below).
Alarm Configuration Summary
The Alarm Configuration Summary is a five character display. Each character in the display is a "-" or the character identifying special treatment of this alarm from the following list:
C: cancelled - alarms will not be sent to the alarm handler
D: disabled - alarms will be ignored by the alarm handler
A: alarms are not required to be acknowledged
T: transient alarms are not required to be acknowledged
L: alarms are not logged
If the current configuration for any parameter in this group or any subgroup is configured to be handled in special way, the corresponding character is displayed, otherwise the character "-" is displayed. For example, the string "--A--" means that at least one parameter is configured not to require the acknowledgment of alarm and that no parameters have any other special configurations.
Alarm Severity Summary
The alarm severity summary allows the user to assess how widespread a particular alarm is. The display shows the total number of parameters with a particular alarm severity for this group and all its subgroups. This summary consists of a set of four entries: (data alarm, major alarm, minor alarm, no alarm)
Each field value indicates the total number of parameters with the specified severity in the associated group and all its subgroups.
The archived data can be viewed from any web browser. To do this in the control room, open a browser on sc by typing the shell command: Netscape
Go to: http://sc3.starp.bnl.gov/archive. This will work from any computer. (Experts can check from home.) Do not view Netscape from the same workstation where you are running the alarm handler or the control display program (tpctop). (Opening screens in the wrong order will mess up your display. It is easier to keep these programs separate to avoid this conflict.) At the website click on the slow controls archive button. Also at the home page are links to the STAR website and the RHIC website where the icons lead directly into those websites. The slow controls button will lead into the main listing of all the database variables listed in the archive. At the main listing choose the variables of the archive, which is needed. From there it either leads directly into that information or into a subdirectory of more variables. If it goes into a subdirectory, continue to narrow the field down until it leads into a list of the information.
This list of information will contain a list of all the different pieces of data contained in the particular directory and times of when they were first and last archived. First, in the box labeled "Names", take out all the variables that are not needed. Next set the start time and end time of the data that is to be reviewed. From there two options can be performed, a graph and a list of data. To get the graph click the plot button, and to get the list of data click the get button. They can only work for ten or less variables so if more are needed then the graphs or spreadsheets will need to be taken more than once for different variables.
Accessing Front End Processors
In the case where a front end processor has a fault it may need to be examined or rebooted. This should only be done on the advice of an expert. This is accomplished by requesting an independent serial connection to a processor located on the platform obtained by typing: telnet scserv xxxx
at a UNIX prompt where xxxx is the port address obtained from the expert. (tip a typed on sc3.starp will do the same thing for the processor in the DAQ room.) The processor can now be tested using VxWorks commands provided by an expert. If you are told to reboot, typing [control]x should initiate the process. Rebooting does not normally affect the front end electronics. It will not shutdown the system. The system can only be shutdown by taking appropriate action on the individual control screens.
and Magnet Commands
VME crates should not be turned on and off except at the advice of an expert. A better option may be to reset the VME bus. This can be done from the controls display. These processes can be carried out remotely for crates on the platform and in the DAQ room using the alarm handler. To obtain a controls screen, click on the program button (P) after the group name of the VME crates (PLATFORM-VME for the platform, or DAQ-VME for the DAQ room). The control program for the VME status for crates in the DAQ room can also be obtained by typing: daq_crate at the UNIX shell prompt. The crate control program for VME crates on the platform can also be accessed by selecting the VME option using tpctop.
The trigger high voltage control program can be accessed by typing: trighv at the UNIX shell prompt.
To restart the link to RHIC and STAR magnet data, type the command: cdev at the Unix shell prompt to get to the correct directory, followed by the commands:
source new.cshrc and MagnetData at the Unix shell prompt.
# Description Port# Crate Location Processor IP address
________________________________________________________________________________
1. CANbus (STAR)
1st 2nd Floor 9003 51 2A9 grant.starp 130.199.61.103
2. CANbus (BARREL)
Barrel crates 9040 100 2C4-1 bemccan.starp 130.199.60.59
3. CANbus (EEMC)
EEMC/QT/West PT 9020 99 2C4-1 vtpc1.starp 130.199.60.189
4. Field Cage 9001 56 2A4 vtpc4.starp 130.199.60.192
5. Gated Grid 9002 54 2A6 vtpc3.starp 130.199.60.191
6. TPC FEE 9004 58 2B5 vtpc2.starp 130.199.60.190
7. Cathode HV 9005 57 2A3 cath.starp 130.199.60.162
8. Inner Anode HV 9006 52 2A7 vtpc7.starp 130.199.61.78
9. BBC HV 9010 77 1A7-1 bdb.starp 130.199.61.218
ZDCsmd, and upVPD
10. Ground 9011 57 2A3 vsc2.starp 130.199.60.217
Plane Pulser
11. Interlock 9012 52 2A7 epics2.starp 130.199.60.149
TPC Temperature
12. Outer Anode HV 9013 59 2A6 vtpc5.starp 130.199.60.193
13. Platform Hygrometer
TPC Gas 9015 58 2B5 hdlc.starp 130.199.60.161
14. Trigger HV 9021 63 1A6 cdb.strarp 130.199.60.40
ZDChv programs
15. SSD 9026 79 1C6 sdvmesc.starp 130.199.60.120
16. SVT not used svtmonitor.starp 130.199.61.50
17. FTPC 9033 71 1B5-1 ftpc.starp 130.199.61.83
Slow Controls
18. EMC TDC 9039 80 2C4-2 creighton5.starp 130.199.60.229
& Slow Controls
19. daq temp & humidity
& gain DAQ room DC2 burton.starp 130.199.61.104
20. CDEV DAQ room DC3-2 vsc1.starp 130.199.60.188
Scalars and Magnet
21. Autoramp anode DAQroom DC2-1 stargate.starp 130.199.61.48
& cathode & testbits
22. TOF_Gas program DAQroom DC3-3 taylor.starp 130.199.60.6
23. CANbus iowritest DAQroom DC3-1 tutor.starp 130.199.60.46
Program
(needs to be rebooted daily)
24. Daq Hygrometer DAQroom DC3-1 medm.starp 130.199.60.49
& GID (PC in daq room)
TPC Lecroy serial session for inner sectors Port 9037
TPC Lecroy serial session for outer sectors Port 9038
FTPC Lecroy serial session Port 9023
SVT?? Lecroy serial session Port 9034
SMD?? Lecroy serial session Port 9035
REMOTE POWER SUPPLIES---requires a telnet
rps1.starp.bnl.gov 130.199.60.26 2A4
rps2.starp.bnl.gov 130.199.60.205 2A3
rps3.starp.bnl.gov 130.199.60.206 2A6
bemcpower.starp.bnl.gov 130.199.60.54 2C4
eemccanpower.starp.bnl.gov 130.199.60.90 1C2
pmdrps2.trg.bnl.local 172.16.128.208 1A3
Name: scdaqpower.starp.bnl.gov Address: 130.199.60.95
computone servers
scserv.starp.bnl.gov 130.199.60.167
Name: scserv2.starp.bnl.gov Address: 130.199.60.96
scserv:notes
telnet scserv
then type help
There is also a web interface http://scserv.starp.bnl.gov/
Name: scserv2.starp.bnl.gov
Address: 130.199.60.96
username and password not assigned yet
- Printer-friendly version
- Login or register to post comments