Intel Xeon Phi
http://software/intel.com/mic-developer
Acronyms to know by heart:
MIC: Many Integrated Core
MPSS: Manycore Platform Software Stack (driver and tools for configuring and communicating with Xeon Phi processors)
SCIF: Symmetric Communications InterFace (part of MPSS; it abstracts the details of communications between Xeon Phi and either other Xeon Phi or Xeon main CPU over a PCI-Express bus)
Intel IMCI: Initial Many Core Instructions (extensions of the Intel 64 architecture-based vector graphic streaming SIMD instructions for use on MIC/Xeon Phi processors)
OFED: OpenFabrics Enterprise Distribution (software stack for Infiniband adapters - since we have no InfiniBand adapters, this isn't relevant)
Loaner 2 notes:
2 x 8 core Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz (16 actual cores total, 32 threads with Hyper-threading)
64GB RAM
dual Intel Xeon Phi co-processors (details to come once MPSS is installed...)
MPSS downloaded: http://registrationcenter.intel.com/irc_nas/3267/mpss_gold_update_3-2.1.6720-15-rhel-6.4.tar
in src, rebuild the kernel module rpm, since we are using an updated kernel:
[root@xeon-phi-dev src]# rpmbuild --rebuild intel-mic-kmod-2.1.6720-15.el6.src.rpm
and replace the original:
[root@xeon-phi-dev mpss_gold_update_3]# mv ~/rpmbuild/RPMS/x86_64/intel-mic-kmod-2.1.6720-15.2.6.32.358.el6.x86_64.rpm ./
[root@xeon-phi-dev mpss_gold_update_3]# yum install --nogpgcheck --noplugins --disablerepo=* *.rpm
micctrl -s indicates both MICs are in 'ready' state, so
[root@xeon-phi-dev mpss_gold_update_3]# "/opt/intel/mic/bin/micflash -update -device all"
no warnings or errors, so reboot.
[root@xeon-phi-dev ~]# /opt/intel/mic/bin/micflash -update -smcbootloader -device all
no warnings or errors, so reboot.
chkconfig mpss on
[root@xeon-phi-dev ~]# /opt/intel/mic/bin/micinfo
MicInfo Utility Log
Created Sat Jul 20 11:21:38 2013
System Info
HOST OS : Linux
OS Version : 2.6.32-358.14.1.el6.x86_64
Driver Version : 6720-15
MPSS Version : 2.1.6720-15
Host Physical Memory : 65943 MB
Device No: 0, Device Name: mic0
Version
Flash Version : 2.1.03.0386
SMC Firmware Version : 1.15.4830
SMC Boot Loader Version : 1.8.4326
uOS Version : 2.6.38.8-g2593b11
Device Serial Number : ADKC30101612
Board
Vendor ID : 0x8086
Device ID : 0x2250
Subsystem ID : 0x2500
Coprocessor Stepping ID : 3
PCIe Width : x16
PCIe Speed : 5 GT/s
PCIe Max payload size : 256 bytes
PCIe Max read req size : 512 bytes
Coprocessor Model : 0x01
Coprocessor Model Ext : 0x00
Coprocessor Type : 0x00
Coprocessor Family : 0x0b
Coprocessor Family Ext : 0x00
Coprocessor Stepping : B1
Board SKU : B1PRQ-5110P/5120D
ECC Mode : Enabled
SMC HW Revision : Product 225W Passive CS
Cores
Total No of Active Cores : 60
Voltage : 990000 uV
Frequency : 1052631 kHz
Thermal
Fan Speed Control : N/A
Fan RPM : N/A
Fan PWM : N/A
Die Temp : 47 C
GDDR
GDDR Vendor : Elpida
GDDR Version : 0x1
GDDR Density : 2048 Mb
GDDR Size : 7936 MB
GDDR Technology : GDDR5
GDDR Speed : 5.000000 GT/s
GDDR Frequency : 2500000 kHz
GDDR Voltage : 1501000 uV
(same for mic1)
From the Intel White Paper "System Administration for the Intel® Xeon Phi™ Coprocessor" http://software.intel.com/sites/default/files/article/373934/system-administration-for-the-intel-xeon-phi-coprocessor.pdf
"Using the offload programming model, it is not necessary for a user to have an account on the coprocessor in order to run a program. Offloaded programs use a generic user, micuser, when executing on the coprocessor. However, if a user wishes to run a native program (one which starts execution on the coprocessor itself) or an MPI program, they will need to be able to log into the coprocessor itself.
Users will automatically be added to the /etc/passwd file for the coprocessor if they have an entry in the host’s /etc/passwd file, the user authentication method is set to local and their user id falls in the range specified in the /etc/sysconfig/mic/default.conf file. (See “User Access” in Intel® Manycore Platform Software Stack (Intel® MPSS) Boot Configuration Guide.) At the same time, a home directory will be created for them in the disk image used to make the coprocessor’s RAM file system.
In addition to being in the host’s /etc/passwd file, users must have a .ssh directory in their home directory on the host containing valid RSA keys. Users should not set a passphrase when they generate their keys; encryption of this passphrase can cause difficulty in logging onto the coprocessor. When a user’s home directory is created in the disk image for the coprocessor, the .ssh directory is copied over. This is the only thing which is copied from the user’s home directory on the host to the user’s home directory on the coprocessor.
Changes to the host’s /etc/passwd file or the user’s .ssh directory only take effect on the coprocessor after the micctrl --resetconfig command is run and the coprocessor is rebooted.
Users can also be added to the coprocessor’s /etc/passwd file using the micctrl --useradd command. Directions for doing this can be found with the command micctrl --help. If you choose to add users in this way, be sure to create a home directory for them as well, and ensure that their .ssh directory is copied over. The easiest way to do this is to use the --home and --sshkeys options on the micctrl --useradd command. If you choose to add users in this way, be careful – some micctrl options such as --cleanconfig and --configuser can completely remove and rebuild the coprocessor’s /etc/passwd file, causing you to lose users who were added manually.
Users can be explicitly removed using the micctrl --userdel command. Again, be careful since some options to micctrl can remove the coprocessor’s /etc/passwd file and remake it from the host’s /etc/passwd file.
In addition to changing the coprocessor’s /etc/passwd file, you can change its /etc/groups file using micctrl --groupadd and micctrl --groupdel."
MPSS commands of note:
micsmc provides a GUI monitor of the state of the Xeon Phi's (load, memory usage, temperature, log messages, system info). (It also has a CLI, but haven't tried it.)
micnativeloadex copies a Xeon Phi binary to a coprocessor and executes it.
Intel compiler stuff at BNL:
http://intranet.bnl.gov/itd/unix/software/intel.asp
http://mirror.bnl.gov/software/intel/
Side note:
RHEL 6 does not install 32-bit libraries by default in 64-bit versions (unlike RHEL 5). See for instance: https://access.redhat.com/site/solutions/36238
To automatically install multiple architectures for future package installations, edit /etc/yum.conf, adding "multilib_policy=all"
To get started using Intel(R) Composer XE 2013 Update 2 located in
/opt/intel/composer_xe_2013.2.146:
- Set the environment variables for a terminal window using one of the following
(replace "intel64" with "ia32" if you are using a 32-bit platform).
For csh/tcsh:
$ source /opt/intel/bin/compilervars.csh intel64
For bash:
$ source /opt/intel/bin/compilervars.sh intel64
To invoke the installed compilers:
For C++: icpc
For C: icc
example:
icpc -mmic -o hw.mic hw.cpp
/opt/intel/mic/bin/micnativeloadex hw.mic
- wbetts's blog
- Login or register to post comments