====== Intro ======
The Spectrum XMC1151 digitizer libraries we have require a specific flavour of Linux to operate.
Kernel from OS install 2016.04.14
Linux lvisf 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
====== Base CentOS ======
Using the **500_00746_r100_SDR7000_Linux_SG.pdf** document, set up the core data system.
===== DVD Boot =====
- Attached the external USB CDROM drive to the computer via the USB3 port
- Insert the Minimal CentOS 6.5 Install Disk
- Boot and set the BIOS to boot off the Plextor drive (our USB based DVD drive)
- At CentOS 6 Welcome, Click -> Next
- Language: English (English)
- Keyboard: U.S. English
- Storage Devices: Basic Storage Devices
- Hostname: lvisf
- Timezone: New York
- root Password: the usual
- Partitioning: Use All Space
- Select the drive (for this case, the 60 GB (or 60000 MB drive) and put that to the right, then Write Changes
- Wait for install to complete (280 packages in ~**4** minutes)
- Drive will eject
- Remove DVD
- Reboot
===== First Boot =====
- Welcome -> Forward
- License: Agree
- Create User: lvis
- Date and Time: accept defaults, do not sync to time server
- Kdump: NO, do **NOT** enable **kdump**
- Reboot
We will attempt to use a 2.6.32 kernel to satisfy the requirements of the Spectrum Signal XMC-1151 digitizer.
====== CentOS Configure ======
===== lvisf Config =====
- Log in as **root** on the main console since ethernet will be disabled by default
- Install Intel ethernet drivers:
- Download {{:subsystem:datasystem:dev:centos65minimal:igb-5.3.4.4.tar.gz|}} to a USB drive
- Uncompress in the home directory
tar xvf igb-5.3.4.4.tar.gz
- Build and install the drvier
cd ~/tmp/igb-5.3.4.4/src/
make
sudo make install
- Modify this module file
sudo nano /etc/sysconfig/modules/lvisf.modules
- Put the following into this file
#!/bin/sh
# start intel network driver
/sbin/modprobe igb
- Change permissions so this script runs at boot time
sudo chmod 755 /etc/sysconfig/modules/lvisf.modules
- Configure our network device
- Change the **eth0** definition
- editing the config file
vi /etc/sysconfig/network-scripts/ifcfg-eth0
- to the following
DEVICE=eth0
HWADDR=00:80:82:1B:E7:04
TYPE=Ethernet
UUID=98bd88d8-8fe8-4b8e-bca0-a58bd3306717
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth0"
IPADDR=10.4.1.2
NETMASK=255.255.255.0
- Add our default gateway
- Edit the network configuration
vi /etc/sysconfig/network
- End up looking like this
NETWORKING=yes
HOSTNAME=lvisf
NETWORKING_IPV6=no
IPV6INIT=no
GATEWAY=10.4.1.1
- Ensure our name server can be found
- Edit **resolv.conf**
vi /etc/resolv.conf
- End up with this
search gsfc.nasa.gov
nameserver 128.183.10.134
nameserver 128.183.50.17
- REMOVE the keyboard and mouse and never use them again (USB interrupts)
- Add **lvis** user to the system (still as root (logged in via SSH for me now))
- Add the user
useradd lvis
- Set password
passwd lvis
- Add **lvis** user to the list of **sudo** users
- Modify the super user file with the following (as logged in as root)
visudo
- After the line that has root and ALL and ALL permissions, insert this line
lvis ALL=(ALL) ALL
- Add **ntfs** drive capability
- Add the **epel** repository
sudo yum install epel-release
- Add **ntfs** to our system
sudo yum install fuse fuse-ntfs-3g
**Remove Unused Services**
- Remove **exim**
- Stop the service
sudo service exim stop
- Remove the package from the system
sudo rpm -e --nodeps exim
- Remove **postfix**
- Stop the service
sudo service postfix stop
- Remove the package from the system
sudo rpm -e --nodeps postfix
===== Spectrum Config =====
Still as root logged into the main desktop
- Disable the Firewall
- Turn off the firewall
sudo service iptables stop
sudo service ip6tables stop
- Make this permanent
sudo chkconfig iptables off
sudo chkconfig ip6tables off
- Disable SELinux
- Edit the configuration file
sudo vi /etc/selinux/config
- Modify the file
- From
SELINUX=enforcing
- To
SELINUX=disabled
- Reboot for all changes to take place
===== Minimize Kernel Modules =====
- Disable **ipv6**, this addition is required along with a line in the modprobe file below
- Modifiy **/etc/sysconfig/network**
sudo vi /etc/sysconfig/network
- And append the following two lines
NETWORKING_IPV6=no
IPV6INIT=no
- Blacklist kernel modules we don't require
- Edit a file **lvisf-blacklist.conf**
sudo vi /etc/modprobe.d/lvisf-blacklist.conf
- Add the following lines
# lvisf kernel module blacklist
# version 1.0 - dlr - 20160418
# ipv6
blacklist ipv6
# NMI watchdog
blacklist iTCO_wdt
blacklist iTCO_vendor_support
blacklist i2c_i801
blacklist igb
blacklist lpc_ich
blacklist shpchp
blacklist dm_mirror
# intel sound
blacklist soundcore
blacklist snd
blacklist snd_seq
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi
# USB controller (still loads)
blacklist xhci_hcd
# this is required to REMOVE ipv6 module, I am not sure why
# http://www.cyberciti.biz/faq/redhat-centos-disable-ipv6-networking/
install ipv6 /bin/true
==== Extras ====
* These two commands will remove some other modules I couldn't figure out how to make permanently go away
# USB driver... if you're using the nousb kernel, this will work
sudo modprobe --remove xhci_hcd
sudo modprobe --remove dm_mirror
==== Minimal Modules ====
Module Size Used by
e1000 170646 0
sg 29350 0
ext4 374902 3
jbd2 93427 1 ext4
mbcache 8193 1 ext4
sd_mod 39069 3
crc_t10dif 1541 1 sd_mod
ahci 42215 2
i915 624511 1
drm_kms_helper 44321 1 i915
drm 280012 2 i915,drm_kms_helper
i2c_algo_bit 5935 1 i915
i2c_core 31084 4 i915,drm_kms_helper,drm,i2c_algo_bit
video 20674 1 i915
output 2409 1 video
dm_mod 84209 9
====== lvisf code ======
===== SVN Repository =====
* Dave Yancich sent me this note back in May:
Repository URL: https://zj.yancich.com:8443/svn/NASA/trunk/lvisF
username "lvis"
password: standard lvis password
This is a cmake project.
To rebuild the make files:
In "build" directory type "cmake ../"
Then "make" to build everything.
"make clean" to clean the programs and kernel modules.
==== Build Document ====
(Yancich sent this to me in March 2016) Original document is {{:subsystem:datasystem:dev:rt:configuration_and_build_procedures_for_lvisf_and_support_libraries.docx|}}
==== Requirements ====
- Install **cmake**
sudo yum install cmake
- Install **htop**
sudo yum install htop
- Install **jed**
sudo yum install ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/jed-0.99.18-5.el5.kb.i386.rpm
- Install **nano**
sudo yum install nano
- Install **p7zip**
sudo yum install p7zip
- Install PCI utilities
sudo yum install pciutils
- Install **telnet**
sudo yum install telnet
- Install **wget**
sudo yum install wget
- Install the development package
sudo yum groupinstall "Development Tools"
- Install kernel development source
cd
mkdir download
cd ~/download
wget http://vault.centos.org/6.5/os/x86_64/Packages/kernel-devel-2.6.32-431.el6.x86_64.rpm
sudo yum --nogpgcheck localinstall ~/download/kernel-devel-2.6.32-431.el6.x86_64.rpm
==== Code Base ====
- Create the repoistory home and check out the source code
- Create a repository home in **/opt**
sudo mkdir /opt/lvisf
- Set proper permissions
sudo chown -R lvis:lvis /opt/lvisf
- Check out the raw source code
cd /opt/lvisf
svn co https://svn.yancich.com:8443/svn/NASA/LVIS/trunk/ src
- Enter the user home directory and create symbolic links
cd /home/lvis
mkdir src
ln -s /opt/lvisf/src/* /home/lvis/src/
- Add the following lines to the **.bashrc** for the **lvis** user
- Edit the file
nano /home/lvis/.bash_profile
- Add these lines
# For SVN commit comment editor
export SVN_EDITOR=nano
# For the Innovative digitizer
export WD_BASEDIR=/home/lvis/src/Innovative/WinDriver
# For the SyncClock timer card
export PLX_SDK_DIR=/home/lvis/src/PLX/PlxSdk
# For the spectrum digitizer
export SSP_INSTALL_ROOT=/home/lvis/src/spectrum
- Edit the file
sudo nano /etc/ld.so.conf.d/lvisf.conf
- Add these lines
/home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib
/home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/libs
/home/lvis/src/spectrum/shared/linux/pc64/lib
- Reload the library path
sudo ldconfig
==== SyncClock32 PLX Drivers ====
- Build all drivers
- Go to the source code home directory
cd /home/lvis/src/PLX/PlxSdk/Driver
- Execute the build script
./buildalldrivers
- Build the samples
- Go to the source directory
cd /home/lvis/src/PLX/PlxSdk
- Make the programs
make
- Load the driver
- To the binary directory
cd /home/lvis/src/PLX/PlxSdk/Bin
- Execute the script for the 9030 card (our syncclock32)
sudo ./Plx_load 9030
# to just see what boards are supported, run the script with no argument
./Plx_load
==== MIC 3753 DIO Drivers ====
**NOTE** These drivers NEED to be built as root, you cannot **sudo** to build these drivers
- Become root all the time
sudo bash
- Go to the code base directory
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/driver_base/src/lnx_ko
- Build the drivers
make
- Go into our specific card directory
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/pci1753_mic3753_pcm3753i/src/lnx_ko
- Build the drivers
make
- Load the drivers
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin
insmod biokernbase.ko
insmod bio1753.ko
==== Innovative Digitizer Drivers ====
=== Innovative OEM Drivers ===
- Make a directory for the Innovative code
sudo mkdir /usr/Innovative
- Copy base Innovative support software into **/usr**
sudo cp /home/lvis/src/INNO-RAW/* /usr/Innovative/
- Fix the permssions so we can execute this script
sudo chmod 755 /usr/Innovative/Setup.sh
- Create this link to the kernel source so the Innovative driver modules can build
sudo ln -s /usr/src/kernels/2.6.32-431.el6.x86_64 /usr/src/linux
- Move into and run the setup script
cd /usr/Innovative
sudo ./Setup.sh
=== Innovative Custom Drivers ===
- Build the **wdapi** support library
- Enter the code directory
cd /home/lvis/src/Innovative/WinDriver/src/wdapi
- Compile the library
make -C LINUX
- Create this symbolic link, but the SVN checkout already has it
ln -s /home/lvis/src/Innovative/WinDriver/src/wdapi/LINUX/libwdapi.so /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib/libwdapi1180.so
- Build the **winCloudDrvr**
- Enter the source directory
cd /home/lvis/src/Innovative/winCloudDrvr
- Compile
make -C linux
- Build our custom driver
- Enter the directory
cd /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist
- Configure and compile
./configure --disable-usb-support
make
- Install
sudo make install
To load the driver:
- **insmod** our custom driver
sudo /opt/lvisf/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist/wdreg winCloudDrvr auto
==== Spectrum Digitizer Drivers ====
- Create the symbolic link for /spectrum
sudo ln -s /opt/lvisf/src/spectrum /spectrum
- Enter the code directory
cd /home/lvis/src/spectrum/shared/linux/pc64/bin
- Execute the setup script
sudo bash ./setup_xmc1151_dvt.sh
- Select the proper option: 1)Install for instance
====== Clean Boot Operations ======
===== Load Drivers =====
- Load the MIC-3753 DIO Drivers
sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/biokernbase.ko
sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/bio1753.ko
- Load the SyncClock32 Timer Drivers
cd /home/lvis/src/PLX/PlxSdk/Bin
sudo ./Plx_load 9030
===== Compile Code =====
- Enter the **build** directory for lvisf
cd /home/lvis/src/lvisF/build
- Use **cmake**
cmake ../
- Build code
make
===== Run Tests =====
==== User Space Latency Test ====
- Enter the testing program
cd /home/lvis/src/lvisF/build/bin/tests/syncclock
- Run the latency program
sudo ./galvMove
====== Latency Tweaking ======
* Set CPU affinity to CPU 3 with something like, made it much more consistent since I guess most things go for CPU 0 by default
CPU_SET(3, &cpus);
* Change the scheduler policy, this information is displayed in the **syncclock** test program
sudo chrt --fifo 99 ./galvMove
* Isolate a CPU, and then have the code attach itself to that cpu
* In **/etc/grub.conf** add this command to the kernel command line, so all user space code will avoid CPU 2 and 3 in this example, leaving 0 and 1 for the OS
isolcpus=2,3
* Change the program such that
CPU_SET(3, &cpus);
* Run the program with **chrt** or not
sudo chrt --fifo 99 ./galvMove
* Set the timer card interrupt affinity to a single CPU (not the prime)
sudo bash -c 'echo 2 > /proc/irq/19/smp_affinity'
* Using **2** here put the interrupt 19 servicing on CPU 1 exclusively
* More low latency items to add to the kernel command line for **/etc/grub.conf**
rhgb nousb quiet isolcpus=2,3 nosoftlockup mce=ignore_ce audit=0
* **nosoftlockup** disables backtrace functionality
* **mce=ignore_ce** ignores corrected errors
* **audit=0** disables kernel components of the audit subsystem
===== Testing =====
* Testing delays
* Install **perf**
sudo yum install perf
* Run this command to record some latency data
cd /opt/lvisf/src/lvisF/build/bin/tests/syncclock
sudo perf sched record -o /dev/shm/perf.data chrt --fifo 99 ./galvMove
* Run the following to read the data
sudo perf sched -i /dev/shm/perf.data latency > /tmp/perf.latency.report
===== Show Interrupt Status =====
* This command shows interrupts for each core / CPU
cat /proc/interrupts
# notice that CPU1 has all the timer card interrupts (PLx9030)
[lvis@lvisf syncclock]$ cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 125 0 0 0 IR-IO-APIC-edge timer
8: 1 0 0 0 IR-IO-APIC-edge rtc0
9: 0 0 0 0 IR-IO-APIC-fasteoi acpi
18: 1968 0 0 0 IR-IO-APIC-fasteoi eth2
19: 0 2021327 0 0 IR-IO-APIC-fasteoi Plx9030
24: 0 0 0 0 DMAR_MSI-edge dmar0
25: 0 0 0 0 DMAR_MSI-edge dmar1
26: 2 0 0 0 IR-HPET_MSI-edge hpet2
27: 0 0 0 0 IR-HPET_MSI-edge hpet3
28: 0 0 0 0 IR-HPET_MSI-edge hpet4
29: 0 0 0 0 IR-HPET_MSI-edge hpet5
34: 235 0 0 0 IR-PCI-MSI-edge i915
35: 0 0 0 0 IR-PCI-MSI-edge ahci
36: 7971 0 0 0 IR-PCI-MSI-edge ahci
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 20569 29227 10924 50493 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 IRQ work interrupts
RES: 324 194 372 353 Rescheduling interrupts
CAL: 47 197 178 211 Function call interrupts
TLB: 142 610 312 1 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 1 1 1 1 Machine check polls
ERR: 0
MIS: 0
===== Test Results =====
* 2016.04.27: Ran a test for 2.7 hours. We missed 46 shots out of 49097902 at 5khz (200 useconds) so as far as actually reacting to the interrupt, we are in good shape.
* Time to make a build 1 data system and get better numbers as we do all our tasks.