david.rabine

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Intro: Minimal CentOS 6.5 ======

The Spectrum XMC1151 digitizer libraries we have require a specific flavour of Linux to operate.

Download the specific version by getting the [[http://vault.centos.org/6.5/isos/x86_64/CentOS-6.5-x86_64-minimal.iso]] which is the "minimal" CentOS 6.5.

  * Actual direct download link for minimal 6.5 CentOS ISO:  [[http://archive.kernel.org/centos-vault/6.5/isos/x86_64/CentOS-6.5-x86_64-minimal.iso]] (md5sum is **0d9dc37b5dd4befa1c440d2174e88a87**  CentOS-6.5-x86_64-minimal.iso)

Kernel from OS install 2016.04.14 <code>
Linux lvisf 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
</code>
====== Base CentOS ======

Using the **500_00746_r100_SDR7000_Linux_SG.pdf** document, set up the core data system.

===== DVD Boot =====

**NOTE:** Press the **Delete** key while booting in order to change the boot to either the USB PLEXTOR drive or the LEXAR CFAST drive
  * BIOS -> Boot -> Boot Option #1
  * Then F4: Save & Exit


  - Attached the external USB CDROM drive to the computer via the USB3 port (USB stick with UNETBOOTIN did NOT WORK for me?!) (UnetBootin will work with later version of CentOS image... 6.5 is broken)
  - Insert the Minimal CentOS 6.5 Install Disk
  - Boot and set the BIOS to boot off the Plextor drive (our USB based DVD drive)
  - At CentOS 6 Welcome, Click -> Next
  - Language:  English (English)
  - Keyboard:  U.S. English
  - Storage Devices:  Basic Storage Devices
  - **NOTE:** If another OS detected, select:  Fresh Installation (SURE YOU WANT TO DELETE THAT OS!?)
  - Hostname:  lvisf
  - Timezone:  New York
  - root Password:  the usual
  - Partitioning:  Use All Space
  - Select the drive (for this case, the 60 GB (or 60000 MB drive) and put that to the right, then Write Changes
  - Wait for install to complete (205 packages in ~**4** minutes)
  - Drive will eject
  - Remove DVD
  - Reboot


====== CentOS Configure ======

===== lvisf Config =====

==== Network Temporary Configuration (DHCP) ====

  - Setup the system for DHCP for now
    - Log in as root
    - Modify **/etc/sysconfig/network-scripts/ifcfg-eth0** so <code>
ONBOOT=yes
</code>
    - Startup network (will take a moment, DHCP) <code>
sudo ifup eth0
</code>


==== Prerequisites for Compiling Kernel Modules ====

  * [[http://www.tldp.org/LDP/lkmpg/2.6/html/x121.html]] <- build kernel module link for an example

  - Install the development tools <code>
yum groupinstall 'Development Tools'
</code>
  - Get some useful programs <code>
yum install nano -y
yum install wget -y
</code>
  - Grab the ethernet driver while you still have access
    - Current driver is here:  [[https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz]] <code>
mkdir ~/down
cd ~/down
wget https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz
</code>

==== Ethernet Driver Install and Configuration ====


  - Log in as **root** on the main console since ethernet will be disabled by default
  - Install Intel ethernet drivers **[Only needed if compiling on Kontron flight hardware]**:
    - Download {{:subsystem:datasystem:dev:centos65minimal:igb-5.3.4.4.tar.gz|}} to a USB drive
    - Current driver is here:  [[https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz]]
    - Uncompress in the home directory <code>
tar xvf igb-5.3.4.4.tar.gz
</code>
    - Build and install the drvier <code>
cd ~/tmp/igb-5.3.4.4/src/
make
sudo make install
</code>
    - Modify this module file <code>
sudo nano /etc/sysconfig/modules/lvisf.modules
</code>
    - Put the following into this file <code>
#!/bin/sh

# start intel network driver
/sbin/modprobe igb
</code>
    - Change permissions so this script runs at boot time <code>
sudo chmod 755 /etc/sysconfig/modules/lvisf.modules
</code>
  - Configure our network device
    - Change the **eth0** definition
        - editing the config file <code>
vi /etc/sysconfig/network-scripts/ifcfg-eth0
</code>
        - to the following <code>
DEVICE=eth0
HWADDR=00:80:82:1B:E7:04
TYPE=Ethernet
UUID=98bd88d8-8fe8-4b8e-bca0-a58bd3306717
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth0"
IPADDR=10.4.1.2
NETMASK=255.255.255.0
</code>
    - Add our default gateway
      - Edit the network configuration <code>
vi /etc/sysconfig/network
</code>
      - End up looking like this <code>
NETWORKING=yes
HOSTNAME=lvisf
NETWORKING_IPV6=no
IPV6INIT=no
GATEWAY=10.4.1.1
</code>
    - Ensure our name server can be found
      - Edit **resolv.conf** <code>
vi /etc/resolv.conf
</code>
      - End up with this <code>
search gsfc.nasa.gov
nameserver 128.183.10.134
nameserver 128.183.50.17
</code>
    - Turn off IPV6 (found here:  [[https://www.centos.org/forums/viewtopic.php?t=7989]]) <code>
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
</code>
    - Change the **eth1** definition
        - editing the config file <code>
vi /etc/sysconfig/network-scripts/ifcfg-eth1
</code>
        - to the following <code>
DEVICE=eth1
HWADDR=00:80:82:1B:E7:05
TYPE=Ethernet
UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth1"
IPADDR=192.168.0.2
NETMASK=255.255.255.0
</code>
    - Change the **eth1:1** definition
        - editing the config file <code>
vi /etc/sysconfig/network-scripts/ifcfg-eth1:1
</code>
        - to the following <code>
DEVICE=eth1:1
HWADDR=00:80:82:1B:E7:05
TYPE=Ethernet
UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth1:1"
IPADDR=129.100.0.2
NETMASK=255.255.255.0
</code>
  - REMOVE the keyboard and mouse and never use them again (USB interrupts)
  - Add **lvis** user to the system (still as root (logged in via SSH for me now))
    - Add the user <code>
useradd lvis
</code>
    - Set password <code>
passwd lvis
</code>
  - Add **lvis** user to the list of **sudo** users
    - Modify the super user file with the following (as logged in as root) <code>
visudo
</code>
    - After the line that has root and ALL and ALL permissions, insert this line <code>
lvis    ALL=(ALL)       ALL
</code>

==== File System Extras ====

  - Add **ntfs** drive capability
    - Add the **epel** repository <code>
sudo yum install epel-release
</code>
    - Add **ntfs** to our system <code>
sudo yum install fuse fuse-ntfs-3g
</code>
  - Add **Exfat** drive capability
    - Add the **epel** repository and this Nux Dextop repository<code>
sudo yum install epel-release
sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el6/x86_64/nux-dextop-release-0-2.el6.nux.noarch.rpm
</code>
    - Add **Exfat** to our system <code>
sudo yum install exfat-utils fuse-exfat
</code>
    - An example mounting an Exfat system is <code>
sudo  mount -t exfat /dev/sdc1 /mnt/tmp
</code>

==== X11 ====

  - This will allow you to run **X11** <code>
# core X11
sudo yum install xorg-x11-xauth xorg-x11-fonts-* xorg-x11-utils xterm -y
# required for tuna
sudo yum install xauth pygtk2-libglade -y
</code>
  - Install **tuna** tuning package <code>
sudo yum install tuna -y
</code>

==== Remove Unused Services ====

  - Remove **exim** (NOTE, this was not on the CentOS Minimal I just installed 2018.01.04 (but leaving for historical))
    - Stop the service <code>
sudo service exim stop
</code>
    - Remove the package from the system <code>
sudo rpm -e --nodeps exim
</code>
  - Remove **auditd** <code>
sudo service auditd stop # now
sudo chkconfig auditd off # after reboot
</code>
  - Remove **postfix**
    - Stop the service <code>
sudo service postfix stop
</code>
    - Remove the package from the system <code>
sudo rpm -e --nodeps postfix
</code>

===== Spectrum Config =====

Still as root logged into the main desktop

  - Disable the Firewall
    - Turn off the firewall <code>
sudo service iptables stop
sudo service ip6tables stop
</code>
    - Make this permanent <code>
sudo chkconfig iptables off
sudo chkconfig ip6tables off
</code>
  - Disable SELinux
    - Edit the configuration file <code>
sudo vi /etc/selinux/config
</code>
    - Modify the file
      - From <code>
SELINUX=enforcing
</code>
      - To <code>
SELINUX=disabled
</code>
  - Reboot for all changes to take place

===== Minimize Kernel Modules =====

  - Disable **ipv6**, this addition is required along with a line in the modprobe file below
    - Modifiy **/etc/sysconfig/network** <code>
sudo vi /etc/sysconfig/network
</code>
    - And append the following two lines <code>
NETWORKING_IPV6=no
IPV6INIT=no
</code>
    - Add these two lines to **sshd_config** <code>
sudo nano /etc/ssh/sshd_config
# lvisf - ipv4 only
AddressFamily inet
ListenAddress 0.0.0.0
</code>
  - Blacklist kernel modules we don't require
    - Edit a file **lvisf-blacklist.conf** <code>
sudo vi /etc/modprobe.d/lvisf-blacklist.conf
</code>
    - Add the following lines <code>
# lvisf kernel module blacklist
# version 1.0 - dlr - 20160418

# ipv6
blacklist ipv6

# NMI watchdog
blacklist iTCO_wdt 
blacklist iTCO_vendor_support

blacklist i2c_i801
# blacklist igb
blacklist lpc_ich
blacklist shpchp
blacklist dm_mirror

# intel sound
blacklist soundcore
blacklist snd
blacklist snd_seq
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi

# USB controller (still loads)
blacklist xhci_hcd


# this is required to REMOVE ipv6 module, I am not sure why
# http://www.cyberciti.biz/faq/redhat-centos-disable-ipv6-networking/
install ipv6 /bin/true
</code>

==== Extras ====

  * These two commands will remove some other modules I couldn't figure out how to make permanently go away <code>
# USB driver... if you're using the nousb kernel, this will work
sudo modprobe --remove xhci_hcd
sudo modprobe --remove dm_mirror
</code>

==== Minimal Modules ====

<code>
Module                  Size  Used by
e1000                 170646  0 
sg                     29350  0 
ext4                  374902  3 
jbd2                   93427  1 ext4
mbcache                 8193  1 ext4
sd_mod                 39069  3 
crc_t10dif              1541  1 sd_mod
ahci                   42215  2 
i915                  624511  1 
drm_kms_helper         44321  1 i915
drm                   280012  2 i915,drm_kms_helper
i2c_algo_bit            5935  1 i915
i2c_core               31084  4 i915,drm_kms_helper,drm,i2c_algo_bit
video                  20674  1 i915
output                  2409  1 video
dm_mod                 84209  9 
</code>

====== lvisf code ======

===== SVN Repository =====

:!: **NOTE:** This is obsolete (the repository link), but left here for posterity.
  * Dave Yancich sent me this note back in May:  <code>
Repository URL:  https://zj.yancich.com:8443/svn/NASA/trunk/lvisF
username "lvis"
password: standard lvis password

This is a cmake project.

To rebuild the make files:
In "build" directory type "cmake ../"
Then "make" to build everything.
"make clean" to clean the programs and kernel modules.
</code>

==== Build Document ====
(Yancich sent this to me in March 2016)  Original document is {{:subsystem:datasystem:dev:rt:configuration_and_build_procedures_for_lvisf_and_support_libraries.docx|}}

==== Requirements ====

  - Install **cmake** <code>
sudo yum install cmake -y
</code>
  - Install **htop** <code>
sudo yum install htop -y
</code>
  - Install **jed** <code>
sudo yum install ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/jed-0.99.18-5.el5.kb.i386.rpm -y
</code>
  - Install **nano** <code>
sudo yum install nano -y
</code>
  - Install **p7zip** and **pigz**<code>
sudo yum install p7zip pigz -y
</code>
  - Install PCI utilities <code>
sudo yum install pciutils -y
</code>
  - Install **screen** <code>
sudo yum install screen -y
</code>
  - Install **telnet** <code>
sudo yum install telnet -y
</code>
  - Install **wget** <code>
sudo yum install wget -y
</code>
  - Install the development package <code>
sudo yum groupinstall "Development Tools"
</code>
  - Install development libraries for md5sum and gzip in real time <code>
sudo yum install openssl-devel -y
</code>
  - Install md5deep so we can check file integrity <code>
sudo yum install md5deep -y
</code>
  - Install man pages <code>
sudo yum install man -y
sudo yum install man-pages -y
</code>
  - Install kernel development source <code>
cd
mkdir download
cd ~/download
wget http://vault.centos.org/6.5/os/x86_64/Packages/kernel-devel-2.6.32-431.el6.x86_64.rpm
sudo yum --nogpgcheck localinstall ~/download/kernel-devel-2.6.32-431.el6.x86_64.rpm -y
</code>

==== Code Base ====

  - Create the repoistory home and check out the source code
    - Create a repository home in **/opt** <code>
sudo mkdir /opt/lvisf
</code>
    - Set proper permissions<code>
sudo chown -R lvis:lvis /opt/lvisf
</code>
    - **[Optional]** Uncompress our locally attached mount with snapshot (faster than network initial checkout)<code>
cd /
tar jxvf /media/LVIS-GH-2TB_WD209/lvisf_build/lvisf_sourcesnapshot_20160816_svn439.tar.bz2
</code>
      - **[Either]** Check out he raw source code from scratch <code>
cd /opt/lvisf
svn co https://svn.yancich.com:8443/svn/NASA/LVIS/trunk/ src
</code>
      - **[Or]** Update it if you used the tar above <code>
cd /opt/lvisf/src
svn update
</code>
    - Enter the user home directory and create symbolic links<code>
cd /home/lvis
mkdir src
ln -s /opt/lvisf/src/* /home/lvis/src/
</code>
  - Add the following lines to the **.bashrc** for the **lvis** user
    - Edit the file <code>
nano /home/lvis/.bash_profile
</code> 
    - Add these lines <code>
# For SVN commit comment editor
export SVN_EDITOR=nano

# For the Innovative digitizer
export WD_BASEDIR=/home/lvis/src/Innovative/WinDriver

# For the SyncClock timer card
export PLX_SDK_DIR=/home/lvis/src/PLX/PlxSdk

# For the spectrum digitizer
export SSP_INSTALL_ROOT=/home/lvis/src/spectrum
</code>
   - Edit the file <code>
sudo nano /etc/ld.so.conf.d/lvisf.conf
</code> 
    - Add these lines <code>
/home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib
/home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/libs
/home/lvis/src/spectrum/shared/linux/pc64/lib
</code>
    - Reload the library path <code>
sudo ldconfig
</code>

==== SyncClock32 PLX Drivers ====

  - Build all drivers
    - Go to the source code home directory <code>
cd /home/lvis/src/PLX/PlxSdk/Driver
</code>
    - Execute the build script <code>
./buildalldrivers
</code>
  - Build the samples
    - Go to the source directory <code>
cd /home/lvis/src/PLX/PlxSdk
</code>
    - Make the programs <code>
make
</code>
  - Load the driver
    - To the binary directory <code>
cd /home/lvis/src/PLX/PlxSdk/Bin
</code>
    - Execute the script for the 9030 card (our syncclock32) <code>
sudo ./Plx_load 9030

# to just see what boards are supported, run the script with no argument
./Plx_load
</code>

==== MIC 3753 DIO Drivers ====

**NOTE**  These drivers NEED to be built as root, you cannot **sudo** to build these drivers

  - Become root all the time <code>
sudo bash
</code>
  - Go to the code base directory <code>
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/driver_base/src/lnx_ko
</code>
  - Build the drivers <code>
make
</code>
  - Go into our specific card directory <code>
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/pci1753_mic3753_pcm3753i/src/lnx_ko
</code>
  - Build the drivers <code>
# to disable IRQs in this driver, just use this code where the IRQ calls are commented out
cp init.c.no_irq init.c
make
</code>
  - Load the drivers (this crashes when i load the bio1753 (IRQ!!! issue) <code>
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin
insmod biokernbase.ko
insmod bio1753.ko
</code>
  - Exit the **sudo bash** <code>
exit
</code>
==== Innovative Digitizer Drivers ====

=== Innovative OEM Drivers ===

  - Make a directory for the Innovative code <code>
sudo mkdir /usr/Innovative
</code>
  - Copy base Innovative support software into **/usr** <code>
sudo cp /home/lvis/src/INNO-RAW/* /usr/Innovative/
</code>
  - Fix the permssions so we can execute this script <code>
sudo chmod 755 /usr/Innovative/Setup.sh
</code>
  - Create this link to the kernel source so the Innovative driver modules can build <code>
sudo ln -s /usr/src/kernels/2.6.32-431.el6.x86_64 /usr/src/linux
</code>
  - Move into and run the setup script <code>
cd /usr/Innovative
sudo ./Setup.sh
</code>

=== Innovative Custom Drivers ===

  - Build the **wdapi** support library
    - Enter the code directory <code>
cd /home/lvis/src/Innovative/WinDriver/src/wdapi
</code>
    - Compile the library <code>
make -C LINUX
</code>
    - Create this symbolic link, but the SVN checkout already has it <code>
ln -s /home/lvis/src/Innovative/WinDriver/src/wdapi/LINUX/libwdapi.so /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib/libwdapi1180.so
</code>
  - Build the **winCloudDrvr**
    - Enter the source directory <code>
cd /home/lvis/src/Innovative/winCloudDrvr
</code>
    - Compile <code>
make -C linux
</code>
  - Build our custom driver
    - Enter the directory <code>
cd /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist
</code>
    - Configure and compile <code>
./configure --disable-usb-support
make
</code>
    - Install <code>
sudo make install
</code>

To load the driver:
  - **insmod** our custom driver <code>
sudo /opt/lvisf/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist/wdreg winCloudDrvr auto
</code>

==== Spectrum Digitizer Drivers ====

  - Create the symbolic link for /spectrum <code>
sudo ln -s /opt/lvisf/src/spectrum /spectrum
</code>
  - Enter the code directory <code>
cd /home/lvis/src/spectrum/shared/linux/pc64/bin
</code>
  - Copy the drivers for our kernel into the binary directory <code>
cp -a /home/lvis/src/spectrum/shared/linux/pc64/bin/`uname -r`/*.ko /home/lvis/src/spectrum/shared/linux/pc64/bin/
</code>
  - Execute the setup script <code>
chmod 755 *.sh   # not sure why these were not already executable
sudo bash ./setup_xmc1151_dvt.sh
</code>
  - Select the proper option:  1)Install

==== Auto-load MIC-3753 Driver ====

  - symlink the drivers into the kernel directory <code>
sudo ln -s /opt/lvisf/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/*.ko /lib/modules/2.6.32-431.el6.x86_64/kernel/drivers/misc/
</code>
  - add to our startup script **/etc/sysconfig/modules/lvisf.modules**
    - modify the file <code>
sudo nano /etc/sysconfig/modules/lvisf.modules
</code>
    - add the following <code>
# start the MIC-3753 drivers
/sbin/modprobe biokernbase
/sbin/modprobe bio1753
</code>

==== Auto-load Plx9030 Driver ====

**NOTE: (in lieu of step 2 below)** I had this not work when I manually edited it the first time.  A copy of the file is in the repository, you could just copy that file in:
<code>
sudo cp /opt/lvisf/src/lvisF/scripts/lvisf.modules /etc/sysconfig/modules/lvisf.modules
</code>

  - symlink the driver into the kernel so it will come up on boot <code>
sudo ln -s /opt/lvisf/src/PLX/PlxSdk/Driver/Source.Plx9000/Output/Plx9030.ko /lib/modules/2.6.32-431.el6.x86_64/kernel/drivers/misc/</code>
  - add to our startup script **/etc/sysconfig/modules/lvisf.modules**
    - modify the file <code>
sudo nano /etc/sysconfig/modules/lvisf.modules
</code>
    - add the following <code>
# start the Plx9030 syncclock32 driver
/sbin/modprobe Plx9030</code>
  - symbolically link in our **rc.local** into **/etc/rc.local**
    - remove the original <code>
# remove the symlink
sudo rm /etc/rc.local
# move the original to something else
sudo mv /etc/rc.d/rc.local /etc/rc.d/rc.local_orig
</code>
    - link in the **lvisf** version <code>
sudo ln -s /opt/lvisf/src/lvisF/scripts/rc.local /etc/rc.d/rc.local
sudo ln -s /etc/rc.d/rc.local /etc/rc.local
</code>

===== Installing Boost Library =====

The **boost** C++ library is required for the data system.  The tarball of the 1.6.2 version is in the repository.  Or download the latest from [[http://www.boost.org/users/download/]].  Here is the latest as of 2018.01.04 [[https://dl.bintray.com/boostorg/release/1.66.0/source/boost_1_66_0.tar.bz2]]

  - Uncompress the tar into **/opt** <code>
cd /opt
sudo tar -jxvf /opt/lvisf/src/tars/boost_1_62_0.tar.bz2
</code>
  - Symbolically link to a generic boost directory <code>
sudo ln -s /opt/boost_1_62_0 /opt/boost
</code>
  - Chown it all to **lvis** user <code>
sudo chown -R lvis:lvis /opt/boost_1_62_0
</code>
  - Set the environment variable for the **BOOST_ROOT** variable
    - Edit the file /home/lvis/.bash_profile <code>
nano /home/lvis/.bash_profile 
</code>
    - Add the line <code>
# Add the boost root directory
export BOOST_ROOT=/opt/boost
</code>
    - Run our profile to load this variable <code>
source /home/lvis/.bash_profile
</code>
  - Compile the **boost** libraries <code>
cd /opt/boost
./bootstrap.sh
./b2
</code>

===== Installing USB Temperature Library =====

Follow the instructions below to compile the USB temperature monitoring device libraries required.

  * [[subsystem:datasystem:dev:hardware:mccdaq:usbtemp:start|Measurement Computing USB-TEMP-AI code instructions.
]]

====== Clean Boot Operations ======

===== Load Drivers =====

  - Load the MIC-3753 DIO Drivers <code>
sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/biokernbase.ko
sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/bio1753.ko
</code>
  - Load the SyncClock32 Timer Drivers <code>
cd /home/lvis/src/PLX/PlxSdk/Bin
sudo ./Plx_load 9030
</code>



===== Compile Code =====

  - Enter the **build** directory for lvisf <code>
cd /home/lvis/src/lvisF/build
</code>
  - Use **cmake** <code>
cmake ../
</code>
  - Build code <code>
make
</code>

===== Run Tests =====

==== User Space Latency Test ====

  - Enter the testing program <code>
cd /home/lvis/src/lvisF/build/bin/tests/syncclock
</code>
  - Run the latency program <code>
sudo ./galvMove
</code>

====== Latency Tweaking ======

  * Set CPU affinity to CPU 3 with something like, made it much more consistent since I guess most things go for CPU 0 by default <code>
CPU_SET(3, &cpus);
</code>
  * Change the scheduler policy, this information is displayed in the **syncclock** test program <code>
sudo chrt --fifo 99 ./galvMove
</code>
  * Isolate a CPU, and then have the code attach itself to that cpu
    * In **/etc/grub.conf** add this command to the kernel command line, so all user space code will avoid CPU 2 and 3 in this example, leaving 0 and 1 for the OS<code>
isolcpus=2,3
</code>
    * Change the program such that <code>
CPU_SET(3, &cpus);
</code>
    * Run the program with **chrt** or not <code>
sudo chrt --fifo 99 ./galvMove
</code>
  * Set the timer card interrupt affinity to a single CPU (not the prime) <code>
sudo bash -c 'echo 2 > /proc/irq/19/smp_affinity'
</code>
    * Using **2** here put the interrupt 19 servicing on CPU 1 exclusively
  * More low latency items to add to the kernel command line for **/etc/grub.conf** <code>
rhgb nousb quiet isolcpus=2,3 nosoftlockup mce=ignore_ce audit=0
</code>
    * **nosoftlockup** disables backtrace functionality
    * **mce=ignore_ce** ignores corrected errors
    * **audit=0** disables kernel components of the audit subsystem

===== Testing =====


  * Testing delays
    * Install **perf** <code>
sudo yum install perf
</code>
    * Run this command to record some latency data<code>
cd /opt/lvisf/src/lvisF/build/bin/tests/syncclock
sudo perf sched record -o /dev/shm/perf.data chrt --fifo 99 ./galvMove
</code>
    * Run the following to read the data <code>
sudo perf sched -i /dev/shm/perf.data latency > /tmp/perf.latency.report
</code>

===== Show Interrupt Status =====

  * This command shows interrupts for each core / CPU <code>
cat /proc/interrupts
# notice that CPU1 has all the timer card interrupts (PLx9030)
[lvis@lvisf syncclock]$ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       
  0:        125          0          0          0  IR-IO-APIC-edge      timer
  8:          1          0          0          0  IR-IO-APIC-edge      rtc0
  9:          0          0          0          0  IR-IO-APIC-fasteoi   acpi
 18:       1968          0          0          0  IR-IO-APIC-fasteoi   eth2
 19:          0    2021327          0          0  IR-IO-APIC-fasteoi   Plx9030
 24:          0          0          0          0  DMAR_MSI-edge      dmar0
 25:          0          0          0          0  DMAR_MSI-edge      dmar1
 26:          2          0          0          0  IR-HPET_MSI-edge      hpet2
 27:          0          0          0          0  IR-HPET_MSI-edge      hpet3
 28:          0          0          0          0  IR-HPET_MSI-edge      hpet4
 29:          0          0          0          0  IR-HPET_MSI-edge      hpet5
 34:        235          0          0          0  IR-PCI-MSI-edge      i915
 35:          0          0          0          0  IR-PCI-MSI-edge      ahci
 36:       7971          0          0          0  IR-PCI-MSI-edge      ahci
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      20569      29227      10924      50493   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RES:        324        194        372        353   Rescheduling interrupts
CAL:         47        197        178        211   Function call interrupts
TLB:        142        610        312          1   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:          1          1          1          1   Machine check polls
ERR:          0
MIS:          0
</code>

===== Test Results =====

  * 2016.04.27:  Ran a test for 2.7 hours.  We missed 46 shots out of 49097902 at 5khz (200 useconds) so as far as actually reacting to the interrupt, we are in good shape.
  * Time to make a build 1 data system and get better numbers as we do all our tasks.