david.rabine

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Intro ======

The Spectrum XMC1151 digitizer libraries we have require a specific flavour of Linux to operate.

Kernel from OS install 2016.04.14 <code>
Linux lvisf 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
</code>
====== Base CentOS ======

Using the **500_00746_r100_SDR7000_Linux_SG.pdf** document, set up the core data system.

===== DVD Boot =====

  - Attached the external USB CDROM drive to the computer via the USB3 port
  - Insert the Minimal CentOS 6.5 Install Disk
  - Boot and set the BIOS to boot off the Plextor drive (our USB based DVD drive)
  - At CentOS 6 Welcome, Click -> Next
  - Language:  English (English)
  - Keyboard:  U.S. English
  - Storage Devices:  Basic Storage Devices
  - Hostname:  lvisf
  - Timezone:  New York
  - root Password:  the usual
  - Partitioning:  Use All Space
  - Select the drive (for this case, the 60 GB (or 60000 MB drive) and put that to the right, then Write Changes
  - Wait for install to complete (280 packages in ~**4** minutes)
  - Drive will eject
  - Remove DVD
  - Reboot

===== First Boot =====

  - Welcome -> Forward
  - License:  Agree
  - Create User:  lvis
  - Date and Time:  accept defaults, do not sync to time server
  - Kdump:  NO, do **NOT** enable **kdump**
  - Reboot


We will attempt to use a 2.6.32 kernel to satisfy the requirements of the Spectrum Signal XMC-1151 digitizer.

====== CentOS Configure ======

===== lvisf Config =====

  - Log in as **root** on the main console since ethernet will be disabled by default
  - Install Intel ethernet drivers:
    - Download {{:subsystem:datasystem:dev:centos65minimal:igb-5.3.4.4.tar.gz|}} to a USB drive
    - Uncompress in the home directory <code>
tar xvf igb-5.3.4.4.tar.gz
</code>
    - Build and install the drvier <code>
cd ~/tmp/igb-5.3.4.4/src/
make
sudo make install
</code>
    - Modify this module file <code>
sudo nano /etc/sysconfig/modules/lvisf.modules
</code>
    - Put the following into this file <code>
#!/bin/sh

# start intel network driver
/sbin/modprobe igb
</code>
    - Change permissions so this script runs at boot time <code>
sudo chmod 755 /etc/sysconfig/modules/lvisf.modules
</code>
  - Configure our network device
    - Change the **eth0** definition
        - editing the config file <code>
vi /etc/sysconfig/network-scripts/ifcfg-eth0
</code>
        - to the following <code>
DEVICE=eth0
HWADDR=00:80:82:1B:E7:04
TYPE=Ethernet
UUID=98bd88d8-8fe8-4b8e-bca0-a58bd3306717
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth0"
IPADDR=10.4.1.2
NETMASK=255.255.255.0
</code>
    - Add our default gateway
      - Edit the network configuration <code>
vi /etc/sysconfig/network
</code>
      - End up looking like this <code>
NETWORKING=yes
HOSTNAME=lvisf
NETWORKING_IPV6=no
IPV6INIT=no
GATEWAY=10.4.1.1
</code>
    - Ensure our name server can be found
      - Edit **resolv.conf** <code>
vi /etc/resolv.conf
</code>
      - End up with this <code>
search gsfc.nasa.gov
nameserver 128.183.10.134
nameserver 128.183.50.17
</code>
  - REMOVE the keyboard and mouse and never use them again (USB interrupts)
  - Add **lvis** user to the system (still as root (logged in via SSH for me now))
    - Add the user <code>
useradd lvis
</code>
    - Set password <code>
passwd lvis
</code>
  - Add **lvis** user to the list of **sudo** users
    - Modify the super user file with the following (as logged in as root) <code>
visudo
</code>
    - After the line that has root and ALL and ALL permissions, insert this line <code>
lvis    ALL=(ALL)       ALL
</code>
  - Add **ntfs** drive capability
    - Add the **epel** repository <code>
sudo yum install epel-release
</code>
    - Add **ntfs** to our system <code>
sudo yum install fuse fuse-ntfs-3g
</code>

**Remove Unused Services**

  - Remove **exim**
    - Stop the service <code>
sudo service exim stop
</code>
    - Remove the package from the system <code>
sudo rpm -e --nodeps exim
</code>
  - Remove **postfix**
    - Stop the service <code>
sudo service postfix stop
</code>
    - Remove the package from the system <code>
sudo rpm -e --nodeps postfix
</code>

===== Spectrum Config =====

Still as root logged into the main desktop

  - Disable the Firewall
    - Turn off the firewall <code>
sudo service iptables stop
sudo service ip6tables stop
</code>
    - Make this permanent <code>
sudo chkconfig iptables off
sudo chkconfig ip6tables off
</code>
  - Disable SELinux
    - Edit the configuration file <code>
sudo vi /etc/selinux/config
</code>
    - Modify the file
      - From <code>
SELINUX=enforcing
</code>
      - To <code>
SELINUX=disabled
</code>
  - Reboot for all changes to take place

===== Minimize Kernel Modules =====

  - Disable **ipv6**, this addition is required along with a line in the modprobe file below
    - Modifiy **/etc/sysconfig/network** <code>
sudo vi /etc/sysconfig/network
</code>
    - And append the following two lines <code>
NETWORKING_IPV6=no
IPV6INIT=no
</code>
  - Blacklist kernel modules we don't require
    - Edit a file **lvisf-blacklist.conf** <code>
sudo vi /etc/modprobe.d/lvisf-blacklist.conf
</code>
    - Add the following lines <code>
# lvisf kernel module blacklist
# version 1.0 - dlr - 20160418

# ipv6
blacklist ipv6

# NMI watchdog
blacklist iTCO_wdt 
blacklist iTCO_vendor_support

blacklist i2c_i801
blacklist igb
blacklist lpc_ich
blacklist shpchp
blacklist dm_mirror

# intel sound
blacklist soundcore
blacklist snd
blacklist snd_seq
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi

# USB controller (still loads)
blacklist xhci_hcd


# this is required to REMOVE ipv6 module, I am not sure why
# http://www.cyberciti.biz/faq/redhat-centos-disable-ipv6-networking/
install ipv6 /bin/true
</code>

==== Extras ====

  * These two commands will remove some other modules I couldn't figure out how to make permanently go away <code>
# USB driver... if you're using the nousb kernel, this will work
sudo modprobe --remove xhci_hcd
sudo modprobe --remove dm_mirror
</code>

==== Minimal Modules ====

<code>
Module                  Size  Used by
e1000                 170646  0 
sg                     29350  0 
ext4                  374902  3 
jbd2                   93427  1 ext4
mbcache                 8193  1 ext4
sd_mod                 39069  3 
crc_t10dif              1541  1 sd_mod
ahci                   42215  2 
i915                  624511  1 
drm_kms_helper         44321  1 i915
drm                   280012  2 i915,drm_kms_helper
i2c_algo_bit            5935  1 i915
i2c_core               31084  4 i915,drm_kms_helper,drm,i2c_algo_bit
video                  20674  1 i915
output                  2409  1 video
dm_mod                 84209  9 
</code>

====== lvisf code ======

===== SVN Repository =====


  * Dave Yancich sent me this note back in May:  <code>
Repository URL:  https://zj.yancich.com:8443/svn/NASA/trunk/lvisF
username "lvis"
password: standard lvis password

This is a cmake project.

To rebuild the make files:
In "build" directory type "cmake ../"
Then "make" to build everything.
"make clean" to clean the programs and kernel modules.
</code>

==== Build Document ====
(Yancich sent this to me in March 2016)  Original document is {{:subsystem:datasystem:dev:rt:configuration_and_build_procedures_for_lvisf_and_support_libraries.docx|}}

==== Requirements ====

  - Install **cmake** <code>
sudo yum install cmake
</code>
  - Install **htop** <code>
sudo yum install htop
</code>
  - Install **jed** <code>
sudo yum install ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/jed-0.99.18-5.el5.kb.i386.rpm
</code>
  - Install **nano** <code>
sudo yum install nano
</code>
  - Install **p7zip** <code>
sudo yum install p7zip
</code>
  - Install PCI utilities <code>
sudo yum install pciutils
</code>
  - Install **telnet** <code>
sudo yum install telnet
</code>
  - Install **wget** <code>
sudo yum install wget
</code>
  - Install the development package <code>
sudo yum groupinstall "Development Tools"
</code>
  - Install kernel development source <code>
cd
mkdir download
cd ~/download
wget http://vault.centos.org/6.5/os/x86_64/Packages/kernel-devel-2.6.32-431.el6.x86_64.rpm
sudo yum --nogpgcheck localinstall ~/download/kernel-devel-2.6.32-431.el6.x86_64.rpm
</code>


==== Code Base ====

  - Create the repoistory home and check out the source code
    - Create a repository home in **/opt** <code>
sudo mkdir /opt/lvisf
</code>
    - Set proper permissions<code>
sudo chown -R lvis:lvis /opt/lvisf
</code>
    - Check out the raw source code  <code>
cd /opt/lvisf
svn co https://svn.yancich.com:8443/svn/NASA/LVIS/trunk/ src
</code>
    - Enter the user home directory and create symbolic links<code>
cd /home/lvis
mkdir src
ln -s /opt/lvisf/src/* /home/lvis/src/
</code>
  - Add the following lines to the **.bashrc** for the **lvis** user
    - Edit the file <code>
nano /home/lvis/.bash_profile
</code> 
    - Add these lines <code>
# For SVN commit comment editor
export SVN_EDITOR=nano

# For the Innovative digitizer
export WD_BASEDIR=/home/lvis/src/Innovative/WinDriver

# For the SyncClock timer card
export PLX_SDK_DIR=/home/lvis/src/PLX/PlxSdk

# For the spectrum digitizer
export SSP_INSTALL_ROOT=/home/lvis/src/spectrum
</code>
   - Edit the file <code>
sudo nano /etc/ld.so.conf.d/lvisf.conf
</code> 
    - Add these lines <code>
/home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib
/home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/libs
/home/lvis/src/spectrum/shared/linux/pc64/lib
</code>
    - Reload the library path <code>
sudo ldconfig
</code>

==== SyncClock32 PLX Drivers ====

  - Build all drivers
    - Go to the source code home directory <code>
cd /home/lvis/src/PLX/PlxSdk/Driver
</code>
    - Execute the build script <code>
./buildalldrivers
</code>
  - Build the samples
    - Go to the source directory <code>
cd /home/lvis/src/PLX/PlxSdk
</code>
    - Make the programs <code>
make
</code>
  - Load the driver
    - To the binary directory <code>
cd /home/lvis/src/PLX/PlxSdk/Bin
</code>
    - Execute the script for the 9030 card (our syncclock32) <code>
sudo ./Plx_load 9030

# to just see what boards are supported, run the script with no argument
./Plx_load
</code>

==== MIC 3753 DIO Drivers ====

**NOTE**  These drivers NEED to be built as root, you cannot **sudo** to build these drivers

  - Become root all the time <code>
sudo bash
</code>
  - Go to the code base directory <code>
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/driver_base/src/lnx_ko
</code>
  - Build the drivers <code>
make
</code>
  - Go into our specific card directory <code>
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/pci1753_mic3753_pcm3753i/src/lnx_ko
</code>
  - Build the drivers <code>
make
</code>
  - Load the drivers <code>
cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin
insmod biokernbase.ko
insmod bio1753.ko
</code>

==== Innovative Digitizer Drivers ====

=== Innovative OEM Drivers ===

  - Make a directory for the Innovative code <code>
sudo mkdir /usr/Innovative
</code>
  - Copy base Innovative support software into **/usr** <code>
sudo cp /home/lvis/src/INNO-RAW/* /usr/Innovative/
</code>
  - Fix the permssions so we can execute this script <code>
sudo chmod 755 /usr/Innovative/Setup.sh
</code>
  - Create this link to the kernel source so the Innovative driver modules can build <code>
sudo ln -s /usr/src/kernels/2.6.32-431.el6.x86_64 /usr/src/linux
</code>
  - Move into and run the setup script <code>
cd /usr/Innovative
sudo ./Setup.sh
</code>

=== Innovative Custom Drivers ===

  - Build the **wdapi** support library
    - Enter the code directory <code>
cd /home/lvis/src/Innovative/WinDriver/src/wdapi
</code>
    - Compile the library <code>
make -C LINUX
</code>
    - Create this symbolic link, but the SVN checkout already has it <code>
ln -s /home/lvis/src/Innovative/WinDriver/src/wdapi/LINUX/libwdapi.so /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib/libwdapi1180.so
</code>
  - Build the **winCloudDrvr**
    - Enter the source directory <code>
cd /home/lvis/src/Innovative/winCloudDrvr
</code>
    - Compile <code>
make -C linux
</code>
  - Build our custom driver
    - Enter the directory <code>
cd /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist
</code>
    - Configure and compile <code>
./configure --disable-usb-support
make
</code>
    - Install <code>
sudo make install
</code>

To load the driver:
  - **insmod** our custom driver <code>
sudo /opt/lvisf/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist/wdreg winCloudDrvr auto
</code>

==== Spectrum Digitizer Drivers ====

  - Create the symbolic link for /spectrum <code>
sudo ln -s /opt/lvisf/src/spectrum /spectrum
</code>
  - Enter the code directory <code>
cd /home/lvis/src/spectrum/shared/linux/pc64/bin
</code>
  - Execute the setup script <code>
sudo bash ./setup_xmc1151_dvt.sh
</code>
  - Select the proper option:  1)Install for instance

====== Clean Boot Operations ======

===== Load Drivers =====

  - Load the MIC-3753 DIO Drivers <code>
sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/biokernbase.ko
sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/bio1753.ko
</code>
  - Load the SyncClock32 Timer Drivers <code>
cd /home/lvis/src/PLX/PlxSdk/Bin
sudo ./Plx_load 9030
</code>



===== Compile Code =====

  - Enter the **build** directory for lvisf <code>
cd /home/lvis/src/lvisF/build
</code>
  - Use **cmake** <code>
cmake ../
</code>
  - Build code <code>
make
</code>

===== Run Tests =====

==== User Space Latency Test ====

  - Enter the testing program <code>
cd /home/lvis/src/lvisF/build/bin/tests/syncclock
</code>
  - Run the latency program <code>
sudo ./galvMove
</code>

====== Latency Tweaking ======

  * Set CPU affinity to CPU 3 with something like, made it much more consistent since I guess most things go for CPU 0 by default <code>
CPU_SET(3, &cpus);
</code>
  * Change the scheduler policy, this information is displayed in the **syncclock** test program <code>
sudo chrt --fifo 99 ./galvMove
</code>
  * Isolate a CPU, and then have the code attach itself to that cpu
    * In **/etc/grub.conf** add this command to the kernel command line, so all user space code will avoid CPU 2 and 3 in this example, leaving 0 and 1 for the OS<code>
isolcpus=2,3
</code>
    * Change the program such that <code>
CPU_SET(3, &cpus);
</code>
    * Run the program with **chrt** or not <code>
sudo chrt --fifo 99 ./galvMove
</code>
  * Set the timer card interrupt affinity to a single CPU (not the prime) <code>
sudo bash -c 'echo 2 > /proc/irq/19/smp_affinity'
</code>
    * Using **2** here put the interrupt 19 servicing on CPU 1 exclusively
  * More low latency items to add to the kernel command line for **/etc/grub.conf** <code>
rhgb nousb quiet isolcpus=2,3 nosoftlockup mce=ignore_ce audit=0
</code>
    * **nosoftlockup** disables backtrace functionality
    * **mce=ignore_ce** ignores corrected errors
    * **audit=0** disables kernel components of the audit subsystem

===== Testing =====


  * Testing delays
    * Install **perf** <code>
sudo yum install perf
</code>
    * Run this command to record some latency data<code>
cd /opt/lvisf/src/lvisF/build/bin/tests/syncclock
sudo perf sched record -o /dev/shm/perf.data chrt --fifo 99 ./galvMove
</code>
    * Run the following to read the data <code>
sudo perf sched -i /dev/shm/perf.data latency > /tmp/perf.latency.report
</code>

===== Show Interrupt Status =====

  * This command shows interrupts for each core / CPU <code>
cat /proc/interrupts
# notice that CPU1 has all the timer card interrupts (PLx9030)
[lvis@lvisf syncclock]$ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       
  0:        125          0          0          0  IR-IO-APIC-edge      timer
  8:          1          0          0          0  IR-IO-APIC-edge      rtc0
  9:          0          0          0          0  IR-IO-APIC-fasteoi   acpi
 18:       1968          0          0          0  IR-IO-APIC-fasteoi   eth2
 19:          0    2021327          0          0  IR-IO-APIC-fasteoi   Plx9030
 24:          0          0          0          0  DMAR_MSI-edge      dmar0
 25:          0          0          0          0  DMAR_MSI-edge      dmar1
 26:          2          0          0          0  IR-HPET_MSI-edge      hpet2
 27:          0          0          0          0  IR-HPET_MSI-edge      hpet3
 28:          0          0          0          0  IR-HPET_MSI-edge      hpet4
 29:          0          0          0          0  IR-HPET_MSI-edge      hpet5
 34:        235          0          0          0  IR-PCI-MSI-edge      i915
 35:          0          0          0          0  IR-PCI-MSI-edge      ahci
 36:       7971          0          0          0  IR-PCI-MSI-edge      ahci
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      20569      29227      10924      50493   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RES:        324        194        372        353   Rescheduling interrupts
CAL:         47        197        178        211   Function call interrupts
TLB:        142        610        312          1   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:          1          1          1          1   Machine check polls
ERR:          0
MIS:          0
</code>

===== Test Results =====

  * 2016.04.27:  Ran a test for 2.7 hours.  We missed 46 shots out of 49097902 at 5khz (200 useconds) so as far as actually reacting to the interrupt, we are in good shape.
  * Time to make a build 1 data system and get better numbers as we do all our tasks.