The Spectrum XMC1151 digitizer libraries we have require a specific flavour of Linux to operate.

Kernel from OS install 2016.04.14

Linux lvisf 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Using the 500_00746_r100_SDR7000_Linux_SG.pdf document, set up the core data system.

1. Attached the external USB CDROM drive to the computer via the USB3 port
2. Insert the Minimal CentOS 6.5 Install Disk
3. Boot and set the BIOS to boot off the Plextor drive (our USB based DVD drive)
4. At CentOS 6 Welcome, Click → Next
5. Language: English (English)
6. Keyboard: U.S. English
7. Storage Devices: Basic Storage Devices
8. Hostname: lvisf
9. Timezone: New York
10. root Password: the usual
11. Partitioning: Use All Space
12. Select the drive (for this case, the 60 GB (or 60000 MB drive) and put that to the right, then Write Changes
13. Wait for install to complete (280 packages in ~4 minutes)
14. Drive will eject
15. Remove DVD
16. Reboot

1. Welcome → Forward
2. License: Agree
3. Create User: lvis
4. Date and Time: accept defaults, do not sync to time server
5. Kdump: NO, do NOT enable kdump
6. Reboot

We will attempt to use a 2.6.32 kernel to satisfy the requirements of the Spectrum Signal XMC-1151 digitizer.

1. Log in as root on the main console since ethernet will be disabled by default
2. Install Intel ethernet drivers:
   1. Download igb-5.3.4.4.tar.gz to a USB drive
   2. Uncompress in the home directory

      tar xvf igb-5.3.4.4.tar.gz

   3. Build and install the drvier

      cd ~/tmp/igb-5.3.4.4/src/
      make
      sudo make install

   4. Modify this module file

      sudo nano /etc/sysconfig/modules/lvisf.modules

   5. Put the following into this file

      #!/bin/sh
      
      # start intel network driver
      /sbin/modprobe igb

   6. Change permissions so this script runs at boot time

      sudo chmod 755 /etc/sysconfig/modules/lvisf.modules

3. Configure our network device
   1. Change the eth0 definition
      1. editing the config file

         vi /etc/sysconfig/network-scripts/ifcfg-eth0

      2. to the following

         DEVICE=eth0
         HWADDR=00:80:82:1B:E7:04
         TYPE=Ethernet
         UUID=98bd88d8-8fe8-4b8e-bca0-a58bd3306717
         ONBOOT=yes
         NM_CONTROLLED=yes
         BOOTPROTO=static
         NAME="System eth0"
         IPADDR=10.4.1.2
         NETMASK=255.255.255.0

   2. Add our default gateway
      1. Edit the network configuration

         vi /etc/sysconfig/network

      2. End up looking like this

         NETWORKING=yes
         HOSTNAME=lvisf
         NETWORKING_IPV6=no
         IPV6INIT=no
         GATEWAY=10.4.1.1

   3. Ensure our name server can be found
      1. Edit resolv.conf

         vi /etc/resolv.conf

      2. End up with this

         search gsfc.nasa.gov
         nameserver 128.183.10.134
         nameserver 128.183.50.17

4. REMOVE the keyboard and mouse and never use them again (USB interrupts)
5. Add lvis user to the system (still as root (logged in via SSH for me now))
   1. Add the user

      useradd lvis

   2. Set password

      passwd lvis

6. Add lvis user to the list of sudo users
   1. Modify the super user file with the following (as logged in as root)

      visudo

   2. After the line that has root and ALL and ALL permissions, insert this line

      lvis    ALL=(ALL)       ALL

7. Add ntfs drive capability
   1. Add the epel repository

      sudo yum install epel-release

   2. Add ntfs to our system

      sudo yum install fuse fuse-ntfs-3g

Remove Unused Services

1. Remove exim
   1. Stop the service

      sudo service exim stop

   2. Remove the package from the system

      sudo rpm -e --nodeps exim

2. Remove postfix
   1. Stop the service

      sudo service postfix stop

   2. Remove the package from the system

      sudo rpm -e --nodeps postfix

Still as root logged into the main desktop

1. Disable the Firewall
   1. Turn off the firewall

      sudo service iptables stop
      sudo service ip6tables stop

   2. Make this permanent

      sudo chkconfig iptables off
      sudo chkconfig ip6tables off

2. Disable SELinux
   1. Edit the configuration file

      sudo vi /etc/selinux/config

   2. Modify the file
      1. From

         SELINUX=enforcing

      2. To

         SELINUX=disabled

3. Reboot for all changes to take place

1. Disable ipv6, this addition is required along with a line in the modprobe file below
   1. Modifiy /etc/sysconfig/network

      sudo vi /etc/sysconfig/network

   2. And append the following two lines

      NETWORKING_IPV6=no
      IPV6INIT=no

2. Blacklist kernel modules we don't require
   1. Edit a file lvisf-blacklist.conf

      sudo vi /etc/modprobe.d/lvisf-blacklist.conf

   2. Add the following lines

      # lvisf kernel module blacklist
      # version 1.0 - dlr - 20160418
      
      # ipv6
      blacklist ipv6
      
      # NMI watchdog
      blacklist iTCO_wdt 
      blacklist iTCO_vendor_support
      
      blacklist i2c_i801
      blacklist igb
      blacklist lpc_ich
      blacklist shpchp
      blacklist dm_mirror
      
      # intel sound
      blacklist soundcore
      blacklist snd
      blacklist snd_seq
      blacklist snd_hda_intel
      blacklist snd_hda_codec_hdmi
      
      # USB controller (still loads)
      blacklist xhci_hcd
      
      
      # this is required to REMOVE ipv6 module, I am not sure why
      # http://www.cyberciti.biz/faq/redhat-centos-disable-ipv6-networking/
      install ipv6 /bin/true

• These two commands will remove some other modules I couldn't figure out how to make permanently go away

  # USB driver... if you're using the nousb kernel, this will work
  sudo modprobe --remove xhci_hcd
  sudo modprobe --remove dm_mirror

Module                  Size  Used by
e1000                 170646  0 
sg                     29350  0 
ext4                  374902  3 
jbd2                   93427  1 ext4
mbcache                 8193  1 ext4
sd_mod                 39069  3 
crc_t10dif              1541  1 sd_mod
ahci                   42215  2 
i915                  624511  1 
drm_kms_helper         44321  1 i915
drm                   280012  2 i915,drm_kms_helper
i2c_algo_bit            5935  1 i915
i2c_core               31084  4 i915,drm_kms_helper,drm,i2c_algo_bit
video                  20674  1 i915
output                  2409  1 video
dm_mod                 84209  9 

• Dave Yancich sent me this note back in May:

  Repository URL:  https://zj.yancich.com:8443/svn/NASA/trunk/lvisF
  username "lvis"
  password: standard lvis password
  
  This is a cmake project.
  
  To rebuild the make files:
  In "build" directory type "cmake ../"
  Then "make" to build everything.
  "make clean" to clean the programs and kernel modules.

(Yancich sent this to me in March 2016) Original document is configuration_and_build_procedures_for_lvisf_and_support_libraries.docx

1. Install cmake

   sudo yum install cmake

2. Install htop

   sudo yum install htop

3. Install jed

   sudo yum install ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/jed-0.99.18-5.el5.kb.i386.rpm

4. Install nano

   sudo yum install nano

5. Install p7zip

   sudo yum install p7zip

6. Install PCI utilities

   sudo yum install pciutils

7. Install telnet

   sudo yum install telnet

8. Install wget

   sudo yum install wget

9. Install the development package

   sudo yum groupinstall "Development Tools"

10. Install kernel development source

    cd
    mkdir download
    cd ~/download
    wget http://vault.centos.org/6.5/os/x86_64/Packages/kernel-devel-2.6.32-431.el6.x86_64.rpm
    sudo yum --nogpgcheck localinstall ~/download/kernel-devel-2.6.32-431.el6.x86_64.rpm

1. Create the repoistory home and check out the source code
   1. Create a repository home in /opt

      sudo mkdir /opt/lvisf

   2. Set proper permissions

      sudo chown -R lvis:lvis /opt/lvisf

   3. Check out the raw source code

      cd /opt/lvisf
      svn co https://svn.yancich.com:8443/svn/NASA/LVIS/trunk/ src

   4. Enter the user home directory and create symbolic links

      cd /home/lvis
      mkdir src
      ln -s /opt/lvisf/src/* /home/lvis/src/

2. Add the following lines to the .bashrc for the lvis user
   1. Edit the file

      nano /home/lvis/.bash_profile

   2. Add these lines

      # For SVN commit comment editor
      export SVN_EDITOR=nano
      
      # For the Innovative digitizer
      export WD_BASEDIR=/home/lvis/src/Innovative/WinDriver
      
      # For the SyncClock timer card
      export PLX_SDK_DIR=/home/lvis/src/PLX/PlxSdk
      
      # For the spectrum digitizer
      export SSP_INSTALL_ROOT=/home/lvis/src/spectrum

3. Edit the file

   sudo nano /etc/ld.so.conf.d/lvisf.conf

   1. Add these lines

      /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib
      /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/libs
      /home/lvis/src/spectrum/shared/linux/pc64/lib

   2. Reload the library path

      sudo ldconfig

1. Build all drivers
   1. Go to the source code home directory

      cd /home/lvis/src/PLX/PlxSdk/Driver

   2. Execute the build script

      ./buildalldrivers

2. Build the samples
   1. Go to the source directory

      cd /home/lvis/src/PLX/PlxSdk

   2. Make the programs

      make

3. Load the driver
   1. To the binary directory

      cd /home/lvis/src/PLX/PlxSdk/Bin

   2. Execute the script for the 9030 card (our syncclock32)

      sudo ./Plx_load 9030
      
      # to just see what boards are supported, run the script with no argument
      ./Plx_load

NOTE These drivers NEED to be built as root, you cannot sudo to build these drivers

1. Become root all the time

   sudo bash

2. Go to the code base directory

   cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/driver_base/src/lnx_ko

3. Build the drivers

   make

4. Go into our specific card directory

   cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/pci1753_mic3753_pcm3753i/src/lnx_ko

5. Build the drivers

   make

6. Load the drivers

   cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin
   insmod biokernbase.ko
   insmod bio1753.ko

1. Make a directory for the Innovative code

   sudo mkdir /usr/Innovative

2. Copy base Innovative support software into /usr

   sudo cp /home/lvis/src/INNO-RAW/* /usr/Innovative/

3. Fix the permssions so we can execute this script

   sudo chmod 755 /usr/Innovative/Setup.sh

4. Create this link to the kernel source so the Innovative driver modules can build

   sudo ln -s /usr/src/kernels/2.6.32-431.el6.x86_64 /usr/src/linux

5. Move into and run the setup script

   cd /usr/Innovative
   sudo ./Setup.sh

1. Build the wdapi support library
   1. Enter the code directory

      cd /home/lvis/src/Innovative/WinDriver/src/wdapi

   2. Compile the library

      make -C LINUX

   3. Create this symbolic link, but the SVN checkout already has it

      ln -s /home/lvis/src/Innovative/WinDriver/src/wdapi/LINUX/libwdapi.so /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib/libwdapi1180.so

2. Build the winCloudDrvr
   1. Enter the source directory

      cd /home/lvis/src/Innovative/winCloudDrvr

   2. Compile

      make -C linux

3. Build our custom driver
   1. Enter the directory

      cd /home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist

   2. Configure and compile

      ./configure --disable-usb-support
      make

   3. Install

      sudo make install

To load the driver:

1. insmod our custom driver

   sudo /opt/lvisf/src/Innovative/winCloudDrvr/winclouddrvr_installation/redist/wdreg winCloudDrvr auto

1. Create the symbolic link for /spectrum

   sudo ln -s /opt/lvisf/src/spectrum /spectrum

2. Enter the code directory

   cd /home/lvis/src/spectrum/shared/linux/pc64/bin

3. Execute the setup script

   sudo bash ./setup_xmc1151_dvt.sh

4. Select the proper option: 1)Install for instance

1. Load the MIC-3753 DIO Drivers

   sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/biokernbase.ko
   sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/bio1753.ko

2. Load the SyncClock32 Timer Drivers

   cd /home/lvis/src/PLX/PlxSdk/Bin
   sudo ./Plx_load 9030

1. Enter the build directory for lvisf

   cd /home/lvis/src/lvisF/build

2. Use cmake

   cmake ../

3. Build code

   make

1. Enter the testing program

   cd /home/lvis/src/lvisF/build/bin/tests/syncclock

2. Run the latency program

   sudo ./galvMove

• Set CPU affinity to CPU 3 with something like, made it much more consistent since I guess most things go for CPU 0 by default

  CPU_SET(3, &cpus);

• Change the scheduler policy, this information is displayed in the syncclock test program

  sudo chrt --fifo 99 ./galvMove

• Isolate a CPU, and then have the code attach itself to that cpu
  • In /etc/grub.conf add this command to the kernel command line, so all user space code will avoid CPU 2 and 3 in this example, leaving 0 and 1 for the OS

    isolcpus=2,3

  • Change the program such that

    CPU_SET(3, &cpus);

  • Run the program with chrt or not

    sudo chrt --fifo 99 ./galvMove

• Set the timer card interrupt affinity to a single CPU (not the prime)

  sudo bash -c 'echo 2 > /proc/irq/19/smp_affinity'

  • Using 2 here put the interrupt 19 servicing on CPU 1 exclusively
• More low latency items to add to the kernel command line for /etc/grub.conf

  rhgb nousb quiet isolcpus=2,3 nosoftlockup mce=ignore_ce audit=0

  • nosoftlockup disables backtrace functionality
  • mce=ignore_ce ignores corrected errors
  • audit=0 disables kernel components of the audit subsystem

• Testing delays
  • Install perf

    sudo yum install perf

  • Run this command to record some latency data

    cd /opt/lvisf/src/lvisF/build/bin/tests/syncclock
    sudo perf sched record -o /dev/shm/perf.data chrt --fifo 99 ./galvMove

  • Run the following to read the data

    sudo perf sched -i /dev/shm/perf.data latency > /tmp/perf.latency.report

• This command shows interrupts for each core / CPU

  cat /proc/interrupts
  # notice that CPU1 has all the timer card interrupts (PLx9030)
  [lvis@lvisf syncclock]$ cat /proc/interrupts
             CPU0       CPU1       CPU2       CPU3       
    0:        125          0          0          0  IR-IO-APIC-edge      timer
    8:          1          0          0          0  IR-IO-APIC-edge      rtc0
    9:          0          0          0          0  IR-IO-APIC-fasteoi   acpi
   18:       1968          0          0          0  IR-IO-APIC-fasteoi   eth2
   19:          0    2021327          0          0  IR-IO-APIC-fasteoi   Plx9030
   24:          0          0          0          0  DMAR_MSI-edge      dmar0
   25:          0          0          0          0  DMAR_MSI-edge      dmar1
   26:          2          0          0          0  IR-HPET_MSI-edge      hpet2
   27:          0          0          0          0  IR-HPET_MSI-edge      hpet3
   28:          0          0          0          0  IR-HPET_MSI-edge      hpet4
   29:          0          0          0          0  IR-HPET_MSI-edge      hpet5
   34:        235          0          0          0  IR-PCI-MSI-edge      i915
   35:          0          0          0          0  IR-PCI-MSI-edge      ahci
   36:       7971          0          0          0  IR-PCI-MSI-edge      ahci
  NMI:          0          0          0          0   Non-maskable interrupts
  LOC:      20569      29227      10924      50493   Local timer interrupts
  SPU:          0          0          0          0   Spurious interrupts
  PMI:          0          0          0          0   Performance monitoring interrupts
  IWI:          0          0          0          0   IRQ work interrupts
  RES:        324        194        372        353   Rescheduling interrupts
  CAL:         47        197        178        211   Function call interrupts
  TLB:        142        610        312          1   TLB shootdowns
  TRM:          0          0          0          0   Thermal event interrupts
  THR:          0          0          0          0   Threshold APIC interrupts
  MCE:          0          0          0          0   Machine check exceptions
  MCP:          1          1          1          1   Machine check polls
  ERR:          0
  MIS:          0

• 2016.04.27: Ran a test for 2.7 hours. We missed 46 shots out of 49097902 at 5khz (200 useconds) so as far as actually reacting to the interrupt, we are in good shape.
• Time to make a build 1 data system and get better numbers as we do all our tasks.