Intro: Minimal CentOS 6.5
Kontron BIOS Changes
Base CentOS
CentOS Package Servers
Hard Drive / Raid
CentOS Configure
lvisf code
Clean Boot Operations
Latency Tweaking

Intro: Minimal CentOS 6.5

- Need to update this since CentOS is no longer installed.

The Spectrum XMC1151 digitizer libraries we have require a specific flavour of Linux to operate.

Download the specific version by getting the http://vault.centos.org/6.5/isos/x86_64/CentOS-6.5-x86_64-minimal.iso which is the “minimal” CentOS 6.5.

Actual direct download link for minimal 6.5 CentOS ISO: http://archive.kernel.org/centos-vault/6.5/isos/x86_64/CentOS-6.5-x86_64-minimal.iso (md5sum is 0d9dc37b5dd4befa1c440d2174e88a87 CentOS-6.5-x86_64-minimal.iso)

Kernel from OS install 2016.04.14

Linux lvisf 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

In case your kernel is wrong, you can download the RPM and install directly

Download the correct one (see above) http://vault.centos.org/6.5/centosplus/x86_64/Packages/kernel-2.6.32-431.el6.centos.plus.x86_64.rpm

cd ~/download
wget http://vault.centos.org/6.5/centosplus/x86_64/Packages/kernel-2.6.32-431.el6.centos.plus.x86_64.rpm
wget http://vault.centos.org/6.5/centosplus/x86_64/Packages/kernel-devel-2.6.32-431.el6.centos.plus.x86_64.rpm

Install

sudo rpm -ivh --oldpackage kernel-2.6.32-431.el6.centos.plus.x86_64.rpm
sudo rpm -ivh --oldpackage kernel-devel-2.6.32-431.el6.centos.plus.x86_64.rpm

I had to do this, but not sure if really I needed to just install the top thing?

cd /usr/src/kernels
ln -s 2.6.32-431.el6.centos.plus.x86_64 2.6.32-431.el6.x86_64

Kontron BIOS Changes

To make these BIOS changes, we need to get to the UEFI shell.
1. Reboot the computer
2. Press <DEL> key while booting to get to BIOS menu
3. Boot → Boot Option #1 → UEFI: Built-in Shell
4. Save and Exit (F4)
5. Press <ESC> while booting to get to the Shell> prompt.

Increase the boot delay so that the Spectrum Digitizers have time to initialise.
1. Enter the following on the Shell>
```
kboardconfig PciCfgDelay 5s
```
2. Double check by running without the argument too see what the parameter is set to
```
kboardconfig PciCfgDelay
# will see [5s] highlighted in green (and that is the longest delay available)
```
~~Disable hyperthreading and whatever VT stands for~~ NOTE: The non-RT version works great with hyperthreading (at least so far: — David Lloyd Rabine 2018/04/27 18:13 )
```
# ONLY RT version needs this DO NOT DO THIS for normal operations with NORT build (build02 / v2)
kboardconfig IntelVT disabled
kboardconfig IntelHT disabled
```
Exit the bios
```
exit
```

What is SpeedStep? What is CpuTurbo? Sounds like things we don't want on!

Base CentOS

Using the 500_00746_r100_SDR7000_Linux_SG.pdf document, set up the core data system.

CentOS Package Servers

Adding Online Resources

https://www.getpagespeed.com/server-setup/how-to-fix-yum-after-centos-6-went-eol (NO LUCK!)

This did work in adding the 6.10 repository to our system: (from) https://www.mark-gilbert.co.uk/fixing-yum-repos-on-centos-6-now-its-eol/

curl http://mark-gilbert-co-uk.s3-website-eu-west-1.amazonaws.com/CentOS-Base.repo --output /etc/yum.repos.d/CentOS-Base.repo
curl http://mark-gilbert-co-uk.s3-website-eu-west-1.amazonaws.com/epel.repo --output /etc/yum.repos.d/epel.repo

You can grab the file here: centos-base.repo.gz
Modified the epel to http (instead of https) in here: epel.repo.gz

Use Archived Packages?

Maybe we should grab what we need? I dunno. Maybe we just make an image that we can expand to a 64 GB drive and be happy.

DVD Boot

NOTE: Press the Delete key while booting in order to change the boot to either the USB PLEXTOR drive or the LEXAR CFAST drive

BIOS (delete key after eSATA RAID Marvel BIOS) → Boot → Boot Option #1
Then F4: Save & Exit

Attached the external USB CDROM drive to the computer via the USB3 port (USB stick with UNETBOOTIN did NOT WORK for me?!) (UnetBootin will work with later version of CentOS image… 6.5 is broken)
Insert the Minimal CentOS 6.5 Install Disk
Boot and set the BIOS to boot off the Plextor drive (our USB based DVD drive)
At CentOS 6 Welcome, Click → Next
Language: English (English)
Keyboard: U.S. English
Storage Devices: Basic Storage Devices
NOTE: If another OS detected, select: Fresh Installation (SURE YOU WANT TO DELETE THAT OS!?)
Hostname: lvisf
Timezone: New York
root Password: the usual
Partitioning: Create Custom Layout (bottom option)
1. Delete any partitions on the new target drive (this 32GB cfast is the demo target)
2. Select the Free disk area → Create a new partition
3. Select → Standard Partition (top option)
  1. Mount Point: /boot/efi
  2. File System Type: EFI System Partition
  3. Size (MB): 1024
  4. Additional Size Options: Fixed Size (default, but make sure)
  5. OK
4. Select → Standard Partition (top option)
  1. Mount Point: /
  2. File System Type: ext4
  3. Size (MB): 15360
  4. Additional Size Options: Fixed Size (default, but make sure)
  5. OK
5. Next
Select the drive (for this case, the 60 GB (or 60000 MB drive) and put that to the right, then Write Changes
Wait for install to complete (205 packages in ~4 minutes)
Drive will eject
Remove DVD
Reboot
Load BIOS on reboot and select the cfast drive as the standard boot device

Hard Drive / Raid

OWC 1U Raid

CentOS Configure

Virtual Box

Some notes on using virtual box

Used RedHat 64 Bit as the base machine, worked fine with the CentOS minimal

Local Network

In order to ssh to your host, you can add a rule to port forward into your VM

Network → Advanced → Port Forwarding
Add a rules that maps
1. Name: Forward SSH (whatever you want)
2. Protocol: TCP
3. Host IP: 127.0.1.1
4. Host Port: 2222
5. Guest IP: 10.0.2.15 (or whatever your machine is)
6. Guest Port: 22
Now you can ssh directly from your host pc to the VM (so I can copy and paste from the wiki, yeah!!)
1. Example:
```
ssh -CY root@127.0.1.1 -p2222
```

Screen Scale

Display settings for the host, you can scale the screen directly now, up to 200%. YAY!

lvisf Config

Filesystem

Add the navigation, internal and external drives to the /etc/fstab file

Edit
```
sudo nano /etc/fstab
```

Add these lines

LABEL=LVISF_INTERNAL	/mnt/internal		ext4	defaults,noatime 	0 0
LABEL=LVISF_EXTERNAL	/mnt/external		ext4	defaults,noatime 	0 0
LABEL=LVISF_NAV		/mnt/nav		ext4	defaults,noatime 	0 0
tmpfs                   /mnt/ramdisk            tmpfs   nodev,nosuid,noexec,nodiratime,size=1M   0 0

Make these directories for mount points

sudo mkdir /mnt/internal /mnt/external /mnt/nav /mnt/tmp /mnt/usb /mnt/nfs /mnt/ramdisk

Note: Ram disk information: https://www.jamescoyle.net/how-to/943-create-a-ram-disk-in-linux

Network Temporary Configuration (DHCP)

Setup the system for DHCP for now
1. Log in as root
2. Use vi to edit /etc/sysconfig/network-scripts/ifcfg-eth0 so
```
ONBOOT=yes
```
3. Startup network (will take a moment, DHCP)
```
sudo ifup eth0
```
Logout and now, you should ssh to gs694-lvisdev.gsfc.nasa.gov and then to the new machine (whatever its DHCP IP address is now on the local LAN)

Prerequisites for Compiling Kernel Modules

http://www.tldp.org/LDP/lkmpg/2.6/html/x121.html ← build kernel module link for an example

Install the development tools
```
yum groupinstall 'Development Tools'
```

Get some useful programs

yum install ftp -y
yum install nano -y
yum install wget -y

Grab the ethernet driver while you still have access
1. Current driver is here: https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz
```
mkdir ~/down
cd ~/down
wget https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz
```

Ethernet Driver Install and Configuration

Log in as root on the main console since ethernet will be disabled by default
Install Intel ethernet drivers [Only needed if compiling on Kontron flight hardware]:
1. Download igb-5.3.4.4.tar.gz to a USB drive
2. Current driver is here: https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz
3. Uncompress in the home directory
```
tar xvf igb-5.3.4.4.tar.gz
```
4. Build and install the drvier
```
cd ~/tmp/igb-5.3.4.4/src/
make
sudo make install
```
5. Modify this module file
```
sudo nano /etc/sysconfig/modules/lvisf.modules
```
6. Put the following into this file
```
#!/bin/sh

# start intel network driver
/sbin/modprobe igb
```
7. Change permissions so this script runs at boot time
```
sudo chmod 755 /etc/sysconfig/modules/lvisf.modules
```

NOTE: you can just do this

sudo cp /opt/lvisf/src/lvisF/scripts/ifcfg-eth* /etc/sysconfig/network-scripts/

Configure our network device

Change the eth0 definition

editing the config file

vi /etc/sysconfig/network-scripts/ifcfg-eth0

to the following

DEVICE=eth0
HWADDR=00:80:82:1B:E7:04
TYPE=Ethernet
UUID=98bd88d8-8fe8-4b8e-bca0-a58bd3306717
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth0"
IPADDR=10.4.1.2
NETMASK=255.255.255.0

Add our default gateway

Edit the network configuration
```
vi /etc/sysconfig/network
```

End up looking like this

NETWORKING=yes
HOSTNAME=lvisf
NETWORKING_IPV6=no
IPV6INIT=no
GATEWAY=10.4.1.1  # or 10.4.1.254 if using Dell Laptop Gateway for WiFi still

Ensure our name server can be found

Edit resolv.conf
```
vi /etc/resolv.conf
```

End up with this

search gsfc.nasa.gov
nameserver 128.183.10.134
nameserver 128.183.50.17

Turn off IPV6 (found here: https://www.centos.org/forums/viewtopic.php?t=7989)

sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

Change the eth1 definition

editing the config file

vi /etc/sysconfig/network-scripts/ifcfg-eth1

to the following

DEVICE=eth1
HWADDR=00:80:82:1B:E7:05
TYPE=Ethernet
UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth1"
IPADDR=192.168.0.2
NETMASK=255.255.255.0

Change the eth1:1 definition

editing the config file

vi /etc/sysconfig/network-scripts/ifcfg-eth1:1

to the following

DEVICE=eth1:1
HWADDR=00:80:82:1B:E7:05
TYPE=Ethernet
UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth1:1"
IPADDR=129.100.0.2
NETMASK=255.255.255.0

REMOVE the keyboard and mouse and never use them again (USB interrupts)
Add lvis user to the system (still as root (logged in via SSH for me now))
1. Add the user
```
useradd lvis
```
2. Set password
```
passwd lvis
```
Add lvis user to the list of sudo users
1. Modify the super user file with the following (as logged in as root)
```
visudo
```
2. After the line that has root and ALL and ALL permissions, insert this line
```
lvis    ALL=(ALL)       ALL
```

File System Extras

Add ntfs drive capability
1. Add the epel repository
```
sudo yum install epel-release
```
2. Add ntfs to our system
```
sudo yum install fuse fuse-ntfs-3g
```

Add Exfat drive capability

Add the epel repository and this Nux Dextop repository

sudo yum install epel-release
sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el6/x86_64/nux-dextop-release-0-2.el6.nux.noarch.rpm

Add Exfat to our system
```
sudo yum install exfat-utils fuse-exfat
```
An example mounting an Exfat system is
```
sudo  mount -t exfat /dev/sdc1 /mnt/tmp
```

X11

This will allow you to run X11

# core X11
sudo yum install xorg-x11-xauth xorg-x11-fonts-* xorg-x11-utils xterm -y
# required for tuna
sudo yum install xauth pygtk2-libglade -y

Install tuna tuning package
```
sudo yum install tuna -y
```
Modify /etc/ssh/sshd_config and add this line to forward X11 (or make sure it is not commented out… this was already enabled last I checked)
```
X11Forwarding yes
```

Remove Unused Services

Remove exim (NOTE, this was not on the CentOS Minimal I just installed 2018.01.04 (but leaving for historical))
1. Stop the service
```
sudo service exim stop
```
2. Remove the package from the system
```
sudo rpm -e --nodeps exim
```

Remove auditd

sudo service auditd stop # now
sudo chkconfig auditd off # after reboot

Remove postfix
1. Stop the service
```
sudo service postfix stop
```
2. Remove the package from the system
```
sudo rpm -e --nodeps postfix
```

eSATA Speed Limit

Our eSATA connection to the raid (and any external ATA devices) was occasionaly running into issues when running at high speed (6.0 GBPS). Dan Caplan found online notes that people put a speed limit on the kernel module that regulates the ATA commuincations, and that solved the types of errors we have been seeing.

Modify /etc/grub.conf by adding this to the kernel starting command line
```
libata.force=3.0
```

/etc/grub.conf

#boot=/dev/sda1
device (hd0) HD(1,800,200000,a70d9e49-77c1-4fc8-b5ce-81ede8517477)
default=0
timeout=5
splashimage=(hd0,1)/boot/grub/splash.xpm.gz
serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=5 serial console
hiddenmenu
title CentOS (2.6.32-431.el6.x86_64)
        root (hd0,1)
        kernel /boot/vmlinuz-2.6.32-431.el6.x86_64 ro root=UUID=2804bc4c-c850-4d83-9fec-7de10cb1e2b4 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarc
yrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb console=tty0 console=ttyS0,115200n8 libata.force=3.0
        initrd /boot/initramfs-2.6.32-431.el6.x86_64.img

Kernel Console Redirection

If you want to send the kernel messages out to the serial port for logging.

Modify /etc/grub.conf so the default kernel command line redirects console to the serial port
```
sudo nano /etc/grub.conf
```
1. Change
```
rhgb quiet
```
2. To
```
rhgb console=tty0 console=ttyS0,115200n8
```

Enable Serial Console Login

And, to make it such that you can log onto the system over that console port…

Modify the grub boot line and add the following lines above all the kernel options
1. Edit /etc/grub.conf
```
sudo nano /etc/grub.conf
```
2. Make the top look like this
```
serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=5 serial console
hiddenmenu
```
3. Add these lines to the kernel that you are using (probably already done, see above!)
```
console=tty0 console=ttyS0,115200n8
```
Add ttyS0 to the /etc/securetty file
1. Edit
```
sudo nano /etc/securetty
```
2. Append this to the bottom of the file
```
ttyS0
```
Modify /etc/init/ttyS0.conf
```
sudo nano /etc/init/ttyS0.conf
```

And fill it with the following code

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345] and (
            not-container or
            container CONTAINER=lxc or
            container CONTAINER=lxc-libvirt)

stop on runlevel [!2345]

respawn
exec /sbin/getty -L 115200 ttyS0 vt102
########

Spectrum Config

Still as root logged into the main desktop

Disable the Firewall

Turn off the firewall

sudo service iptables stop
sudo service ip6tables stop

Make this permanent

sudo chkconfig iptables off
sudo chkconfig ip6tables off

Disable SELinux
1. Edit the configuration file
```
sudo vi /etc/selinux/config
```
2. Modify the file
  1. From
```
SELINUX=enforcing
```
  2. To
```
SELINUX=disabled
```
Reboot for all changes to take place

Minimize Kernel Modules

Disable ipv6, this addition is required along with a line in the modprobe file below
1. Modifiy /etc/sysconfig/network
```
sudo vi /etc/sysconfig/network
```
2. And append the following two lines
```
NETWORKING_IPV6=no
IPV6INIT=no
```
3. Add these two lines to sshd_config
```
sudo nano /etc/ssh/sshd_config
# lvisf - ipv4 only
AddressFamily inet
ListenAddress 0.0.0.0
```

Blacklist kernel modules we don't require

Edit a file lvisf-blacklist.conf

sudo vi /etc/modprobe.d/lvisf-blacklist.conf

Add the following lines

# lvisf kernel module blacklist
# version 1.0 - dlr - 20160418

# ipv6
blacklist ipv6

# NMI watchdog
blacklist iTCO_wdt 
blacklist iTCO_vendor_support

blacklist i2c_i801
# blacklist igb
blacklist lpc_ich
blacklist shpchp
blacklist dm_mirror

# intel sound
blacklist soundcore
blacklist snd
blacklist snd_seq
blacklist snd_hda_intel
blacklist snd_hda_codec_hdmi

# USB controller (still loads)
blacklist xhci_hcd


# this is required to REMOVE ipv6 module, I am not sure why
# http://www.cyberciti.biz/faq/redhat-centos-disable-ipv6-networking/
install ipv6 /bin/true

Extras

These two commands will remove some other modules I couldn't figure out how to make permanently go away

# USB driver... if you're using the nousb kernel, this will work
sudo modprobe --remove xhci_hcd
sudo modprobe --remove dm_mirror

Minimal Modules

Module                  Size  Used by
e1000                 170646  0 
sg                     29350  0 
ext4                  374902  3 
jbd2                   93427  1 ext4
mbcache                 8193  1 ext4
sd_mod                 39069  3 
crc_t10dif              1541  1 sd_mod
ahci                   42215  2 
i915                  624511  1 
drm_kms_helper         44321  1 i915
drm                   280012  2 i915,drm_kms_helper
i2c_algo_bit            5935  1 i915
i2c_core               31084  4 i915,drm_kms_helper,drm,i2c_algo_bit
video                  20674  1 i915
output                  2409  1 video
dm_mod                 84209  9

lvisf code

SVN Repository

NOTE: This is obsolete (the repository link), but left here for posterity.

Dave Yancich sent me this note back in May:

Repository URL:  https://zj.yancich.com:8443/svn/NASA/trunk/lvisF
username "lvis"
password: standard lvis password

This is a cmake project.

To rebuild the make files:
In "build" directory type "cmake ../"
Then "make" to build everything.
"make clean" to clean the programs and kernel modules.

Build Document

(Yancich sent this to me in March 2016) Original document is configuration_and_build_procedures_for_lvisf_and_support_libraries.docx

Requirements

Install cmake
```
sudo yum install cmake -y
```
Install htop
```
sudo yum install htop -y
```

Install jed

sudo yum install ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/jed-0.99.18-5.el5.kb.i386.rpm -y

Install nano
```
sudo yum install nano -y
```
Install p7zip and pigz
```
sudo yum install p7zip pigz -y
```
Install PCI utilities
```
sudo yum install pciutils -y
```
Install screen
```
sudo yum install screen -y
```
Install telnet
```
sudo yum install telnet -y
```
Install wget
```
sudo yum install wget -y
```

Install the development package

sudo yum groupinstall "Development Tools"

Install development libraries for md5sum and gzip in real time
```
sudo yum install openssl-devel -y
```
Install the ncurses development library assuming our console gets written using it
```
sudo yum install ncurses-devel -y
```
Install md5deep so we can check file integrity
```
sudo yum install md5deep -y
```

Install man pages

sudo yum install man -y
sudo yum install man-pages -y

Install kernel development source

cd
mkdir download
cd ~/download
wget http://vault.centos.org/6.5/os/x86_64/Packages/kernel-devel-2.6.32-431.el6.x86_64.rpm
sudo yum --nogpgcheck localinstall ~/download/kernel-devel-2.6.32-431.el6.x86_64.rpm -y

Install gdisk for partition manipulation on modern drives
```
sudo yum install gdisk -y
```
Install the TIFF I/O Library for the CIMG library used for the DEM library
```
sudo yum install libtiff-devel -y
```
Install the X11 Development library also used for the DEM library
```
sudo yum install libX11-devel -y
```

Debugging

This looks like a great article on remote debugging: https://www.cnx-software.com/2012/01/31/debugging-embedded-linux-with-gdbserver-and-insight-gdb-gui/

Install gdb and gdbserver
```
sudo yum install gdb gdb-gdbserver
```
Install debug information (found this https://stackoverflow.com/questions/10389988/missing-separate-debuginfos-use-debuginfo-install-glibc-2-12-1-47-el6-2-9-i686)
1. Add these yum utility
```
sudo yum install yum-utils
```
2. Modify this file and enable the repo by setting enable to = 1
```
sudo /etc/yum.repos.d/CentOS-Debuginfo.repo
# and set
# enabled=1
```
3. Install this
```
sudo debuginfo-install glibc
```

3rd Party Libs to Build from Source

GSL (GNU Scientific Library)

Download version 2.5 and follow install instructions. This will build libgsl and libgslcblas need for the DEM library.

Grab the library and uncompress

cd ~/download
wget http://mirror.rit.edu/gnu/gsl/gsl-2.5.tar.gz
cd ~/code
tar -xvf ~/download/gsl-2.5.tar.gz

Compile

cd ~/code/gsl-2.5
./configure
make -j 4
sudo make install

Code Base

Create the repoistory home and check out the source code
1. Create a repository home in /opt
```
sudo mkdir /opt/lvisf
```
2. Set proper permissions
```
sudo chown -R lvis:lvis /opt/lvisf
```
3. [Optional] Uncompress our locally attached mount with snapshot (faster than network initial checkout)
```
cd /
tar jxvf /media/LVIS-GH-2TB_WD209/lvisf_build/lvisf_sourcesnapshot_20160816_svn439.tar.bz2
```
  1. [Either] Check out he raw source code from scratch
```
cd /opt/lvisf
svn co https://svn.yancich.com:8443/svn/NASA/LVIS/trunk/ src
```
  2. [Or] Update it if you used the tar above
```
cd /opt/lvisf/src
svn update
```
4. Enter the user home directory and create symbolic links
```
cd /home/lvis
mkdir src
ln -s /opt/lvisf/src/* /home/lvis/src/
```

Add the following lines to the .bashrc for the lvis user

Edit the file
```
nano /home/lvis/.bash_profile
```

Add these lines

# For SVN commit comment editor
export SVN_EDITOR=nano

# For the Innovative digitizer
export WD_BASEDIR=/home/lvis/src/Innovative/WinDriver

# For the SyncClock timer card
export PLX_SDK_DIR=/home/lvis/src/PLX/PlxSdk

# For the spectrum digitizer
export SSP_INSTALL_ROOT=/home/lvis/src/spectrum

Load this environment (re-run this script)
```
source ~/.bash_profile
```

Edit the file

sudo nano /etc/ld.so.conf.d/lvisf.conf

Add these lines

/home/lvis/src/Innovative/winCloudDrvr/winclouddrvr_installation/lib
/home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/libs
/home/lvis/src/spectrum/shared/linux/pc64/lib

Reload the library path
```
sudo ldconfig
```

SyncClock32 PLX Drivers

Build all drivers
1. Go to the source code home directory
```
cd /home/lvis/src/PLX/PlxSdk/Driver
```
2. Execute the build script
```
./buildalldrivers
```
Build the samples
1. Go to the source directory
```
cd /home/lvis/src/PLX/PlxSdk
```
2. Make the programs
```
make clean
make
```

Load the driver

To the binary directory
```
cd /home/lvis/src/PLX/PlxSdk/Bin
```

Execute the script for the 9030 card (our syncclock32)

sudo ./Plx_load 9030

# to just see what boards are supported, run the script with no argument
./Plx_load

MIC 3753 DIO Drivers

NOTE These drivers NEED to be built as root, you cannot sudo to build these drivers

Become root all the time
```
sudo bash
```

Go to the code base directory

cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/driver_base/src/lnx_ko

Build the drivers
```
make
```

Go into our specific card directory

cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/pci1753_mic3753_pcm3753i/src/lnx_ko

Build the drivers

# to disable IRQs in this driver, just use this code where the IRQ calls are commented out
cp init.c.no_irq init.c
make

Make symbolic links to these drivers in the kernel module

ln -s /opt/lvisf/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/*.ko /lib/modules/2.6.32-431.el6.x86_64/kernel/drivers/misc/

Load the drivers (this crashes when i load the bio1753 (IRQ!!! issue)

cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin
insmod biokernbase.ko
insmod bio1753.ko

Exit the sudo bash
```
exit
```

Innovative Digitizer Drivers

Described in LinuxNotes.pdf (might be this /usr/Innovative/MalibuRed-1.3/KerPlug/LinuxNotes.pdf)

Link our Innovative directory to /usr/Innovative

sudo ln -s /opt/lvisf/src/Innovative /usr/Innovative

Untar the latest driver

tar -zxvf /opt/lvisf/src/tars/64WinDriver-12.3-0.X86_64.tgz.tar.gz --directory=/usr/Innovative/

Build 12.3 Windriver

Link 12.3 as our Windriver

cd /usr/Innovative
rm /usr/Innovative/WinDriver
ln -s WinDriver-12.3 WinDriver

Go here

cd /usr/Innovative/WinDriver/redist

Configure

./configure
make
sudo make install # they say do not do this, we live on the edge

Kerplug

Build (builds in the LINUX directory for the RT kernel)
```
cd /usr/Innovative/KerPlug/linux
./configure
make
```
OK, this SHOULD fail.

Edit the Makefile with

nano Makefile

Add seven dot dot slashes to the line:

Convert this

MOD_CSRCS = $(WD_BASEDIR)/redist/linux_wrappers.c

to this

MOD_CSRCS = ../../../../../../../$(WD_BASEDIR)/redist/linux_wrappers.c

Change 1180 to 1230

Run make again
```
make
```

Load the driver

sudo insmod LINUX.2.6.32-431.el6.x86_64.x86_64/kp_malibu_module.ko

Need to link the generic library name with the compiled library
```
cd /usr/Innovative/KerPlug
sudo ./quicklink wdapi 1230
```

Final license issue with changed authorization mode
1. Install qt development
```
sudo yum install qt4 -y
sudo yum install qt4-devel -y
```
2. Rebuild (and this will also copy the files where they need to go) (NOTE: I believe this is unnecessary, but it does not hurt anything to do this)
```
cd /home/lvis/src/Innovative/Malibu/Qt/Os_Mb
qmake-qt4
make
```

Historical Log

All of this below is just for historical records. Do NOT need to do any more Innovative.

Raw code from /etc/rc.local from the primary machine to load this

sudo bash
cd /usr/Innovative/WinDriver-12.3/redist && ./wdreg LINUX.2.6.32-431.el6.x86_64.x86_64/windrvr1230 auto
#/usr/Innovative/winCloudDrvr/winclouddrvr_installation/redist/wdreg winCloudDrvr auto
#chmod 666 /dev/winCloudDrvr
insmod /usr/Innovative/KerPlug/linux/LINUX.2.6.32-431.el6.x86_64.x86_64/kp_malibu_module.ko
exit

Field Addendum

In the field we noticed that the driver was saying unregistered, and would only run for 60 minutes (sometimes it ran longer, but this was an issue). Dave found these instructions from an email:

FIRST!

Make sure this folder exists:

/usr/Innovative/Lib/Dll (and it is linked to the Innovative folder in src)

Make sure LicenseLoader.so exists in that folder.

(I have attached that file and a tar of the 3 others needed to rebuild)

Hopefully its just that the .so did not make it to that system disk. 

If is does exist:

Copy all the files in the tar to:

/home/lvis/src/Innovative/Malibu

Cd to /home/lvis/src/Innovative/Malibu/Qt/Os_Mb

“qmake-qt4”
“make”
 
This will make the Os library and copy it to the correct folder.
 
“That is all I got” 😊
 
Dave

Spectrum Digitizer Drivers

Create the symbolic link for /spectrum

sudo ln -s /opt/lvisf/src/spectrum /spectrum

Enter the code directory

cd /home/lvis/src/spectrum/shared/linux/pc64/bin

Copy the drivers for our kernel into the binary directory

cp -a /home/lvis/src/spectrum/shared/linux/pc64/bin/`uname -r`/*.ko /home/lvis/src/spectrum/shared/linux/pc64/bin/

Execute the setup script

chmod 755 *.sh   # not sure why these were not already executable
sudo bash ./setup_xmc1151_dvt.sh

Select the proper option: 1)Install

Auto-load MIC-3753 Driver

symlink the drivers into the kernel directory

sudo ln -s /opt/lvisf/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/*.ko /lib/modules/2.6.32-431.el6.x86_64/kernel/drivers/misc/

add to our startup script /etc/sysconfig/modules/lvisf.modules

modify the file

sudo nano /etc/sysconfig/modules/lvisf.modules
sudo chmod 755 /etc/sysconfig/modules/lvisf.modules

add the following

# start the MIC-3753 drivers
/sbin/modprobe biokernbase
/sbin/modprobe bio1753

Auto-load Plx9030 Driver

NOTE: (in lieu of step 2 below) I had this not work when I manually edited it the first time. A copy of the file is in the repository, you could just copy that file in:

sudo cp /opt/lvisf/src/lvisF/scripts/lvisf.modules /etc/sysconfig/modules/lvisf.modules

symlink the driver into the kernel so it will come up on boot

sudo ln -s /opt/lvisf/src/PLX/PlxSdk/Driver/Source.Plx9000/Output/Plx9030.ko /lib/modules/2.6.32-431.el6.x86_64/kernel/drivers/misc/

add to our startup script /etc/sysconfig/modules/lvisf.modules

modify the file

sudo nano /etc/sysconfig/modules/lvisf.modules

add the following

# start the Plx9030 syncclock32 driver
/sbin/modprobe Plx9030

symbolically link in our rc.local into /etc/rc.local

remove the original

# remove the symlink
sudo rm /etc/rc.local
# move the original to something else
sudo mv /etc/rc.d/rc.local /etc/rc.d/rc.local_orig

link in the lvisf version

sudo ln -s /opt/lvisf/src/lvisF/scripts/rc.local /etc/rc.d/rc.local
sudo ln -s /etc/rc.d/rc.local /etc/rc.local

verify that the system will not try to launch on boot (unless that is what you want!)
```
# look for this near the top
AUTOSTART_LVISF="false"
```

Installing Boost Library

The boost C++ library is required for the data system. The tarball of the 1.6.2 version is in the repository. Or download the latest from http://www.boost.org/users/download/. Here is the latest as of 2018.01.04 https://dl.bintray.com/boostorg/release/1.66.0/source/boost_1_66_0.tar.bz2

Uncompress the tar into /opt

cd /opt
sudo tar -jxvf /opt/lvisf/src/tars/boost_1_62_0.tar.bz2

Symbolically link to a generic boost directory
```
sudo ln -s /opt/boost_1_62_0 /opt/boost
```

Chown it all to lvis user

sudo chown -R lvis:lvis /opt/boost_1_62_0

Set the environment variable for the BOOST_ROOT variable
1. Edit the file /home/lvis/.bash_profile
```
nano /home/lvis/.bash_profile 
```
2. Add the line
```
# Add the boost root directory
export BOOST_ROOT=/opt/boost
```
3. Run our profile to load this variable
```
source /home/lvis/.bash_profile
```

Compile the boost libraries

cd /opt/boost
./bootstrap.sh
./b2 -j $(nproc --all)  # should figure out number of cpus automagically

Installing Ethernet to Serial MOXA Drivers

Follow the instructions here:

Moxa NPort 5250A Series

Installing USB Temperature Library

Follow the instructions below to compile the USB temperature monitoring device libraries required.

Measurement Computing USB-TEMP-AI

LVIS User Binary Path

Create this directory
```
mkdir ~/bin
```

Symbolically link in programs into this path from the scripts directory

ln -s /opt/lvisf/src/lvisF/scripts/applanix* ~/bin/.
ln -s /opt/lvisf/src/lvisF/scripts/build* ~/bin/.
ln -s /opt/lvisf/src/lvisF/scripts/javad* ~/bin/.
ln -s /opt/lvisf/src/lvisF/scripts/lvis* ~/bin/.
ln -s /opt/lvisf/src/lvisF/scripts/query* ~/bin/.
ln -s /opt/lvisf/src/lvisF/scripts/run* ~/bin/.

Clean Boot Operations

Load Drivers

Load the MIC-3753 DIO Drivers

sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/biokernbase.ko
sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/bio1753.ko

Load the SyncClock32 Timer Drivers

cd /home/lvis/src/PLX/PlxSdk/Bin
sudo ./Plx_load 9030

Compile Code

Enter the build directory for lvisf
```
cd /home/lvis/src/lvisF/build
```
Use cmake
```
cmake ../
```
Build code
```
make
```

Run Tests

User Space Latency Test

Enter the testing program

cd /home/lvis/src/lvisF/build/bin/tests/syncclock

Run the latency program
```
sudo ./galvMove
```

Latency Tweaking

Set CPU affinity to CPU 3 with something like, made it much more consistent since I guess most things go for CPU 0 by default
```
CPU_SET(3, &cpus);
```
Change the scheduler policy, this information is displayed in the syncclock test program
```
sudo chrt --fifo 99 ./galvMove
```
Isolate a CPU, and then have the code attach itself to that cpu
- In /etc/grub.conf add this command to the kernel command line, so all user space code will avoid CPU 2 and 3 in this example, leaving 0 and 1 for the OS
```
isolcpus=2,3
```
- Change the program such that
```
CPU_SET(3, &cpus);
```
- Run the program with chrt or not
```
sudo chrt --fifo 99 ./galvMove
```
Set the timer card interrupt affinity to a single CPU (not the prime)
```
sudo bash -c 'echo 2 > /proc/irq/19/smp_affinity'
```
- Using 2 here put the interrupt 19 servicing on CPU 1 exclusively
More low latency items to add to the kernel command line for /etc/grub.conf
```
rhgb nousb quiet isolcpus=2,3 nosoftlockup mce=ignore_ce audit=0
```
- nosoftlockup disables backtrace functionality
- mce=ignore_ce ignores corrected errors
- audit=0 disables kernel components of the audit subsystem

Testing

Testing delays

Install perf
```
sudo yum install perf
```

Run this command to record some latency data

cd /opt/lvisf/src/lvisF/build/bin/tests/syncclock
sudo perf sched record -o /dev/shm/perf.data chrt --fifo 99 ./galvMove

Run the following to read the data

sudo perf sched -i /dev/shm/perf.data latency > /tmp/perf.latency.report

Show Interrupt Status

This command shows interrupts for each core / CPU

cat /proc/interrupts
# notice that CPU1 has all the timer card interrupts (PLx9030)
[lvis@lvisf syncclock]$ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       
  0:        125          0          0          0  IR-IO-APIC-edge      timer
  8:          1          0          0          0  IR-IO-APIC-edge      rtc0
  9:          0          0          0          0  IR-IO-APIC-fasteoi   acpi
 18:       1968          0          0          0  IR-IO-APIC-fasteoi   eth2
 19:          0    2021327          0          0  IR-IO-APIC-fasteoi   Plx9030
 24:          0          0          0          0  DMAR_MSI-edge      dmar0
 25:          0          0          0          0  DMAR_MSI-edge      dmar1
 26:          2          0          0          0  IR-HPET_MSI-edge      hpet2
 27:          0          0          0          0  IR-HPET_MSI-edge      hpet3
 28:          0          0          0          0  IR-HPET_MSI-edge      hpet4
 29:          0          0          0          0  IR-HPET_MSI-edge      hpet5
 34:        235          0          0          0  IR-PCI-MSI-edge      i915
 35:          0          0          0          0  IR-PCI-MSI-edge      ahci
 36:       7971          0          0          0  IR-PCI-MSI-edge      ahci
NMI:          0          0          0          0   Non-maskable interrupts
LOC:      20569      29227      10924      50493   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RES:        324        194        372        353   Rescheduling interrupts
CAL:         47        197        178        211   Function call interrupts
TLB:        142        610        312          1   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:          1          1          1          1   Machine check polls
ERR:          0
MIS:          0

Test Results

2016.04.27: Ran a test for 2.7 hours. We missed 46 shots out of 49097902 at 5khz (200 useconds) so as far as actually reacting to the interrupt, we are in good shape.
Time to make a build 1 data system and get better numbers as we do all our tasks.

Table of Contents

Intro: Minimal CentOS 6.5

Kontron BIOS Changes

Base CentOS

CentOS Package Servers

Adding Online Resources

Use Archived Packages?

DVD Boot

Hard Drive / Raid

CentOS Configure

Virtual Box

Local Network

Screen Scale

lvisf Config

Filesystem

Network Temporary Configuration (DHCP)

Prerequisites for Compiling Kernel Modules

Ethernet Driver Install and Configuration

File System Extras

X11

Remove Unused Services

eSATA Speed Limit

/etc/grub.conf

Kernel Console Redirection

Enable Serial Console Login

Spectrum Config

Minimize Kernel Modules

Extras

Minimal Modules

lvisf code

SVN Repository

Build Document

Requirements

Debugging

3rd Party Libs to Build from Source

GSL (GNU Scientific Library)

Code Base

SyncClock32 PLX Drivers

MIC 3753 DIO Drivers

Innovative Digitizer Drivers

Historical Log

Field Addendum

Spectrum Digitizer Drivers

Auto-load MIC-3753 Driver

Auto-load Plx9030 Driver

Installing Boost Library

Installing Ethernet to Serial MOXA Drivers

Installing USB Temperature Library

LVIS User Binary Path

Clean Boot Operations

Load Drivers

Compile Code

Run Tests

User Space Latency Test

Latency Tweaking

Testing

Show Interrupt Status

Test Results