This is an old revision of the document!

RT Kernel on CentOS

Tried compiling my own PREEMPT kernel, but it had many issues and ran worse than the generic kernel. I only enabled PREEMPT and did not change any other options. I assume the RT kernel they are packaging is tweaked for better real time. I noticed the CPU wasn't scaling down with the CERN RT kernel.

NOTE: To change eth0 if you move drives, simply change HWADDRESS to match ifconfig of eth0 and leave the generated UUID as the OS created it. This UUID is random and means nothing, but it might appear in other scripts and have to stay the same.

Links

RT Kernel

http://linux.web.cern.ch/linux/scientific6/
http://glitesoft.cern.ch/ ← RT kernel repository
https://rt.wiki.kernel.org/index.php/Main_Page ← rt main page
https://rt.wiki.kernel.org/index.php/Cyclictest ← cyclic testing software
- To resolve compile dependency
  - Install the libnuma-dev package (on Debian-based systems)
  - Install the numactl-devel package (on RHEL/CentOS systems)
https://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO ← PREEMPT HowTo
https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application ← PREEMPT How to build an application

Latency Testing

https://www.osadl.org/fileadmin/dam/rtlws/12/Brown.pdf (local copy Brown.pdf) (Page 9 is VERY interesting!)
https://lwn.net/Articles/139784/
https://lwn.net/Articles/139403/ ← I would love to get these response times!

IPC

http://www.kohala.com/start/unpv22e/unpv22e.chap12.pdf ← excellent shared memory chapter (IPC)
https://courses.engr.illinois.edu/cs241/sp2012/lectures/29-IPC.pdf ← another good IPC presentation
http://menehune.opt.wfu.edu/Kokua/More_SGI/007-2478-008/sgi_html/ch03.html ← more good information
http://www.cse.psu.edu/~deh25/cmpsc473/notes/OSC/Processes/shm.html ← REALLY good simple C example of producer / consumer code

Cyclic Testing

Ran this in a second login to create disk load:

time sudo sh -c "dd if=/dev/zero of=/mnt/data/ddfile bs=8k count=2000000 && sync" # writes 16GB

SUCCESS! (rt kernel)

[lvis@lvisf-centos65 rt-tests]$ uname -a
Linux lvisf-centos65 2.6.33.9-rt31.79.el6rt.x86_64 #1 SMP PREEMPT RT Wed Jan 11 13:54:22 CET 2012 x86_64 x86_64 x86_64 GNU/Linux

[lvis@lvisf-centos65 rt-tests]$ sudo ./cyclictest -a -t -n -p99 -f -b100
[sudo] password for lvis: 
# /dev/cpu_dma_latency set to 0us
INFO: debugfs mountpoint: /sys/kernel/debug/tracing/
policy: fifo: loadavg: 1.81 1.79 1.12 2/220 11904           

T: 0 (11851) P:99 I:1000 C: 231860 Min:     20 Act:   27 Avg:   26 Max:      85
T: 1 (11852) P:99 I:1500 C: 154572 Min:     22 Act:   28 Avg:   28 Max:      72
T: 2 (11853) P:99 I:2000 C: 115929 Min:     22 Act:   26 Avg:   27 Max:      65
T: 3 (11854) P:99 I:2500 C:  92743 Min:     22 Act:   28 Avg:   28 Max:      58

FAIL! (Spectrum Default Kernel)

^C[lvis@lvisf rt-tests-1.0]$ uname -a
Linux lvisf 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux


[lvis@lvisf rt-tests-1.0]$ sudo ./cyclictest -a -t -n -p99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.28 0.35 0.15 1/133 2044          

T: 0 ( 1897) P:99 I:1000 C: 288733 Min:      1 Act:    3 Avg:    7 Max:   66553
T: 1 ( 1898) P:99 I:1500 C: 192622 Min:      1 Act:    3 Avg:    2 Max:     543
T: 2 ( 1899) P:99 I:2000 C: 144466 Min:      2 Act:    3 Avg:    3 Max:       7
T: 3 ( 1900) P:99 I:2500 C: 115573 Min:      2 Act:    3 Avg:    3 Max:       8
^C[lvis@lvisf rt-tests-1.0]$ sudo ./cyclictest -a -t -n -p99
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.08 0.35 0.20 1/132 2083          

T: 0 ( 2061) P:99 I:1000 C: 226186 Min:      1 Act:    2 Avg:    7 Max:  270059
T: 1 ( 2062) P:99 I:1500 C: 150977 Min:      1 Act:    3 Avg:    2 Max:     603
T: 2 ( 2063) P:99 I:2000 C: 113232 Min:      2 Act:    3 Avg:    3 Max:       8
T: 3 ( 2064) P:99 I:2500 C:  90586 Min:      2 Act:    3 Avg:    3 Max:      10

Installing

Kontron BIOS Changes

To make these BIOS changes, we need to get to the UEFI shell.
1. Reboot the computer
2. Press <DEL> key while booting to get to BIOS menu
3. Boot → Boot Option #1 → UEFI: Built-in Shell
4. Save and Exit (F4)
5. Press <ESC> while booting to get to the Shell> prompt.

Increase the boot delay so that the Spectrum Digitizers have time to initialise.
1. Enter the following on the Shell>
```
kboardconfig PciCfgDelay 5s
```
2. Double check by running without the argument too see what the parameter is set to
```
kboardconfig PciCfgDelay
# will see [5s] highlighted in green (and that is the longest delay available)
```

Disable hyperthreading and whatever VT stands for

kboardconfig IntelVT disabled
kboardconfig IntelHT disabled

Exit the bios
```
exit
```

What is SpeedStep? What is CpuTurbo? Sounds like things we don't want on!

USB Memory Stick Install

Version 6.8 of CentOS will boot from a UnetBootin created flash disk.
1. Downloaded CentOS-6.8-x86_64-minimal.iso (md5sum 0ca12fe5f28c2ceed4f4084b41ff8a0b ) and Win32DiskImager on Windows 10 Pro laptop
2. Insert a clean and formatted (FAT32) USB stick of at least 1GB size
3. Browse to the ISO image, select the correct drive (SERIOUSLY, MAKE SURE YOU ARE OVERWRITING THE CORRECT DRIVE!)
4. Write the image to the USB (are you sure that is the USB! )
Install the minimal CentOS base
1. Insert the USB stick, and connect the mouse, keyboard and ethernet (you want DHCP available) to the Kontron system computer.
2. Boot and press <DELETE> on boot to get BIOS menu
3. Change boot option to USB flash memory as Option #1
4. F4 to save and ext and then boot from the USB
~~Install the GH ethernet switch, as the e1000 driver is on the DVD, the igb on the SBC is not.~~
~~Boot and set the BIOS to boot off the Plextor drive (our USB based DVD drive)~~
~~At CentOS 6 Welcome, Click → Next~~
At Boot: Install or Upgrade Existing System
Language: English (English)
Keyboard: U.S. English
Installation Method → Hard drive
1. /dev/sdc1 was mine today — David Lloyd Rabine 2017/03/27 09:01
Storage Devices: Basic Storage Devices
Hostname: lvisf
Timezone: New York
root Password: the usual
Partitioning: Use All Space
Select the drive (for this case, the 60 GB (or 60000 MB drive)
1. Select the 64GB drive and put that to the right
2. Click “Boot Loader” in front of it (over in the right column)
3. Next to accept
4. Write changes to disk (which FORMATS the destination )
Wait for install to complete (280 packages in ~4 minutes)
Reboot

Network Temporary Configuration (DHCP)

YES, the igb driver is a part of the CentOS minimal package, so no need for GH Ethernet CPCI card.

Setup the system for DHCP for now
1. Log in as root
2. Modify /etc/sysconfig/network-scripts/ifcfg-eth0 so
```
ONBOOT=yes
```
3. Restart network service
```
service network restart
```

Add lvis User

Add lvis user to the system (still as root (logged in via SSH for me now))
1. Add the user
```
useradd lvis
```
2. Set password
```
passwd lvis
```
Add lvis user to the list of sudo users
1. Modify the super user file with the following (as logged in as root)
```
visudo
```
2. After the line that has root and ALL and ALL permissions, insert this line
```
lvis    ALL=(ALL)       ALL
```

Prerequisites for Compiling Kernel Modules

http://www.tldp.org/LDP/lkmpg/2.6/html/x121.html ← build kernel module link for an example

Update the system to the latest packages
```
yum update
reboot
```
Install the development tools
```
yum groupinstall 'Development Tools'
```
Get wget
```
yum install wget -y
```
Grab the ethernet driver while you still have access (or put on a memory stick (FAT32 is native readable))
1. Start here to check if there are newer versions: http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005767.html
2. Current driver is here: https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz
```
mkdir ~/download
cd ~/download
wget https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz
```

Scientific Linux Kernel

Install the SL MRG Repository

cd /etc/yum.repos.d/
sudo wget http://glitesoft.cern.ch/cern/mrg/slc6-mrg.repo
cd /etc/pki/rpm-gpg/
sudo wget http://ftp.scientificlinux.org/linux/scientific/5x/x86_64/RPM-GPG-KEYs/RPM-GPG-KEY-cern

Install the MRG Group with yum
```
sudo yum groupinstall "MRG Realtime"
```
Install the RT kernel source as well (this installed the latest, which is kernel-rt-devel-3.10.0-327.rt56.176.el6rt.x86_64)
```
sudo yum install kernel-rt-devel
```

Went to run the Latency Test and just as bad as ever.

Install 2.6 RT Kernel

Grab the kernel from this list: http://glitesoft.cern.ch/cern/mrg/slc6X/x86_64/RPMS/repoview/kernel-rt.html

cd ~/download
wget http://glitesoft.cern.ch/cern/mrg/slc6X/x86_64/RPMS/kernel-rt-2.6.33.9-rt31.79.el6rt.x86_64.rpm

And the source

wget http://glitesoft.cern.ch/cern/mrg/slc6X/x86_64/RPMS/kernel-rt-devel-2.6.33.9-rt31.79.el6rt.x86_64.rpm

Install them

sudo rpm -i --force kernel-rt-2.6.33.9-rt31.79.el6rt.x86_64.rpm
sudo rpm -i --force kernel-rt-devel-2.6.33.9-rt31.79.el6rt.x86_64.rpm

Reboot and select the new kernel (should be at top) and you won't have internet

Compile 2.6 RT Kernel from Source

SKIP THIS, NOT NECESSARY

Full Kernel Build

Installed packages based on this link: https://wiki.centos.org/HowTos/I_need_the_Kernel_Source

Did this:

sudo yum install rpm-build redhat-rpm-config asciidoc bison hmaccalc patchutils perl-ExtUtils-Embed xmlto 
sudo yum install audit-libs-devel binutils-devel elfutils-devel elfutils-libelf-devel
sudo yum install newt-devel python-devel zlib-devel

Grab the full source

cd ~/download
wget http://glitesoft.cern.ch/cern/mrg/slc6X/x86_64/SRPMS/kernel-rt-2.6.33.9-rt31.79.el6rt.src.rpm

Install them

rpm -i ~/download/kernel-rt-2.6.33.9-rt31.79.el6rt.src.rpm

Prepare the source

cd ~/rpmbuild/SPECS
rpmbuild -bp --target=$(uname -m) kernel-rt.spec

Modify the configuration file to build the USB3 module

Build the kernel

cd ~/rpmbuild/BUILD/kernel-2.6.33/linux-2.6.33.x86_64
make -j4  # build with 4 cores

Compiling Modules Matching Magic Version

Found this information here: http://www.tldp.org/LDP/lkmpg/2.6/html/x380.html

TRY THIS!

Install and Configure Ethernet Driver

Log in as root on the main console since ethernet will be disabled by default

Install Intel ethernet drivers [Only needed if compiling on Kontron flight hardware]:

Uncompress in the home/code directory

mkdir ~/code
cd ~/code
tar xvf igb-5.3.4.4.tar.gz

Build and install the drvier

cd ~/code/igb-5.3.4.4/src/
nano kcompat.h  # see below the section to remove (this define does not appear to work)

## DELETE THIS SECTION OF THE SOURCE CODE ###
#if (!(RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(5,7)) || \
     (RHEL_RELEASE_CODE == RHEL_RELEASE_VERSION(6,0)))
static inline __be16 vlan_get_protocol(const struct sk_buff *skb)
{
        if (vlan_tx_tag_present(skb) ||
            skb->protocol != cpu_to_be16(ETH_P_8021Q))
                return skb->protocol;

        if (skb_headlen(skb) < sizeof(struct vlan_ethhdr))
                return 0;

        return ((struct vlan_ethhdr*)skb->data)->h_vlan_encapsulated_proto;
}
#endif /* !RHEL5.7+ || RHEL6.0 */

Build and install the drvier
```
make
sudo make install
```

Modify this module file

sudo nano /etc/sysconfig/modules/lvisf.modules

Put the following into this file

#!/bin/sh

# start intel network driver
/sbin/modprobe igb

Change permissions so this script runs at boot time

sudo chmod 755 /etc/sysconfig/modules/lvisf.modules

Configure our network device

Change the eth0 definition

editing the config file

vi /etc/sysconfig/network-scripts/ifcfg-eth0

to the following

DEVICE=eth0
HWADDR=00:80:82:1B:E7:04
TYPE=Ethernet
UUID=98bd88d8-8fe8-4b8e-bca0-a58bd3306717
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth0"
IPADDR=10.4.1.2
NETMASK=255.255.255.0
DNS1=128.183.10.134
DNS2=128.183.50.17
DOMAIN="gsfc.nasa.gov ndc.nasa.gov"
GATEWAY="10.4.1.1"

Add our default gateway

Edit the network configuration
```
vi /etc/sysconfig/network
```

End up looking like this

NETWORKING=yes
HOSTNAME=lvisf
NETWORKING_IPV6=no
IPV6INIT=no
GATEWAY=10.4.1.1

Ensure our name server can be found (NOTE: Probably redundant if you modified the script, but check after rebooting that DNS and domain search are in resolv.conf)
1. Edit resolv.conf
```
vi /etc/resolv.conf
```
2. End up with this (NOTE: You don't need to do this if you modified ifcfg-eth0 to contain these values)
```
search gsfc.nasa.gov
nameserver 128.183.10.134
nameserver 128.183.50.17
```

Turn off IPV6 (found here: https://www.centos.org/forums/viewtopic.php?t=7989)

sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

Change the eth1 definition

editing the config file

vi /etc/sysconfig/network-scripts/ifcfg-eth1

to the following

DEVICE=eth1
HWADDR=00:80:82:1B:E7:05
TYPE=Ethernet
UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth1"
IPADDR=192.168.0.2
NETMASK=255.255.255.0

Change the eth1:1 definition

editing the config file

vi /etc/sysconfig/network-scripts/ifcfg-eth1:1

to the following

DEVICE=eth1:1
HWADDR=00:80:82:1B:E7:05
TYPE=Ethernet
UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
NAME="System eth1:1"
IPADDR=129.100.0.2
NETMASK=255.255.255.0

Install lvisf Source

Add NTFS Capability

Add ntfs drive capability
1. Add the epel repository
```
sudo yum install epel-release
```
2. Add ntfs to our system
```
sudo yum install fuse fuse-ntfs-3g
```

Add Exfat Capability

Add Exfat drive capability

Add the epel repository and this Nux Dextop repository

sudo yum install epel-release
sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el6/x86_64/nux-dextop-release-0-2.el6.nux.noarch.rpm

Add Exfat to our system
```
sudo yum install exfat-utils fuse-exfat
```

An example mounting an Exfat system is

sudo  mount -t exfat /dev/sdc1 /mnt/tmp

Requirements

Install cmake
```
sudo yum install cmake
```
Install htop
```
sudo yum install htop
```

Install jed

sudo yum install ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/jed-0.99.18-5.el5.kb.i386.rpm

Install nano
```
sudo yum install nano
```
Install p7zip
```
sudo yum install p7zip
```
Install PCI utilities
```
sudo yum install pciutils
```
Install telnet
```
sudo yum install telnet
```
Install wget
```
sudo yum install wget
```

Install the development package

sudo yum groupinstall "Development Tools"

Install development libraries for md5sum and gzip in real time
```
sudo yum install openssl-devel
```
Install md5deep so we can check file integrity
```
sudo yum install md5deep
```

Install man pages

sudo yum install man
sudo yum install man-pages

X11 Minimal

This will allow you to run X11

# core X11
sudo yum install xorg-x11-xauth xorg-x11-fonts-* xorg-x11-utils xterm
# required for tuna
sudo yum install xauth pygtk2-libglade

Software Tweaking

Let's remove EVERYTHING on this system we don't care about.

Remove postfix we don't need email
```
sudo yum remove postfix
```

Disable the Firewall

Turn off the firewall

sudo service iptables stop
sudo service ip6tables stop

Make this permanent

sudo chkconfig iptables off
sudo chkconfig ip6tables off

Disable SELinux
1. Edit the configuration file
```
sudo vi /etc/selinux/config
```
2. Modify the file
  1. From
```
SELINUX=enforcing
```
  2. To
```
SELINUX=disabled
```

Remove auditd

sudo service auditd stop # now
sudo chkconfig auditd off # after reboot

Disable ipv6

Add these two lines to sshd_config

# lvisf - ipv4 only
AddressFamily inet
ListenAddress 0.0.0.0

Run these two lines

sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1

Reboot for all changes to take place

Code Base

Create the repoistory home and check out the source code
1. Create a repository home in /opt
```
sudo mkdir /opt/lvisf
```
2. Set proper permissions
```
sudo chown -R lvis:lvis /opt/lvisf
```
3. [Optional] Uncompress our locally attached mount with snapshot (faster than network initial checkout)
```
cd /
tar jxvf /media/LVIS-GH-2TB_WD209/lvisf_build/lvisf_sourcesnapshot_20160816_svn439.tar.bz2
```
  1. [Either] Check out he raw source code from scratch
```
cd /opt/lvisf
svn co https://svn.yancich.com:8443/svn/NASA/LVIS/trunk/ src
```
  2. [Or] Update it if you used the tar above
```
cd /opt/lvisf/src
svn update
```
4. Enter the user home directory and create symbolic links
```
cd /home/lvis
mkdir src
ln -s /opt/lvisf/src/* /home/lvis/src/
```

Add the following lines to the .bashrc for the lvis user

Edit the file
```
nano /home/lvis/.bash_profile
```

Add these lines

# For the Innovative digitizer
export WD_BASEDIR=/home/lvis/src/Innovative/WinDriver

# For the SyncClock timer card
export PLX_SDK_DIR=/home/lvis/src/PLX/PlxSdk

# For the spectrum digitizer
export SSP_INSTALL_ROOT=/home/lvis/src/spectrum

# SVN editor
export SVN_EDITOR=nano

Edit the file

sudo nano /etc/ld.so.conf.d/lvisf.conf

Add these lines

/home/lvis/src/spectrum/shared/linux/pc64/lib
/home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/libs

Reload the library path
```
sudo ldconfig
```

Add hooks into the run scripts for the lvis user (this is just convenient)
1. Create a local bin directory in the case there isn't one
```
mkdir ~/bin
```
2. Symbolically link all the run scripts into this directory
```
ln -s /opt/lvisf/src/lvisF/scripts/run* ~/bin/
```

SyncClock32 PLX Drivers

Build all drivers
1. Go to the source code home directory
```
cd /home/lvis/src/PLX/PlxSdk/Driver
```
2. Execute the build script
```
./buildalldrivers
```
Build the samples
1. Go to the source directory
```
cd /home/lvis/src/PLX/PlxSdk
```
2. Make the programs
```
make
```

Load the driver

To the binary directory
```
cd /home/lvis/src/PLX/PlxSdk/Bin
```

Execute the script for the 9030 card (our syncclock32)

sudo ./Plx_load 9030

# to just see what boards are supported, run the script with no argument
./Plx_load

MIC 3753 DIO Homebrew Drivers

NOT REQUIRED We have reverted to the original factory drivers

~~; compile and insert the kernel module~~

cd /opt/lvisf/src/lvisF/ko
make
sudo insmod /opt/lvisf/src/lvisF/ko/lvisf_mic3753.ko
sudo mknod /dev/lvisf_mic3753 c 100 0

MIC 3753 DIO Drivers

NOTE These drivers NEED to be built as root, you cannot sudo to build these drivers

Become root all the time
```
sudo bash
```

Go to the code base directory

cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/driver_base/src/lnx_ko

Build the drivers
```
make
```

Go into our specific card directory

cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/pci1753_mic3753_pcm3753i/src/lnx_ko

Build the drivers
```
make
```

Load the drivers

cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin
insmod biokernbase.ko
insmod bio1753.ko

Exit the sudo bash
```
exit
```

Spectrum Digitizer Drivers

Create the symbolic link for /spectrum

sudo ln -s /opt/lvisf/src/spectrum /spectrum

Enter the code directory

cd /home/lvis/src/spectrum/shared/linux/pc64/bin

Copy the drivers for our kernel into the binary directory

cp -a /home/lvis/src/spectrum/shared/linux/pc64/bin/`uname -r`/*.ko /home/lvis/src/spectrum/shared/linux/pc64/bin/

Execute the setup script

chmod 755 *.sh   # not sure why these were not already executable
sudo bash ./setup_xmc1151_dvt.sh

Select the proper option: 1)Install

Auto-load MIC-3753 Driver

symlink the drivers into the kernel directory

sudo ln -s /opt/lvisf/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/*.ko /lib/modules/2.6.33.9-rt31.79.el6rt.x86_64/kernel/drivers/misc/

add to our startup script /etc/sysconfig/modules/lvisf.modules

modify the file

sudo nano /etc/sysconfig/modules/lvisf.modules

add the following

# start the MIC-3753 drivers
/sbin/modprobe biokernbase
/sbin/modprobe bio1753

Auto-load Plx9030 Driver

NOTE: (in lieu of step 2 below) I had this not work when I manually edited it the first time. A copy of the file is in the repository, you could just copy that file in:

sudo cp /opt/lvisf/src/lvisF/scripts/lvisf.modules /etc/sysconfig/modules/lvisf.modules

symlink the driver into the kernel so it will come up on boot

sudo ln -s /opt/lvisf/src/PLX/PlxSdk/Driver/Source.Plx9000/Output/Plx9030.ko /lib/modules/2.6.33.9-rt31.79.el6rt.x86_64/kernel/drivers/misc/

add to our startup script /etc/sysconfig/modules/lvisf.modules

modify the file

sudo nano /etc/sysconfig/modules/lvisf.modules

add the following

# start the Plx9030 syncclock32 driver
/sbin/modprobe Plx9030

symbolically link in our rc.local into /etc/rc.local
1. remove the original
```
sudo rm /etc/rc.local
```
2. link in the lvisf version
```
sudo ln -s /opt/lvisf/src/lvisF/scripts/rc.local /etc/rc.local
```

Installing Boost Library

The boost C++ library is required for the data system. The tarball of the 1.6.2 version is in the repository.

Uncompress the tar into /opt

cd /opt
sudo tar -jxvf /opt/lvisf/src/tars/boost_1_62_0.tar.bz2

Symbolically link to a generic boost directory
```
sudo ln -s /opt/boost_1_62_0 /opt/boost
```

Chown it all to lvis user

sudo chown -R lvis:lvis /opt/boost_1_62_0

Set the environment variable for the BOOST_ROOT variable
1. Edit the file /home/lvis/.bash_profile
```
nano /home/lvis/.bash_profile 
```
2. Add the line
```
# Add the boost root directory
export BOOST_ROOT=/opt/boost
```
Compile the boost libraries
```
cd /opt/boost
./bootstrap.sh
./b2
```

Installing USB Temperature Library

Follow the instructions below to compile the USB temperature monitoring device libraries required.

Measurement Computing USB-TEMP-AI code instructions.

Installing Innovative Digitizer Drivers

Described in LinuxNotes.pdf (might be this /usr/Innovative/MalibuRed-1.3/KerPlug/LinuxNotes.pdf)

Link our Innovative directory to /usr/Innovative

sudo ln -s /opt/lvisf/src/Innovative /usr/Innovative

Untar the latest driver

tar -zxvf /opt/lvisf/src/tars/64WinDriver-12.3-0.X86_64.tgz.tar.gz --directory=/usr/Innovative/

Build 12.3 Windriver

Link 12.3 as our Windriver

cd /usr/Innovative
rm /usr/Innovative/WinDriver
ln -s WinDriver-12.3 WinDriver

Go here

cd /usr/Innovative/WinDriver/redist

Configure

./configure
make
sudo make install # they say do not do this, we live on the edge

Kerplug

Build (builds in the LINUX directory for the RT kernel)
```
cd /usr/Innovative/KerPlug/linux
./configure
make
```
OK, this SHOULD fail.

Edit the Makefile with

nano Makefile

Add seven dot dot slashes to the line:

Convert this

MOD_CSRCS = $(WD_BASEDIR)/redist/linux_wrappers.c

to this

MOD_CSRCS = ../../../../../../../$(WD_BASEDIR)/redist/linux_wrappers.c

Change 1180 to 1230

Run make again
```
make
```

Load the driver

sudo insmod LINUX.2.6.33.9-rt31.79.el6rt.x86_64.x86_64/kp_malibu_module.ko

Need to link the generic library name with the compiled library
```
cd /usr/Innovative/KerPlug
sudo ./quicklink wdapi 1230
```

Raw code from /etc/rc.local from the primary machine to load this

sudo bash
cd /usr/Innovative/WinDriver-12.3/redist && ./wdreg LINUX.2.6.33.9-rt31.79.el6rt.x86_64.x86_64/windrvr1230 auto
#/usr/Innovative/winCloudDrvr/winclouddrvr_installation/redist/wdreg winCloudDrvr auto
#chmod 666 /dev/winCloudDrvr
insmod /usr/Innovative/KerPlug/linux/LINUX.2.6.33.9-rt31.79.el6rt.x86_64.x86_64/kp_malibu_module.ko
exit

Latency Tweaks on Kernel

Low latency items to add to the kernel command line for /etc/grub.conf

After
```
rhgb quiet 
```

Add the following

nosoftlockup mce=ignore_ce audit=0 maxcpus=2

Final line looks like this

        kernel /vmlinuz-2.6.33.9-rt31.79.el6rt.x86_64 ro root=/dev/mapper/vg_lvisf-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=vg_lvisf/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_lvisf/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet nosoftlockup mce=ignore_ce audit=0 maxcpus=2

nosoftlockup disables backtrace functionality
mce=ignore_ce ignores corrected errors
audit=0 disables kernel components of the audit subsystem
maxcpus=2 uses only the two actual CPUs on the system disabling any hyperthreading

Compile Code

Enter the build directory for lvisf
```
cd /home/lvis/src/lvisF/build
```
Use cmake
```
cmake ../
```
Build code
```
make
```

Drive Configuration

~~Create our disk partition (using the proper settings for fdisk to use 4096 byte blocks)~~
```
sudo fdisk -H 224 -S 56 /dev/sdb
```
1. Delete whatever is on there (use gparted below)
2. Create a new partition with the default settings (use gparted below)

Install and use gparted to configure the drive.
1. Delete existing partitions (on the correct drive!!)
Format ( BE SURE YOU ARE FORMATTING THE CORRECT PARTITION )
```
sudo mkfs.ext2 /dev/sdb1 -L LVISF_INTERNAL
```
To show attached drives by label
```
ls -l /dev/disk/by-label
```
Add the EKF based internal data drive to the system
1. Create a mount point
```
sudo mkdir /mnt/internal
```
2. Modify /etc/fstab to add it by label
```
sudo nano /etc/fstab
```
3. Add the following line:
```
LABEL=LVISF_INTERNAL    /mnt/internal           ext4    defaults        0 0
```
4. Symbolically link to /mnt/data/
```
sudo ln -s /mnt/internal /mnt/data
```
Share the drive via nfs so the GUI and other systems can mount (read only) the raw data drive
1. Add the required software
```
sudo yum install nfs-utils nfs-utils-lib
```
2. Start the nfs services
```
sudo chkconfig nfs on 
sudo service rpcbind start
sudo service nfs start
```
3. Add our data directory as a shared data drive to /etc/exports
  1. Edit our shares
```
sudo nano /etc/exports
```
  2. Add this line
```
/mnt/internal          10.4.1.4(ro,sync,no_root_squash,no_subtree_check)
```
4. Export our share
```
sudo exportfs -a
```

On the client side, add the shared drive with

Add this to /etc/fstab

10.4.1.2:/mnt/internal   /mnt/nfs/internal   nfs    ro,bg,intr,soft,rsize=8192,wsize=8192,timeo=5,retrans=5,actimeo=10,retry=5  0  0

Timing Tweaks

Interrupt Timing

Interrupts are BACK! timing03 test by isolating cpu3, and then putting the IRQ for the timer card on that CPU appears to have worked. I think I was using the wrong value in the smp_affinity for /proc/irq/17/ entry. I was putting the actual CPU, but it appears to be a mask of some kind. The tuna program put the correct value in there (7 = 0-2 and 8 = 3 ?) I think.

KEYTABLE=us rd_NO_DM rhgb quiet noht isolcpus=3 nosoftlockup mce=ignore_ce audit=0 intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll

Tuna Timing Tuning

Trying this option on the command line: acpi_irq_nobalance

Isolate ODD cpus, so that is probably a single hyperthreading core.

# isolate CPU 1 and 3 (which are physical CPU 1 leaving the primary for regular OS)
/usr/bin/tuna --cpus=1,3 --isolate

CPU affinity is 0,2 (which is mask 5 (1 and 4))

Move the 9030 timer card IRQ to CPU 1 (primary CPU of secondary core) (need to isolate this) This is IRQ 11 on the development system (with no Spectrum boards)
```
# move the timer card IRQ (17 for 0x9030 syncclock32 card)
/usr/bin/tuna --irqs=17 --cpus=1 --move
```

FYI, you can save the configuration with this command line

sudo /usr/bin/tuna --save=/tmp/tuna.config

Run Build01

Start shell one via a ssh connection and run

run1  # does the full 40 second Spectrum XMC-1151 calibration

or

run1nocal  # if you want to skip the 40 second Spectrum XMC-1151 calibration

Start shell two via a ssh connection and run
```
run2
```
Start shell three via a ssh connection and run
```
run3
```

Timing Tests

Timing considerations

163,840 bytes is two channels of 4k samples for 10 shots (4096 * 2 * 2 * 10) /2*1024*1024*1024 = 76 microseconds

76 microseconds every 2.5 milliseconds is NOT A LOT!

or all of it (for 30,000 feet) (32768*2*2) / 2*1024*1024*1024 = 61 microseconds per shots (or everyon 250 microseconds

timing03

Resurrected timing03 to investigate interrupts — David Lloyd Rabine 2017/02/09 11:55

To run timing03 (as of 2017.02.09) it can use the same receiver (aka lvisf)

Short instructions

Terminal One
```
run1nocal
```
Terminal Two
```
runtiming03
```

tuna might have saved the day!

Isolate CPU 3 (so nothing is running on there)
Affine the interrupt for the 9030 board to CPU 3
Profit! It appears that solves the missing triggers issue for the moment (been fooled before)

timing04

Load Drivers

sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/biokernbase.ko
sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/bio1753.ko
cd /home/lvis/src/PLX/PlxSdk/Bin
sudo ./Plx_load 9030

Run Test: timing04

cd /opt/lvisf/src/lvisF/build/
sudo chrt --fifo 99 ./bin/tests/timing04/timing04

2016.07.19 Overnight Test Results

SVN Revision 378 timing04
Total interrupts = 214748056

00 11 (0.000005) 
01 10 (0.000005) 
02 6 (0.000003) 
03 8 (0.000004) 
04 8439881 (3.930279) 
05 9181555 (4.275661) 
06 8656159 (4.030995) 
07 14232072 (6.627583) 
08 166745360 (77.649887) 
09 7277561 (3.389011) 
10 191284 (0.089077) 
11 6471 (0.003013) 
12 4103 (0.001911) 
13 5352 (0.002492) 
14 34 (0.000016) 
15 18 (0.000008) 
16 15 (0.000007) 
17 18 (0.000008) 
18 19 (0.000009) 
19 6 (0.000003) 
20 6 (0.000003) 
21 17 (0.000008) 
22 9 (0.000004) 
23 13 (0.000006) 
24 2 (0.000001) 
25 0 (0.000000) 
26 0 (0.000000) 
27 0 (0.000000) 
28 0 (0.000000) 
29 0 (0.000000)

2016.07.20 One Hour Test Results

SVN Revision 379 timing03 (added stats to PLX interrupt test)

FYI, thread priority wasn't even HIGH!
galvThread: Scheduler settings of DoWork thread
    policy=SCHED_FIFO, priority=5


00 0 (0.000000) 
01 0 (0.000000) 
02 0 (0.000000) 
03 0 (0.000000) 
04 0 (0.000000) 
05 0 (0.000000) 
06 0 (0.000000) 
07 2 (0.000014) 
08 2033 (0.014115) 
09 2967 (0.020599) 
10 1384397 (9.611487) 
11 12906360 (89.605298) 
12 106019 (0.736061) 
13 372 (0.002583) 
14 280 (0.001944) 
15 184 (0.001277) 
16 375 (0.002604) 
17 358 (0.002485) 
18 206 (0.001430) 
19 15 (0.000104) 
20 0 (0.000000) 
21 0 (0.000000) 
22 0 (0.000000) 
23 0 (0.000000) 
24 0 (0.000000) 
25 0 (0.000000) 
26 0 (0.000000) 
27 0 (0.000000) 
28 0 (0.000000) 
29 0 (0.000000) 
30 0 (0.000000) 
31 0 (0.000000)

2016.07.20 One Hour Test 2 Results

Says ZERO missed shots, but there are a couple of times that are suspect… BUT, overall, looking great. Acceptable lost shots in my opinion for a non hard real time OS.

SVN Revision 380 and timing03

Total Shots = 14404168 (and 14404170)
Total Missed Shots = 0
00 2 (0.000014) 
01 0 (0.000000) 
02 2 (0.000014) 
03 3 (0.000021) 
04 3 (0.000021) 
05 5 (0.000035) 
06 1 (0.000007) 
07 4 (0.000028) 
08 1935 (0.013434) 
09 2676 (0.018578) 
10 1980021 (13.746167) 
11 12304306 (85.421845) 
12 113162 (0.785620) 
13 542 (0.003763) 
14 293 (0.002034) 
15 210 (0.001458) 
16 396 (0.002749) 
17 362 (0.002513) 
18 216 (0.001500) 
19 20 (0.000139) 
20 3 (0.000021) 
21 1 (0.000007) 
22 2 (0.000014) 
23 1 (0.000007) 
24 2 (0.000014) 
25 0 (0.000000) 
26 0 (0.000000) 
27 0 (0.000000) 
28 0 (0.000000) 
29 0 (0.000000) 
30 0 (0.000000) 
31 0 (0.000000)

2016.07.20 Overnight Test Results

00 13 (0.000007) 
01 4 (0.000002) 
02 6 (0.000003) 
03 6 (0.000003) 
04 7959297 (4.079181) 
05 8363128 (4.286146) 
06 7973271 (4.086342) 
07 12581677 (6.448174) 
08 151407385 (77.597063) 
09 6616644 (3.391064) 
10 203955 (0.104528) 
11 5439 (0.002788) 
12 3784 (0.001939) 
13 5238 (0.002685) 
14 45 (0.000023) 
15 14 (0.000007) 
16 12 (0.000006) 
17 13 (0.000007) 
18 6 (0.000003) 
19 12 (0.000006) 
20 6 (0.000003) 
21 14 (0.000007) 
22 12 (0.000006) 
23 11 (0.000006) 
24 2 (0.000001) 
25 0 (0.000000) 
26 0 (0.000000) 
27 0 (0.000000) 
28 0 (0.000000) 
29 0 (0.000000)

Table of Contents

RT Kernel on CentOS

Links

RT Kernel

Latency Testing

IPC

Cyclic Testing

Installing

Kontron BIOS Changes

USB Memory Stick Install

Network Temporary Configuration (DHCP)

Add lvis User

Prerequisites for Compiling Kernel Modules

Scientific Linux Kernel

Install 2.6 RT Kernel

Compile 2.6 RT Kernel from Source

Full Kernel Build

Compiling Modules Matching Magic Version

Install and Configure Ethernet Driver

Install lvisf Source

Add NTFS Capability

Add Exfat Capability

Requirements

X11 Minimal

Software Tweaking

Code Base

SyncClock32 PLX Drivers

MIC 3753 DIO Homebrew Drivers

MIC 3753 DIO Drivers

Spectrum Digitizer Drivers

Auto-load MIC-3753 Driver

Auto-load Plx9030 Driver

Installing Boost Library

Installing USB Temperature Library

Installing Innovative Digitizer Drivers

Latency Tweaks on Kernel

Compile Code

Drive Configuration

Timing Tweaks

Interrupt Timing

Tuna Timing Tuning

Run Build01

Timing Tests

timing03

timing04

2016.07.19 Overnight Test Results

2016.07.20 One Hour Test Results

2016.07.20 One Hour Test 2 Results

2016.07.20 Overnight Test Results