====== RT Kernel on CentOS ====== Tried compiling my own PREEMPT kernel, but it had many issues and ran worse than the generic kernel. I only enabled PREEMPT and did not change any other options. I assume the RT kernel they are packaging is tweaked for better real time. I noticed the CPU wasn't scaling down with the CERN RT kernel. :!: **NOTE:** To change eth0 if you move drives, simply change **HWADDRESS** to match **ifconfig** of **eth0** and leave the generated **UUID** as the OS created it. This **UUID** is random and means nothing, but it might appear in other scripts and have to stay the same. ===== Links ===== ==== RT Kernel ==== * [[http://linux.web.cern.ch/linux/scientific6/]] * [[http://glitesoft.cern.ch/]] <- RT kernel repository * [[https://rt.wiki.kernel.org/index.php/Main_Page]] <- rt main page * [[https://rt.wiki.kernel.org/index.php/Cyclictest]] <- cyclic testing software * To resolve compile dependency * Install the **libnuma-dev** package (on Debian-based systems) * Install the **numactl-devel** package (on RHEL/CentOS systems) * [[https://rt.wiki.kernel.org/index.php/RT_PREEMPT_HOWTO]] <- PREEMPT HowTo * [[https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application]] <- PREEMPT How to build an application === Latency Testing === * [[https://www.osadl.org/fileadmin/dam/rtlws/12/Brown.pdf]] (local copy {{:subsystem:datasystem:dev:cernscientificlinux:brown.pdf|Brown.pdf}}) (Page 9 is VERY interesting!) * [[https://lwn.net/Articles/139784/]] * [[https://lwn.net/Articles/139403/]] <- I would love to get these response times! ==== IPC ==== * [[http://www.kohala.com/start/unpv22e/unpv22e.chap12.pdf]] <- excellent shared memory chapter (IPC) * [[https://courses.engr.illinois.edu/cs241/sp2012/lectures/29-IPC.pdf]] <- another good IPC presentation * [[http://menehune.opt.wfu.edu/Kokua/More_SGI/007-2478-008/sgi_html/ch03.html]] <- more good information * [[http://www.cse.psu.edu/~deh25/cmpsc473/notes/OSC/Processes/shm.html]] <- REALLY good simple C example of producer / consumer code ===== Cyclic Testing ===== **NOTE!!** **This is where I got the idea to run our thread at priority 99, which is directly contradicted by the following page which makes clear there are some things that need to run at higher priority than our program** [[https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application]] which states: Do not configure your application to run with priority 99. There are a few management threads which need to run with higher priority then your application, e.g. watchdogs threads. Ran this in a second login to create disk load: time sudo sh -c "dd if=/dev/zero of=/mnt/data/ddfile bs=8k count=2000000 && sync" # writes 16GB SUCCESS! (rt kernel) [lvis@lvisf-centos65 rt-tests]$ uname -a Linux lvisf-centos65 2.6.33.9-rt31.79.el6rt.x86_64 #1 SMP PREEMPT RT Wed Jan 11 13:54:22 CET 2012 x86_64 x86_64 x86_64 GNU/Linux [lvis@lvisf-centos65 rt-tests]$ sudo ./cyclictest -a -t -n -p99 -f -b100 [sudo] password for lvis: # /dev/cpu_dma_latency set to 0us INFO: debugfs mountpoint: /sys/kernel/debug/tracing/ policy: fifo: loadavg: 1.81 1.79 1.12 2/220 11904 T: 0 (11851) P:99 I:1000 C: 231860 Min: 20 Act: 27 Avg: 26 Max: 85 T: 1 (11852) P:99 I:1500 C: 154572 Min: 22 Act: 28 Avg: 28 Max: 72 T: 2 (11853) P:99 I:2000 C: 115929 Min: 22 Act: 26 Avg: 27 Max: 65 T: 3 (11854) P:99 I:2500 C: 92743 Min: 22 Act: 28 Avg: 28 Max: 58 FAIL! (Spectrum Default Kernel) ^C[lvis@lvisf rt-tests-1.0]$ uname -a Linux lvisf 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [lvis@lvisf rt-tests-1.0]$ sudo ./cyclictest -a -t -n -p99 # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 0.28 0.35 0.15 1/133 2044 T: 0 ( 1897) P:99 I:1000 C: 288733 Min: 1 Act: 3 Avg: 7 Max: 66553 T: 1 ( 1898) P:99 I:1500 C: 192622 Min: 1 Act: 3 Avg: 2 Max: 543 T: 2 ( 1899) P:99 I:2000 C: 144466 Min: 2 Act: 3 Avg: 3 Max: 7 T: 3 ( 1900) P:99 I:2500 C: 115573 Min: 2 Act: 3 Avg: 3 Max: 8 ^C[lvis@lvisf rt-tests-1.0]$ sudo ./cyclictest -a -t -n -p99 # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 0.08 0.35 0.20 1/132 2083 T: 0 ( 2061) P:99 I:1000 C: 226186 Min: 1 Act: 2 Avg: 7 Max: 270059 T: 1 ( 2062) P:99 I:1500 C: 150977 Min: 1 Act: 3 Avg: 2 Max: 603 T: 2 ( 2063) P:99 I:2000 C: 113232 Min: 2 Act: 3 Avg: 3 Max: 8 T: 3 ( 2064) P:99 I:2500 C: 90586 Min: 2 Act: 3 Avg: 3 Max: 10 ===== Installing ===== ==== Kontron BIOS Changes ==== * To make these BIOS changes, we need to get to the UEFI shell. - Reboot the computer - Press key while booting to get to BIOS menu - Boot -> Boot Option #1 -> UEFI: Built-in Shell - Save and Exit (F4) - Press while booting to get to the **Shell>** prompt. - Increase the boot delay so that the Spectrum Digitizers have time to initialise. - Enter the following on the **Shell>** kboardconfig PciCfgDelay 5s - Double check by running without the argument too see what the parameter is set to kboardconfig PciCfgDelay # will see [5s] highlighted in green (and that is the longest delay available) - Disable hyperthreading and whatever VT stands for kboardconfig IntelVT disabled kboardconfig IntelHT disabled - Exit the bios exit - What is SpeedStep? What is CpuTurbo? Sounds like things we don't want on! ==== USB Memory Stick Install ==== - Version 6.8 of CentOS will boot from a UnetBootin created flash disk. - Downloaded **CentOS-6.8-x86_64-minimal.iso** (md5sum 0ca12fe5f28c2ceed4f4084b41ff8a0b ) and Win32DiskImager on Windows 10 Pro laptop - Insert a clean and formatted (FAT32) USB stick of at least 1GB size - Browse to the ISO image, select the correct drive (SERIOUSLY, MAKE SURE YOU ARE OVERWRITING THE CORRECT DRIVE!) :!: - Write the image to the USB (are you sure that is the USB! :!::?::!:) - Install the minimal CentOS base - Insert the USB stick, and connect the mouse, keyboard and ethernet (you want DHCP available) to the Kontron system computer. - Boot and press on boot to get BIOS menu - Change boot option to USB flash memory as Option #1 - F4 to save and ext and then boot from the USB - Install the GH ethernet switch, as the **e1000** driver is on the DVD, the **igb** on the SBC is not. - Boot and set the BIOS to boot off the Plextor drive (our USB based DVD drive) - At CentOS 6 Welcome, Click -> Next - At Boot: Install or Upgrade Existing System - Language: English (English) - Keyboard: U.S. English - Installation Method -> Hard drive - **/dev/sdc1** was mine today --- //[[david.rabine@nasa.gov|David Lloyd Rabine]] 2017/03/27 09:01// - Storage Devices: Basic Storage Devices - Hostname: lvisf - Timezone: New York - root Password: the usual - Partitioning: Use All Space - Select the drive (for this case, the 60 GB (or 60000 MB drive) - Select the 64GB drive and put that to the right - Click "Boot Loader" in front of it (over in the right column) - Next to accept - Write changes to disk (which FORMATS the destination :!: ) - Wait for install to complete (280 packages in ~**4** minutes) - Reboot ==== Network Temporary Configuration (DHCP) ==== 8-) YES, the **igb** driver is a part of the CentOS minimal package, so no need for GH Ethernet CPCI card. - Setup the system for DHCP for now - Log in as root - Modify **/etc/sysconfig/network-scripts/ifcfg-eth0** so ONBOOT=yes - Restart network service service network restart ==== Add lvis User ==== - Add **lvis** user to the system (still as root (logged in via SSH for me now)) - Add the user useradd lvis - Set password passwd lvis - Add **lvis** user to the list of **sudo** users - Modify the super user file with the following (as logged in as root) visudo - After the line that has root and ALL and ALL permissions, insert this line lvis ALL=(ALL) ALL ==== Prerequisites for Compiling Kernel Modules ==== * [[http://www.tldp.org/LDP/lkmpg/2.6/html/x121.html]] <- build kernel module link for an example - Update the system to the latest packages yum update reboot - Install the development tools yum groupinstall 'Development Tools' - Get **wget** yum install wget -y - Grab the ethernet driver while you still have access (or put on a memory stick (FAT32 is native readable)) - Start here to check if there are newer versions: [[http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005767.html]] - Current driver is here: [[https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz]] mkdir ~/download cd ~/download wget https://downloadmirror.intel.com/13663/eng/igb-5.3.5.3.tar.gz ==== Scientific Linux Kernel ==== - Install the SL MRG Repository cd /etc/yum.repos.d/ sudo wget http://glitesoft.cern.ch/cern/mrg/slc6-mrg.repo cd /etc/pki/rpm-gpg/ sudo wget http://ftp.scientificlinux.org/linux/scientific/5x/x86_64/RPM-GPG-KEYs/RPM-GPG-KEY-cern - Install the MRG Group with **yum** sudo yum groupinstall "MRG Realtime" - Install the RT kernel source as well (this installed the latest, which is **kernel-rt-devel-3.10.0-327.rt56.176.el6rt.x86_64**) sudo yum install kernel-rt-devel Went to run the [[https://gs490lviswiki.ndc.nasa.gov/doku.php?id=subsystem:datasystem:dev:centos65:start#user_space_latency_test|Latency Test]] and just as bad as ever. ==== Install 2.6 RT Kernel ==== - Grab the kernel from this list: [[http://glitesoft.cern.ch/cern/mrg/slc6X/x86_64/RPMS/repoview/kernel-rt.html]] cd ~/download wget http://glitesoft.cern.ch/cern/mrg/slc6X/x86_64/RPMS/kernel-rt-2.6.33.9-rt31.79.el6rt.x86_64.rpm - And the source wget http://glitesoft.cern.ch/cern/mrg/slc6X/x86_64/RPMS/kernel-rt-devel-2.6.33.9-rt31.79.el6rt.x86_64.rpm - Install them sudo rpm -i --force kernel-rt-2.6.33.9-rt31.79.el6rt.x86_64.rpm sudo rpm -i --force kernel-rt-devel-2.6.33.9-rt31.79.el6rt.x86_64.rpm - Reboot and select the new kernel (should be at top) and you won't have internet ==== Compile 2.6 RT Kernel from Source ==== FIXME SKIP THIS, NOT NECESSARY FIXME === Full Kernel Build === * Installed packages based on this link: [[https://wiki.centos.org/HowTos/I_need_the_Kernel_Source]] * Did this: sudo yum install rpm-build redhat-rpm-config asciidoc bison hmaccalc patchutils perl-ExtUtils-Embed xmlto sudo yum install audit-libs-devel binutils-devel elfutils-devel elfutils-libelf-devel sudo yum install newt-devel python-devel zlib-devel - Grab the full source cd ~/download wget http://glitesoft.cern.ch/cern/mrg/slc6X/x86_64/SRPMS/kernel-rt-2.6.33.9-rt31.79.el6rt.src.rpm - Install them rpm -i ~/download/kernel-rt-2.6.33.9-rt31.79.el6rt.src.rpm - Prepare the source cd ~/rpmbuild/SPECS rpmbuild -bp --target=$(uname -m) kernel-rt.spec - Modify the configuration file to build the USB3 module - Build the kernel cd ~/rpmbuild/BUILD/kernel-2.6.33/linux-2.6.33.x86_64 make -j4 # build with 4 cores === Compiling Modules Matching Magic Version === * Found this information here: [[http://www.tldp.org/LDP/lkmpg/2.6/html/x380.html]] FIXME TRY THIS! ==== Install and Configure Ethernet Driver ==== - Log in as **root** on the main console since ethernet will be disabled by default - Install Intel ethernet drivers **[Only needed if compiling on Kontron flight hardware]**: - Uncompress in the home/code directory mkdir ~/code cd ~/code tar xvf igb-5.3.4.4.tar.gz - Build and install the drvier cd ~/code/igb-5.3.4.4/src/ nano kcompat.h # see below the section to remove (this define does not appear to work) ## DELETE THIS SECTION OF THE SOURCE CODE ### #if (!(RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(5,7)) || \ (RHEL_RELEASE_CODE == RHEL_RELEASE_VERSION(6,0))) static inline __be16 vlan_get_protocol(const struct sk_buff *skb) { if (vlan_tx_tag_present(skb) || skb->protocol != cpu_to_be16(ETH_P_8021Q)) return skb->protocol; if (skb_headlen(skb) < sizeof(struct vlan_ethhdr)) return 0; return ((struct vlan_ethhdr*)skb->data)->h_vlan_encapsulated_proto; } #endif /* !RHEL5.7+ || RHEL6.0 */ - Build and install the drvier make sudo make install - Modify this module file sudo nano /etc/sysconfig/modules/lvisf.modules - Put the following into this file #!/bin/sh # start intel network driver /sbin/modprobe igb - Change permissions so this script runs at boot time sudo chmod 755 /etc/sysconfig/modules/lvisf.modules - Configure our network device - Change the **eth0** definition - editing the config file vi /etc/sysconfig/network-scripts/ifcfg-eth0 - to the following DEVICE=eth0 HWADDR=00:80:82:1B:E7:04 TYPE=Ethernet UUID=98bd88d8-8fe8-4b8e-bca0-a58bd3306717 ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static NAME="System eth0" IPADDR=10.4.1.2 NETMASK=255.255.255.0 DNS1=128.183.10.134 DNS2=128.183.50.17 DOMAIN="gsfc.nasa.gov ndc.nasa.gov" GATEWAY="10.4.1.254" # GATEWAY="10.4.1.1" - Add our default gateway - Edit the network configuration vi /etc/sysconfig/network - End up looking like this NETWORKING=yes HOSTNAME=lvisf NETWORKING_IPV6=no IPV6INIT=no GATEWAY=10.4.1.254 # GATEWAY=10.4.1.1 - Ensure our name server can be found (**NOTE:** Probably redundant if you modified the script, but check after rebooting that DNS and domain search are in **resolv.conf**) - Edit **resolv.conf** vi /etc/resolv.conf - End up with this (**NOTE:** You don't need to do this if you modified **ifcfg-eth0** to contain these values) search gsfc.nasa.gov nameserver 128.183.10.134 nameserver 128.183.50.17 - Turn off IPV6 (found here: [[https://www.centos.org/forums/viewtopic.php?t=7989]]) sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1 sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1 - Change the **eth1** definition - editing the config file vi /etc/sysconfig/network-scripts/ifcfg-eth1 - to the following DEVICE=eth1 HWADDR=00:80:82:1B:E7:05 TYPE=Ethernet UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95 ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static NAME="System eth1" IPADDR=192.168.0.2 NETMASK=255.255.255.0 - Change the **eth1:1** definition - editing the config file vi /etc/sysconfig/network-scripts/ifcfg-eth1:1 - to the following DEVICE=eth1:1 HWADDR=00:80:82:1B:E7:05 TYPE=Ethernet UUID=5eb0ea0f-b1e8-43ab-a64d-83d3d9e1dc95 ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=static NAME="System eth1:1" IPADDR=129.100.0.2 NETMASK=255.255.255.0 ==== Install lvisf Source ==== === Add NTFS Capability === - Add **ntfs** drive capability - Add the **epel** repository sudo yum install epel-release - Add **ntfs** to our system sudo yum install fuse fuse-ntfs-3g === Add Exfat Capability === - Add **Exfat** drive capability - Add the **epel** repository and this Nux Dextop repository sudo yum install epel-release sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el6/x86_64/nux-dextop-release-0-2.el6.nux.noarch.rpm - Add **Exfat** to our system sudo yum install exfat-utils fuse-exfat An example mounting an Exfat system is sudo mount -t exfat /dev/sdc1 /mnt/tmp === Requirements === - Install **cmake** sudo yum install cmake - Install **htop** sudo yum install htop - Install **jed** sudo yum install ftp://ftp.pbone.net/mirror/centos.karan.org/el5/extras/testing/i386/RPMS/jed-0.99.18-5.el5.kb.i386.rpm - Install **nano** sudo yum install nano - Install **p7zip** sudo yum install p7zip - Install PCI utilities sudo yum install pciutils - Install **telnet** sudo yum install telnet - Install **wget** sudo yum install wget - Install the development package sudo yum groupinstall "Development Tools" - Install development libraries for md5sum and gzip in real time sudo yum install openssl-devel - Install **screen** so we can launch programs into detached terminals! sudo yum install screen - Install md5deep so we can check file integrity sudo yum install md5deep - Install man pages sudo yum install man sudo yum install man-pages === X11 Minimal === - This will allow you to run **X11** # core X11 sudo yum install xorg-x11-xauth xorg-x11-fonts-* xorg-x11-utils xterm # required for tuna sudo yum install xauth pygtk2-libglade === Software Tweaking === Let's remove EVERYTHING on this system we don't care about. - Remove **postfix** we don't need email sudo yum remove postfix - Disable the Firewall - Turn off the firewall sudo service iptables stop sudo service ip6tables stop - Make this permanent sudo chkconfig iptables off sudo chkconfig ip6tables off - Disable SELinux - Edit the configuration file sudo vi /etc/selinux/config - Modify the file - From SELINUX=enforcing - To SELINUX=disabled - Remove **auditd** sudo service auditd stop # now sudo chkconfig auditd off # after reboot - Disable ipv6 - Add these two lines to **sshd_config** # lvisf - ipv4 only AddressFamily inet ListenAddress 0.0.0.0 - Run these two lines sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1 sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1 - Reboot for all changes to take place === Code Base === - Create the repoistory home and check out the source code - Create a repository home in **/opt** sudo mkdir /opt/lvisf - Set proper permissions sudo chown -R lvis:lvis /opt/lvisf - **[Optional]** Uncompress our locally attached mount with snapshot (faster than network initial checkout) cd / tar jxvf /media/LVIS-GH-2TB_WD209/lvisf_build/lvisf_sourcesnapshot_20160816_svn439.tar.bz2 - **[Either]** Check out he raw source code from scratch cd /opt/lvisf svn co https://svn.yancich.com:8443/svn/NASA/LVIS/trunk/ src - **[Or]** Update it if you used the tar above cd /opt/lvisf/src svn update - Enter the user home directory and create symbolic links cd /home/lvis mkdir src ln -s /opt/lvisf/src/* /home/lvis/src/ - Add the following lines to the **.bashrc** for the **lvis** user - Edit the file nano /home/lvis/.bash_profile - Add these lines # For the Innovative digitizer export WD_BASEDIR=/home/lvis/src/Innovative/WinDriver # For the SyncClock timer card export PLX_SDK_DIR=/home/lvis/src/PLX/PlxSdk # For the spectrum digitizer export SSP_INSTALL_ROOT=/home/lvis/src/spectrum # SVN editor export SVN_EDITOR=nano - Edit the file sudo nano /etc/ld.so.conf.d/lvisf.conf - Add these lines /home/lvis/src/spectrum/shared/linux/pc64/lib /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/libs - Reload the library path sudo ldconfig - Add hooks into the run scripts for the **lvis** user (this is just convenient) - Create a local **bin** directory in the case there isn't one mkdir ~/bin - Symbolically link all the run scripts into this directory ln -s /opt/lvisf/src/lvisF/scripts/run* ~/bin/ ==== SyncClock32 PLX Drivers ==== - Build all drivers - Go to the source code home directory cd /home/lvis/src/PLX/PlxSdk/Driver - Execute the build script ./buildalldrivers - Build the samples - Go to the source directory cd /home/lvis/src/PLX/PlxSdk - Make the programs make - Load the driver - To the binary directory cd /home/lvis/src/PLX/PlxSdk/Bin - Execute the script for the 9030 card (our syncclock32) sudo ./Plx_load 9030 # to just see what boards are supported, run the script with no argument ./Plx_load ==== MIC 3753 DIO Homebrew Drivers ==== :!: NOT REQUIRED :!: We have reverted to the original factory drivers ; compile and insert the kernel module cd /opt/lvisf/src/lvisF/ko make sudo insmod /opt/lvisf/src/lvisF/ko/lvisf_mic3753.ko sudo mknod /dev/lvisf_mic3753 c 100 0 ==== MIC 3753 DIO Drivers ==== **NOTE** These drivers NEED to be built as root, you cannot **sudo** to build these drivers - Become root all the time sudo bash - Go to the code base directory cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/driver_base/src/lnx_ko - Build the drivers make - Go into our specific card directory cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/pci1753_mic3753_pcm3753i/src/lnx_ko - Build the drivers make - Load the drivers cd /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin insmod biokernbase.ko insmod bio1753.ko - Exit the **sudo bash** exit ==== Spectrum Digitizer Drivers ==== - Create the symbolic link for /spectrum sudo ln -s /opt/lvisf/src/spectrum /spectrum - Enter the code directory cd /home/lvis/src/spectrum/shared/linux/pc64/bin - Copy the drivers for our kernel into the binary directory cp -a /home/lvis/src/spectrum/shared/linux/pc64/bin/`uname -r`/*.ko /home/lvis/src/spectrum/shared/linux/pc64/bin/ - Execute the setup script chmod 755 *.sh # not sure why these were not already executable sudo bash ./setup_xmc1151_dvt.sh - Select the proper option: 1)Install ==== Auto-load MIC-3753 Driver ==== - symlink the drivers into the kernel directory sudo ln -s /opt/lvisf/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/*.ko /lib/modules/2.6.33.9-rt31.79.el6rt.x86_64/kernel/drivers/misc/ - add to our startup script **/etc/sysconfig/modules/lvisf.modules** - modify the file sudo nano /etc/sysconfig/modules/lvisf.modules - add the following # start the MIC-3753 drivers /sbin/modprobe biokernbase /sbin/modprobe bio1753 ==== Auto-load Plx9030 Driver ==== **NOTE: (in lieu of step 2 below)** I had this not work when I manually edited it the first time. A copy of the file is in the repository, you could just copy that file in: sudo cp /opt/lvisf/src/lvisF/scripts/lvisf.modules /etc/sysconfig/modules/lvisf.modules - symlink the driver into the kernel so it will come up on boot sudo ln -s /opt/lvisf/src/PLX/PlxSdk/Driver/Source.Plx9000/Output/Plx9030.ko /lib/modules/2.6.33.9-rt31.79.el6rt.x86_64/kernel/drivers/misc/ - add to our startup script **/etc/sysconfig/modules/lvisf.modules** - modify the file sudo nano /etc/sysconfig/modules/lvisf.modules - add the following # start the Plx9030 syncclock32 driver /sbin/modprobe Plx9030 - symbolically link in our **rc.local** into **/etc/rc.local** - remove the original # remove the symlink sudo rm /etc/rc.local # move the original to something else sudo mv /etc/rc.d/rc.local /etc/rc.d/rc.local_orig - link in the **lvisf** version sudo ln -s /opt/lvisf/src/lvisF/scripts/rc.local /etc/rc.d/rc.local sudo ln -s /etc/rc.d/rc.local /etc/rc.local ===== Installing Boost Library ===== The **boost** C++ library is required for the data system. The tarball of the 1.6.2 version is in the repository. - Uncompress the tar into **/opt** cd /opt sudo tar -jxvf /opt/lvisf/src/tars/boost_1_62_0.tar.bz2 - Symbolically link to a generic boost directory sudo ln -s /opt/boost_1_62_0 /opt/boost - Chown it all to **lvis** user sudo chown -R lvis:lvis /opt/boost_1_62_0 - Set the environment variable for the **BOOST_ROOT** variable - Edit the file /home/lvis/.bash_profile nano /home/lvis/.bash_profile - Add the line # Add the boost root directory export BOOST_ROOT=/opt/boost - Compile the **boost** libraries cd /opt/boost ./bootstrap.sh ./b2 ===== Installing USB Temperature Library ===== Follow the instructions below to compile the USB temperature monitoring device libraries required. * [[subsystem:datasystem:dev:hardware:mccdaq:usbtemp:start|Measurement Computing USB-TEMP-AI code instructions. ]] ===== Installing Innovative Digitizer Drivers ===== Described in **LinuxNotes.pdf** (might be this **/usr/Innovative/MalibuRed-1.3/KerPlug/LinuxNotes.pdf**) - Link our Innovative directory to **/usr/Innovative** sudo ln -s /opt/lvisf/src/Innovative /usr/Innovative - Untar the latest driver tar -zxvf /opt/lvisf/src/tars/64WinDriver-12.3-0.X86_64.tgz.tar.gz --directory=/usr/Innovative/ - Build 12.3 Windriver - Link 12.3 as our Windriver cd /usr/Innovative rm /usr/Innovative/WinDriver ln -s WinDriver-12.3 WinDriver - Go here cd /usr/Innovative/WinDriver/redist - Configure ./configure make sudo make install # they say do not do this, we live on the edge - Kerplug - Build (builds in the LINUX directory for the RT kernel) cd /usr/Innovative/KerPlug/linux ./configure make - OK, this SHOULD fail. - Edit the Makefile with nano Makefile - Add seven dot dot slashes to the line: - Convert this MOD_CSRCS = $(WD_BASEDIR)/redist/linux_wrappers.c - to this MOD_CSRCS = ../../../../../../../$(WD_BASEDIR)/redist/linux_wrappers.c - Change 1180 to 1230 - Run **make** again make - Load the driver sudo insmod LINUX.2.6.33.9-rt31.79.el6rt.x86_64.x86_64/kp_malibu_module.ko - Need to link the generic library name with the compiled library cd /usr/Innovative/KerPlug sudo ./quicklink wdapi 1230 Raw code from **/etc/rc.local** from the primary machine to load this sudo bash cd /usr/Innovative/WinDriver-12.3/redist && ./wdreg LINUX.2.6.33.9-rt31.79.el6rt.x86_64.x86_64/windrvr1230 auto #/usr/Innovative/winCloudDrvr/winclouddrvr_installation/redist/wdreg winCloudDrvr auto #chmod 666 /dev/winCloudDrvr insmod /usr/Innovative/KerPlug/linux/LINUX.2.6.33.9-rt31.79.el6rt.x86_64.x86_64/kp_malibu_module.ko exit ==== Field Addendum ==== In the field we noticed that the driver was saying unregistered, and would only run for 60 minutes (sometimes it ran longer, but this was an issue). Dave found these instructions from an email: FIRST! Make sure this folder exists: /usr/Innovative/Lib/Dll (and it is linked to the Innovative folder in src) Make sure LicenseLoader.so exists in that folder. (I have attached that file and a tar of the 3 others needed to rebuild) Hopefully its just that the .so did not make it to that system disk. If is does exist: Copy all the files in the tar to: /home/lvis/src/Innovative/Malibu Cd to /home/lvis/src/Innovative/Malibu/Qt/Os_Mb “qmake-qt4” “make” This will make the Os library and copy it to the correct folder. “That is all I got” 😊 Dave ===== Latency Tweaks on Kernel ===== * Low latency items to add to the kernel command line for **/etc/grub.conf** * After rhgb quiet * Add the following nosoftlockup mce=ignore_ce audit=0 maxcpus=2 * Final line looks like this kernel /vmlinuz-2.6.33.9-rt31.79.el6rt.x86_64 ro root=/dev/mapper/vg_lvisf-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=vg_lvisf/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_lvisf/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet nosoftlockup mce=ignore_ce audit=0 maxcpus=2 * **nosoftlockup** disables backtrace functionality * **mce=ignore_ce** ignores corrected errors * **audit=0** disables kernel components of the audit subsystem * **maxcpus=2** uses only the two actual CPUs on the system disabling any hyperthreading ===== Compile Code ===== - Enter the **build** directory for lvisf cd /home/lvis/src/lvisF/build - Use **cmake** cmake ../ - Build code make ===== Drive Configuration ===== - Create our disk partition (using the proper settings for **fdisk** to use 4096 byte blocks) sudo fdisk -H 224 -S 56 /dev/sdb - Delete whatever is on there (use **gparted** below) - Create a new partition with the default settings (use **gparted** below) - Install and use **gparted** to configure the drive. - Delete existing partitions (on the correct drive!!) - Format ( :!: BE SURE YOU ARE FORMATTING THE CORRECT PARTITION :!: ) sudo mkfs.ext2 /dev/sdb1 -L LVISF_INTERNAL - To show attached drives by label ls -l /dev/disk/by-label - Add the EKF based internal data drive to the system - Create a mount point sudo mkdir /mnt/internal - Modify **/etc/fstab** to add it by label sudo nano /etc/fstab - Add the following line: LABEL=LVISF_INTERNAL /mnt/internal ext2 defaults 0 0 - Symbolically link to **/mnt/data/** sudo ln -s /mnt/internal /mnt/data - Share the drive via **nfs** so the GUI and other systems can mount (read only) the raw data drive - Add the required software sudo yum install nfs-utils nfs-utils-lib - Start the **nfs** services sudo chkconfig nfs on sudo service rpcbind start sudo service nfs start - Add our data directory as a shared data drive to **/etc/exports** - Edit our shares sudo nano /etc/exports - Add this line /mnt/internal 10.4.1.4(ro,sync,no_root_squash,no_subtree_check) - Export our share sudo exportfs -a ===== Timing Tweaks ===== ===== Interrupt Timing ===== Interrupts are BACK! **timing03** test by isolating cpu3, and then putting the IRQ for the timer card on that CPU appears to have worked. I think I was using the wrong value in the **smp_affinity** for **/proc/irq/17/** entry. I was putting the actual CPU, but it appears to be a mask of some kind. The **tuna** program put the correct value in there (7 = 0-2 and 8 = 3 ?) I think. KEYTABLE=us rd_NO_DM rhgb quiet noht isolcpus=3 nosoftlockup mce=ignore_ce audit=0 intel_idle.max_cstate=0 processor.max_cstate=0 idle=poll ===== Tuna Timing Tuning ===== * FIXME Trying this option on the command line: **acpi_irq_nobalance** - Isolate ODD cpus, so that is probably a single hyperthreading core. # isolate CPU 1 and 3 (which are physical CPU 1 leaving the primary for regular OS) /usr/bin/tuna --cpus=1,3 --isolate - CPU affinity is 0,2 (which is mask 5 (1 and 4)) - Move the 9030 timer card IRQ to CPU 1 (primary CPU of secondary core) FIXME (need to isolate this) This is IRQ 11 on the development system (with no Spectrum boards) # move the timer card IRQ (17 for 0x9030 syncclock32 card) /usr/bin/tuna --irqs=17 --cpus=1 --move FYI, you can save the configuration with this command line sudo /usr/bin/tuna --save=/tmp/tuna.config ===== Run Build01 ===== - Start shell **one** via a ssh connection and run run1 # does the full 40 second Spectrum XMC-1151 calibration or run1nocal # if you want to skip the 40 second Spectrum XMC-1151 calibration - Start shell **two** via a ssh connection and run run2 - Start shell **three** via a ssh connection and run run3 ===== Timing Tests ===== Timing considerations 163,840 bytes is two channels of 4k samples for 10 shots (4096 * 2 * 2 * 10) /2*1024*1024*1024 = 76 microseconds 76 microseconds every 2.5 milliseconds is NOT A LOT! or all of it (for 30,000 feet) (32768*2*2) / 2*1024*1024*1024 = 61 microseconds per shots (or everyon 250 microseconds ==== timing03 ==== Resurrected **timing03** to investigate interrupts --- //[[david.rabine@nasa.gov|David Lloyd Rabine]] 2017/02/09 11:55// To run **timing03** (as of 2017.02.09) it can use the same receiver (aka **lvisf**) Short instructions - Terminal One run1nocal - Terminal Two runtiming03 **tuna** might have saved the day! - Isolate CPU 3 (so nothing is running on there) - Affine the interrupt for the 9030 board to CPU 3 - Profit! It appears that solves the missing triggers issue for the moment (been fooled before) ==== timing04 ==== * Load Drivers sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/biokernbase.ko sudo insmod /home/lvis/src/MIC3753/linux_driver_source_3.2.7.0_64bit/drivers/bin/bio1753.ko cd /home/lvis/src/PLX/PlxSdk/Bin sudo ./Plx_load 9030 * Run Test: **timing04** cd /opt/lvisf/src/lvisF/build/ sudo chrt --fifo 99 ./bin/tests/timing04/timing04 === 2016.07.19 Overnight Test Results === SVN Revision 378 timing04 Total interrupts = 214748056 00 11 (0.000005) 01 10 (0.000005) 02 6 (0.000003) 03 8 (0.000004) 04 8439881 (3.930279) 05 9181555 (4.275661) 06 8656159 (4.030995) 07 14232072 (6.627583) 08 166745360 (77.649887) 09 7277561 (3.389011) 10 191284 (0.089077) 11 6471 (0.003013) 12 4103 (0.001911) 13 5352 (0.002492) 14 34 (0.000016) 15 18 (0.000008) 16 15 (0.000007) 17 18 (0.000008) 18 19 (0.000009) 19 6 (0.000003) 20 6 (0.000003) 21 17 (0.000008) 22 9 (0.000004) 23 13 (0.000006) 24 2 (0.000001) 25 0 (0.000000) 26 0 (0.000000) 27 0 (0.000000) 28 0 (0.000000) 29 0 (0.000000) === 2016.07.20 One Hour Test Results === SVN Revision 379 timing03 (added stats to PLX interrupt test) FYI, thread priority wasn't even HIGH! galvThread: Scheduler settings of DoWork thread policy=SCHED_FIFO, priority=5 00 0 (0.000000) 01 0 (0.000000) 02 0 (0.000000) 03 0 (0.000000) 04 0 (0.000000) 05 0 (0.000000) 06 0 (0.000000) 07 2 (0.000014) 08 2033 (0.014115) 09 2967 (0.020599) 10 1384397 (9.611487) 11 12906360 (89.605298) 12 106019 (0.736061) 13 372 (0.002583) 14 280 (0.001944) 15 184 (0.001277) 16 375 (0.002604) 17 358 (0.002485) 18 206 (0.001430) 19 15 (0.000104) 20 0 (0.000000) 21 0 (0.000000) 22 0 (0.000000) 23 0 (0.000000) 24 0 (0.000000) 25 0 (0.000000) 26 0 (0.000000) 27 0 (0.000000) 28 0 (0.000000) 29 0 (0.000000) 30 0 (0.000000) 31 0 (0.000000) === 2016.07.20 One Hour Test 2 Results === Says ZERO missed shots, but there are a couple of times that are suspect... BUT, overall, looking great. Acceptable lost shots in my opinion for a non hard real time OS. SVN Revision 380 and timing03 Total Shots = 14404168 (and 14404170) Total Missed Shots = 0 00 2 (0.000014) 01 0 (0.000000) 02 2 (0.000014) 03 3 (0.000021) 04 3 (0.000021) 05 5 (0.000035) 06 1 (0.000007) 07 4 (0.000028) 08 1935 (0.013434) 09 2676 (0.018578) 10 1980021 (13.746167) 11 12304306 (85.421845) 12 113162 (0.785620) 13 542 (0.003763) 14 293 (0.002034) 15 210 (0.001458) 16 396 (0.002749) 17 362 (0.002513) 18 216 (0.001500) 19 20 (0.000139) 20 3 (0.000021) 21 1 (0.000007) 22 2 (0.000014) 23 1 (0.000007) 24 2 (0.000014) 25 0 (0.000000) 26 0 (0.000000) 27 0 (0.000000) 28 0 (0.000000) 29 0 (0.000000) 30 0 (0.000000) 31 0 (0.000000) === 2016.07.20 Overnight Test Results === 00 13 (0.000007) 01 4 (0.000002) 02 6 (0.000003) 03 6 (0.000003) 04 7959297 (4.079181) 05 8363128 (4.286146) 06 7973271 (4.086342) 07 12581677 (6.448174) 08 151407385 (77.597063) 09 6616644 (3.391064) 10 203955 (0.104528) 11 5439 (0.002788) 12 3784 (0.001939) 13 5238 (0.002685) 14 45 (0.000023) 15 14 (0.000007) 16 12 (0.000006) 17 13 (0.000007) 18 6 (0.000003) 19 12 (0.000006) 20 6 (0.000003) 21 14 (0.000007) 22 12 (0.000006) 23 11 (0.000006) 24 2 (0.000001) 25 0 (0.000000) 26 0 (0.000000) 27 0 (0.000000) 28 0 (0.000000) 29 0 (0.000000)